He lives in a tower on the courtyard.
The only way in is that doorway above the garden.
Ever read Jack and The Beanstalk?