Those guards are stopping you.
Ever heard of creating a distraction?
You'll need to create an emergency, but don't stand around.
Fires could be good distractions.
Notice any source of fire in the castle?
Look in the dining hall.
There's a burning lamp.
Knock it over.
(Chi will not do it unless he is with Lucky.)