The key is not in the castle.
Use the drawbridge to find the key.
Stand on the drawbridge and 'rock' the control hole.