I'm not really sure how players are supposed to figure this out.
The game's hint suggests that you're somehow offending the dwarves.
You've just left the sewers. Maybe it's that nasty smell.