Why did my workers leave me?

Hint 1 of 2:

Did you read your diary (in the game box)?

Hint 2 of 2:

Did you read the note that they left you?