Of course you need to. That's the basic rule of the Simon the Sorcerer games: All that can be done, has to be done. No red herrings, no waste of time.