I can't "mine" any of the gems on the walls.

Your "thought process" tells you that "there must be a better way".