Your "thought process" tells you that "there must be a better way".
Surely you don't just expect to pick ore out of clusters on the walls!
You'll need some kind of tool to mine the various gems you find.
General Outfitters offered a Rock Filet for this purpose, but they were sold out.