One person on the island knows.
Talk to the storekeeper.
Aren't you curious where he goes to talk to the Sword Master?
Follow the storekeeper when he goes to see the Sword Master.