Just try walking back into his mansion and see if you can get it from him. You can talk your way past his guard fairly trivially.