Just try walking back into his mansion and see if you can get it from him. You can talk your way past his guard fairly trivially.
Pick up the book and see what happens.
Phatt notices the difference in weight when you try picking up the book.
So replace the book with something of equal weight.