You can't take a fruit. All you need to do is eat one.
Have you tried to go to the inn after you ate a fruit?