You'll have to make him not want to go with you.
Would pleading insanity work here?
Try telling him that parents can be a pain (#1).