Oh dear. A bit of an own goal, I fear.
Keeping a spreadsheet on your movements? Yes, pretty agressive, weird, stalkerish behaviour.
He's going to make himself look controlling, unreasonable, aggressive. Trying to make out that you are - what? Going for a run to keep fit? That you have friends? Erm...
I wouldn't worry about it, but now that you know about it, if you wanted to make absolutely sure that you spike his guns on this one, talk about it to your solicitor. Tell them how intimidated and stalked this makes you feel. You could also innocently draw up your own very basic account of time spent, which of course takes into account the fact that you work part-time, that you do pick-ups, that you spend time (presumably) investing in your child's social life, etc. All the things he doesn't do... Just in the interests of working out if he has a point, of course.
I can picture you now, sadly pointing out that throughout the entire relationship, you of course were at pains to make sure that the family had other adult friends, for the sake of your son's social development as much as anything else. Given the strange, friendless, stalkerish, intense behaviour of his father, who by contrast never socialised or had adult friends of his own, the court will understand that you worked very hard to maintain some normality for your child...