You have a disability and this is bullying and harassment.
I strongly recommend talking to ACAS - only when you’re back! enjoy your leave NOW - and getting clued up with your legal rights regarding the Working Time Regulations 1998 and the Equality Act 2010. They will probably recommend copying in your HR in communications with your manager.
I would ignore and when you are back brief HR and/or talk to your union rep to get them to attend and document your next 121 to ensure that (a) it won’t happen again and (b) you get your leave back. Also contact your organisation’s Disability Network if they have one, or ask to speak to a disability rep from the union or similar.
As you have a disability I would also look at unpacking your response thus far. When you have a disability you often work twice as hard to match your peers, because you’re having to use coping strategies, which are in and off themselves quite hard work.
I know, because I’ve been there.
This can go two ways: you become less assertive of your rights because you want to fit in and don’t want to rock the boat or draw attention to your disability for fear of bullying. And managers take advantage of it. I think what he has said to you - as you’ve documented upthread - is a threat that is bullying in itself. And taking advantage of your desire to fit in. You’re not being treated equally and fairly.
Or it can mean you mean you become assertive, set boundaries and point out as a disabled person you arguably need your legally entitled downtime even more to become fully refreshed, and to harass a disabled person while on leave - for minor matters no less - meets the criteria for bullying and specifically would be considered against your protected characteristics as set out in the Equality Act.
If he is indeed male and senior to you then invoke your second protected characteristic as well - sex as well as disability - and suddenly it looks a whole lot more serious.
Never mind the fact that you are not exempted from the Working Time Regulations 1998.
This lays the organisation wide open to a complaint and grievance process on two grounds. Pretty serious.
HR should rightly be horrified. And just stating your entitlement to a complaint on legal grounds to HR will no doubt concentrate minds.
I’ve done it - not quite in the same circumstances - but I pointed out disabled staff were disproportionately affected by a corporate decision. I consulted ACAS, quoted the law and won my case - ie a different decision was taken - without it ever going to grievance. My threat was deadly serious and they knew it. I have always been a diligent and compliant employee and I spoke up not only for myself but others affected with tons of evidence.
Use your work time for all of this. It’s your right. Enjoy your leave and talk to ACAS/HR/union when you’re back or you’ll be doing more work on your leave to make your case.
Log the times and dates so far then forget it till you’re back (which will be easier said than done as it’s going to hang over the rest of your leave, probably). Which you would be reasonable to point out to HR! Park the boss and the organisation for now and just enjoy yourself.