I am quite used to always being seen as ‘at fault’ in my family but I am genuinely starting to think maybe I am being unreasonable here so wanted to get external opinions. It’s long so I apologize but I’d really appreciate any input.
backstory -
my brother has ASD (an Asperger’s diagnosis when that was still a thing) and although is very ‘high functioning’ is prone to being selfish, devoid of any empathy and at times violent.
My entire childhood revolved around him, and making sure he wasn’t overwhelmed due to the violent outbursts that would occur.
As he has gotten older the violence has started to become less frequent, but since being with my DH (11 years) he has witnessed 2-3 violent incidents from my brother towards me which has made DH not like my brother at all. He has ASD himself and because he isn’t violent doesn’t think my brother’s diagnosis is an excuse for that behavior. So I try to keep them separate, DH is polite and civil when around my brother but doesn’t enjoy spending time with him (although he does a very good job of masking it, as my brother is under the impression that he and DH are cool)
My brother lives 3 hours drive away from me.
(this is relevant) - additionally 8 years ago my father moved abroad for work and comes back to the UK every 2 years.
Now onto the issue at hand.
In November last year I was told my father was going to come back to the UK to visit for 3 weeks. This would be the first trip back since I had my DS. So it would be his first chance to meet his grandson. He was due to spend 2 weeks with my brother and one week with me.
DHs birthday is also in November and we had planned a city break for a long weekend. The city was about 40 mins away from where my brother lives. So I spoke to DH and said since we had limited time with my father I was thinking about making a detour to have maybe a nice lunch with my brother and dad on the way to the city break. To maximize the time as a family. He said of course as he also misses my dad (they have a great relationship) and wanted DS to get as much time with him as possible. I also hadn’t seen my brother for a few months at this point (as he won’t get a bus or coach to come and see us due to anxiety, and I had a newborn so wasn’t in the mood for 6 hour round trips)
Spoke to my brother, he was excited to see us all, great.
Unfortunately dad had to pull out of the trip due to medical issues a month before coming back. As that would just then leave the afternoon as just me, my brother, DH and DS I cancelled the detour to see my brother on the way to DHs birthday trip. As the only reason we were doing it was to maximize time with my father and tbh I felt bad expecting DH to spend a day around a man he dislikes on his birthday trip.
I understand why my brother is upset with this, but now I am public enemy number 1 in my family for being ‘so cruel’ to him and ‘making him feel like an afterthought’ - quite honestly he was an afterthought, he hasn’t bothered to come and visit us, he has never made the effort to come to me, and is now getting annoyed I wouldn’t do a 40 min detour to see him, I’m apparently a narcissist and almost as awful as Hitler (his words)
I’m not sure if since having DS a lot of trauma from my childhood and having to bend to my brothers will so much is coming up and making me a bit of an asshole, but honestly speaking - in this situation was I being unreasonable? Should I apologize?