Your child will not even know it's their birthday, will not remember it or have any clue who was there or not.
This was just you putting on a performance for a day that has no meaning to the person it was really about so YABU.
If your child was old enough to actually understand it was their special day it might be different, although if he left at 6am to go to the gym and was back by 9.30am it still wouldn't really matter, so it depends what time of the day he was at the gym even then.
It was your childs birthday, not yours.
It wasn't about you.