Totally agree with cornish.
Even if the child is young, it is likely to still have a significant impact. My DD was supposed to see her dad every Wednesday at a contact centre between the ages of 2 and 3.
Every time he missed it, she'd be devastated. I quickly learnt not to tell her where she was going until the contact centre phoned me 10 minutes prior to the session to let me know if he had showed up (luckily we lived very close, though this was tricky as she's always bounded out of bed in the morning and wants to know what the plan for the day is).
But then she picked up days of the week, which days she goes to nursery, classes, nannys etc and so began to know if it was contact day without me telling her. Bloody broke her heart when he didn't show, and mine.
It was such an awful time, and I worry every day at the effect it may have had on her self esteem. She's a really sensitive, emotional kid at times. So I worry that this has caused permanent damage to her MH and I go to huge efforts to bolster her self esteem and make sure she knows that she is so so cherished and adored by all of our family.
Anyway, please go. Ask your nursery if you can collect your DS an hour later, or pick him up earlier and take him too. Our contact centre was lovely and I wouldn't worry about your child being there if yours is the same. Just wanted to share as some PPs seem to think that it doesn't matter so much with a younger child.