Your house, your home, your rules.
Boundaries matter and you have the absolute right not to admit a man who battered yours into your space.
But your son has picked up on an inconsistency, which could look like, unfairness, hypocrisy - that you go into his father's house.
I'd say stay out.
Role model the behaviour you expect to see from his father.
It will make it easier for you to reinforce your own house rules.
And 'We don't go into each others' houses' is easier for him to comprehend than 'I go into his, but he can't come into mine'.
Your son could take a video, which you'd approve, that he could then send or take with him to show. Allows him to show what he's proud to have created, while still keeping your boundaries.