I did more or less the same when DD was just under 4. We were in a domestic abuse situation and needed an escape route. DD was protected from it all so she wasn't aware as such that anything was wrong as I distracted her constantly. My life was directly threatened once and that was once too often, on top of the rest which had happened intermittently over the years.
To cut a long story short, a fairytale began - a man on a white horse came to rescue us, took fair maiden and DD to safety and left the nasty abuser crying in a pile just like he fecking deserved.
We all lived happily ever after except the nasty abuser started emotionally abusing DD during contact time - we listened to every word when she came home, she spilled everything in total innocence that her dad had said to her and how he'd treated her.
So in my experience it's not necessarily anything you & your partner are doing, it might be what is being said and done when he's at the other house.
I have taken on board everything DD has said since she started talking, ex is an evil manipulator who, when confronted, always says she's lying.
Be observant, listen and re-assure him xx