"it was apparently difficult last year with their dads girlfriend and not very relaxed although"
Sooo...you knew Christmas at their dad's last year was difficult because of the girlfriend being there, (perhaps other things)....
but knowing this ...a short time span after (had to be a short span of time if you've been seeing him under a year now) Christmas...you started seeing this man, and having him in your house and around your children?? During their visits??
You and your children don't even know him well enough for these things to happen.
Why do you feel like he needs to be around during visits at all??
They don't want him there at all. They are uncomfortable around him and the house when he's there. Your eldest probably only agreed to him coming to dinner because he knows you want him there that it would mean much to you, or he was coerced into it.
Even if they all wanted him there genuinely, it would still be inappropriate, and something concerning.
Don't force your children to accept a stranger.
Even though it was years ago when you and their father ended things, they may still be having a difficult time.
I personally don't feel parents should start dating...seriously...until their children leave the nest.
Also, this guy that you're seeing doesnt sound like a quality man. He shouldn't be hanging around a woman and her children during their visitation. Nor should he be wanting to insert himself into Christmas/family dynamics.
Honestly if he was of much quality, it would be a turn off that a woman that he barely knows has had him in her home (while kids there) and around her kids (whom he hasn't formed a bond with).
Just be careful with him....you don't know him well...and him allowing himself to be around your children is concerning.