Hi OP
How coincidental, I wrote a very similar post to this last week (the title was identical)
In my case, it's because I'm pregnant and didn't want DP bringing his children here when they're ill as I'm vulnerable and have suffered 2 previous losses.
I posted in AIBU and was ripped to shreds for a good 30 pages, fortunately I had a number of people jump on to the thread in my defence who didn't think I was being unreasonable at all.
In short, no you're not being unreasonable. It's your house (if that's the case, as it is with me) and nothing is stopping him seeing his children elsewhere. His contact should not depend on having your home to facilitate it.
Why can't he take them out if you're not feeling up to a house full?
If you're struggling with the nuances of a blended family come over to the childlessstepmum forums, it's not just for women with no children of their own btw, you'll find they're much more understanding than most on MN where there is an awful bias against step mothers and girlfriends of the NRP dad.
I haven't read all of the replies here but I already know you'll have received much of the same vitriol as I did, don't take it to heart - there is ALOT of projection on here.
Feel free to PM me anytime x