It depends, if your toddler is whiny, upset and hard work, does dealing with that fall to you or will your DH step up and take over so you can have a glass of wine and enjoy a leisurely lunch with your friends? I'm guessing it's the former if you are the one on edge.
Quite frankly, if your DH isn't prepared to deal with the extra work that comes from not sticking to DS's routine, then he shouldn't expect you to fit fun things for him around it. that's actually very selfish of him, he's decided what's fun for him, is more important than you and DS having a nice time.
I'd point it out like that - for special occasions or things that can't be at other times, then fine, you'll suck it up, but every time he wants to put something that involves you being out of the house over nap time, point out clearly that you and DS will be having a really, really shit day. Be very clear, there is no way you and DS will have fun, he is making arrangements that will involve the two people he cares the most about having a crap time. Obviously, he might wish that wasnt the case, but if you have a child that needs to stick to a routine to avoid meltdowns, you have a choice, stick to the routine or don't and deal with the concequences, pretending you can just flex the routine and it'll be just as nice as if you didn't is really just ignoring the reality.
in summery, YANBU