Think he is BU - because it sounds like he has come to you with a fait acompli for that weekend, rather than saying 'what shall we do that weekend, colleague has invited (some of?) us to go camping, what do you think...
Just because he was invited doesn't mean he is obliged to say yes - he should have at least said I'll check with matchsticks and see what plans we currently have for the weekend.
It's not unreasonable to want to do something for your DS's first birthday and he should have realised that. Had you already started making plans for any sort of party/gathering and did your dh know that this is how you hoped to spend your ds's birthday? Do you think he feels obliged to do the camping thing with his dsds because he doesn't always get full access and this is clouding his judgement (doesn't want them to think that their new little brother is more important that they are)?
how old are your dsds and dd? Do they desperately want to go camping or can you pull the 'but they'll miss ds's birthday' line on their behalf?
And if this was any other weekend - would you have wanted to go along to a local guest house while they camped or would you have stayed at home - think that is a separate issue from the birthday one.
Can you ring work colleague directly and say that not possible to go that weekend so how about changing to another weekend?
Sorry, lots of questions and not many answers but definitely think you need to make your Dh realise that he is the one that is BU - or at least that you are both being as U as each other for making plans for that weekend without discussing them with each other much earlier...
next year, remember to start talking about your ds's party/whatever you want to do for him for his birthday - just after christmas so you get in there first and plans can't be changed later 
DS1 has his birthday in the spring bank holiday weekend and although he's only 5 at the moment, I already foresee that it's only going to get trickier as parties will tend to be shifted as people are away for the weekend or people want to do things because it is the bank holiday and he will be expected to shift his birthday celebrations. you have my sympathies!