No... YANBU- think I'm going against the crowd here...
My DH doesn't drive yet either (hasn't passed) so I would have to do all the driving too...
5 hours is too long to keep my children in the car for- but they are still very young, and the toddler hates being in the car as she is too old to sleep, but too small to reach much and drops her toys all the time, then howls. The baby would be bored when he was awake, staring at the back seat poor thing! Plus they always want Mama's attention, which makes things difficult if I have to drive for 5 hours...
Plus- it isn't really a free trip is it? You'd have to pay petrol costs or train fares so hardly negligible, probably a tank and a half all together. You'd probably be eating out as no-one wants to cook when they're on 'holiday' so that adds up too- and no doubt you'd be offering to pay for PIL as it is FIL's birthday. Also, it's hardly a holiday when you have a toddler and an infant... so personally, I wouldn't be up for going.
I don't know what your PIL are like, but it sounds like they wanted to go somewhere, and could only find a largish house, so invited you along too as they were already paying for it. If they had planned it from the start, surely they'd have picked somewhere between the two of you, around halfway?
It wouldn't be a fun road trip for us, that's for sure!
[curmudgeon emoticon]