If he's refused an invitation of yours to come to something because it's too far and this is much further then he obviously only believes that family is more important when it suits him.
Tell him that you would love to come but it's just too far and remind him that he has used the same response when he had much less distance to travel and was without a 4 yr old in tow.
and definitely do as gorionine suggests and say that you would come if he could come and pick you up (is it a much shorter journey by car?) - or would you go if your dh would go and drive you, in which case say that too. Is he expecting you to pay towards the cottage and is that why he's really insistent you go? But that he has booked one that doesn't have enough beds for your dh or would there be a bed there if you turned up with your dh (and a campbed for your dd if necessary)?
Or point out that if he was really serious about family etc then he would have checked with you first about whether you were free for the whole weekend or a long weekend and booked somewhere that was a reasonable distance for you to travel to too. If he's your dad he must know that you can't drive and you'd be having to come by train so it was a fairly mean thing for him to do (I guess unless you have other family who are having to come a 6 hour journey in an opposite direction so it's bad for everyone).
In fact, turn it around on him and say that you would have loved to have come but given the way that he has organised it you don't think that he wants you to come, and he is just using the family line to guilt trip you so you don't notice if you don't notice.
practise what you are going to say in your head lots before you talk to him and work the conversation through in lots of different ways so you have all the objections worked out to his demands, then ring him up, stick to your guns and say that you're sorry but it's just not possible for you to do that for a weekend.
YAdefinitelyNBU