I suppose it's all a matter of compromise. You want to see them sometimes, but still have time to be a family.
To cut down on long trips to see them, could your say that your son is becoming a very bad travellor on long car journeys. Always being sick, continual crying etc etc., so you are avoiding these as much as possible.
We stay with Mil and Fil a lot and sometimes I too wish we had more time to be alone together. However, when we do the in-laws thing, we go our own way during the day. The in-laws get to see us for a long, leisurly breakfast with the children and for a long, leisurly supper without them. So we are all happy - kind of!
We tend to talk through our plans when we first arrive, so they get to see their grandchildren for some day trips, while also babysitting in the evening so my husband and I can go out together.
When you stay with your in-laws, do they expect to see you all day, every day, or do they accept that you will do things alone? I would find the former very stressful.