The youngest any of my children have been away is Y3. She was not very sure about going, but her elder sister was on the same trip so that made it better. When that child was in Y3 or Y4 and the opportunity for the same trip came up, I declined.
Other than that, Y5 or Y6 is when the opportunities have arisen for my older kids and we had no qualms about their readiness.
Apart from a week in France, these have all been activity courses (eg PGL-type) at that age.
Once they are in senior school, they go on visits directly connected to the curriculum, eg Geography fieldwork.
We've never had the opportunity to go on trips longer than a week. I think I would want to visit at the weekend if it were a two week holiday (which I did myself as a child).
I don't think it is a big deal to the child to spend their nights away from you abroad vs somewhere in the UK, especially if going by tunnel or ferry. It might be an issue for parents, but the school risk assessment for overseas trips is very rigorous - there is really no need to worry.
I think a bigger issue is if they go on an activity holiday and can't swim 25m - that means they can't take part in the watersports.