if he does this at school he will get into a lot of trouble - especially if it continues until he is 7 or 8 - is that what your DH wants for him, that he is known as the little boy who exposes himself and grabs people's bottoms ? Other children won;t want to be around him for fear of being humiliated by having their pants pulled down.
And what about your porr DD - does your Dh want her to think that actually a boy doing something completely innapropriate and embarrassing to her is OK and her parents wont; protect her from it. How would he feel if a boy outside of your family pulled her pants down - would he want that boy 'deprived' or punished.
You have to come down zero tolerance on this 9obviously Granny is going to be a bit of an eception if she has dementia she can;t be expected to be on programme with the approach, but I imagine one of you is always with him at Granny's ?)
As soon as he does it anywhere any time you get take him away from his target and get to his level and say very firmly in a quiet but very cross tone 'No. We do not do that, You have been told.' And follow with an instant sanction (not a delayed one - like no TV tonight or no screen for a week) he needs to absolutely connect instant repercussion with the behaviour.
So if you are out, you go home - no second chances, no hesitation, - the pther parent can saty wih your ittle girl so she gets her day out. if it's during play, he is removed form the play and made to stand or sit away from the action. and so on. and you continue absolutely no tolerance. no engaging in a chat about it, no grabbing of hands or any kind of tussle - that's all attention for him.
Just remove, firm voice of disapproval, then he can return and see if he can manage to play without doing it.
No one laughs, no one thinks it's funny. no one lets him get away with it at all. Your DH has to be on board with this or he is seriously letting down your DS and your DD.
It might mean you get a few days where you are constantly removing him, telling him no and stopping activities. But he will get the message eventually. At the moment he's getting amixed message, you try to stop it, but your H is not on the same p age and is letting him do it. So he's not understanding - he just thinks 'Dad doesn't mind. Mum's making a fuss'.
He might grow out of it, but he might not. And at six he should have grown out of it (it's not unusual for 6yr olds to find bottoms and everything to do with them funny, but to contiually harrass others and keep touching other people when they don't want it, is more unusual)