CalleighDoodle I've drank some too but have just eaten so much it's absorbed any alcohol! 
Can I ask how often your ex looks after dd and normally for how long?
Re I do feel uneasy about ex being in charge of dd, this is his first overnight contact he would if had with her and the only reason I have agreed is because it will be at his brothers and SIL house so I know they will be there to help Is the over night part going to be at the ex wife?
If so and you are not happy just say, presumably you agreed to visit because you thought it was just ex partner with his brother and his wife and not another woman who you presumably do not know know at all well.
You say I do feel uneasy about ex being in charge of dd I am just wondering where your ex will be, won't he be there too, won't he be in charge of dd? Is there any chance he would go out and leave her with ex? You said on the way he is going to pop into ex wife's with dd to see his son. Which sounds a bit like stopping by for a short time, not leaving her with his ex wife.
Do you drive, could you go with him for the part to ex or would that be really awkward?
If you are not happy just say no if you don't want it to happen. It is unsettling enough for a young child to be away from mum for an overnight, even if with dad, unless she knows dad well, which it appears she does not. Without including a home and family the baby does not know. Your ex may not agree or may not follow what you say/suggest but you can at least express your concerns.
Just make sure it is all about your dd's feelings and welfare, and by your I mean plural, you and your ex, so he feels you are not trying to undermine him but you are both looking out for how dd will feel. It sounds like he is more interested in his son's 'needs' and this may be that he is trying to rebuild the relationship with his son. Which is a good thing, presumably, as long as he is also taking your (plural) dd's needs into account.
In your shoes I would at least explain to your ex that this visit could be unsettling for your daughter and if he wants the visit to go well he will take special care of her and make sure the stop off to see her half brother is well supervised etc.