I can't comment usefully on your relationship with your brother, but I do sympathise a lot with your feelings about his attitude towards women.
There are almost 8 years between me and my husband, and in the early days of our relationship he had a friend who to this day I dislike. (They're still friends, but back then were very close.) This friend is slightly older again than my husband, and he was single at the time and constantly turned up to events I was hosting or attending, essentially to try and get in my friends' pants. I was about 23-24, husband 30-32, and I think friend was around 34-36 (if I ever knew his exact age I've forgotten it). He never even pretended to like me very much, so it was completely transparent why he was attending. He was actually generally very dismissive and impatient with me, clearly because there was no point in extending pleasantries to women who were not sexually available to him. He came to the most unlikely events just to scout out my female friends. I could never put my finger on exactly why I found this quite so gross and enraging - although I'd hold my hands up to resenting the dismissive attitude towards me coupled with the complete exploitation of me as a gateway to women in their twenties - but to this day it gives me the rage to remember him swanning around my house, or turning up at my birthday parties, and sliming all over my friends. In theory it makes sense as a mid-thirties singleton to target younger women who are statistically far more likely to be single, or maybe even just to attend any and all party you can attend regardless of who's throwing it, but he wasn't just a fun, lonely social butterfly. There was a definite sense of single-mindedness about it. He wasn't trying to find a partner, he was specifically trying to find a young partner, and gravitated towards women who were stunningly out of his league, and often not even single. It was so painful to watch him make a beeline for women he wouldn't have had a cat's chance in hell with, and I found myself enjoying some very unedifying schadenfreude when he failed comprehensively to appeal to any of my friends, and readily confess that when asked about him afterwards, I enjoyed apologising for the desperate old weirdo that followed my boyfriend around.
I also sympathise with this unease carrying over into anxiety about your own marriage. It's now 10 years later and the friend has been married and swiftly divorced, and I sometimes worry about his presence in my husband's life in the future. I think he was a very bad influence before, and though my husband has grown up a lot, the friend doesn't seem to have at all. He still has a palpable sense of entitlement and the disastrous marriage (and it was disastrous) doesn't seem to have altered his attitude in the slightest. He even stayed in our house a while ago when he needed (divorce related) emergency temporary accommodation, and even then he didn't really manage to be more than loftily tolerant of me, even slightly snide when we'd opened our doors to him, and didn't display any gratitude. I just have to hope my husband's outgrown his influence - a while ago we had a couple over who have never met him, and as I was describing him (I forget why he came up), the woman asked 'do you think he's a bit of a misogynist?' and my husband immediately nodded and agreed, with an ease and enthusiasm which surprised me.