I think a lot of it has to do with your particular family. For instance, I stopped going to meetings between about 21 and 27 (it's called inactive) and it had no effect of my family stopping talking to me etc. Old friends would email me for a catchup or if I was in the same country/town I'd visit, they'd visit no problem. My Dbro is disfellowshipped. I keep in contact with him, I make sure he is okay and well. I see him maybe once every couple of years (we live in different countries). He has a wife and I have a niece (wonderful!) and send gifts for her and met her last year. If the elders had a problem with that so be it. I make sure he knows I love him, I think of him, and that if EVER he needed me, he could call me. Some men may not think that is okay, but I sure hope that God would see that my motivations are good and not judge me for it. I also hope it is encouraging to my Dbro.
I also see my non JW family, I love them, they love me. They don't care about my religious persuasion.
I grew up in the 80s when it was very much, you must not go to university, to the pub, clubs, gyms etc... It was the NO generation. Nowadays it is recognised that people have free will and can make decisions and take reponsibility for those decisions themselves. I know many that have had further education and have good jobs, my DH is one of them. We go to pubs for a pint, I've been out clubbing with my friends from the "church" and am happy to join a gym for exercise. My DC will be going to university (whether they like it or not! 
). I believe now to have a decent job and support a family well, this is necessary. It is all about balance. You won't see me raging drunk hanging out of clubs on a saturday night, but I can go with a group of friends and have a dance and a drink and enjoy myself without any bother.
It is when behaviours become extreme, that it comes to the attention of the elders.
I also have some social contact with people at work, occasional after work drinks, visit a couple of good friends from work for cups of teas, have been away on a weekend trip with one of them or ski trips etc.
We are normal people who love our families and are just trying to get on as best we can in life. We don't believe in the same things as everyone else, and that is okay, it's not wrong, it's just different.
I do feel for you WhiteShores, you are in a difficult place. I think though, you get to a certain age and just think, sod it, this is what I'll do, it's my life, if people don't like it, tough.