Is it more the choice of activity or the initial exclusion that bugs you more?
I'm a guy, and I'd hate this sort of thing, partly because golf isn't for me, and I'm a massive introvert, but I'd still go.
This is a bonding activity and if you don't go, you may find that decisions have been made without you, and they'll return an even stronger unit, excluding you further! Golf clubs are not cheap, so even if you don't enjoy the golf, the sun, bar and food is usually worth the trip on its own.
Make a joke of your playing, or better, make a joke of theirs! With guys, it's all about the banter, don't be shy. You'll play, and suck, as any new person will, and you'll probably get eye rolls... but that's fine! There is your opportunity to ask for pointers, as men LOVE to show off their knowledge! It's all about bravado. If there are any comments, remind them of the phrase, if you can't teach it to someone inexperienced, you don't know it well enough yourself... make your training a challenge! They'll be fighting over you by the end :)