How old are you, and what do you do? If you're 23 and go for quiet hacks I would be less bothered than if you're 55 and do cross country eventing. The reason I mention age is because generally, younger people fall better or can avoid falls more successfully. I wouldn't personally advise anyone to not wear one, but my friend who is 23 and only hacks (albeit on a slightly crackers pony) doesn't wear one and I find it easier to not nag her than my godmum who is considerably older and less flexible in terms of staying on and landing.
They're rather warm and I quite like wearing one in winter. You really need one that fits you properly though, I've never bought one off the hanger. I don't typically wear one if I am schooling in summer (I do dressage) because I like to be aware of holding myself correctly instead of feeling held by the body protector, and its very hot! But that's my choice and there are lots on the market now, some thicker than others. I would wear one if I was on roads etc before I got to the school.
You get used to wearing them too, they feel rather odd to start with but that does wear off. I think they're becoming compulsory for driving soon. Just be aware though, some have this thing where you fall off and they inflate, but they don't meet the regulations coming in for the likes of driving because all they do is protect you from the bump, not properly covering your body/organs (or so I was told at HOYS by the stall selling them).
This is one of those things where I can let myself not wear 100% of the time and its fine, but if my niece or nephew were going riding there is no question they would be wearing one from the off. As far as my niece and nephew are aware I wear one all the time etc. I'm only 23, but body protectors weren't common where and when I was learning to ride and it was only when I fell off rather badly that my Mum insisted I get one. Not that I came off too bad, I had been knocked out but nothing broken etc thankfully. I hear broken ribs are horrific, so its a risk for you to decide to take or not really.
I really do think it is important to get one which is well fitted though. I have a long back and ones off the hanger ie. standard sizes don't come down low enough over my ribs, nor do they cover my spine as much as they need to. Pop into a shop and ask if you're not sure what to get/try.