YANBU at ALL @Sittinonthesand
Slightly different situation, but at a place I worked at some 20 years ago, there was a man in his late 30s there. He was in lower management, so had 'power' over the 'ordinary hourly-paid' staff. He was a HORRIBLE man.
He was misogynistic, and rude, and bad-tempered ALL THE TIME. Whenever anyone needed the safe key to get change, or vouchers or something, he would sigh so loudly that the roof tiles lifted, and if it was a girl/woman asking, 'he'd say 'bloody women!' and would scowl, and then huff and puff with frustration at being 'bothered.' 
I dreaded having to speak to him, and so did some of the others. With his lower management and middle management colleagues, he was not quite so vile, but he was less patient and more arsey with the FEMALE ones. He'd call hem dim,' thick, stupid, and all sorts when they got the slightest thing wrong...
He also refused to let people go for a break on their 6 hour shift, or to the loo when they needed to. He was a bully. He reduced many girls (and even a few boys) to tears quite often, and even drove a few of them out of the job as he was so vile.
So long story short, he died aged 41 of a sudden heart attack. Had it at home (he was single,) and wasn't discovered for 2 days when a colleague went to see where he was on the Tuesday, as he hadn't been in on the Monday, and hasn't answered any calls. (Apparently he had died on the Sunday.)
When told by the Manager of the place, that this man's funeral was 'next Wednesday,' myself, and a number of others (maybe 10 of us,) said we wouldn't be going. Most people went (about 30 people,) even though half of those 30 couldn't stand him, but 10 of us didn't go...
When asked why, we all said 'because he was not a friend, we were not close, he rarely spoke to me etc...' The Manager said 'but he was your colleague!' One (brave) woman said 'listen Steve! We're not going, because we couldn't stand the man. He was rude, moody, misogynistic, and treated the lower grade staff like shit. We're not gonna be standing there pretending we care that he's dead.'
The Manager was stunned, when this woman said this. Had no reply though, as he knew we had a right to refuse.
Weirdly, most people didn't say anything to us, (about not attending the funeral,) and we think that was because they knew why. Basically, they knew what he was like...