wrt the 'easier/harder' thing, I think it's swings and roundabouts (and, of course, dependent on the personality etc of the actual child/ren)
In some ways I can see that my lfie with just one DD IS a lot 'easier' than the lives of friends with more than one (specifically, actually, more than one child when they are all under 5ish). The sleepless nights diminish. If there is illness, it's just one of them getting sick (and then maybe you) as opposed to all of them going down with it one after the other. Mealtimes are a calmer affair - just one fissy eater, perhaps, to cater for - that kind of thing.
In other ways, it's harder. You are that child's everything (specifically when they're too young for many playdates etc) and ime it is a rare only child that 'plays nicely by themselves' in an idyllic fantasy world while the parent tidies up, stacks the dishwasher, cooks supper, answers the phone etc. In almost every case i know - mine included - all these things are done, if they're done at all, with a whining or stropping or just cheerfully demanding small person hanging off your leg or following you round like a shadow insisting that you take your part in the Cinderella game she HAS TO PLAY RIGHT NOW...
When DD has her older cousins round to play, it is bliss. They're 2, 3 and 4 years older than DD and I get the proper experience of being a mum of more than one in the sense that they keep each other occupied for HOURS and I actually get things done in my day - a total revelation.
It's the sheer relentlessness of having to be company for an only that is hard work.
That said, if there's one thing and one thing alone I don't envy when I see parents with more than one child, it's the refereeing of sibling fights. So maybe it's in my head that it's 'easier' when they occupy each other, because perhaps for every blissful half hour, there's another half hour of warfare.
My point, OP - sorry to ramble - is that it isn't neccessarily easier with one, and it isn't neccesarily easier with 2 or 3 or 4... There are bad times and good times, easier days and harder days and - again - different personalities. I have it much harder with my one spirited DD than my friend has it with her two very very mild and quiet little boys.
The basic point is that it DOESN'T matter and it SHOULDN'T matter to anyone else. They may well not mean to be unpleasant or let any jealousy show. Or maybe they do. The only thing you can control is how you feel about it. And IF it's blissfully easy with 'just' one - so what?! There may be a tonne of things about your life that aren't so bloody easy, and heaven forfend anything shoudl ever be easy in life... much better to be a martyr and struggle through every single good thing that comes your way, right? 
Sorry, this became accidentally epic.