I'm an only and I have just one DD and will probably be keeping that way.
Obviously from a pure numbers point of view, logic dictates that more than 1 child will be harder, but there are so many other factors to consider.
I do think that as a parent of an only child you do feel that you need to be their playmate/entertainer, as well as factoring in all the other parental duties, whereas if you have more than 1 child and they can play together that isn't likely to be an issue, (not saying you wouldn't want to play however many kids you have - but the guilt level if you don't is probably not quite so intense if you at least know there is someone else there for them).
Personally I also like to try and arrange things with freinds and their kids so she has as much opportunity to develop her social skills as possible, which is not so pressing if they have a brother or sister to practise them on!
I had a similar thing crop up with a friend of mine who felt that the things I had planned for me and my DD over the summer were unachievable for her with a 3.6 and 1 year old (I'm talking watching Pocahontis, then making a wigwam in the garden and a totem pole out of painted boxes so nothing massively strenuous).
She works one day a week, has the older child in nursery 2 1/2 days a week, little one in on the day she works, family help 1 day a week and a husband working 9-5 from home.
At the time I was working 3 days a week, the only time DD was in nursery was when I was working and my DH was working an average of nearly 70 hours a week, so how her life was so much harder with 2 I cannot fathom, but it's all down to perspective.
All children are different and everyone's circumstances are different, and how we all deal with parenting is different. I just don't think it's possible to say anyone has it easier than anyone else, purely based on the number of kids they have.