OP, to answer your qu, I think the honest answer is that people with 2+ kids do probably look at you with one and think "piece of cake". My DS is now 9 weeks and I have a DD of 2.7. On the days when I am alone with DS, I now think "what on earth was so challenging when it was just me and DD!!" To be fair DS is a dream baby though, days when it is just me and him are very relaxing as he just feeds, plays about a bit, then sleeps.
But people who think having one is easy from the vantage point of having two, are not comparing like with like. Its easy to have one baby when you have already gone through the baby stage and are used to a toddler.
But with your first you are making the transition to parenthood and responsibility and it is this, more than the actual demands on your time, that is the real burden with your first.
With your second baby, you don't have to carry that burden again - you have already adjusted to being a parent, always coming second to the children, never having time for yourself, the responsibility of ALWAYS having to think about a little person, 24/7. Plus the baby seems easy compared to your toddler, ha ha! There are challenges when you have two, but they are logistical (you become MUCH more busy and you often have two of them screaming for you simultaneously which is slightly mind bending) rather than psychological, if you get my meaning.
I think when your baby is a toddler you will probably think that toddlerdom is harder than you have it now. Toddlers ARE hard - OK they do sleep through the night usually, but oh the endless negotiations, dramas, tantrums, irrational behaviour- such an emotional rollercoaster and its really tiring having to be the adult rather than just come down to their level and scream! At the same time they are still pretty dependent - you can't yet just "direct" a toddler to have a wash/clean teeth/eat food. You still have to do it all with them just like with a baby.
But don't panic, toddlerdom doesn't happen overnight, its a gradual change, and you will be able to cope when it happens! The challenges will appear and you will step up, because you have to! Mothers are forged out of steel, after all ...
I nurture a fantasy that once mine are both at school life is going to be much easier, but I am sure the challenges will just be different ones!
Best of luck!