Darling just because you had no sugar should in no way make you feel bad, with a baby you can't do what your kids want all of the time, stop sloshing so much presure on yourself. even if you had he maight have cried because you were mixing it and he wasn't or the sprinkles were the wrong colour.
From what you have said, you seem pretty busy socially, so maybe when the baby is asleep do something really simple, like read a book together, make a train track, simple and easy stuff to do in the house together, even if it's just 20 mins.
He will then know he will get you and only you at home for some special time. And as you are feeling peed off with others reactions (and fuck them by the way, like they have a medal with my kid has never cried written on it!) maybe you could change your social life, try a new group, we have a lovley church one here, and before you think it, they are so not the God Squad!
Does he go to pre-school? maybe even one morning a week, he would learn how to play and interact with other children, see how others behave and how others don't behave etc, give him a good rounder view of other kids other than family. It might be worth a try.
And remember he's 3, it's hard, it's hard with a baby too, tomorrow is another day, you are doing A BRILLIANT JOB!