If you are out with your family, why aren't your family helping ?
However, things like "Oh, if you have had a challenging baby the 2nd will be calm" are just platitudes people say for small talk.
You haven't done anything wrong, it's just the way it is sometimes - some babies a livelier than others, some are grumpier than others, some are more placid than others, some sleep well, some feed well, some have reflux, some can't latch on, some are sickly, some are well.
Have you spoken to your health visitor ?
Might they be intolerant to milk (or something else - not sure what ages we are talking)
Do you get a break at all?
Do they go to Nursery or is there someone else (partner perhaps) who you can hand them over to ? It is true that once you are tense babies seem to sense that and find it harder to settle sometimes.
Tell us a bit more about them and I'm sure someone will come along with some advice that is more up to date than I can suggest. 