The relentlessness in the newborn stage was hard, but there was a chance to learn from a mistake and do a bit better every couple of hours.
The anxiety about her reflux (which she mercifully outgrew), and about ear infections being potentially more sinister than they turned out to be, was hard. So was being verbally patted on the head by a few GPs, while I tried to sort those things out. But through these events, I've been able to learn more about what's worth getting firm with a GP, and what's worth watching and waiting.
Sleep deprivation was hard. So was learning to accept help, because I thought I was being undermined somehow. But over time, I grew to understand that my MIL did respect me, and that her intentions were genuine.
The effect it had on my relationship with my husband was hard. There were many stages of DD's infancy where literally everything was my domain, and literally nothing was his domain. Like, her constant waking up: it was me who woke with her, me who fed her, me who settled her back down, me who read the sleep training books, me who tried to memorize all the different techniques (hard to do when sleep deprived), me who tried to carry them out, me who failed. I would try to show my husband the latest book I was reading, and he'd say, "Oh, just tell me about what you've read. I'm sure you can handle it." And then he'd feel hurt that I was too exhausted to have sex.
There are some things about having children that will always be hard, and may not change the second time around, but the last one, the relationship one ... we could have made that much easier on ourselves by splitting the physical and emotional labour that an infant requires more equally. A bath a few nights a week does not an involved dad make. Even if you're breastfeeding and taking mat leave for a year, there are ways to share the sleep burden, and I wish we had during DD's first year.
(^FWIW, DH and I have talked a lot about the above, and are in a much better place. We would have saved ourselves a lot of trouble if we'd just done all that talking before DD was born.)