YANBU.
I had infertility and miscarriages too so when I had my first it was like a dream come true, when I got pregnant with DD it was like all of my dreams come true. There was almos exactly two years between them and it was everything I ever wanted yet I couldn't work out why I wasn't happy. DS was in that screaming phase so I felt guilty because I felt like I'd caused it, I ruined his life by making him share it with another child. DD was sickly so I was afraid to love her in case anything happened to her, she was only ever sick with treatable things (thankfully) but I suffered awfully from anxiety and read this series of bugs and nasties as symptoms of something far more sinister. More than once I considered leaving DH to it and just heading off into the sunset to start a new life, one where I wasn't being pulled in twenty different directions and had no responsibilities.
It passes. It really, really does. By about three months in things were different again, by six months it was all good. They're now 4 and 2, they have their moments but mostly it's lovely. They play together and they really do love each other, DD tells me every day that DS is her best friend and DS is always shouting for her to come look at this/let's play that/I need to show you something awesome (usually just before he leads her into trouble!).
We've recently had DC3 so I'm back in that sleep deprived, hormonal, knackered stage. I've had days where I've thought to myself "what the hell have we done?!" and then I feel like shit for even thinking it. But I keep plugging away because that's what we do, we keep going and things change and they get better.
Just keep going, it won't always be this hard.
If you're anywhere near me I don't mind sitting with your baby while you have a shower or a nap or even a hot cup of tea, I'll bring my baby, they can have a play date :)