As I struggled to conceive my second pregnancy and I remember looking at DS1 and feeling a yearning for another baby, and guilt that in some way DS1 might feel he might not have been "enough." it was upsetting and my desperation drove me to more IVF. (Which worked, thankfully.)
But nothing nothing nothing, for me, could ever compare to the utter misery, heartache and terror I felt when trying to have DS1. I'd wake in the night, drowning in panicked sweat, shaking that my parents might never me grandparents, that my brothers might never be uncles, that my husband might stay with me out of pity, or leave me for someone who could give him a child. That my friends with children would think my life was self indulgent and irrelevant. That my friends without children might think this was a choice we'd made. That the fucking burning ghastly RAGE at every single woman who had conceived easily and naturally, might consume me totally and sour every fibre. That we'd run out of money, out of time, out of patience and affection for each other. That all this impotent love I felt, would never have anywhere to go. And that I would never be a mum.
I thank God, and modern medicine that IVF worked for us both times and now we have 3 children. But I still feel sick to the guts when I remember what if was like Before. So, in my experience and opinion, YANBU. Not a bit.