Can relate to a lot of the OP....also had fertility tx, and 4 miscarriages, before having dd1 and felt almost disloyal having another. Equally, and partly because of this, felt it would be great for her if I did!
Anyway...for myself, I half wanted another and half didn't, and am an older mum and worried about that too, etc etc. Finally decided to try just once, and now have dd2 who is nearly 22 months and utterly beloved.
dh can testify to my ambivalence throughout pregnancy (mostly to do with anxiety about coping, and anxiety about not being as much 'there' for dd1 as I had been) right down to driving to hospital saying 'oh god I'm not sure about this '.
Agree with all thats said about second child being different, and the only way I can describe it is this: when dd1 was born I burst into tears of wonder and said 'oh my god its a baby' - when dd2 was born I shed a tear and said 'oh my god, its you'.
For about a year or so after that, I still wasn't 'sure' whether we were right to have a second child, or whetehr I could cope (she had health issues to boot), but that doesn't mean I didn't love her throughout. And now she's nearly two and I'm so, so glad we went ahead - dd1 too, she adores her (and loathes her on occasion, but not for long ).
Good luck deciding, there are no rights and wrongs and its not easy...maybe ask yourself how you would feel in five years time if you a) didn't try and b) did and succeeded.