You're absolutely right, it's not a nice decision to have to make and it's not one that anyone else can make for you because ultimately it's your responsibility.
Personally I would be looking at the rest of our lives, mine, DH's and DD's and making the decision based on that, however, when emotions kick in, it complicates even the clearest of plans.
If you go ahead how do you feel DH will be as a new father again and will he support you throughout the pregnancy and beyond? or do you feel it would put a strain on your relationship. I mean your personal relationship not your family and parenting one.
Go to the appointment and talk things through with a view to reaching the right decision for you and for your family, don't be swayed by what you feel you should decide either way and don't forget that although this exceptionally personal, you're not in a unique position, women are making this kind of decision every day and carrying on whatever they decided. You'd still mourn if you have a scan and find the pregnancy isn't going to continue anyway, I'm afraid it's always going to tear at your heartstrings either way
Iwish is right, it's a lonely place to be right now.
By the way I hope the above doesn't come across as harsh as it's really not meant that way, I wanted to say it's meant as prosaic, but that's not the right word. I mean not judgmental and more matter of fact or something like that.