Right. Lets start with a disclaimer - you need to do what you think is best - I'm not a medical person this is just my opinion, but ...
While it s possible as Fluffyforlife says to get pregnant before the man comes, from little bits of sperm which are released (or so I've heard), I always thought this was only just before he came (please correct me if I'm wrong people.)
The chances of you getting pregnant from him just being inside you for a minute are IMO very, very slim indeed!
I regularly used withdrawal as a method of contraception with my ex for 10 years and I never got pregnant (both of us have gone on to have kids, so we were fertile).
And we're not even talking classic withdrawal here (where the guy pulls out just before he comes). Here (it seems) he wasn't even near coming. Personally if it was me I wouldn't give it a second's thought and the morning after pill wouldn't cross my mind.
But perhaps you are both super fertile. (Worried that you might be back here in a month saying "Oi lowrib, i'm pregnant!" But really because I'm more worried about sod's law than the actual chance of you being pregnany IYSWIM!)
To be honest I'm more worried about your attitude to the morning after pill. For it to be effective, you need to take it as soon as possible. You can't just wait till the 72 hours is nearly up to make up your mind - the chances of it working go down as time passes, and if you leave it till the last minute you may well take it and find you are pregnant anyway. (May mate's 2yo DS was conceived this way).
If I was you, I'd keep the Drs appointment and use it as a chance to talk over your contraception for the future. It sounds like you've had a crappy time with contraception and finding one which suits you. I'd be honest with him/her about why you think you might be pregnant and go on his/her advice.
One last thing - are you breastfeeding? Have your periods come back yet? Again, while it is possible to get pregnant while breastfeeding, if your period hasn't come back yet, it is much less likely that you have conceived.
HTH. Try not to worry too much!