I totally understand this. I don't think it is cold to say you would never have started a family with him had you known this to be the situation from the outset.
If, in the early stages of the relationship, he had said "I only want one child, I would never consider having more than one", then there's a chance to consider, before you're years down the line, whether this is the man for you in terms of shared goals and whether you love him enough to want to make a life together even if you have to compromise on your own dreams - or whether, actually, what you want from life is too different to make this a viable relationship proposition.
The problem is, in most cases, we don't find out these things until we are at least more emotionally involved. And sometimes people change their minds (as they are entitled to do).
And sometimes, people string other people along until they are confident enough that the other person won't leave them when they find out that actually, they don't share the same dreams and ambitions. "Oh, actually, darling, yes, I know I agreed it would be lovely to have a dog but I actually hate dogs and wouldn't ever be willing to have one." (OK, bit of a difference but you get the idea!)
I think it does depend, to some extent, on whether your DH has genuinely only ever wanted one child (and you hoped to persuade him otherwise, assuming you were aware of this), or whether he has changed his mind.
If he always knew he only wanted one - or didn't want kids at all but thought he'd concede to having one in order to keep you - but nodded along to the idea of two/three/four kids, that is something I personally would find it very difficult to move on from. I would feel deceived.
I guess it depends on the extent to which you would be willing and able to leave him and break up your family unit in pursuit of IVF, or a second relationship. And whether you, as a couple, can move on from this.
I wouldn't judge you if you couldn't get past this. It is a huge thing. It's not like not having a third or fourth child, in my opinion - there is a difference between one child and "children".
Especially in view of your age. After all, if he wants children in ten years time, he could remarry. You won't have that option. I think this is why this dilemma (which, sadly, is not uncommon but rarely discussed) is so hurtful.
Obviously you take your marriage vows seriously. And obviously your first responsibility is to your existing child. But this is a huge thing.