Where you are now, you are kind of choosing between taking a risk now that you have kids but won't enjoy being a parent, and taking a risk later, that you will feel surer of what you want, but if that surer is surer you do want children, there is an increased risk it won't be possible. If that isn't too garbled?
How awful that would be varies for different people - some are just willing to see what happens, others feel devastated. I wonder if it's possible to sense in yourself which of those you would be, and which of those your DH would be.
I think the fact you are waking up anxious about it suggests you ought to talk it though with someone who will help you get to the root of what is making you so anxious.
No idea if this perspective will help, but here's my story: I had my 2 pretty late (40+), mainly because opportunity didn't really present and I wans't looking for one - no-one really tried to talk me into it until DH - who I only married at 40. I still wasn't convinced, but wanted to stay with him, realised it would be an adventure, and that I would never really know one way or the other (perhaps that makes me a bit odd, or perhaps just honest - I don't know!). It kind of felt like throwing myself off a cliff - no going back and all that.
And now? I adore my children, they have enriched my life hugely - but they are leaving home now and that is going to hurt. As an anxiety prone person I have had very severe bouts of anxiety related to them (as well as to other things) and worry about them a little, most of the time. And I worry about their futures too, with rising fascism and irreversible climate change, for example. So in a way I also kind of think I shouldn't have had them, that having them was self-indulgent and it will come back to bite me if/when things go wrong in their lives.
I suppose what I'm trying to say is that you don't have to have children, or even to want them - not everyone does. And certainly not everyone needs to have them. And even people who do want them, may not be devastated if they end up not being able to (though of course some are). My sister wanted children but it didn't happen and she really seems at peace with that.
Good luck, and remember there probably isn't a simple, "right" answer, and even if there is, there is no way to be sure you have it!! Life eh?