No, OP it isn't bonkers to me, I think it is very natural and understandable.
My kids are adults now but ex moved in with OW when they were mid teens and I won't lie, it bloody hurt knowing they were with ex's partner and him. But I have learnt that it's ok. Because she isn't me and never will be, because I'm their mum and nothing or no one will ever come close.
I wish our marriage had lasted, but it didn't. I am happy on my own, but didn't expect to be. I adapted, the kids adapted. Ex might stay with his partner, he might not. It doesn't matter. Our kids have two parents who put them first, she's as superfluous as she was before she came on the scene.
Yes your husband may move on, may meet someone who is part of your kids' lives. You will adapt. You might meet someone who will be like a dad to the kids, again, you'll all adapt.
You can't base your current unhappiness on future 'what ifs'.
You must never, ever, underestimate your role as their mum. Realise this, and all your concerns are gone.