This is going to sound harsh and it's not meant to be but blaming other people for our choices isn't the path that moves us forward. You weren't robbed of having more children, you chose to stay in a relationship with a man who didn't want more, that was a choice. You gave/ give him power by suggesting that you didn't have any, when you did. Staying in an unsatisfactory relationship is a choice, your investment didn't pay off and that can be incredibly disappointing but it doesn't mean it was a waste.
I ended my relationship with my Ex after he assaulted me, I should have ended it years before but I was stuck waiting for the person I wanted him to be to show up, instead of accepting him for who he was. The saddest part, if it wasn't for the assault I would have stayed longer. In a bizarre twist, being momentarily powerless actually made me stronger because in my moment of powerlessness, my survival instinct kicked in.
Everyone has their role to play in a relationship dynamic and what I learned from my relationship post-mortem, was that the answers to why I stayed/ hid in an unsatisfactory relationship for so long, were in me. The assault and his crap behaviour that followed the assault were on him, despite his best efforts to make me culpable. The truth was I was more invested in the relationship than the person I was having a relationship with and that enabled me to overlook his many shortcomings and our incompatibility.
I was with my Ex for ten years but I don't consider it a waste because if we hadn't of been together, I wouldn't have had my awesome DD or my new found general kick-assness. I did love him once but love is not an excuse to tolerate unhealthy behaviour/ relationship dynamic.
If you want to move forward, you first need to accept your behaviour in your relationship dynamic and learn from them. The 'power' this man had over you was surrendered to him and the good news is, you can take it back anytime you like.