OP, remember , you only have one life . You can't come back and do it all again . There is no second chance . Yes , you may have agreed to something , yes you made vows , but goal posts do change . That is how we grow and develope as individuals . Live your life in a way that is of your choosing , not somebody else's .
You do have a responsibility to your LO , however as long as you love , provide and guide, don't feel guilty about any other decision you may make .
For what its worth , it has also been decided I will only have one , and I am pregnant with my first . I love my husband dearly , and couldn't envision growing old without him. How do you feel about your DH ? Has some of the magic been lost in the birth of your first ? is it something that needs revisiting and recapturing , without any pressure regarding another child, it may help you make sense of your feelings and your DHs his .
I believe it is very important that we get what we want out of life , that you have no major regrets . You potentially could have two big ones here , a broken family , or not having another child . Only you know which would be the worst , but do take time to work out your feelings , things may change .