To be honest, I think it sounds like you have lost trust for your dh. He abandoned you when you were at your most vulnerable, many women comfort themselves that when they are ready they can try again. Your dh just upped and left, taking with him any hope you had for another child.
You have been working tirelessly, building up enough financial stability, to afford you to be a SAHM after the birth of your next child. Your life has been carefully planned around the choices that you made together years ago. Your husband has been dishonest with you, if he had been more open years ago then you could have planned things differently, allowing yourself more time at home to enjoy your daughter whilst she was small.
Your husband obviously left for a reason last year, has he opened up about what that reason was? I would bet money that he knew before returning to you that he didn't want another child. Has he even justified his reasoning as to why? You made plans for a life together, promises to each other and he has just pulled the rug out from under your feet!
I would be very angry, not so much for his deciding that he wants only one child. I would be angry because you have been saving and planning for this for years. It's like you got engaged, saved up for a wedding, booked a venue, saved the date, then a few months been the wedding he turns round and says ' by the way, I forgot to tell you, I don't want to get married, I still want to be with you but I've decided I don't want to marry you.' You would be fuming and rightly so, ignoring the meaning of not wanting to marry you, as this is more about the fact that you have been working overtime for years. You have picked a venue, saved the date, what's more you haven't had a holiday together in two years! You have been saving for this wedding (which he said he wanted!), you could have had lovely holidays, spent more time with each other, you could even have decorated your house, put in a new kitchen . Or you know, you could have cut back on the hours, hours that left you exhausted, hours that took you away from your family and friends.
If you had come on here telling us that your dh had done the above, you would have been getting a whole lot of LTB's. I am angry for you, he has known that you have been working towards your plans for a long time. I would be telling him he can work the overtime, he can support you so you can spend the precious early years with your daughter. Find your part time job and enjoy your little girl. I would also have a long talk with your dh, what else has he changed his mind on? My Uncle decided he was leaving my Aunt for someone else the week she retired, they spent years planning their retirement, now she is left alone and he has swanned off to another country with the ow. I would want an honest conversation, is it that he doesn't want a second child, or is he too cowardly to tell you that he doesn't want a second child with you.