Of course you will be worried, and the logistical reasons you mentioned are valid. Space etc. The baby is tiny and will stay tiny for quite some time, so there is always a chance that at some point you can move into somewhere larger.
Pregnancy could be an opportunity to really invest in yourself, you sound exhausted and worn out to me. I think now is the time to start to make time for yourself, you sound like a wonderful mother who really works hard to keep her children happy and well, but you also need to look after yourself.
Finances are clearly important, but could you start with warm baths and candles, and massages your husband could do them for you every night that time is just for you and for him. Join a weekly pilates class to meet other new mothers and also to enjoy looking after your body and your bump as it grows.
Perhaps going out for long walks on your own, would clear your mind and give you some space.
It sounds like you are still in shock finding out you were expecting again, and so possibly you need more time before announcing it to everyone around you, but when you do make sure you do so positively. I think you will be surprised how many people will be delighted for you. Pregnant women are truly beautiful, at their best, but you will never feel that way unless you start to really prioritise yourself.
I did go through a stage in my life like this, and once I started eating well, making time to relax, to spend with friends. I felt A LOT better. Can you not treat yourself to something beautiful to wear, or even a blow dry? Or both?
Being a good mother means looking after ourselves. Truly it does. So go to the hairdresser, go and buy a flattering maternity dress, and enjoy your day and book a film with some friends one evening. Talk to your husband and GP if you feel any worse. Good luck!