Hi, I've really struggled with anxiety in the past so I do understand why you're worried. My advice would be: Firstly, put the Doppler away. It's really not recommended that you use one at home, and while it can tell you there's a heartbeat, it's not the best way to check on your baby. If you are really worried about your baby's health, talk to your midwife and maybe go in and see her.
Secondly, the baby is really well insulated from stress. Inside you they are well protected from all of that, and your own body will protect them from the vast vast majority of things that are upsetting you. Put it this way, I had norovirus and was sick as a dog for nearly a week at about 30 weeks, I was worried sick, and according to all the checks and scans I and baby are both fine. But again, if you're worried, please talk to your midwife. They can refer you to the maternity mental health team for some extra support if needs be, and there is no shame in taking advantage of that.
Finally, you need to talk to your other half in a calm and sensible way. He's not being very supportive at the moment, but may well be finding your anxiety rubbing off on him and not coping very well. Sit him down and talk, and try and find practical ways you can manage this time together.
Good luck, and please do talk to your midwife, they really can help.