I used to contact nap. I wake when DS wakes in the morning, even though I naturally prefer to wake up earlier, because he wakes whenever I get up and he's miserable if he wakes too early. I understand.
What I would suggest, however, is letting one of the naps be 30 minutes so you can AT LEAST have a hot cup of tea by yourself each day (don't do housework or work work or life admin, seeing as if you'd stayed with him you wouldn't be doing it anyway).
AND/OR plan to do SOMETHING for yourself at least one evening a week. Meet a friend (or even just call/facetime one), start a crafty hobby, do a free online course in something that interests you, read if you don't much already, plan to watch a film, take a bath, go for a run, etc. I often fall into the "just get into bed and scroll until I fall asleep" trap, and its fine most of the time so long as I also take one evening to force myself to do something different.
When DH isn't working, he takes DS out for a couple of hours to the park etc and I'll meet a friend for brunch or go swimming alone or just stay home and watch a trashy movie in bed. It makes the world of difference. I don't do the same regularly for DH, honestly, but usually every 4-6 weeks I'll take DS and stay at my mums for a few days to a week so that I can see family etc, so he gets all the time he isn't working on those days (he's self employed and weekends mean nothing to his job, but he often takes days or mornings "off" when we aren't there specifically to have "me time" which I support because it is so important!).