He doesn't forget what he says. He just wants you to believe he forgets it. Abusers like to do this. It's part of gaslighting.
Do you work at all?
Any possibility you could rent a small flat? Even if it's just 1 bedroom. That way your older kids could also escape him from time to time.
That being said, the fact that an emergency shelter has been suggested, probably means they think you should leave ASAP. Could you go for a bit and then look into renting?
Speak with a solicitor ASAP regarding divorcing him. Also look into getting proof of his earnings for child support services in future, before letting on you are leaving...eg...if there's any wage slips lying around.
Once you get your share in the divorce you may find that makes renting somewhere possible. If not before. Look into the benefits you are entitled to.
Focus on getting safe and getting the kids somewhere safe to be too (you don't need full custody but they need an environment they can visit for days at a time that is totally free of him).
Be careful to not be followed when returning to the social housing. I know the thought of sharing could be off putting but...you might make new, life long friends. And it'll be good to meet others who know what you've been through. So I wouldn't let yourself be put off regarding sharing a bathroom or even a room.