OP I was in a long abusive marriage too. I am still digesting some of the things that happened in my marriage, and that was over 5 years ago. I am not saying someone can never go into a relationship after abuse, but some things that you have posted seem to spring out to me. Firstly, you started seeing him really quick after the end of your marriage, other pp's have mentioned this.
But the other thing that stood out to me is that he isn't available 100% of the time to you. He is available for you when you need him. I bet if you don't fancy having a chat, then you just don't have one, and get on with whatever it is you would rather be doing. So, you get the attention you crave from him, but only when you need it. I bet you plenty that even if you met this man and he lived right next door to you, you would not have pursued this quite as much.... as a neighbour would just be too available. This guy is available for your emotional needs, but from a distance, which you are completely in control of.
So, yes there are red flags that stand out to me, but the fact you say you will never marry him or allow him a visa, seems to scream to me that he is a rebound, and you are taking from this what you need. You saw him 4 times in 18 months? Sounds ideal to me to be honest.
Maybe he is just what you need WHILE you process everything else you need to? As long as you are hyper aware that he could very well be a in this for the wrong reasons, you may also just be using him for what you need right now. Which I do get btw.
I fully believe that people processing trauma go for unavailable people. By that I mean married (I am not saying you are, but a lot of people do), long distance, age gaps... just enough distance to not get into a 100% full time relationship. I also do believe the saying that people come into your life for a lesson, and then leave again. It has happened to me and I complete admit that I tend to gravitate towards people who are not completely available to me. My previous 2 (short) relationships post marriage were both with men that work away a lot. It suited me, but when things started to get a bit heavy, then I was no longer so keen.
So, I would not say its doomed as I think you are just taking what you need from it. And I dare say, he say some motive too.
I can understand your older teens not being overly happy about this, but they won't understand the situation for as how I seem to see it as an outsider.