Ok this is where I wheel out my Harem theory.
There is an instinct in men - particularly strong in alpha or beta plus males - to 'conquer' as many females as possible. Some religions even formalise and promote this - or require it! - in law.
So in some societies it's the respected norm. And in others, where it isn't legitimate, it is mostly considered morally repugnant. However, the fact that it has been pushed 'underground' doesn't make this instinct go away.
These men like to feel they have a number of women under their belt, as it were. Woman who favour them, trust them, love them. But going around having physical affairs, etc isn't practical, or even necessary. The men can gather their personal harem just by engaging with women they encounter and like. It doesn't have to be an actual bending over the couch - the look of acquiescence in her eye, the eager text message reply, is enough. He knows that she adores him, that indeed he's her rock, that he's her go-to confidant, that he's the first and probably only person she thinks of when she goes to sleep, and again when she wakes up.
She is safely and willingly in his virtual harem.
Just look around you. It's going on everywhere. For example - there is a very very charismatic guy in one of our local small supermarkets. He has a quality which somehow commands immediate adoration in women. He is very polite. Respectful. Sensible. Helpful. Charming. Handsome. A man of substance. And I've noticed how all women change when they talk to him. Look into his eyes and you can see he's noting 'ah yes, come on in, join the gang ...' - he's got you. I was intrigued when I first saw him, disarmed, wondered where my sudden rush of liking for him came from, and by the second encounter was resolutely in his harem. It is just an acknowledgement, only in the eyes. The eyes of every woman that enters the shop!!! 😂
This is going on everywhere. But it is more elaborate and involved in some cases.
As with your situation, OP. So, this guy has successfully colonised you. You're a favoured new member of the virtual harem.
Yes, he does rate you, does like you, does want you to be only his. But he knows he can't actually set you up in a tent in his garden. So he's flagged up that he isn't going to run off with you. He wants you in his life as you are, and his wife. He'd probably add more to his harem if the chance arose. May even have more. He's put it on the table: you're welcome but you're not to make trouble or want more.
For some women, that suits their circumstances. For others, it's not on.
Think about it. Either way, your husband doesn't seem to feature at all. But as you sound really quite vulnerable, maybe you're better off retreating to the relative comfort and security of your marriage and just deciding what you want and need. If it's this guy, you'll have to accept the 'friendship' (aka being his virtual second wife) is not ever going beyond what it is now.