OP, you're going to get some very projected responses on this thread, just so you're aware of that. OW are not generally tolerated here because many women have been on the receiving end and blame the OW as much, if not more, than the errant husband/partner.
That said, I know what your situation is like in reality and it sounds very sad indeed. I'm not going to talk about the risks of discovery because they may or may not be realised but I'll talk about how you're feeling because that's the bit I can relate to.
So, your situation sounds like a 'frying pan to fire' scenario to me. The attention and reciprocated desire that you wanted just isn't there. There's a reason why he 'wins you around' and that is that OW are not as commonplace as someone would think. I mean this kindly but, you could be any woman. The only difference was that many women would not 'go there' and you/we did. I know that's a bit of slap in the face but it's probably something that you realise deep down anyway.
An affair is not going to give you what you need/want here. Your only real salvation is to think about whether you want your marriage to work and if you do, put some energy and attention into it. It's not ok for your husband to be inattentive/not want you (and vice versa). You two have a contract - and children - so you will either need to make it work properly - or end it. You're using the excuse of finances but it's not real because, if your husband left you, you'd have to make it work somehow. You're entitled to end your marriage if you want to; only you know how difficult and intolerable the situation really is but at the moment, you're distracting yourself with an (unsatisfying) affair and that's actually stopping you from taking affirmative action to sort out your life.
Going back to your affair... OM is using you; even worse than that, you're allowing him to. Yes, you're both being extremely selfish but he's not even giving you the feelings of attention that you crave and that's still acceptable to you. Just how low is your self-esteem that you would allow somebody to be so dismissive and cavalier of you? He takes what he wants and leaves you and your feelings by the wayside. That is not ok.
I do feel for you, I know what it's like but this is not a solution for you. You hate it because it's making you feel crappy and so it would anybody. You have absolutely nothing to lose by ending this 'thing' with the OM because it's doing nothing but reinforcing the message that you're unlovable and undesirable and that's very unhealthy.
I think you know that he would end it in a heartbeat and walk away without looking backwards if his marriage were in any way threatened... that is a very real prospect. How would you feel if your husband found out and ended your marriage, with all the upset to your children in the wake of that? How badly would your esteem take a hit?
The OM is really not worth it. Your marriage might be worth saving. One thing that would give you an immediate boost to your sense of self-preservation, protection, and self-esteem would be in ending the affair. It's not giving you anything, it's taking, just taking - and the end result could be something you really didn't want. Don't let it get that far. For your own sake, end the affair.