Where I'm from he would be referred to as the "fancy man" and she would be referred to as the "fancy piece". 
I have only associated the term Mistress as someone bonking a male rich guy, who puts up a woman in a nice apartment where he can have unlimited sexual access and dinner, and not have to worry about booking a hotel or paying for extras.
I imagine he's calculated the long term expenditure and ease (apartment, clothes, food, holidays) versus the short term cost and hassle (high class prostitutes, 5 star hotels, risk) and chosen option A. I also always imagine that there is some kind of sexual function mistress is willing to perform - I.e. nappy changing, spanking, rubbing bollocks on a cheese grater, that his upbringing would never demand of his wife.
To this I always think of Joanna Lumley in that "Chips and egg" film.
The rest of us, should we enter into sex without but recompense of lust and love, would be OW.
If I was obscenely rich I still wouldn't put up a gorgeous young man in an apartment for my bidding. I'm too bloody tight. I might get a tennis instructor, gardener or regular masseur as there's a tremendous sense of value for money in that.