What on earth is this obsession with pools and spas?! If I was in OP's situation, I could imagine passing a couple of hours with a massage in a spa, but that's it for the week - it's not something that's going to fill the time. And going for a swim every day, no thanks!
Loads of ski resorts have very little else to do if you're not skiing. Some are purpose built and are a long way from 'real' places; some are purpose built and have other activities built into that (but generally pools/other sports). And some are lovely little towns, which might have enough to occupy you for a day, but I have never been to a ski town where there's enough to do to keep you busy. You could get public transport to other towns etc, but I can totally see how OP maybe doesn't want to do that.
Yes - there are coffee shops and bars, but how much fun is going from coffee shop to bar, by yourself, on repeat for days on end.
Yes - you could get a gondola up the mountain perhaps, and meet them for lunch, but that relies on them wanting to have lunch at/near the gondola station (whereas most skiers might use that as a start point for the day and be miles away by lunchtime).
I totally get you OP. A ski holiday without skiing would be grim.
But your DH needs to get over himself and go alone. Skiing with couples isn't like being on a beach holiday with couples. We regularly ski with one half of a couple (a DH of a good friend of mine); and last year we skied with a single person. In a group of skiers, there are often splits according to who wants to do what/go where, and I'd be amazed if that always involved each couple being a single entity. We often split into boys/girls, or people that want to do a certain run/people that don't, or people with tired legs/people who want more. If you know them well enough to go on holiday, I really can't see the issue.