I'm doing this too, EWCI. I have hypermobility syndrome and early onset, multi-site osteoarthritis, and CFS and fibromyalgia have been suggested too. :( I am now on the waiting list for a total hip replacement, probably in May, and I know my recovery will be much better if I am fitter.
I can't exercise at all on dry land. I have a blue badge because walking at all hurts so much. If I really, really push myself I can walk up to about 1km in 2 hours, with rests, but then I am crippled for several days.
Instead, I am combining swimming with sauna/jacuzzi. I discovered that heat helps my pain and helps my muscles relax. I do a sort of 'heat sandwich' - i.e. A bit of sauna/jacuzzi, then swimming, then more heat.
I started in October, and without heat, I could only swim about 4 lengths. With heat, I could immediately double this to 8. At first I went twice a week, then 3 times, and I have now built up to 5 or even occasionally 6 times a week. I have also built up from 8 lengths to usually around 20, although I have 'bad days' where I only manage 14, and have had some 'good days' when I have managed 24, and once even 28! I swam over 2 miles last week, and have now set myself a 'one mile (64 lengths) per week' minimum. This feels like amazing progress - up from 8 lengths/week in 4 months. :) It is definitely easier to swim 4x per week than twice, I guess because your muscles get used to the unusual movements.
I swim an odd combination of strokes: normal breast-stroke is hard on my back and hip, so I alternate with lying on my back and kicking my legs, 'cycling' under water, and doing a sort of slow underwater crawl, where I punch through the water rather than lifting my arms above it.
I still hurt, but it is a different sort of pain and doesn't feel unmanageable. Amazingly, in the water and in the sauna, my pain drops so much that that is an incentive in itself: I now have an hour or two, most days each week, when I don't hurt much at all, which is amazing! :) My mental health has definitely improved, and I have lost a bit of weight, which will obviously help too.
Go for it! I'd recommend 'little and often', and I'd definitely recommend the heat method! :)