Have you read this:
www.tommys.org/toxo?gclid=CI2lg8T_orECFcgntAodU2rgsQ
I had toxo when I was pregnant but did not realise at the time. When I found out, I called Tommys for advice and, if I remember, correctly, they said 99% of cases are not due to domestic cats but due to food contamination and their link above reflects that I think.
I too had a cat but she did not have a cat litter so I was not handling cat faeces and we had a patioed garden so I was not infected because of gardening. I suspect I contracted it on holiday when I stayed at my Mums finca in Spain from food that wa not cooked through properly.
Anyway, perhaps investigate for yourself the likely causes to put your own mind at rest with regards to keeping the cats or not. Whatever you decide to do, you are just making the best decision you can, there is no right or wrong.
More importantly, I am so sorry this has happened to you and your baby. I really do empathise with your situation as I have had a baby with an abnormality, a premature baby and then my daughter from the pregnancy affected by toxo so I feel for you on all levels. I would imagine the worst thing is the not knowing. Tommys do have a leaflet about toxo in pregnancy (causes, symptoms, treatment, explanation of fetal monitoring etc) which I can post here or email you if you want?
Be kind to yourself and if you want to 'talk' then PM me xx