Yeah I would go too - to see how they are together and to present a united front. Maggie is right, that is probably the best thing to do to send the message, it's over, no chance etc. BUT and it's a big but, your husband has not come clean and has not provided any evidence it's over has he?
Trust your instincts, when my hubby did come clean he said every time I got suspicious, I was always right, he said somehow I always knew. Your paranoia is probably spot on too.
I feel perhaps the reason he is not disclosing anything is a lingering loyalty to the OW and that is why he is not revealing any details or the truth.
Until he becomes completely transparent about any contact he has with her and shows it to you if an e-mail or text and his replies etc. You cannot start to even try and build up some trust again.
This situation is not healthy for you or your children due to the underlying tensions and open conflict there maybe. He may prefer to put the topic of the affair in a locked box - but secrecy fuels obsession, obsessive thoughts about her maybe intensified by refusing to discuss the affair. Sharing information about the affair, will allow you both to let go. Until you take steps to grapple with shattered assumptions and construct a story about the affair that make sense to you, you will be prone to obsessing until healing is complete.
In an e-mail to my husband who had an affair I included a lot of the above and it was that which finally made him come clean with the solicitors appointment I'd booked as well. I'd got tired of him disappearing and coming home with an excuse every time. So finally got firm with him. That was the breakthrough. It may not be the same for you but I think you need to somehow get him to have a reality check as well.
I'm sure you've been recommended this book already but the Shirley Glass book called Just good friends is very helpful. Until I read this I didn't realise so many things. Its eye opening. You need some clear boundaries now.
Btw the state would take care of you with 3 kids! I should know, I have 2 and I'm separated now! Sure your standard of living maybe a little different but it can be done! Don't stay for money if you don't want to try and save your marriage. That's not a good enough reason to. Kids will pick up on it and it may do them more harm than good.
Anyway it's taken me ages to type this, so I hope the conversation hasn't moved on too much in the meantime.