People saying 'it's just a cat'... some cats are bastards!
My mum had an indoor cat. Had him from a kitten, he had no trauma. He was never mistreated, he was loved and petted and treated well at all times.
He was a total mean little fucker! We used to call him the attack cat. He'd come to you all sweet looking for a fuss, lie down for you to rub his belly and love it for a few minutes. Then all of a sudden he'd wrap all of his legs around your arm, dig his claws in/scratch and bloody bite you! I've got scars!
He'd also sit on the sofa and swipe to scratch you as you walked past, and wait outside bedroom doors to pounce and bite your feet when you emerged.
Nothing helped - he was neutered, he had all the sprays and plug ins in case he was anxious, my mum tried everything to make him nicer. Nothing worked, he was just a little shit.
No one was bothered when it was the adults going to her house because we could look after ourselves. Once I had my babies though I insisted that my mum shut the cat upstairs when we visited. She didn't like it - it too, was her 'baby' and 'he'd never hurt little ones he knows better they can sense it'.
Told her that was fine, we wouldn't visit at her house then, and I meant it. She could come to us instead but I wasn't having my tiny kids around that feral beast.
She did shut him out in the end for grandchildren visits (were local so it was never overnights) and he lived to a ripe old age very happily.
OP your SIL is batshit. I don't think you're OTT, a 2.5 year old can be starting to learn 'leave the cat well alone' but a cruising 11 month old can't they're just a bloody baby! Yes of course you'll need to be watching your children regardless - I'm sure you do that at home too - but unless the cat is either not there or shut away you're going to spend your visit on pins.
Shame your DH is so wussy. So time for you not to be - just tell them (including your DH) no. Should you have to? Nope, DH should. But he won't, and these people are your family too, so you'll need to. Either the cat doesn't go (or is shut away sensibly) or you're not going. It's sad that your SIL doesn't have children if she wants them, but regardless, she and others don't get to put an animal over a tiny child's welfare. It's mental.
I love cats btw. I even loved my mums utterly psychotic one, I cried when he had to be put to sleep when he was 15. But I'd still choose keeping babies and toddlers out of harms way over any pet.