YANBU - children are very hard on a relationship if you aren't both 100% committed. Not to done if you're not sure.
Lots of people have taken the decision not to have DCs, but these people are therefore not PIL to anyone - and often don't socialise or live in child friendly places to friends with people like your PIL. Your PIL also will be from a generation where fertility treatment wasn't as available/successful so are more likely to just assume those amongst their social group who don't have DCs weren't able too IYSWIM.
Personally, I'd talk to your DH about it again, be 100% certain he doesn't want them, then get him to let it slip to his Dad that he thinks he can't have DCs, that will probably shut your FIL up for a while, which is all you want (don't let it be you that officially has the problem, as then you might find your FIL starts pressuring your DP to go find a woman who can have DCs)
But yes, be very certain your DP doesn't want DCs, you have only a few years left of being able to have DCs, he's easily got a couple of extra decades. I know a few dads who had DCs in their 50s, and their 30-something DWs were not their first long term relationship. It might not be a deal breaker now, but it might become a deal breaker in his 40s. Best to be certain his "I don't mind either way" doesn't mean "I don't mind either way now but probably will want them at some point." (If that's the case, best to leave him now rather than wait for the inevitable)