Barnier's comments were made at a meeting in Nîmes yesterday, and I've not actually heard exactly what he said. A lot of commentary is secondhand.
Le Monde is reporting that "Brussels" is stunned, though, so Clav isn't alone in struggling to dissociate Barnier from his role in the negotiations.
Barnier brought out a book this summer, which I've not seen and which didn't get much media coverage so I'm not sure of its exact content. The only info I can find right now on his immigration ideas is in Le Figaro behind a firewall which I'd cut my own ear off before paying to see behind. In the bit I can read, he clearly refers to terrorism, Islamic State, etc. so it's probably not FOM per se he's referring to.
Without being anti-FOM, he may perhaps be in favour of more screening and increased border controls - and the court case brought by victims and relatives of victims of the Paris terror attacks that killed 130 and wounded 350 in 2015 has just begun, so some French voters may be looking with concern at possible repercussions of events in Afghanistan. Also, those terrorists skipped back and forth a lot between Belgium and France, as well as the UK, so that may be a factor in some voters' minds, too.
However, as far as I can tell, Covid has slightly shifted issues like that off the radar, I think. The health minister who was in the post when the pandemic began has just been in court, too, facing charges of endangering others' lives and "knowingly failing to take action to fight a disaster" (my loose translation). Other key political figures are being investigated as well by the court that looks into the conduct of those currently holding ministerial positions. Not something Boris et al. have to worry about, obviously, as they handled it all so well.
None of it is simple, however satisfying a few gotcha c&p's may appear.