Gah, this is all wrong!
The centile they are born on is thought to be irrelevant now, because it's much more related to gestation, mum's size, fluid levels and gestational diabetes. They are supposed to take the centile they're on at 2 weeks as a starting point - so it sounds like he was ALWAYS supposed to be on the 9th, he's still on the 9th. There's nothing wrong with that, everyone has to be somewhere on the chart.
The health visitor is using older methods which aren't thought to be accurate any more, and causing you undue stress and worry.
The cold would have impacted weight gain and is totally normal, plus it's normal for weight gain to slow down at around 4, 5 months - you'll see the line flattens out a bit if you look at the book.
Follow your baby's cues. If he's refusing more then he's having enough. It might even be harmful to force him up to the next centile line. I think their advice is dangerous, not to mention upsetting.
Other signs your baby is getting enough milk - if he's alert when he's awake, he's weeing at least 6-7 times a day and his poos are soft like porridge, not hard.
You sound like you're doing fantastically, please don't feel like you should stop or that you've failed.