Speak to your GP expressing your concerns and ask for a referral for Speech and Language Therapy.
I am a retired SaLT myself, but don't want to offer any advice other than that because I can neither hear your son, nor examine his speech apparatus so I can't say anything more than others are suggesting. (And I specialised speech after stroke and head injury, so am stretching into the far reaches of my memory - but communication is too important to leave to chance. Also if you get anxious, he will pick up on it and it will make him anxious, too.)
In my experience HVs are rubbish at anything to do with speech and language unless there is an obvious autism (Just don't get me started. No. Really. Don't.) and the tiny example you have given suggests that you have nothing to worry about on that score.
As Wenda says, it could well be a hearing problem (HVs are useless at picking these up, too - the distraction test they use is a complete waste of time.) Or it may be that he has a small tongue tie or even a palatal cleft (some are tiny - just a minute hole in the hard palate, but it is enough to make free and easy speech difficult).
Has he any problems (now or in the past) with eating and/or drinking? Is he able to "pucker up" for a kiss? these will give some clues as to how his speech apparatus is developing.
He is likely to cope when he gets to nursery, but he really needs to be sorted out before primary school, and although some problems "grow"out, many e.g. lisps, don't (not suggesting that he has a lisp BTW).
Velvain
HV said she no longer met the the criteria for help
Frankly, I have never heard such shite - but it doesn't surprise me. HVs aren't the most helpful individuals in my experience. You are right to suggest private therapy if you can afford it. Speech and Language are incredibly important, for, as you so rightly say, social acceptance and confidence as well as daily communication. And the older a child gets before any difficulties are sorted, then the more likely s/he is to be teased and bullied, and then they won't want to talk.
I would like to point out to everyone who will be complaining about me HV bashing, that I am speaking from my own experience, and that of friends who are still working, many of whom work in paediatric SLT, and that despite what you may think, no Health Visitors were harmed in the making of this post.