I'm not a SALT, but my 3 yr old has four words and we've seen two salts, for two blocks of therapy.. We're well an truly in the system thank goodness.
So, as a parent and not an expert, I'm just giving you my impression from what you say about your son.....he clearly has the inclination to communicate, and the urge to communicate is a very important factor. He already has in his head a 'label' (or word approximation) for a lot of nouns/words which shows he has gone beyond a certain stage of development. For ages my son couldn't attach a photo of something to that item iyswim.
So in that respect I can see what your friend means.
But from all the questions and tests we've had (baylee/griffiths/chat you name it) I think your son would be considered a priority if at 3 yrs old if his speech is unclear to everybody but his mother! He'd definitely benefit from SALT.
Take him to see a doctor as soon as you can. Probably, by the time you are refered, everything will have fallen into place. It can take a while to be refered, but just in case, I'd get the ball rolling if I were you. Afterall, if he has great speech at 5 he'll be away and flying at school, but if his speech doesn't 'normalise' 'til he's about 7, he could act quieter at school than he really is. It'd be better to act now really.
I was lucky because my son's late speech was picked up at 18 months and it all rolled from there, without me having to go looking for help.