Our son has been diagnosed with GDD, he is also delayed with his speech, toilet training etc. He is also very small for his age (he is now 4yrs 4mths and still in 18-24mth clothing, only now are some of the trousers starting to look a bit short!)
He was delayed with all the milestones as a baby; didn't crawl until he was a year old, didn't walk until he was nearly 2 etc. At 2 he didn't have any words and only a few very baby-ish sounds.
As our first child I knew he was behind in comparison to our friends children but nursery hadn't raised any concerns and he had been going there since he was 9mths old. It wasn't until his 2 year review that the HV bought up her concerns about how delayed he was and then she asked nursery their views and then they finally said "well actually...", she started the ball rolling with paediatric, slt, hearing referrals etc. His hearing is fine and so is his understanding - we never had any concerns about these! We ended up being lucky enough to be seen by the amazing team at our local child development centre and it was there he was assessed over a month and they gave the diagnosis of GDD.
He has been having regular speech and language sessions over the last 2 years and is now saying lots more single simple words & learnt more concepts like on/under, big/small, but even at the age he is at now, he only puts 2 words together usually and not a proper sentence.
If you have concerns I would speak to your HV, ours seemed to be able to pull strings and arrange things really quickly in comparison to going to the GP. For example I was worried about his eyesight and the GP said it could just be hay fever or "a tick he had developed...although he is a bit young for a tick", the HV quickly referred us to the eye hospital and low and behold he needed glasses!
Ask your HV for the skills matrix for his age range (and perhaps the one below) and then work through it and see how many he can do/sometimes does/ can't do - you may have to practice some of them with him before you can answer them. They are a good starting point to get an idea of if he is a little bit behind where he should be in 1 or 2 areas (and hv might say that's perfectly normal!) or if he is way behind where he should be. My DS was behind in all areas even on the younger forms when I did them for him.
Does your LO go to nursery? Can you ask them if they have any concerns about their development?