We had the 'we think your child may need assessing' conversation in nursery for very similar reasons. He would play alone or alongside others but not really joining in, he wouldn't sit for circle time he would be off doing his own thing. His speech was excellent and had an advanced vocabulary for his age but that 'hid' his receptive language difficulties.
I was like you, "no, he's ok" no need to refer until he moved into reception and it was pointed out to me how many 5 year olds can tell you the day you were born, in seconds from your date of birth.
He was referred that day, that led to two years of SALT and OT intervention before he was assessed as having high functioning Autism, and high Anxiety. Nursery are seeing your child in a different setting to you, they are noticing their difficulties compared to their peer group. The sooner you know the sooner the right help can be implemented.
My eldest is now 17, in 6th form and because he had the help and support he needed he is thriving not surviving, he has a lovely friend group and is generally happy. This took years of help and support and without that, from early on he would, I'm sure, have struggled more.
Contrast this to my youngest son. I was asking for years in Primary school if they noticed anything, they said no, he is now in year 7 and struggling, he 'masks' so much better than his brother, he is a quiet, well behaved child and has 'flown under the radar' so a lot was missed. We're currently struggling with school based anxiety and burnout. Luckily we have been able to have him assessed quickly under right to choose so that we can now get more support from school but I wonder if having more support in Primary school would have helped him cope better with the transition to secondary school where lots of undiagnosed children begin to really struggle.
Listen to the Nursery staff, its better to know now than later on.