In answer to your questions:
My first question is AIBU that this not appropriate to say especially if you don't know someone.
It's a tricky question. On the one hand we shouldn't treat SEN as if it's something shameful, to be hidden away and never discussed. And if it's socially unacceptable to ever speculate on if someone might be on the spectrum then it's basically saying it's a taboo subject.
On the other hand, I remember telling a mum that my DS had been diagnosed with ADHD and she was very forceful in her opinion that the entire family was ND. I was angry and indignant what I felt was intrusive and not true, so I do empathise, I remember that feeling well. FWIW, many years later I suspect she was right, though.
My second question is should I be taking notice? My ds was observed at nursery for SEN when he was younger as he used to play alone but this sorted itself out and he has lots of friends now and nursery said no concerns or referrals needed.
I'm sorry to say I wouldn't take the nursery's lack of concern to mean your child isn't ND. What they mean is, he behaves well and gets on at nursery well enough that they don't need any interventions in order to be able to cope with him. They are not experts, however, and a lack of concern is not the same thing as going for a diagnosis and being told by a suitably qualified medical professional that he is NT.
I delayed pursuing diagnosis for my DS because teachers in years 1 & 2 AND the SENCO told me they had no concerns, that I had nothing to worry about, even though my gut was telling me something was up, and we have ASD in the family. I trusted them. They had met lots of DC and I assumed they knew what they were talking about. But they didn't.
When we eventually did pursue diagnosis - with the support of his year 3 teacher, he was indeed diagnosed with ASD. It was a stranger, however, who spurred my to get a diagnosis - a nurse at a hearing test appointment. She asked if he has SEN, and was surprised when I said no, and suggested I get him checked.
None of us can say if your DS is on the spectrum or not. I would suggest keeping it in the back of your mind, it may be that the mum did recognise something that indicates ASD, annoying though her approach may have been from your point of view, just in case. He either is, or he isn't, you'll find out in time.
FWIW, my DS is now doing his GCSEs. He's a bright, friendly, academic, quirky child with a big personality and a strong sense of fairness. I am so proud of him. I suspect he will struggle with some aspects of personal relationships, but I also expect he'll find a job that uses his neuro diversity to his advantage, just as my ASD mum did. She's crap with small talk and lots of aspects of our relationship are tricky, but professionally she was very successful in her field (science related).