Hi, firstly I'm sending a big hand hold- it sound alike quite a complicated situation emotionally.
Can I clarify, is that issue with getting the passport legal or emotional? It sounds like you have the legal right to get a passport from what you are saying, but possibly that your family would disagree?
I'm also not quite sure what you mean when you say never wanted it- did you mean you never wanted the responsibility of the SGO order, or that the the family never wanted the SGO order to go through?
I would say that if you have an SGO, your main focus has to be the child's wellbeing, and if that means making sure they are able to access the same level of family life and standards you would give a biological child, then applying for a passport and going on holiday is a perfectly reasonable thing to do. Maybe if you aren't sure about your rights (for example, you can't take them out of the country for more than 3 months without birth parents/social services permission) you could contact social services and ask for more clarification and support with the specific application you want. I believe in theory you can apply for their passport, but may need some documentation that you might not have access to. I did a quick Google and this blog post gives a bit more information:
https://www.crackednailsandsplitends.co.uk/blog/applying-for-a-passport-for-a-child-under-a-special-guardianship-order
Essentially it seems like you may need to persevere and keep calling the passport office until you get through to someone who actually understands special guardianship.
If the problem is potentially causing ructions in your family, it would be helpful to know why you think this.
The final thing I would say is that you mention the child found out he was with you because a relative told you he said they were his mum- it sounds like he might be being confused and some life story work might be beneficial. You can apply through social services (or in theory should be able to if they ever get round to renewing the funding) for the Adoption and special guardians support fund that could help him to access therapy and emotional support to help his understanding o the situation.