Am I going to be flagged as difficult for this?
Nope. They've said it's fine. What you could also do is to write them a letter, thanking them for offering you and DS support and, explaining some of what you've said here about why you find it difficult to access their support right now. And even asking if you could visit them instead.
Because you never know your luck - they might even offer help in a way that meets your needs better, if you tell them what your needs are. (Well we can dream!)
I've also refused some therapy sessions for DS when they weren't practical for us and they assured me that this wouldn't decrease DS's chance of being offered the same sessions again, at a later date. Different services may vary on this.
I was treated for depression as a teenager due to some issues that have now been resolved.
I don't know if it's any comfort but I've had more therapy than you can shake a stick at, as a teen and as an adult. When DS had his first interview with the clinical psych for an ASC assessment the psych started asking about family background, I started telling him, and he airily waved a hand and said "ah yes, potty family syndrome". I laughed and agreed. I didn't even need to carry on. And DS got his diagnosis and nothing was "blamed" on me. Every now and again the psych did ask how I was coping and checked that I wasn't in a bout of unmanaged depression, but that was fine. The clinical psych was also very approving that I was seeing a counsellor myself while DS was being diagnosed. I'm not being dismissive about your fears but what I mean is that the professionals do see these things regularly and they - well, at least some of them - do understand.
And I'm really sorry about your bad experience in the past. Of course that will shake your trust in the profession, and if you are already anxious that would only make things worse!
One thing that strikes me is that you do sound quite anxious, and even if the anxiety does reduce temporarily, it might be good to get some long-term help for it, because it sounds as if you have long-term social anxiety issues which aren't just to do with the prem birth. And it also sounds as if your own anxieties might be interfering with your ability to access help for your DS, at least on this occasion.
And the more open you can be about what's bothering you - by which I mean, not saying "nasty professionals intruding into my home" but saying "I'm anxious and my family have a lot of additional needs which I'm working flat-out to manage and I'd really like a different kind of help" - the more likely you are to get the right help.
Wishing you and your children the very best 