"Incidentally, I don't think I've ever met someone who was a size 6, let alone a 4 - that sounds awfully small to me!!!*
I am. And I genuinely don't look that small because of my shape. Interestingly, I have several items of clothing that are a 6 that are loose round the waist, bought in recent times, and my old clothes (I'm terrible for throwing things away) that are a size 10, from 15-20 years ago that I can't even do up anymore.
I've got bigger, my dress size has gone down two sizes.
I still wouldn't call myself slim then be surprised at the reaction in the OPs context though because what she was saying with her... "surprised high street fashion could be so hit and miss on sizing and that it was so demoralising ‘even for someone relatively slim like me’."... Was actually quite snide in itself as it's a leading sentence.
As in, "Well, if someone relatively slim like me has trouble, just spare a thought for those poor women who aren't slim like me, ohhhh the trouble they must have" 
I think the other woman was more addressing that, by acknowledging OP isn't less relatively slim, and more relatively standard, so falls into the category that she's insinuating must be really struggling to find clothes.