The other mother should have invited you as well, and made it clear, if they weren't prepared to have your DD unsupervised.
Fwiw, I've sent my nursery age kids unaccompanied, when reliably toilet-trained and if not too shy. If the other parent is happy to pick them up, give lunch and let them play for a couple of hours, then why would I decline? I'm on the end of a phone if needed to come early. Never had any problems.
And similarly I reciprocate with those who are happy to have me look after their nursery-age DC for a playdate. If it's the first time, I say "Would So-and-So like to come and play, I'll make some lunch. Would you like to come over too, or will they be OK on their own? I'm happy to pick up straight from nursery if you think they'll be OK with it, but you're more than welcome to come too if you prefer."
Crystal clear. That's how it's always been offered to me too.
OP, I think you've had an undeserved kicking on this score.
However, I would say that if you decline the cup of tea, and don't make the effort early on, then why would they include you at a party? It's hard if you're not comfortable socialising, but a little bit of effort goes a long way.
Do some return invites, and offer them the chance for lunch or tea if they'd like, or for you to pick up.