If you say "I used to go" pronouncing the final "ed", then you're using the pronunciation for the verb "to use" not the verb "used to" so you're the one who is wrong.
There is no such thing as the verb "Used to".
Verbs have an infinitive form, such as "to play" or "to go" or "to see".
"Used to" is a common phrasal verb, using the verb "to use" and the preposition "to". The two main iterations of it are "To be used to something" and "Something I used to do". Examples:
I'm used to my partner elbowing me in the face when we sleep together.
I used to be a rock climber, but these days I don't have time.
The correct pronunciation of Phrasal Verbs is completely unchanged from the pronunciation of their component verbs and prepositions - if you have the verb "to look" and the preposition "out" to make the phrasal verb "Look out" e.g. "Look out for some decent cheese when you go shopping today" then you don't pronounce either of the words any differently to how they're normally pronounced.
It's exactly the same with "Used to". While it's common to pronounce it as "Yoos tuh" rather than "Yoost tuh" it isn't empirically correct. You should pronounce both the D and the T separately. However we're lazy, and so we don't. It's simply common practice, even if it's technically incorrect.
There's no excuse for missing out the D in written form apart from ignorance.
However in the negative form, you don't use the D. It's "Didn't use to" not "Didn't used to" because you wouldn't say "Did not used to".
For the record, I'd say that skipping the pronoun "Am going to the shops" is less of a mistake than skipping the contraction apostrophe in "I'm" which is why I sometimes write the former, but would never write "Im going to the shops".