Hello. I'd say persevere, but not in a 'I will crack this meditating thing' way, but in a 'it's normal to get distracted by my busy mind and I'm going to be kind to myself' way.
I too suffered back from anxiety for most of my life, but I don't anymore. I believe in the Divine, as a part of all of us, and as something that connects us all. If I had the experience you had, I would have thought it was my inner voice, or it her self, having the chance to speak once I had managed to still my chattering egoic mind.
Have you read A New Earth by Ekhart Tolle? That explains the ego/self thing well.
And don't worry about getting distracted - it's totally normal and part of the process. I wonder if you're in some way telling yourself off for your mind wandering? The best thing to do is to notice your mind wandering - just notice, don't judge; no huffing and saying 'oh for god's sake, I'm doing it again! I'm crap at meditating,' - and lovingly and kindly draw your attention back to your breath.
When we start to learn that we have a higher self, and that the ego is false, our ego doesn't like it. It does everyone it can to protect itself and shouts even louder than before at first. What you've done is, after years of living trapped by your ego, discovered and opened the door to your soul - the beautiful outside full of light and love and truth - but your ego, your captor, has realised and has double locked that door. What you need to do now is to learn how to distinguish between your ego and your soul/higher self (Christians would call this the voices of the Devil and of God) and keep at it until you are able to still your egoic anxiety attacks by calling on your higher self for strength.
What have you done to learn meditation? Is it worth trying a different book, or a course? Or maybe you need other ways to connect with the Divine - ritual, chanting, yoga (not the physical centred Western yoga, but the deeper heart of yoga), prayer, visualisation or journeying?
Whereabouts do you live? Is there anything local going on that you could connect with to learn other ways to connect with your higher self? You may need to step outside of your comfort zone, but it's probably worth it.
On the other hand, trusting that each stage - the seemingly backward ones, as well as the forward ones - are part of your journey and that the right things will step into your path when you are ready for them is probably really helpful too.