This all sounds very familiar to me. We're you able to get any counselling in hospital?
(We were very fortunate that the NICU my daughter ended up in arranged for us to see one of their psychologists before they agreed to discharge my daughter. This was helpful to give us some tips for helping manage the guilt we felt about her getting so ill from lack of milk. Down the line, I actually feel slightly more angry that the issues were overlooked, given we saw midwives every day from birth until she was admitted to NICU. It's past now, though, and I can't change any of it, and she's perfectly healthy.)
It sounds like you are done with breastfeeding, but it isn't too late to work on encouraging your baby to latch. This can be nice for comfort, for you to feel like they are getting some breastmilk benefits, and so on, even if you it isn't working out for food. However, if you want to, I would try to see a feeding counsellor for good, in person advice. The gist of it will be lots of skin to skin time, but also, switch your bottle teats to ultra slow flow premmie baby ones, and tilt the bottle away so they dont get milk immediately it goes in the mouth - let it out once they've started sucking (as it would be during let down for breasrfeeding). The little blighters are quite lazy, and getting milk from a bottle with a faster flowing teat is less work than boob.
It's fine to be done with breastfeeding too, though. On the plus side, at least you won't turn into me, stuck with a toddler who can now call "Boo-bie! Boo-bie!" in public. You'll also likely find your formula fed baby sleeps through the night sooner (there's no evidence for this, but anecdotally, that's what I've observed: I think it's that it's easier to pack enough milk into them in the day with bottle feeding, for them to need less, and wake less, at night). If you ditch the pumping you'll have more time and energy and enthusiasm for your baby!
If you choose to let your milk supply dry up, watch out that this can cause more baby blues / a mood dip, because of the hormone changes.
There will be many choices in how you do parenting. How your baby us fed is only the first, so try to keep it in perspective. It has been a life and death choice that was forced on you - formula feeding was needed, which makes it feel very intense . Once you start weaning, though, your baby will be eating all sorts - and many of them will actually be bad, unlike formula! (Cake. Biscuits. Dirt...)