Hi Cotton and congrats on your new DS.
You should be able to drop the formula but you probably need to talk to someone about a plan to reduce it gradually.
How much were you pumping in the first couple of weeks? You would need to pump the same amount of times as the baby would have fed to maintain your supply (8 -10 times roughly). That said it's not impossible to get it back.
Also, don't assume that because you couldn't pump anything and he guzzled the formula that you are starving him. Babies will often guzzle a bottle down just because it is easier. Even if you don't pump much milk, it doesn't 't matter because the pump is doing the job of sending the signal to make more milk.
Have you tried switch nursing? You feed on one side, then the other, then back again (and again if needed). This ups the volume of milk going in.
Good idea to get the latch checked again as something was obviously not quite right after birth. Is the feeding comfortable now? Perhaps ask for someone to check for tongue tie if they are able to.
Some women find that they can pump more milk in the mornings or if they leave a break between a breast feed and a pumping session. You could also try some hand massage before pumping and some hand expression afterwards too -this has been shown to increase the amount of milk removed.
If you are able, you might like to try and have a 'baby moon' - lots of rest, skin to skin and just feed to feed. Of course this will only be a good idea if the milk is being removed properly so make the attachment is good and he has a big mouthful of breast.
Hope that helps for now, I can't give you any idea of timescales but just keep an eye on the weight and nappy output so you know that if your reduce the formula your milk can be enough.