I can see both sides...
With my first baby I needed to pump to boost supply and give him expressed top ups due to jaundice/tongue tie/poor weight gain. So that required a breast pump, bottles and teats. We also paid for a second tongue tie division privately as the NHS waiting list was too long and we could afford to pay to bypass it. In addition, we paid for cranial osteopathy (recommended by an NHS IBCLC midwife, but not available on the NHS) to improve latch.
With my second baby, breastfeeding was a dream and I didn't even touch my breast pump until around 8 weeks (convenience to allow me to go out, rather than absolute necessity).
So. If breastfeeding is going brilliantly, I'd say it can definitely be free. However if there are issues establishing breastfeeding, yes, it can be expensive initially... however I'd still say that the long term cost (assuming a mother breastfeeds long term) is still less than formula feeding (taking into account the cost of formula, plus bottles and replacing teats as they wear out/faster flow is required).
A note on breast pumps, those such as the Elvie are heavily marketed at the moment, however they are very expensive (around £250) and in my opinion spending that much is unnecessary when you only need a basic model and could even use a hand operated pump (costs around £20)... most mothers I've met who exclusively breastfeed tend not to pump very often as it's such a faff, so spending lots on a pump isn't worth it (different story if you are, for example, going back to work early or exclusively pumping).
During antenatal education I recall "breast is best" with health benefits for mother and baby being the key message... followed by convenience and cost benefits. I do feel that antenatal education (especially that provided by the NHS in my area) doesn't cover nearly enough information about what to expect, troubleshooting and how to access support.
I'm lucky in that I live in an area with really good breastfeeding support provided by the NHS; in the early weeks I had access to specialist infant feeding assistants by appointment on demand, plus drop in feeding clinics run by an IBCLC midwife. Sadly I know of people who live in areas without funding for this and I think it's a real shame. If "breast is best" is going to be promoted, then there needs to be adequate support to ensure that women can follow the guidelines to breastfeed if they choose.
In summary, I guess I'm saying that breastfeeding can be free, however I agree that breastfeeding support provision should be better.