I would say it's an animal product as we are animals are we not.
I suppose it depends on your reason for being vegan in the first place. If you don't like/tolerate the idea of eating any part/product of an animal, than maybe. But I think, for most vegans, it's the ethics and the idea that you're exploiting/killing an animal for your own selfish benefit.
If an animal could somehow talk and unequivocally tell you that it wanted you to kill it and eat it and make clothes from its skin, or to take its milk and drink it yourself, then I suppose the ethical point would be moot.
In the same way, by actively putting a baby to your breast and encouraging him/her to drink milk from your body, you are expressly giving your consent for the baby to take what you are freely offering.
In fact, you could argue that you've turned things around consent-wise, as you are freely consenting to give the milk but the baby isn't old enough to make an informed choice as to whether they truly want to take it and is merely acting on instinct.
Of course, if somebody were forcing or coercing you to breastfeed, that could potentially take us back and give weight to the original proposition: that your breastmilk is neither vegan nor being ethically/non-exploitatively consumed (although, of course, you wouldn't personally blame the baby).