YANBU and please don't let anyone tell you that you are.
You've posted in AIBU and people like to have a go IMO. If you'd posted in breast & bottle feeding, parenting or employment issues you would undoubtably have got a more balanced response.
At 11 months, your employer would by law need to keep your job open if you had decided your baby was too young for you to come to work at all. Being away for 3 days is a bigger ask IMO. Some mums and babies might be fine with this but some may not.
I left DS with MIL for New Years Eve when he was 12 months. She said he was fine, but she later admitted that he screamed for pretty much 6 hours 
I can't believe the unsupportiveness of the comments on this thread (well actually I can, this is AIBU). You're not saying you can't go because you;re a mother, full stop. You're saying you can't go because you're a mother of a very young baby!
You're instincts are right IMO. If your baby is breastfed and used to you being around, 3 days away from you all of a sudden could be very distressing indeed. And it could mess with BFing, which could affect the long term health of your child, ultimately.
In your shoes I'd ask about being allowed not to go this year (reminding them that you are back to work early - you did not take the full year, as you are entitled to). If they said no, I'd start looking for a new job asap.
And next time you have a serious question, please post it somewhere other than AIBU!