Hang on a minute, added to the mix your husband also works part time? Is there a bit of that story missing (additional study he's doing, ill health, etc). If not, why are you assuming it's fair for you to do more of the childcare? The reason that's common is because often the mother takes a part time job and the father remains full time.
I think you need to focus a bit on the question you asked - which is whether it was reasonable to say you didn't want to travel or whether you should make more of an effort to go.
Unless travel is considered mandatory to your role, you can obviously refuse to go. That's always the case with optional duties, and if that's what you want to do go ahead.
BUT
This is travel maybe once a year, for something important. Your employer will think less of you if, two years in, you don't want to do that. They just will. It's not like a young baby. Only you can make the decision about whether that's a price worth paying. You seem to want people on here to tell you that you can refuse to go, that's totally reasonable, and your employer is being unreasonable to expect you to go. I'm afraid life's not that simple. We all have to balance and make sacrifices.
FWIW, I'm an extended breastfeeder and would hate the thought of being away from my kids overnight. Although, TBH, I don't actually see what breastfeeding has to do with it at this stage. Your child will be fine, you will be fine, there are ways of keeping the breastfeeding relationship going at that stage. I don't think you are any more or less tied to your child than a mother who isn't breastfeeding at that point. So, perhaps I should just say "as a mother, I would hate the thought of being away overnight", but you just have to balance it. You can't expect to opt out of important meetings and not have it have an impact on your career.
So by all means say you won't go. If your employer wants to force the issue you will have to decide whether you think they are being reasonable to make it mandatory. But that's a choice like any other, and it will probably have consequences for how you're perceived. You just have to decide if that price is worth paying...