Absolutely, I think so personally, yes. I remember planting 25 stargazers lilies that I purchased in a bank holiday sale many years ago - they were gorgeous and smelled amazing.
I came into the house to wash up having planted them all and just so happened to turn on the radio - ironically it was a gardening programme and even more ironically they were talking about lily pollen being poisonous to cats. I had no idea and I had three cats!
I locked them in the house with food and then spent another hour pulling the damned things up and putting them in the green waste. Lesson learned! As you say, labelling plants would help a lot. I would have felt so unbelievably guilty if I had left them there and they had adversely affected my pets in any way.