Maybe a bit...I'm sure your baby will be fine! Especially if it wasn't actually that brown bit, but a bit from elsewhere on the orange. If it makes you feel better, I once made my 3yo a pate sandwich, only to discover later that day that the peel-back bit of the packaging was green with mould. He's now 12, no apparent harm done...