Just to explain I clicked YANBU to be worried about something you've been told for weeks in big bold news headlines is a pandemic and is now on your street.
That said, the main concern is that it's a virus that is difficult to treat if your immunity/health is already poor. Should you be worried? I guess not.
Would I be in your shoes? Probably yes. You are being groomed by the media to be hysterical about it. Take it easy on yourself and your young family. Remind yourself the flu is dangerous and you manage ok with those risks and concerns. Do whatever you need to do to feel comfortable and at ease with the current concerns.