You are surely aware that ai chatbots regularly (yes, even now) get information wrong, even basic information is often simply wrong. They fabricate sources, quotations, and more.
I have a friend who works on legal reports (not a lawyer) who recently asked for some cases to back up his matter, basically speeding up his process rather than hunt and peck through searching.
Luckily, he always backchecks and discovered one of the cases had simply been invented. It didn't exist at all.
Back when I was still trying ai out I asked it for a Scottish poet, gave the name and a few lines of a specific poem I was trying to track down. It actually invented one, on the spot, and claimed it was the real deal. It looked quite genuine too at first glance. Fortunately, I always check sources too.
The last time I tried to use Chat GPT for anything more than a simple internet search was about a year ago. I gave it a list of places in a particular city and asked it to copy a basic map of the city and highlight just those places I had named.
It simply fabricated streets, it placed some destinations in completely the wrong place and the "map" it gave me was totally useles, it was certainly not from this universe.
This is a well known, heavily mapped city, there is a lot of information available about this place online.
In these three instances it admitted it was simply inventing things when challenged. Oopsie, yes, sorry about that, is very common with chatbots. This is a serious problem that they cannot seem to overcome.
Language matters a lot, the naming of a thing contributes to the trust we put in it, we really need to go back to calling them chatbots.
There is no way at all to tell if a chatbot is hallucinating without seeking verification.
Anyway, I suppose if the chatbot was your only source of information when trying to flee a danger zone it would be better than just guessing, and you might not have time to find a credible source in these circumstances, so not actually necessarily stupid, but certainly not ideal.