New York City's AI-powered chatbot, criticized for dispensing flawed advice to small businesses, remains on the official website despite glaring inaccuracies. Experts warn against the potential dangers of unchecked AI deployment by governments, emphasizing the necessity for robust oversight and accountability. The incident underscores broader concerns about the reliability of large language models and the potential consequences of their misuse.