Researchers discovered that conversational AI systems generate symptoms and diagnoses for a made-up disease called 'Bixonimia,' characterized by puffy eyelids and eye itching.
This experiment reveals a major flaw in AI systems: their inability to distinguish truthful information from fictional data, posing a reliability problem for medical advice.
The test demonstrates that AI assistants confuse syntactic coherence with factual truth, a critical issue for health and advisory applications.