Study Reveals a Decrease in ChatGPT’s Accuracy

In their evaluation, the team discovered that the behavior of the same LLM service can change significantly in a relatively short period, underscoring the importance of continuously monitoring LLM quality. For instance, in March 2023, GPT-4 showed an impressive 98 percent accuracy in identifying prime numbers, but by June, its accuracy dropped drastically to less than 3 percent for the same task.

Meanwhile, GPT-3.5 in June 2023 showed improvement in identifying prime numbers compared to its March 2023 version. However, when it came to generating computer code, both versions performed worse between March and June.

These variations could have real-world consequences, especially in fields like healthcare. A recent paper by researchers from NYU revealed that ChatGPT’s responses to healthcare-related queries were almost indistinguishable from those of human medical professionals in terms of tone and phrasing.

Participants had difficulty telling whether responses came from a human healthcare provider or the OpenAI language model, raising concerns about AI’s ability to handle medical data privacy and its potential to provide inaccurate information.

Academics and users have also taken note of ChatGPT’s declining performance. OpenAI’s developer forum has been discussing the LLM’s progress, or lack thereof, with users expressing their disappointment & requesting an official response to address the issue.

One user even compared the LLM’s performance to going from being a great assistant sous chef to dishwasher, highlighting frustration among payed customers.