OpenAI finds 30% of popular AI coding benchmark tasks flawed
OpenAI reviewed SWE-Bench Pro, a widely used benchmark for assessing AI programming skills, and found nearly 30% of its tasks to be flawed or invalid.
Published 1sem1 source
Lire en français
≈ 29s
The fact
The company is withdrawing its endorsement of the benchmark following this finding, casting doubt on its reliability for evaluating AI model performance.
This discovery raises concerns about the robustness of current assessments for LLM capabilities in software development.
Click the link to read an article on the topic: