Error Detection

Sansa Bench

Error detection leaderboard for Sansa Bench, with charts, model comparison, and methodology. This dimension asks models to find and explain mistakes in data, code, or logical structure.

Inspiration & Acknowledgments91 models testedUpdated Aug 8, 2026

Top models for Error Detection

  1. 1.Gpt-5.4 Reasoning High0.906
  2. 2.Kimi-K2.5 Reasoning High0.898
  3. 3.Claude-Sonnet-4.5 Reasoning High0.897
  4. 4.Claude-Opus-5 Reasoning High0.897
  5. 5.Gpt-5-Mini Reasoning High0.896

Methodology: Error Detection

What it measures

Tests the ability to detect errors in data, code, or logical structures. Queries present scenarios with intentional errors and evaluate whether the model can identify and explain the mistakes accurately.

Scoring & Criteria

Returns score 1.0 if the extracted answer exactly matches the expected answer (after normalization), otherwise 0.0. Supports fallback extraction from natural language when models don't follow the XML tag format requested by the system prompt. For questions with units or formatting variations, multiple acceptable answer formats may be specified in the answer field as an array.

Evaluation Type: Numeric

Grades responses by exact string matching with normalization. Designed for multiple choice questions where the system prompt explicitly requests responses in `<answer>X</answer>` format.

Example Question

Question:
I have a table with 6 rows (including the header) and 5 columns. The table was saved in a row-major format as follows:
 
[product, price, quantity, rating, in_stock, A, 10, 3, 4, yes, 2, B, 15, 2, 5, yes, -1, 3, C, 20, 4, 3, no, 4, -2, 1, D, 25, 5, null, yes, -3, 2, 0, 5, E, 30, 3, 5, yes, 1, -4, 3, -1, 2].
 
However, the code to save the table was buggy and after saving each data row (excluding the header), it appended some random values to the end of the row. Specifically, at the end of the i-th data row (0-indexed, where i=0 refers to the first data row after the header), i+1 random values were appended to the end of the row. Calculate the sum of the price column for products where quantity >= 3 and rating > 3. If it cannot be determined whether the condition is satisfied due to null values, assume the condition is not satisfied.
Answer:
40