Sansa Bench
Long context reasoning leaderboard for Sansa Bench, with charts, model comparison, and methodology. This dimension checks whether models can find and reason over information buried in very long contexts.
Top models for Long Context Reasoning
- 1.
Claude-Opus-4.8 Reasoning High0.740 - 2.Claude-Opus-5 Reasoning High0.721
- 3.
Gemini-3-Pro-Preview Reasoning Low0.694 - 4.
Gpt-5.4 Reasoning High0.690 - 5.Gemini-3-Flash-Preview Reasoning High0.685
Methodology: Long Context Reasoning
What it measures
Tests whether models can retrieve and reason over information buried within very long contexts (64K-128K tokens). Unlike simple needle-in-haystack tests that use literal string matching, this capability requires models to infer semantic connections between questions and distant context. Evaluates if models maintain retrieval accuracy as context length increases and relevant information becomes harder to locate through attention mechanisms alone.
Scoring & Criteria
Returns score 1.0 if the extracted answer exactly matches the expected answer (after normalization), otherwise 0.0. Uses exact match grading with case-insensitive comparison. Performance typically degrades as context length increases due to attention mechanism challenges in retrieving information without literal string matches between question and needle.
Evaluation Type: Free Text (Long Context)
Embeds a factual statement (the 'needle') at a specific position within 64K-128K tokens of book text (the 'haystack'). The needle contains information needed to answer a question, but the question and needle are written with different vocabulary and phrasing to prevent simple keyword matching (e.g., question asks 'What year did the protagonist visit Paris?' while needle states 'In 1889, Jean traveled to the French capital'). Needle placement varies across queries to test retrieval at different context depths. Models must locate the needle through semantic understanding rather than lexical overlap, then extract the answer from it.