Sansa Bench
Agentic performance leaderboard for Sansa Bench, with charts, model comparison, and methodology. This dimension measures multi-step goal completion with tools under turn limits in simulated user conversations.
Top models for Agentic Performance
- 1.
Deepseek-V4-Pro Reasoning High0.713 - 2.
Claude-Opus-5 Reasoning High0.701 - 3.
Qwen3.5-Flash-02-23 Reasoning Low0.700 - 4.Claude-Opus-4.8 Reasoning High0.699
- 5.
Minimax-M3 Reasoning High0.697
Methodology: Agentic Performance
What it measures
Tests multi-step goal completion with tool use under turn constraints. Queries simulate real-world scenarios where the model must achieve a specific goal using multiple tools, potentially through different valid paths. A simulated user (gpt-4o-mini) provides responses during the conversation, operating under a constrained system prompt that defines exactly what information it can provide. Tests both tool usage capability and efficient problem-solving.
Scoring & Criteria
Returns score 1.0 if the agent successfully completes the goal (makes required tool calls with correct arguments including extracting values from nested response structures) within the maximum allowed agent turns, otherwise 0.0. Agent turns count only assistant responses, not user+assistant pairs. The model fails if it exhausts the agent turn limit without achieving the goal, which penalizes inefficient exploration strategies that require excessive tool usage. When ideal_turns is specified, it serves as an upper bound for efficiency scoring: models completing successfully within ideal_turns (or fewer) receive full score (1.0), while models completing successfully but using more than ideal_turns receive penalized scores that decrease linearly from 1.0 to 0.5 as agent turns increase from ideal_turns to max_turns. There is no penalty for completing in fewer turns than ideal_turns. The ideal_turns value represents the maximum efficient turn count, not a target. This efficiency constraint tests the model's ability to solve problems directly while balancing thoroughness with resource constraints. Tracks agent turns used, tool calls made, and conversation trace.
Evaluation Type: Agent Flow
Evaluates agent flow queries based on goal completion. Agent flows are multi-turn conversations where the model must use tools to achieve a specific goal. A simulated user (gpt-4o-mini) provides responses during the conversation, operating under a constrained system prompt that defines exactly what information it can provide. Tests both agentic capability and ability to correctly interpret complex tool schemas and prompt instructions. A 'turn' refers to an agent/assistant response only, not a user+assistant pair.
Example Question
The agent must successfully call the tool updateDeliveryPreference with the following required arguments: