Sansa Bench
Tool use leaderboard for Sansa Bench, with charts, model comparison, and methodology. This dimension scores selecting the right tools, formatting calls correctly, and using tools to complete tasks.
Top models for Tool Use
- 1.
Qwen3.5-9b Reasoning High0.805 - 2.Qwen3.5-9b Reasoning Low0.790
- 3.
Gemini-3.1-Pro-Preview Reasoning High0.707 - 4.
Claude-Opus-4.8 Reasoning High0.703 - 5.Gemini-3.1-Flash-Lite-Preview Reasoning High0.701
Methodology: Tool Use
What it measures
Tests tool usage capability including simple, multiple, parallel, and language-specific tool use. Queries require the model to correctly identify when tools are needed, select appropriate tools, format tool calls correctly, and use tools effectively to accomplish tasks.
Scoring & Criteria
Returns score 1.0 if the number of tool calls matches expected (within min_calls/max_calls range) AND each call matches expected function name and all arguments match valid values. Returns 0.0 otherwise. Handles both string and dict formats for arguments (including double-encoded JSON from some providers). When the answer field contains arrays for argument values (e.g., {"accountNumber": ["FF123456789"]}), this indicates multiple acceptable values for matching purposes. The model does not need to return an array, but its argument value must match one of the values in the array. Parameters with default values in the tool schema may be included or omitted by the model without penalty. If the model explicitly includes a parameter set to its default value (e.g., including "includeExpirationInfo": true when the schema default is true), this is treated as semantically equivalent to omitting that parameter and does not result in a scoring penalty. Supports flexible value matching including type coercion, string normalization, keyword matching, and list subset matching.
Evaluation Type: Tool Call
Grades tool calls by comparing against expected function names and parameters. Accepts tool calls in the standard OpenAI/OpenRouter API format (structured JSON with function.name and function.arguments fields).