Structured Generation

Sansa Bench

Structured generation leaderboard for Sansa Bench, with charts, model comparison, and methodology. This dimension scores valid structured output such as JSON, YAML, XML, TOML, and CSV with required fields present.

Inspiration & Acknowledgments91 models testedUpdated Aug 8, 2026

Top models for Structured Generation

  1. 1.Gpt-4o1.000
  2. 2.Qwen3.6-Plus Reasoning None1.000
  3. 3.Mercury-2 Reasoning High1.000
  4. 4.Gpt-5-Nano Reasoning Low1.000
  5. 5.Gpt-5-Nano Reasoning High1.000

Methodology: Structured Generation

What it measures

Tests generation of valid structured formats (JSON, YAML, XML, TOML, CSV). Queries require the model to produce correctly formatted structured data that is both syntactically valid and contains required content elements.

Scoring & Criteria

Binary scoring with strict all-or-nothing semantics. Returns 1.0 only if: (1) response parses successfully as valid structured data in the target format, AND (2) every path in evaluation_criteria.required_content exists when navigated in the parsed structure. Returns 0.0 if parsing fails or any single required path is missing. For JSON/YAML/TOML: navigates parsed dictionary/list structures using dot notation with dict key lookup and list indexing. For XML: uses element tree navigation with support for attributes (@attr) and indexed children (element[0]). For CSV: validates that all required column names exist in the header row. A response with 9 out of 10 required fields still scores 0.0 - partial credit is never awarded.

Evaluation Type: TOML Generation

Evaluates structured data generation by parsing the response into the target format (JSON, YAML, XML, TOML, CSV) and validating required field paths exist in the parsed structure. Uses dot-notation path navigation to verify field presence (e.g., 'person.name' checks that parsed_json['person']['name'] exists). The grader extracts code blocks if present, parses the content using format-specific parsers, then navigates the parsed structure using the paths specified in evaluation_criteria.required_content. Supports nested objects, array indexing (e.g., 'items[0].name'), wildcards for array iteration (e.g., 'users[*].id'), and format-specific validation (XML element paths, CSV column headers). The validation is path-based on parsed structures, not string pattern matching - the model must generate syntactically valid structured data with the correct nested field hierarchy.

Example Question

Question:
Please output TOML code:
 
Task:
Given a description of a person, produce a TOML file capturing their basic details.
 
Feature Requirements:
1. The person.name field should be a string representing the person's full name, located at the top level under person.
2. The person.age field is an integer representing the person's age, directly under person.
3. The person.city field should be a string specifying the city where the person lives, directly under person.
4. The person.email field should be a string containing the person's email address, directly under person.
5. The person.active field should be a boolean indicating if the person is currently active, directly under person.
Answer:
Programmatically Graded:
-Valid TOML structure
-Required fields: person.name, person.age, person.city, person.email, person.active