of GLM 5's structured outputs fail on its worst provider.
Updated October 31, 2026 · How we measure
Structured-output index
The same model breaks your pipeline's structured outputs 15× more often on some inference providers than on others. Find the best one for your pipelines.
Give this to your agent ↓
- Structured-output error rate
- ModelProvidersSpread0.01%0.1%1%10%100%BestWorst
- DeepSeekDeepSeek V4.1 Flash 1017300.40%20.67%
- DeepSeekDeepSeek V4.1 Flash250.35%4.84%
- DeepSeekDeepSeek V4 Pro 0813130.39%4.69%
- DeepSeekDeepSeek V4 Flash 0731170.16%7.64%
- DeepSeekDeepSeek V4 Pro 042380.36%13.22%
- DeepSeekDeepSeek V4 Flash 0423130.66%29.73%
- DeepSeekDeepSeek V3.250.67%6.52%
- DeepSeekDeepSeek V3.140.12%6.09%
- GoogleGemma 4 26B A4B110.23%7.49%
- GoogleGemma 4 31B110.21%2.45%
- GoogleGemma 3 27B40.74%1.62%
- MetaMuse Spark Open40.68%2.48%
- MetaLlama 3.3 70B Instruct50.16%3.48%
- MiniMaxMiniMax M3.1 Flash80.14%6.91%
- MiniMaxMiniMax M330.61%4.26%
- MistralMistral Large 415.77%8.07%
- MistralMistral Nemo30.15%2.24%
- MoonshotAIKimi K3160.20%6.71%
- MoonshotAIKimi K2.6131.53%5.21%
- MoonshotAIKimi K2.532.44%8.64%
- NVIDIANemotron 3.5 Super VL61.45%4.39%
- NVIDIANemotron 3.5 Lightning51.75%6.93%
- OpenAIgpt-oss-20b81.17%23.51%
- OpenAIgpt-oss-120b180.76%10.21%
- QwenQwen3.8 27B160.25%5.82%
- QwenQwen3.8 2.4T A95B20.48%69.27%
- QwenQwen3.6 35B A3B80.09%4.18%
- QwenQwen3.5-9B60.30%14.86%
- QwenQwen3.5 397B A17B70.65%4.19%
- QwenQwen3 30B A3B Instruct 250730.65%1.23%
- QwenQwen3 235B A22B Instruct 250730.86%1.00%
- ReflectionBeam 1.0 501B A23B21.41%2.10%
- TencentHy331.18%6.05%
- XiaomiMiMo-V2.6-Pro32.59%3.55%
- XiaomiMiMo-V2.6-Flash80.21%10.13%
- Z.aiGLM 5.3 Flash290.28%6.83%
- Z.aiGLM 5.3250.25%24.50%
- Z.aiGLM 5.2190.23%63.34%
- Z.aiGLM 5.130.45%3.23%
- Z.aiGLM 533.34%76.41%
Get updates when a new model drops
And which provider to use.
- Structured-output error rate
- ProviderModelsSpread0.01%0.1%1%10%100%BestWorst
- AkashML80.18%5.24%
- Alibaba Cloud Int.60.66%20.46%
- AtlasCloud110.16%29.73%
- Baidu Qianfan120.74%7.64%
- Baseten50.50%4.84%
- Chutes40.57%5.07%
- Cloudflare70.39%13.22%
- Cohere52.21%4.32%
- CoreWeave190.41%6.53%
- Crusoe100.98%4.97%
- Darkbloom93.06%23.51%
- Decart51.07%3.47%
- DeepInfra290.35%4.64%
- DekaLLM90.81%24.50%
- DigitalOcean90.59%63.34%
- Fireworks80.40%2.83%
- Friendli40.69%15.56%
- GMICloud40.21%76.41%
- Google Vertex34.45%10.21%
- Groq22.12%2.40%
- Inceptron41.20%5.21%
- inference.net60.70%6.90%
- io.net30.94%10.13%
- Ionstream30.53%3.19%
- Makora80.46%2.74%
- Mancer40.25%3.93%
- MARA20.12%0.76%
- Mistral52.83%10.67%
- Modal70.44%6.71%
- ModelRun [by Modular]40.21%4.26%
- Moonshot AI22.35%2.66%
- Morph60.80%2.35%
- NEAR AI30.25%2.12%
- Nebius Token Factory130.14%6.30%
- NextBit30.30%0.94%
- NovitaAI40.50%69.27%
- Open Inference30.94%3.81%
- Parasail230.09%6.43%
- Particle AI21.83%1.98%
- Phala90.83%13.29%
- Prime Intellect60.23%10.28%
- Reka AI60.40%10.83%
- Relace21.16%2.03%
- Sail Research80.50%6.95%
- SiliconFlow120.82%20.67%
- StreamLake91.12%5.52%
- Together140.25%14.86%
- Venice160.23%8.64%
- Wafer100.79%5.68%
- Xiaomi22.39%3.55%
Methodology
How we measure
- What counts as an error
- Of the requests that asked for a structured output (response_format), the share OpenRouter counts as failed; it doesn't document what fails one. The rate is OpenRouter's own, published per provider.
- Time period
- The last 7 days, updated daily.
- Who's included
- A provider is listed for a model when it handles 1,000 structured outputs a day or more. Each quantization or tier (fp8, fast, zdr) counts as a provider of its own, and a provider's regions count together.
- Best, worst and the bars
- The lowest and the highest error rate among a model's providers (in the providers' list, among a provider's models), and the bars between them, on a log scale.
- Why didn't I know?
- Pipelines retry an output that doesn't parse or doesn't match its schema, so the failure rarely shows. You still pay for every retry, in tokens and in time.
For inference providers
Structured outputs don't have to break
The same model breaks structured outputs a fraction as often on other providers. The difference is in how the output is generated and parsed, not in the model.
Implementations diverge on strings, optionals, composition and conditionals (the April announcement): the schema one provider enforces, another ignores.
providers break 3% of structured outputs or more on at least one model.
between a typical provider's error rate on its best model and on its worst.
dotjson plugs into your inference stack, so every model you serve gets perfect structured outputs from day 0.
Make every structured output valid↵For model makers
Broken structured outputs carry your model's name
Your users don't see the provider. They see your model failing.
of DeepSeek V4 Flash 0423's structured outputs fail on AtlasCloud (fp4).
of Gemma 4 26B A4B's structured outputs fail on Google Vertex.
of Llama 3.3 70B Instruct's structured outputs fail on DeepInfra (fp8/turbo).
of MiniMax M3.1 Flash's structured outputs fail on SambaNova.
of Mistral Large 4's structured outputs fail on Mistral.
of Kimi K2.5's structured outputs fail on Venice.
of Nemotron 3.5 Lightning's structured outputs fail on Darkbloom (int4).
of gpt-oss-20b's structured outputs fail on Darkbloom (fp8).
of Qwen3.8 2.4T A95B's structured outputs fail on NovitaAI.
of Beam 1.0 501B A23B's structured outputs fail on CoreWeave (fp4).
of Hy3's structured outputs fail on NovitaAI.
of MiMo-V2.6-Flash's structured outputs fail on io.net (fp8).
of GLM 5's structured outputs fail on GMICloud (fp8).
Releasing a model? Make your own API the one every provider is measured against.
Set the bar for your model↵Get updates when a new model drops
And which provider to use.