The quality you get from an open-weight model depends on more than the model itself

When Ubicloud started serving Kimi K2.6 with the vLLM inference engine, the numbers looked reasonable at first glance: the model responded quickly, consumed GPUs efficiently, and delivered coherent results. But there was a silent problem that only appeared under pressure — only 92% of tool calls were valid. That means that, for every 25 attempts, two failed because the model generated malformed JSON, a wrong parameter name, or a schema that simply didn't match what the orchestration code expected.

For a traditional chat application, 92% sounds like success. For an autonomous AI agent that needs to execute multiple calls in sequence, it's a critical failure. Each invalid tool call generates incorrect context, which leads to more wrong calls, and the model falls into a spiral of confusion — what Ubicloud's engineers call "nonsensical context buildup."

What Ubicloud discovered, and published on October 20, 2026, is that the inference engine matters as much as the model for the end-user experience. And the solution — a technique called constrained decoding, which vLLM introduced as an official feature in version 0.31 — could change the relationship between open-weight providers and the models they serve.

Why tool-calling breaks so frequently

Here's the paradox: models like Kimi K2 were specifically trained to make tool calls with high fidelity. During training, they saw thousands of examples of synthetic tool calls across multiple formats. When you run the model on Moonshot AI's own API, the success rate is close to 100%. But when you run the same model with vLLM — one of the most popular inference engines for open-source models — the rate drops significantly.

Ubicloud isolated the problem down to the token level. What happens is that the model, when generating a tool call, needs to follow a JSON schema that was provided in the prompt. But because LLMs are probabilistic — they don't run a compiler or a type checker — the model can generate a tool name that exists in the training set but isn't in the current list. A field called `city_name` when the schema expects `location`. A JSON block wrapped in markdown code blocks when the parser doesn't expect it.

The problem compounds when you mix models. The vLLM blog documented a similar case testing Kimi K2 on their own engine: the initial success rate fell below 20%. Three issues were identified: the `add_generation_prompt` parameter being silently discarded because Kimi's tokenizer declared it via `**kwargs`, which vLLM ignores for security; an empty `content` field breaking the prompt formatting; and a tool-call ID parser so rigid that it rejected valid IDs in edge cases.

Each of these issues was fixed individually — with contributions directly to the Moonshot AI team and PRs sent to vLLM. The final result was an 83.57% F1 score, far from the 100% of the native API. But still not enough for production-grade agentic workloads.

What constrained decoding is and why it matters

Constrained decoding is a decoding technique that forces the model to generate only tokens that are syntactically valid according to a schema. Instead of letting the model generate freely and then trying to validate the result afterward, constrained decoding operates at the tokenizer level: at each step, it restricts the vocabulary of allowed tokens to what's compatible with the JSON schema of the tool.

Imagine you're dictating something to someone, but instead of letting the person say anything, you give them an exact list of words they can use at each position. That's exactly what constrained decoding does with JSON: instead of "generate a valid JSON object," it says "the next word must be this, and this, and this."

Ubicloud tested constrained decoding with Kimi K2.6 and DeepSeek V4 Flash on the K2VV ToolCall benchmark from Moonshot AI — a 2,000-request benchmark that measures both when the model decides to call a tool and whether the call is syntactically correct. With constrained decoding enabled, both models achieved 100% schema accuracy. The exact number the native API achieved.

This isn't just a technical win. It's a paradigm shift in how we think about the quality of open-source models. Historically, the assumption was that the difference between a vendor's API and the open-weight version was marginal. But Ubicloud shows that, for agentic workloads, the difference is visible and measurable.

What this means for the ecosystem

The tool-calling story in vLLM reveals something few were willing to admit: inference infrastructure is not neutral. Two deployments of the same model, with the same weights, can produce dramatically different results depending on the inference engine, the tool-call parser, and the decoding configurations.

For companies building AI agents with open-source models, this is a practical lesson: choosing the inference engine is as critical as choosing the model. If you're serving Kimi K2.6, DeepSeek V4 Flash, or any other model with robust agentic capabilities, running without constrained decoding could be costing you up to 8% tool call success — and that's just the first round.

Constrained decoding still has limitations: there are interaction bugs with other vLLM features, and not all models support the feature natively. But the trajectory is clear — as inference engines mature, the line between the native API and the open-weight deployment should become increasingly indistinguishable.

What remains is the realization that, in the world of LLMs, the quality that reaches the end user is the product of the model, the engine, the parser, and the schema — and optimizing only one of those components is no longer enough.

Sources: Ubicloud, vLLM Blog, GitHub — vLLM Issue

✓ Independent sources cross-checked and verified before publishing