If you've shipped an LLM feature past the demo stage, you've hit this: the model works fine for a week, then one morning JSON.parse() throws on a response that just... stops. Mid-array. No closing bracket.

Most teams try to fix this by yelling at the prompt harder — "return ONLY valid JSON, I mean it this time." That buys you a few days. It doesn't fix the actual problem, because the problem isn't the prompt. It's architecture.

The three ways structured output breaks

LLMs used for extraction, classification, or transformation rarely fail randomly — they fail in one of three specific, predictable ways.

Mid-stream truncation happens when a large array output hits the model's max output token limit and gets cut off mid-string, leaving you with something like {"items": [{"id": 1}, {"id": 2... and a parser that has no idea what to do with it.

Schema drift is the quieter failure: the model omits a required field, returns a string where you expected a number, or invents a nested attribute nobody asked for — usually on the exact edge case your test suite didn't cover.

Markdown pollution is the most common and the easiest to miss in dev: the model wraps its JSON in ```json fences, or opens with "Sure, here's the analysis you requested:" before the actual payload. Your parser sees that preamble and dies on line one.

None of these are prompt-wording problems at their core. They're what happens when you ask a probabilistic text generator to behave like a deterministic API without giving it the constraints to actually do that.

Stop prompting for JSON — constrain the decoding instead

The reliable fix is to stop asking the model nicely and start making invalid output physically impossible to generate. That's what constrained decoding does.

Older prompt-based approaches ("You are an API, return only valid JSON") are inherently probabilistic — they raise your success rate, they don't guarantee it. Modern APIs — OpenAI's Structured Outputs, Gemini's response_schema, Anthropic's tool calling — instead use Context-Free Grammars to constrain the model's token sampling at the logit level. The model literally cannot emit a token that would violate your schema. This isn't a stronger suggestion; it's a different mechanism entirely.

Pair that with a strict schema on your side, and you close off the two most common escape hatches models use to drift:

import { z } from 'zod';

// Strict schema - no unknown attributes allowed through
export const ArticleAnalysisSchema = z.object({
  topic: z.string().min(5).max(100),
  intent: z.enum(['informational', 'commercial', 'transactional']),
  opportunityScore: z.number().int().min(0).max(100),
  keyTakeaways: z.array(z.string()).min(2).max(5),
  metadata: z.object({
    wordCountTarget: z.number().int().positive(),
    targetAudience: z.string(),
  }),
}).strict();

That .strict() call matters more than it looks. Without it, Zod silently drops or ignores keys it doesn't recognize instead of rejecting them — which means schema drift can sail through validation undetected for months.

Fixing truncation means changing the request, not the prompt

If you're extracting 50 line items from a financial document in one shot, you're going to hit the token ceiling eventually — no amount of prompt tuning changes that math. The fix is to stop asking for the whole thing at once.

I've found two patterns that actually hold up under load:

Schema chunking. Split the source document into semantic chunks and cap extraction at something like 10 items per call. It's more requests, but each one finishes cleanly instead of racing the token limit.

Two-stage extraction. First call returns a lightweight list — just IDs or headlines, nothing else. Second stage fans out parallel requests to expand each item into its full schema individually. This scales better than chunking when items vary wildly in size, since a giant item and a tiny one no longer compete for the same token budget.

As a last line of defense — not a primary strategy — add a JSON repair layer using something like json-repair in Node or dirtyjson in Python. It'll close unclosed brackets and salvage a partially truncated response rather than throwing it away entirely. Treat this as a safety net, not a fix; if you're leaning on it constantly, your chunking strategy is undersized.

What to do when you can't use constrained decoding

Not every model or provider gives you grammar-constrained decoding — plenty of open-source weights and older API tiers don't. When you're stuck prompting for JSON the old-fashioned way, precision in the prompt itself is the next best lever.

The pattern that's worked best for me is what I'd call schema anchoring: spell out the exact JSON Schema in the prompt, then follow it immediately with blunt, unambiguous constraints.

# TASK
Analyze the provided user query and output a structured analysis JSON object.

# SCHEMA SPECIFICATION
Adhere strictly to this JSON Schema structure:
{
  "category": string (One of: ["AI", "Backend", "Frontend", "DevOps"]),
  "confidenceScore": number (Float between 0.0 and 1.0),
  "actionItems": [
    {
      "priority": "HIGH" | "MEDIUM" | "LOW"
    }
  ]
}

# CONSTRAINTS
- Return PURE raw JSON only. Do not include markdown formatting or backticks.
- If an attribute is unknown, populate it with null rather than omitting the key.
- Maintain camelCase formatting for all field names.

The "populate with null rather than omitting the key" line does more work than it looks like it should. Models left to their own devices will happily just skip a field they're unsure about — and an omitted key is a much harder bug to catch than a null one, because your downstream code might not even notice it's missing until three steps later.

Matching the failure to the fix

Every one of these failures has a specific root cause, and reaching for the wrong fix — usually "write a better prompt" — is why teams end up re-solving the same bug every few months.

If you only take one thing from this: don't reach for prompt tweaks to fix what's actually an architecture problem. A stricter sentence in your system prompt might buy you a better success rate this week. Constrained decoding, chunking, and strict validation are the difference between "usually works" and "hasn't broken in production in six months."