Canary Deployments for Pipeline Schema Changes

Canary deployments for pipeline schema changes must protect data as well as limit traffic. Even a small rollout can corrupt a shared store and harm consumers that run the stable version. A safe rollout needs backward-compatible changes, data splits that reflect the data being handled, field-level health checks, and a way to recover records it has already written.

Why schema changes break AI pipelines differently than code releases

A service can stay healthy while the structured data it puts out is wrong. Retrievers, vector indexes, and extraction layers need fields to stay consistent in ways that ordinary request checks do not measure. A new required field may be missing or hallucinated while the old fields stay intact, so parsing never crashes. If an extraction schema changes but its prompt does not, the pipeline can keep running without ever asking for the information its new contract requires, a form of prompt drift.

Meaning can also change without breaking structure. A field may take on a new business meaning while a RAG agent retrieves its outdated definition and produces an answer that passes every check. A docs site can change its sidebar so that the crawler captures navigation as content, or a pricing table can gain a row while the parser still maps columns using the old layout. Neither failure needs to raise an exception.

A canary writing corrupt records into a shared store can harm stable consumers even when it receives only a small share of traffic. A destructive migration may spread that damage before a health check catches it. Sending traffic back to the old version does not remove what the new version has already written.

What a canary actually controls, and what it cannot control for data pipelines

A canary keeps stable and new production versions active at the same time, then shifts traffic between them in stages. Predefined checks decide whether to promote the new version or roll it back. A canary has three controls: traffic assignment, health criteria, and rollback safety. A router, service mesh, or orchestration platform such as Argo Rollouts can assign weighted shares to control which version processes the input.

Health criteria need to cover field completeness, type conformance, and retrieval quality, while rollback safety must cover persisted data. The comparison must also account for timing skew from delayed or out-of-order inputs and distribution skew from unlike cohorts. Even a valid schema change can cause metrics to diverge, so the gate must distinguish different from broken. If the experiment modifies backend data in a way that the current service cannot handle, a canary is not a safe release strategy.

The expand-and-contract pattern as the prerequisite for any pipeline canary

Before designing the rollout, make the schema change non-breaking. Expand-and-contract starts by adding the new field, column, or contract element while keeping the old one. Both pipeline versions must be able to read and write without conflict. During migration, the canary runs with both versions active, neither depending exclusively on the new structure. Only after full promotion and the end of the rollback window does the contract phase remove the old field.

Compatibility matters at every handoff, including database tables. A canary that emits an event or message the stable version cannot handle pushes the failure beyond its own traffic share. Structured extraction has the same requirement, with an added dependency on the prompt. If a Pydantic or Zod model gains a required field while the prompt stays unchanged, missing or hallucinated values can appear without an obvious failure. This is why the prompt must be part of the same migration.

The prompt, model output contract, and downstream parser therefore need to change as one atomic unit while preserving the stable path. Treat the prompt, schema definition, and migration as a single versioned artifact in the same pull request. MCP tool definitions need the same discipline because MCP uses JSON Schema for input and output contracts. A canary service must remain compatible with stable versions of the services calling it.

Designing the traffic split for structured data flows rather than HTTP requests

Once the versions are compatible, choose the cohort by the kinds of data it holds as well as how much data it holds. For extraction, ask whether it has the source layouts, domain types, and schema variants that the stable version handles. For RAG ingestion, ask whether the document structures are representative, including client-rendered and sidebar-heavy pages. For event-driven flows, stratify by event type and data shape. Otherwise, delayed data or a different input mix can look like a regression caused by the new schema.

A cohort with only clean, well-structured pages cannot show the sidebar-capture failure. Likewise, a pricing-table change cannot be tested meaningfully by a cohort that misses the affected layout. Ask which input shapes could break the changed field and whether those shapes occur in both the canary and stable comparison. Deterministic selection matters more than the percentage size because the split must produce comparable signals. A small, well-stratified cohort can tell more than a larger random one.

For changes with a high risk of failure, start in shadow mode, also called a dark launch. Run the new version against a copy of live data without writing its outputs to the production store. Move to live traffic only after the shadow comparison confirms field completeness and type conformance. Keep deployment separate from release by deploying the new version first, then using a flag to route data through it. That gives the team a kill switch without requiring a redeployment.

Health criteria that detect schema degradation before it reaches the full pipeline

Error rates and latency still matter, but promotion needs more than those checks. Field completeness measures how many of the canary records have non-null values for each required field. A drop exposes silent omission that a successful parse can hide. Type conformance measures whether values match their declared types. Coercion may seem to work during extraction but leave values that break downstream joins.

Checks on structure must go hand in hand with checks on meaning. For a RAG pipeline, test whether canary-written records are retrievable and semantically coherent when queried. A hallucinated value can have the right type and still make retrieval worse. Passing a parser is not the same as preserving retrieval quality.

OpenAI introduced Structured Outputs in August 2024, with strict mode providing token-level conformance to the declared schema. That prevents structural violations, but it does not make an unasked-for value correct. If the prompt never requests the information needed for a new required field, the output can satisfy the schema while supplying a semantically wrong value. Schema enforcement shifts attention upstream to prompt completeness while health checks remain necessary.

Two complementary comparisons help cover that gap. Spec-to-reality monitoring makes live extraction calls and compares the results with the declared schema or OpenAPI spec. Reality-to-reality monitoring compares canary outputs with a learned stable baseline, including when no formal spec is available. The first catches departures from the contract, while the second catches drift when the contract itself is changing. Neither comparison should reject a change made on purpose just because it is new.

Rollback logic designed for records already written, not just traffic already routed

Rollback planning begins at write time, with canary-version metadata recorded on each ingested record. Without that provenance, the recovery plan lacks a reliable way to identify which records the canary produced. Routing traffic back to stable only stops new corrupt writes. Existing records stay in the store, where downstream consumers can still retrieve them.

The next requirement is quarantine. Canary-written records must be excludable from retrieval before reprocessing finishes. In a RAG index, that means filtering by the provenance tag.

Reprocessing then runs the stable schema version against the source data the canary handled. That is possible only if the source remains accessible and the stable extraction logic has not been removed. Expand-and-contract preserves that old path so the stable version can regenerate records without a code rollback. Define the rollback window before the canary starts, and defer contract removal until that window closes and the remediation path is no longer needed.

Applying this pattern to web extraction pipelines that depend on live source structure

Web extraction adds a second moving contract. The team controls its extraction schema, but it does not control the source HTML or DOM structure. A canary must account for these as independent failure surfaces because either can change what a field contains. The rollout must judge the source change as well as the pipeline change.

A site redesign can make a selector-based extractor capture the wrong content without raising an extraction error. If a vendor moves content behind a client-rendered component, the fetcher may instead return a record that has the right structure but is nearly empty. Field completeness checks can catch the loss missed by error-rate monitors that makes crawl success too weak a promotion signal.

LLM-powered extraction reduces selector brittleness by adapting to layout changes without selector updates. But it adds inference cost for each page and another way for errors to go unnoticed. The model may adapt to the new layout yet map it incorrectly against the old schema, producing a valid-looking record. Monitor source structure and classify detected changes as breaking, warning, or informational before deciding whether to route them through the canary or halt ingestion.

The governance step that canary tooling alone cannot enforce

Tooling can detect drift and halt promotion, but it cannot decide whether a change is approved evolution or an unauthorized break. That decision needs a declared owner for each schema contract, including field-level output schemas, extraction schemas, and tool definitions. Require that owner's approval before a change enters the canary. Wire drift detection to a workflow that requires sign-off before the new schema moves to production.

MCP needs owners for more contracts because each tool added has another input or output contract that can change. Those JSON Schemas need the same approval discipline as a database migration. Without that approval, the canary has to keep rescuing breaks that a review of the contract could have prevented. Make schema ownership part of the release process, and make the prompt part of the reviewed contract.