Schema Drift Reaches the Tool Definition

Everyone who sells a context layer talks about freshness. Fresh rows, streaming updates, sub-second sync. We wrote one of those posts ourselves. This post is about the staleness nobody measures: the shape of the data changed, the meaning of a field changed, or the rule about who sees it changed, and the tool your agent calls still describes the old world. Nothing errors. The answers are just wrong.

Three weeks of quiet wrong answers

Here is the failure that made us write this, with names removed.

A logistics team ran an agent over an orders data product. One of its tools, list_orders, took a status filter with four allowed values: NEW, PAID, SHIPPED, CANCELLED. The tool was generated once, reviewed once, and shipped. It worked. The agent answered “how many orders are stuck this week” correctly every day for two months.

Then the order management system added a fifth status, ON_HOLD, for orders waiting on a compliance check. Nobody told the agent team. Why would they? The column did not change type. No pipeline broke. No dashboard went red.

The agent kept answering. When a user asked about stuck orders, the model looked at its tool, saw four statuses, and built its query from the ones that fit. Orders on hold were never NEW, PAID, SHIPPED, or CANCELLED, so the agent never asked for them. The “stuck orders” count was short by every on-hold order in the system. Week one it was short by a few hundred. By week three it was short by over two thousand, and the on-hold queue was exactly the set of orders someone should have been looking at.

It took an analyst running the SQL by hand to notice. Not an alert. Not a test. A person who happened to compare two numbers.

That is the shape of this problem. Schema drift does not break agents. It bends them, and bent agents keep talking.

Three kinds of drift, and the one this post is not about

Atlan uses the term context drift for the gap that opens between what the metadata says about data and what the data actually is, and we will use it the same way rather than pretend we coined it. Atlan’s framing stays at the catalog. We want to follow the drift one step further, to the tool definition the agent actually reads, because that is where it does damage.

Row freshness is a different thing. Our earlier post on real-time context for AI agents covers the case where the rows are hours old and the agent answers from yesterday. That is a value problem and you fix it with streaming or faster sync. Drift is a structure problem. You can have rows that are two seconds old and a tool definition that is two months old, and the second one is what the model reasons from.

There are three kinds of drift you need to catch, and each shows up differently at the tool.

The common thread: the tool definition is a snapshot. The world it describes is not. Any context layer that generates grounding once and stores it has built a cache with no invalidation. Caches without invalidation are bugs waiting for a date.

This is not the memory staleness problem

If you have read the agent memory vendors, you have seen the word “staleness” already. Mem0 and Zep both write about it. Their version of the problem is this: the agent remembered “user prefers email” in March and “user prefers Slack” in June, and now it has two conflicting facts and has to pick. That is a real problem. It is also a different problem.

Memory staleness is about the agent’s own notes going out of date. Context drift is about the ground truth changing under the agent’s tools. In the first case the truth is fine and the recollection is wrong. In the second case the recollection is fine and the truth moved. You fix the first with recency weighting and conflict resolution inside the memory store. You fix the second by watching the source and regenerating everything downstream of it, and no amount of clever retrieval over the old grounding will help, because the old grounding is the thing that is wrong.

This matters when you are evaluating vendors. A memory layer that advertises “temporal reasoning” and “fact invalidation” is talking about conversation history. Ask it what happens when a column is renamed in Snowflake. The honest answer from a memory product is “nothing, that is not our layer.” Fine. Then something else has to own it, and that something is the context layer.

Detection: know the shape changed before the agent does

You cannot propagate a change you did not notice. Four detection mechanisms cover most of the surface, and you want all four, because each catches a kind of drift the others miss.

Schema fingerprinting on every run

Take the schema the tool was generated from (column names, types, nullability, nested structure) and hash it. Store the hash on the tool. On every ingestion, or on every tool call if the source is cheap to introspect, recompute and compare. A mismatch is a drift event. This is the cheapest check there is and it catches renames, drops, retypes, and additions within one run.

This is where Nexla starts from a head start. Nexsets already compare the incoming schema against the known schema on each ingestion and flag changes, and that comparison is what lets a Nexset tell you “a new column arrived in this feed” without anyone writing a test for it. The work described in this post is wiring that signal to the tool layer, which we come back to below.

Sample-based enum discovery

Fingerprints catch shape. They do not catch a fifth value appearing in a column whose type is still string. For that you sample. On each run, pull distinct values for every column the tool exposes as an enum (or any low-cardinality column) and compare the set against the enum baked into the tool. A new value is a drift event. A value that disappears for N consecutive runs is a probable deprecation, which is also a drift event, just a gentler one.

Sampling has a cost. Cap the number of columns you treat this way, cap the distinct count at which you stop (a column with 4,000 distinct values is not an enum), and run it on the ingested batch rather than on the whole table. In the hook story above, this one check would have fired on the first on-hold order.

Lineage-triggered invalidation

A tool sits on a data product. The data product sits on sources, transforms, and joins. When any upstream node changes, every tool downstream of it is suspect. You do not need to know what changed to know that the tool needs a re-check. This is the mechanism that catches semantic drift in derived fields: net_revenue is recomputed from a new transform, the transform is upstream of the tool, the tool is invalidated.

Practically, this means your context layer needs lineage that reaches the tool, not just the table. Most catalogs stop at the table or the dashboard. A tool definition is a downstream asset like any other, and it should appear in the lineage graph so it can be invalidated like one.

Doc and glossary change hooks

Semantic drift often starts in a document, not a schema. Someone edits the glossary entry, updates the runbook, or changes the KPI definition in the metrics layer. If those documents feed the tool descriptions (and they should), then a change to the document is a change to the tool. Subscribe to the change events. A glossary edit on a term that appears in a tool description is a drift event for that tool.

Four mechanisms, four kinds of signal. Put them in a table if it helps your team:

Propagation: regenerate the tool, do not edit it

Detection without propagation is an alert nobody reads. The question is what happens after the drift event fires.

The answer that does not scale is “someone edits the tool definition.” That is how most internal MCP servers are maintained today: a JSON file in a repo, a person who knows it exists, and a Slack message when something breaks. It works for five tools. It fails at fifty, and it fails silently, because the person who owns the source system does not know the tool exists and the person who owns the tool does not watch the source.

The answer that scales is to never hand-edit the tool in the first place. Generate it from the data product. When the data product changes, regenerate. The tool definition becomes a build artifact, the data product is the source, and drift propagation is just “rebuild on change.” This is the same move that made infrastructure manageable: stop editing servers, edit the template and redeploy.

This is the link that the current writing on tool versioning misses. The “Evolvable MCP” piece on Medium makes a good case for treating MCP tool contracts like API contracts, with semantic versions and deprecation policies. It stops there. It treats the tool as the thing that changes. In practice the tool changes because the data under it changed, and if you version the tool without connecting it to the data, you have a versioning scheme with no trigger. The data product is the trigger.

Three parts of this need more than a box on a diagram.

Version every tool

Give every generated tool a version and carry it in the definition, not just in a changelog. Use a scheme the agent and the humans can both read. We use semantic versioning with a simple rule: additive schema changes (new optional parameter, new enum value, new output field) bump the minor version. Anything that removes, renames, retypes, or changes meaning bumps the major version. Patch versions are for description wording that does not change behavior.

Deprecation windows for pinned agents

Some agents pin to a tool version, and they should be able to. A production workflow that was tested against list_orders version 3 should not wake up one morning talking to version 4 with a parameter it never saw. So the old version keeps being served, for a stated window, with a deprecation date in the definition and a note saying what replaced it. After the window, calls to the old version fail loudly with the same note. Loud failure after a grace period is the point. Quiet success forever is the failure mode we started with.

Tell the agent what changed

MCP already has a mechanism for this: servers send a tool list changed notification and clients re-fetch. Use it. But also put the change into the definition itself, because the model reads the definition and does not read your changelog. A one-line change note in the tool, in plain language, lets the model adjust its behavior on the next call: “ON_HOLD was added on 2026-08-04, include it when the user asks about stuck or blocked orders.” That sentence is worth more than any version number.

Here is the same tool before and after the drift in the hook story, generated from the data product both times.

{
  "name": "list_orders",
  "description": "List orders from the orders data product, filtered by status and date range. Use this for questions about order volume, backlog, and stuck orders.",
  "inputSchema": {
    "type": "object",
    "properties": {
      "status": {
        "type": "string",
        "enum": ["NEW", "PAID", "SHIPPED", "CANCELLED"],
        "description": "Order lifecycle status."
      },
      "placed_after": { "type": "string", "format": "date" },
      "placed_before": { "type": "string", "format": "date" }
    },
    "required": []
  },
  "x-context": {
    "version": "3.2.1",
    "generated_from": "nexset://orders_governed",
    "schema_fingerprint": "sha256:9f1c...a47e",
    "generated_at": "2026-06-02T09:14:00Z"
  }
}

{
  "name": "list_orders",
  "description": "List orders from the orders data product, filtered by status and date range. Use this for questions about order volume, backlog, and stuck orders. Orders with status ON_HOLD are waiting on a compliance check and count as stuck or blocked.",
  "inputSchema": {
    "type": "object",
    "properties": {
      "status": {
        "type": "string",
        "enum": ["NEW", "PAID", "SHIPPED", "CANCELLED", "ON_HOLD"],
        "description": "Order lifecycle status. ON_HOLD added 2026-08-04."
      },
      "placed_after": { "type": "string", "format": "date" },
      "placed_before": { "type": "string", "format": "date" }
    },
    "required": []
  },
  "x-context": {
    "version": "3.3.0",
    "supersedes": "3.2.1",
    "change_type": "additive",
    "change_note": "Enum status gained ON_HOLD (first observed 2026-08-04, 412 rows in sample). Description updated to say ON_HOLD counts as stuck. No parameters removed or renamed.",
    "deprecation": {
      "3.2.1": { "served_until": "2026-09-04", "reason": "Superseded by 3.3.0. Calls after this date fail with this note." }
    },
    "review": { "mode": "auto", "rule": "additive-enum", "approved_at": "2026-08-04T11:02:37Z" },
    "generated_from": "nexset://orders_governed",
    "schema_fingerprint": "sha256:2b77...0d19",
    "generated_at": "2026-08-04T11:02:35Z"
  }
}

The x-context block is not part of the MCP spec. It is an extension block, and any client that does not understand it ignores it. The model does not ignore it, because the model reads the whole definition. That is the trick: metadata the host treats as optional is context the model treats as instruction.

In Nexla the detection half of this already runs: a Nexset compares the schema on every ingestion and flags what changed. MCP Studio generates tools from Nexsets, so regeneration is rerunning generation against the updated data product rather than editing a file. The part we are describing as the pattern, and asking you to hold us to, is the connection between the two: the drift event on the Nexset driving the rebuild and the version bump without a person in the loop for additive changes.

Review: treat context like code

Regeneration without review is how you ship a tool that exposes a column someone just classified as restricted. Every regenerated tool should produce a diff, and the diff should go through a gate. The gate can be a person or a rule, depending on what changed.

Diff the right things. A tool schema diff is a JSON diff and tools for that exist. A glossary diff is a text diff on the definition. A policy diff is a diff on the projection (which columns and rows the tool returns to which role). Put all three in the same review, because a single source change often touches more than one.

Then route by change type. The rule we use:

A natural question is whether you should just put all of this in git. Kaelio’s “Beyond the Semantic Layer” makes the case for a git-native context layer: definitions in a repo, pull requests for changes, CI on merge. For a small number of sources this is right, and it is better than anything that lives in a UI with no history. The diff, the review, and the rollback come for free.

It breaks in two places at scale. First, git is a store for human-authored changes. Drift is machine-detected and happens whenever the source feels like it. At a few hundred sources you are generating dozens of automated commits a day, and a PR queue of machine-authored diffs that humans are supposed to review becomes a queue nobody reads. Second, git holds the text of the definition but not the live link to the source. It cannot tell you that a tool is stale. It can only tell you that someone changed it. The fingerprint, the sample set, the lineage edge, and the policy reference have to live somewhere that talks to the source on every run, and that somewhere is the context layer, with git as an audit trail behind it rather than the system of record in front of it.

Our position: version control semantics, yes. A repo as the place drift is detected, no.

Context CI: the checklist

If your context layer were a codebase, this is the CI you would expect on it. Run through these before you trust an agent on a data product that changes, which is every data product.

1. Every tool definition carries the fingerprint of the schema it was generated from and the timestamp it was generated at.
2. Fingerprints are recomputed on every ingestion (or every call, for cheap sources) and a mismatch raises a drift event automatically.
3. Every enum in a tool is discovered from data, not typed by hand, and re-sampled on a schedule you can name.
4. Tools appear in the lineage graph as downstream assets, so an upstream change invalidates them.
5. Glossary and KPI definitions that feed tool descriptions are subscribed to, and an edit raises a drift event for every tool that uses the term.
6. Tool definitions are generated from the data product, never hand-edited, and regeneration is a single command or an automatic step.
7. Every tool has a version, a supersedes pointer, and a deprecation window for the previous version.
8. Every regeneration produces a diff on schema, description, and policy projection, reviewed together.
9. Additive changes auto-approve by rule. Semantic and policy changes require a named human. Restricted-field exposure blocks and pages.
10. Agents receive a tool list change notification and the definition itself carries a change note the model reads.
11. Calls against a deprecated version are counted, and the count goes to zero before the window closes, or someone finds out who is still pinned.
12. The regression suite for the agent (see the evaluation post in this series) reruns on every context change, not only on every model change.

Twelve items. If you can tick the first five you can detect drift. If you can tick the next four you can propagate it. The last three are how you know it worked.

Drift metrics worth tracking

Freshness has a metric everyone understands: data age. Drift needs its own, and we have not seen anyone publish a set. Here is ours, with definitions tight enough to compute. The targets are where we would start, not numbers we have published.

Time to detect is the one to put on a dashboard first. It is the number that turns “we found out from an analyst three weeks later” into “we found out from the ingestion log the same morning,” and it is the one that your data product owners will understand without a briefing.

Closing

The context layer conversation has been about what to put in front of the model: schemas, samples, lineage, glossary, policy. That is the right list. It is also a snapshot the moment you generate it, and every source in your company is busy making it wrong. Freshness of rows is solved by moving data faster. Freshness of structure and meaning is solved by treating the tool definition as a build artifact of the data product, detecting drift at the source, regenerating with a version, and reviewing the diff like code.

Nobody selling a context layer today describes how they do this. We are describing how we think it should work and what we have built toward it, and we would rather be held to that than be vague. If your agent has ever been wrong for three weeks without an error, you already know why it matters.

Read next: The Tool Description Is the Context and Real-time context for AI agents: why batch data fails.