Schema Drift Reaches the Tool Definition
Context layer series Everyone who sells a context layer talks about freshness. Fresh rows, streaming…
Aug 20, 2026
Context layer series
Two MCP servers sit in front of the same warehouse. You ask both “how many active customers did we have in EMEA last quarter?” Both answer 1,284. Both score 100 percent on your benchmark. One of them understood the question. The other one guessed, and the guess happened to land. Your scoreboard cannot tell them apart. This post is about building one that can.
We have published three MCP server benchmarks at Nexla. They measured tokens, tool calls, turns, and pass rate. That was the right first step and the numbers were real. It was also an incomplete picture, and this post is partly a correction of our own method. The memory vendors have the same problem from the other side: recall on conversation benchmarks tells you whether the agent remembered what you said, not whether it understood your data.
Same score, different truth
Here are the two traces behind that 1,284. Hover over each card (or tab into it) and the step where the model guessed lights up.
Server A: generic warehouse server pass
1 list_tables returns 212 tables
2 describe_table(customers) status, is_active, region_code, geo, created_at, updated_at, plus 41 more columns
3 model reasoning: “is_active sounds like the right flag. region_code probably starts with EU for EMEA. I will filter on both.”guessed
4 run_sql: SELECT COUNT(*) FROM customers WHERE is_active = true AND region_code LIKE ‘EU%’ AND created_at < ‘2026-07-01’
5 answer: 1,284
Why it passed: is_active and the governed definition of active (a paid subscription with status ACTIVE or PAUSED_BILLING in the quarter) agreed for this quarter. The Middle East rows are in region_code values that begin with EU because of a 2019 migration. The time filter is on the wrong column and ignores the quarter start. Three accidents cancelled out.
Server B: task-specific server pass
1 active_customer_count(region=”EMEA”, period=”2026-Q2″)
2 tool description read by the model: “active means subscription_status in (ACTIVE, PAUSED_BILLING) at any point in the period. region maps to the governed sales_region field, not region_code. period resolves to fiscal quarter bounds.”
3 answer: 1,284
Why it passed: the definition, the region field, and the window came from the context layer, and the trace shows it. Nothing to guess. The next quarter, when is_active drifts from the real definition, this server still passes. Server A silently fails and keeps reporting a green pass rate on the old test set.
A benchmark that reports pass rate treats both traces as equal. The difference only shows up if you look at how the answer was produced, score the steps, and check them against what the data actually means. That is the whole idea of evaluating a context layer: you are not scoring the answer, you are scoring the grounding.
What the current measures miss
Three families of metrics dominate the conversation today. Each one measures something real. None of them measures whether the agent understood the data.
Token counts
Tokens are a cost proxy. Our HubSpot benchmark showed 21,682 tokens against 708,973 for the same task, and that gap matters for the bill and for context window health. But a server that loads fewer tokens and guesses the column is not better than one that loads more and gets the column right. Several 2026 posts on model economics (DoiT’s “cost per task, not per token” among them) already make the case that tokens are the wrong denominator for model spend. The same argument applies one layer down. Tokens per call tells you what a call costs. It tells you nothing about whether the call was correct for the right reason.
Pass rate on hand-written questions
Every benchmark we have run, and every vendor benchmark we have read, uses a question set written by the people running the test. That set is clean, unambiguous, and small. Production questions are not. Real users say “active” and mean four different things. They ask for “last quarter” on the third day of a new quarter. They reference a customer by a name that matches two accounts. A hand-written set hides exactly the cases where grounding matters, which means a 100 percent pass rate on it is a floor, not a result.
Memory recall benchmarks
LoCoMo and LongMemEval test whether an agent can recall facts from a long conversation. Mem0, Zep and Letta report against them. These are useful numbers for a memory layer. They say nothing about a context layer, because the question they answer is “did the agent remember what the user said” and the question you need answered is “did the agent understand what the data means.” Different layer, different failure modes, different test.
| Measure | What tokens per call tells you | What trace-scored evaluation tells you |
|---|---|---|
| Cost | Spend per call | Spend per correct answer, including retries and clarification turns |
| Correctness | Nothing | Whether the answer is right and whether the call used the right fields |
| Definitions | Nothing | Whether “active,” “revenue,” “churned” matched the governed definition |
| Freshness | Nothing | Whether the data read was inside the task’s freshness bound |
| Policy | Nothing | Whether any row or column leaked past the caller’s entitlement |
| Drift | Nothing | Which cases break when a schema, definition, or policy changes |
A failure taxonomy for grounded tool calls
The Berkeley, UW and Hasura Data Agents Benchmark (DAB, March 2026) is the closest prior art. It scores data agents across five failure modes, and one of those modes is “wrong data selection.” When we labeled our own production traces, almost every grounding failure landed in that one bucket, which made it useless for deciding what to fix. So we split it. Five classes, each with a distinct trace symptom and a distinct fix. Future AGI’s MCP evaluation metrics (tool selection precision and recall, argument correctness) sit one level above this: they tell you the agent chose the right tool and filled the arguments. This taxonomy tells you whether the arguments meant the right thing.
Pick a class to see a real example, what it looks like in the trace, and what fixes it.
The call refers to the wrong thing: wrong identifier type, wrong record, wrong object.
Example A support agent is asked to pull “the open tickets for Acme.” The CRM has account_id (CRM), billing_account_no (billing), and cust_id_v2 (canonical). The model passes the CRM account ID into a ticketing tool that keys on cust_id_v2. The tool returns zero rows, and the agent tells the user Acme has no open tickets.
Trace symptom An argument whose format matches one ID scheme passed into a tool that documents another, or no documentation of which scheme at all. Often followed by an empty result the agent treats as a true negative.
Fix Every identifier parameter states which identifier it is, and the context layer supplies the crosswalk. Covered in P2.
The call uses the right entity but the wrong meaning: a business term resolved to the wrong predicate.
Example “Active customers” in finance means a paid subscription in the period. In product it means logged in within 30 days. In the warehouse there is is_active, status, and last_login_at. The model picks is_active. The finance user gets a product number and reports it to the board.
Trace symptom A reasoning step that contains “sounds like,” “probably,” or “I will assume.” A WHERE clause that uses a column whose name resembles the business term instead of the governed definition.
Fix Definitions travel with the tool (the predicate is in the description or the tool encodes it), and the eval checks the predicate, not just the number.
Right entity, right meaning, wrong shape: a bad join, a missed pagination, a fan-out, a unit mismatch.
Example Revenue by customer. The model joins orders to order_lines and sums orders.total after the join. Every order with three lines counts three times. Revenue triples for multi-line accounts and the top ten list is wrong. In a second case, a list tool returns 100 rows per page, the model reads page one and reports “100 results.”
Trace symptom Aggregation after a one-to-many join. A paginated response with next_cursor set and no follow-up call. Amounts in cents compared to amounts in dollars.
Fix Tools that expose the grain and the pagination contract, and governed data products (Nexsets) that pre-resolve the join so the model never does it.
Right everything, wrong time: a stale snapshot, the wrong window, the wrong timestamp column.
Example “Pipeline at risk this week” is answered from a nightly snapshot that is 14 hours old, after a morning in which three deals slipped. Or “last quarter” is computed on created_at instead of closed_at, which is what the metric is defined on. Or the quarter is calendar when finance runs fiscal.
Trace symptom No freshness metadata in the tool result. A date filter on a column other than the one the metric definition names. A window computed from today’s date with no fiscal calendar lookup.
Fix Tool results carry as_of, tools declare which timestamp the metric uses, and the eval has a freshness bound per task. See why batch data fails agents.
The answer is correct and the caller was not allowed to have it.
Example A regional sales manager asks for “deals closing this month.” The tool runs with a service credential and returns every region. The number is right. Rows the user has no entitlement to are sitting in the agent’s context and in the answer.
Trace symptom A tool call made with a credential that does not carry the end user’s identity. A result set whose row count exceeds what the same user gets in the source UI. A column in the result that is masked in the source.
Fix User identity pushed down to the connector and enforced there. The eval runs each case as a named user and compares against a policy oracle. See MCP security: identity, credentials, policies.
The five classes are ordered by where the fix lives. Entity and semantic errors are fixed in the tool definition. Structural errors are fixed in the data product. Temporal errors are fixed in the result contract. Policy errors are fixed in the gateway. When you label a failure, you are also assigning it to the team that owns the fix, and that is what makes the taxonomy worth the labeling cost.
Build the eval set from traces, not from imagination
You do not write evaluation cases. You harvest them. Hand-written questions are the reason every vendor benchmark, ours included, hits 100 percent. Production traces contain the ambiguity, the near-miss identifiers, and the half-specified windows that expose whether the context layer is doing its job.
Traces
Every tool call, argument, result, and model reasoning step from production runs
Sample
Stratify by tool, by user role, by outcome. Oversample retries and clarifications
Label
Tag each failed or lucky call with one of five classes plus the gold fields and definition
Regression suite
Keep the cases that expose grounding. Drop the ones any server passes
Rerun
On every schema, definition, policy, or tool change
Four steps, each with a rule that keeps the set honest.
Sample real runs, stratified. Take a week of production traces. Stratify by tool, by user role, and by outcome (clean pass, retry, clarification, failure). Oversample the retries and clarifications: that is where grounding failures hide, because the agent usually recovers and the top-line metric never sees it. Two hundred traces is enough to start. Fewer than fifty is not.
Label with the taxonomy. For each trace, a human (or a model with a human checking) records: was the final answer correct, which fields the call used, which fields the governed definition requires, which failure class applies if any, and whether the answer was correct for the right reason. That last flag matters most. Server A in the opening example gets “correct: true, grounded: false, class: semantic.”
Keep the cases that expose grounding. If every server you test passes a case, it is not telling you anything. Keep the cases where at least one configuration fails or passes ungrounded. This is the opposite of how most teams build test sets, and it is why the sets stay small and sharp.
Turn it into a regression suite. The suite reruns on every change to the context: a schema change detected on a data product, a glossary edit, a policy update, a regenerated tool definition. P3 covers how drift reaches the tool. This is how you find out whether the regenerated tool still grounds the same way. The suite is the test in “context CI.”
Here is one labeled case from the opening example, in the format the harness reads.
cases.jsonl one labeled trace case
{
"case_id": "emea-active-q2-017",
"source_trace": "trace_9f31c2",
"user_role": "finance_analyst",
"as_user": "fin.analyst.02",
"question": "How many active customers did we have in EMEA last quarter?",
"gold": {
"answer": 1284,
"tolerance": 0,
"required_fields": ["customers.subscription_status", "customers.sales_region", "subscriptions.period_start", "subscriptions.period_end"],
"forbidden_fields": ["customers.is_active", "customers.region_code", "customers.created_at"],
"definition": "active = subscription_status in (ACTIVE, PAUSED_BILLING) at any point in the fiscal quarter",
"window": { "start": "2026-04-01", "end": "2026-06-30", "calendar": "fiscal" },
"freshness_bound_hours": 24,
"policy_scope": "all_regions"
},
"label": {
"correct": true,
"grounded": false,
"failure_class": "semantic",
"note": "Used is_active and region_code LIKE 'EU%'. Matched by coincidence this quarter."
},
"cost": { "input_tokens": 41210, "output_tokens": 1880, "tool_calls": 4, "turns": 3, "usd": 0.0912 }
}The forbidden_fields list is what catches Server A. The as_user field is what makes the policy class testable at all. The cost block is what makes the last metric possible.
The metrics
Six numbers. Each has a precise definition so two teams computing it on the same traces get the same result. The dashboard below uses illustrative values so you can see the shape of a report. They are not measured numbers.
Illustrative values. Not measured. Replace with your own run.
Grounding precision
0.91
Grounding recall
0.84
Definition match rate
0.78
Freshness pass rate
0.95
Policy violation rate
0.03
Clarification loops per task
0.40
Grounding precision and recall
For each call, compare the set of fields the call actually used (columns in the query, arguments in the tool call, joins performed) against the gold set of required fields for that task.
Grounding precision = fields used that are in the required set / all fields used. A call that touches is_active when the definition needs subscription_status loses precision.
Grounding recall = required fields that were used / all required fields. A call that forgets the period bound loses recall.
Report both at the task level (averaged across calls in the task) and at the suite level. A server that is correct for the right reason has precision and recall of 1.0 on that case. Server A scored 0.25 and 0.25: one field right (the count target) out of four used, one out of four required.
Definition match rate
Definition match rate = tasks where every business term in the question resolved to its governed predicate / tasks containing at least one business term. “Resolved to” means the predicate in the query or tool arguments is equivalent to the definition string in the gold record. This is the metric that separates the two servers in the opening. Pass rate said 100 and 100. Definition match rate says 0 and 100 on that case.
Freshness pass rate
Freshness pass rate = tasks where the as_of timestamp of every result was within the task’s freshness_bound_hours / tasks with a freshness bound. If the tool result carries no timestamp, the task fails freshness, because an agent that cannot tell the user how old the data is has failed the user even when the number is right.
Policy violation rate
Policy violation rate = calls that returned at least one row or column the as_user is not entitled to / all calls. You need a policy oracle: a function that, given a user and a result set, says which rows and columns are allowed. In practice this is the source system’s own permission check, run on the result after the fact. Any nonzero value is a release blocker, and the metric exists so that you see it before the auditor does.
Clarification loops per task
Clarification loops per task = turns in which the agent asked the user a question instead of calling a tool / tasks. Some clarification is good: an agent that asks “fiscal or calendar quarter” is grounded enough to know the question is ambiguous. A high number is a symptom that the context layer is not resolving terms the definitions already resolve. Track it alongside definition match rate and you learn whether the agent is asking because it does not know, or because nothing told it.
Cost per correct answer
Cost per correct answer = total cost across all tasks / number of tasks answered correctly and grounded. Total cost includes every token (input and output, priced at the model’s rate), every tool execution, every retry, and every clarification turn the user had to answer. The denominator is not “answered” and not “correct.” It is correct and grounded, because an ungrounded correct answer is a liability you have not paid for yet.
This is the number a buyer should demand. Tokens per call rewards a server that loads little and guesses. Pass rate rewards a test set that is too clean. Cost per correct answer is the only one of the three that gets worse when a server is cheap and wrong, or expensive and right, and better only when it is both grounded and efficient.
How this differs from a catalog-centric list
Atlan’s context quality post proposes metrics like certified asset share and quality warning coverage. Those measure the catalog: what fraction of assets have an owner, a certification, a description. They are inputs. The six metrics here measure outcomes at the tool call: did the agent use the certified asset, did it apply the definition the catalog holds, did the user get only what they were allowed to see. You need both. A high certified asset share with a low definition match rate tells you the context exists and is not reaching the agent, which is a pipeline problem, not a catalog problem.
Applying it to our own benchmark
We are rerunning the HubSpot benchmark with the labeler and the six metrics. The four numbers we published stay as they were. The six new columns are blank until the run is done, and we will publish them whether or not the task-specific server wins every one. The honest expectation is that it wins on grounding precision and definition match (the tool carries the definition) and on cost per correct answer (fewer turns), and that clarification loops and freshness are closer than the token gap suggests, because both servers read the same HubSpot API with the same staleness.
| Metric | Generic HubSpot MCP | Nexla task-specific server | Status |
|---|---|---|---|
| Total tokens | 708,973 | 21,682 | published |
| Tool calls | 18 | 6 | published |
| Turns | 10 | 2 | published |
| Pass rate | 100% | 100% | published |
| Grounding precision / recall | TBD | TBD | rerun |
| Definition match rate | TBD | TBD | rerun |
| Freshness pass rate | TBD | TBD | rerun |
| Policy violation rate | TBD | TBD | rerun |
| Clarification loops per task | TBD | TBD | rerun |
| Cost per correct answer | TBD | TBD | rerun |
One thing the rerun already changed: the original HubSpot question set was eleven hand-written tasks. The rerun adds traces sampled from real MCP Studio sessions against HubSpot, labeled with the taxonomy, and keeps only the ones that expose grounding. That set is harder, and the 100 percent on both sides is not going to survive it. That is the point.
An open eval harness you can run
The harness is four files and no framework. It runs against any MCP server, not only ours, because the thing being scored is the trace, and every MCP client can produce one.
cases.jsonl holds one case per line in the schema below. run.py connects to the server under test, replays each question as the named user, captures the full trace (tool calls, arguments, results, model reasoning when the client exposes it), and writes traces/. label.py scores each trace against its gold record: field overlap for grounding precision and recall, predicate equivalence for definition match, as_of against the freshness bound, the policy oracle for violations, and the cost block from the client’s usage report. It writes labels.jsonl, with a flag for any case the automatic scorer is unsure about so a human reviews only those. report.py aggregates into the six metrics plus the per-class failure counts.
Case schema what each line of cases.jsonl has to carry
{
"case_id": "string, unique",
"source_trace": "string, optional, id of the production trace this came from",
"user_role": "string",
"as_user": "string, identity the runner impersonates, required for policy scoring",
"question": "string, the user's words, unedited",
"gold": {
"answer": "number | string | array",
"tolerance": "number, absolute tolerance for numeric answers",
"required_fields": ["schema.table.column or tool.argument"],
"forbidden_fields": ["fields whose use marks the call ungrounded"],
"definition": "string, the governed predicate for each business term in the question",
"window": { "start": "date", "end": "date", "calendar": "fiscal | calendar" },
"freshness_bound_hours": "number, null if not applicable",
"policy_scope": "string, the entitlement the oracle checks against"
},
"label": {
"correct": "boolean",
"grounded": "boolean",
"failure_class": "entity | semantic | structural | temporal | policy | null",
"note": "string"
},
"cost": { "input_tokens": "int", "output_tokens": "int", "tool_calls": "int", "turns": "int", "usd": "number" }
}The report has a fixed set of fields so you can diff two runs (two servers, or one server before and after a context change).
report.json fields, one object per server under test
{
"server": "string",
"run_id": "string",
"cases": "int",
"correct": "int",
"correct_and_grounded": "int",
"grounding_precision": "float, mean over tasks",
"grounding_recall": "float, mean over tasks",
"definition_match_rate": "float",
"freshness_pass_rate": "float",
"policy_violation_rate": "float",
"clarification_loops_per_task": "float",
"cost_total_usd": "float",
"cost_per_correct_answer_usd": "float",
"failures_by_class": { "entity": "int", "semantic": "int", "structural": "int", "temporal": "int", "policy": "int" },
"needs_human_review": "int"
}Two things the harness will not do for you. It will not write your gold records: someone who knows the data has to say which fields a task requires and what “active” means, and that is the work the context layer is supposed to make reusable. And it will not build the policy oracle: you point it at the source system’s permission check, or you write one. Both are one-time costs, and both pay back the first time a schema change silently breaks a tool and the suite catches it before a user does.
The repository link is a placeholder until the rerun is published: github.com/REPO-LINK.
What to do tomorrow
Pull a week of traces from whatever agent you have in production. Sample fifty. Label them with the five classes. Count how many were correct and not grounded. If the number is zero, your context layer is working and your test set was already hard. If it is not zero, you now have a regression suite, a failure distribution that tells you which team owns the fix, and a denominator for the only cost number that matters.
We built Helix and MCP Studio to make the grounding step cheap: schemas, samples, lineage, definitions, and policies attached to every tool call instead of left for the model to guess. Whether that works is an empirical question, and tokens saved was never the right way to answer it. This is.
Run the harness against your MCP servers
Point it at a generic server and a task-specific one built in MCP Studio, label fifty traces, and compare cost per correct answer. If the task-specific server does not win on grounding, we want to see the trace.
Read next: Schema drift reaches the tool definition and Nexla MCP server vs HubSpot MCP: the original benchmark.
