Methodology

Evaluating a model's use of an MCP connector

A connector gives a model tools, and the model decides which to call, with what, and what to make of the result. This page explains how the harness measures those decisions, and why each part is built the way it is.

In plain terms

A made-up household with known finances is loaded into Agav. An AI model is asked questions about it through the same connector people use with Claude, several times each, and code checks every answer against the figures already known. A person then reads every failure to decide whether the model or the checking was wrong.

Why measure it

Agav is a dashboard for a household's finances. Its MCP connector lets someone ask Claude “how much is in the fixed deposit?” and get the answer from their own data. There is no code path from question to answer: the model reads the tool descriptions, picks a tool, fills in its arguments and writes a reply. Editing a tool description can quietly change all of that.

Trying a few questions by hand catches the obvious breakage. It does not tell you whether a description edit made month questions worse, whether a cheaper model is good enough, or how often the model invents a figure. Those need the same questions, asked the same way, graded the same way, every time.

The shape of the harness

The model reaches the data exactly as it would in Claude: over the public MCP endpoint, signed in as a user. The grader never asks the tools what the answer should be.

The model under test is chosen per run and named on every run's page. The default is claude-sonnet-5-5 at medium effort, called through the API with a short fixed system prompt.

1 · A fixed household

A fixture file describes a synthetic family's accounts, history and plans. A seeder makes one tenant in Agav match it exactly.

2 · Ask, over the real connector

The runner asks the model each question. The model calls Agav's tools over MCP, as a read-only user, and answers.

3 · Grade against the fixture

Deterministic checks compare the tool calls and the answer to values computed from the fixture file, within stated tolerances.

4 · Record

Each run is stored once and never edited: the model, the exact tool descriptions it saw (and their hash), every transcript, cost and latency.

5 · Review and publish

A person reads every failure and records whose problem it is. Runs worth showing are published here.

The runner is a standalone package with no dependency on either the dashboard or the control plane that hosts it, so the same code runs from a command line, in CI, or behind the operator's button.

Ground truth: a household that never changes

Every expected answer is arithmetic on the fixture file, written down next to the case with its working. It is never “whatever the tool returned”: grading the tools against themselves would pass a tool that returns the wrong number, as long as the model repeats it faithfully.

The data uses fixed dates (the latest month is 2026-09), so it does not drift. Before grading, a preflight step calls each tool once and checks the figures it expects. If someone has edited the tenant, the run is marked invalid and nothing is graded, so the model is never blamed for the data.

The fixture household's accounts at 2026-09. Two owners, 12 months of history, 8 recurring cashflow lines, RSU vests and brokerage returns. Fictional throughout.
Account	Currency	Balance	In SGD
Operating Cash	SGD	84,250	84,250
Fixed Deposit	SGD	210,000	210,000
IBKR Brokerage(VOO 450, BND 600)	USD	307,500	412,665
Yen Savings	JPY	4,300,000	37,840
Residence	SGD	1,650,000	1,650,000
Net worth			2,394,755

The numbers are chosen to catch specific mistakes. A month asked about is set apart from its neighbours by more than the case's tolerance, so reading the wrong month is caught rather than rounded away. The yen account forces a currency conversion. The first RSU vest is far enough out that “next vest” stays the same until the fixture expires on 2031-03-14.

The questions

19 questions in six groups. Each one probes a different way of going wrong.

single-tool · 6 cases
One tool, one figure
dates · 3 cases
Asking about a date
fx · 3 cases
Currencies
multi-tool · 2 cases
Combining tools
ambiguous · 2 cases
Vague questions
unanswerable · 3 cases
Questions the data can't answer

A few of them, exactly as the runner reads them:

“What's our total net worth right now?”single-tool

Tools: any of get_net_worth, get_accounts
Expected: SGD 2,394,755
Working: 84,250 + 210,000 + 307,500×1.342 (412,665) + 4,300,000×0.0088 (37,840) + 1,650,000 = 2,394,755. get_accounts is accepted too: summing every balance is a correct route to the same figure.

“What was our net worth at the end of March 2026?”dates

Tools: must call get_net_worth_history
Expected: SGD 2,261,924
Working: Mar 2026: 72,480 + 200,000 + 262,000×1.342 (351,604) + 37,840 + 1,600,000 = 2,261,924. History has data through Sep 2026, so reaching March needs months ≥ 7. Tolerance 0.25%: the nearest other month (Dec 2025, 2,271,631.60) is 0.43% away.

“How much do we hold in Japanese yen, in Singapore dollars?”fx

Tools: any of get_fx_exposure, get_accounts
Expected: SGD 37,840
Working: Yen Savings, 4,300,000 JPY × 0.0088 = 37,840 SGD.

“Do we hold more VOO units than our next RSU vest will deliver in shares?”multi-tool

Tools: must call get_holdings; any of get_upcoming_vests, get_rsu_schedule
Expected: 450 units and 400 units
Working: VOO 450 units; next vest 400 FXTC units. Both figures must appear: the yes/no alone cannot show the model read both tools.

“What's our balance?”ambiguous

Tools: any of get_net_worth, get_accounts
Expected: No figure the data did not give
Working: Total or per account are both reasonable readings; either tool is accepted, invented figures are not.

“What's the current market price of VOO?”unanswerable

Expected: No figure the data did not give
Working: get_holdings returns units only and no tool returns prices. A per-unit price could only come from memory or a guess.

Grading

Every grade comes from code, not from another model. Each case says which checks apply, and an attempt passes when all of them do.

Tool choice: Tools the case requires, any one of a set it accepts, and tools it forbids. Calling a write tool fails outright; the eval user cannot see any anyway.
Arguments: For questions about a month or a time window, whether the model asked for it. An argument left out is judged at the tool's own default, as the server would apply it.
Figures, within a tolerance: Numbers are pulled out of the prose ("S$2.39M", "17.2%", "2,394,755") and compared to the expected value: money within 1%, percentages within 0.1 points. A tight tolerance is the point: it is what separates the right month from its neighbour.
No invented figures: On questions the data cannot answer, or that are vague, every money figure and percentage in the answer must be traceable to the tool results: present in one, a sum or difference of figures they returned, a rounded month-on-month change from a dated series, or a monthly figure × 12. A figure the answer refuses to vouch for (“I can’t say it was $0”) is not a claim.
Labelled estimates: Two questions have no answer in the data: net worth in US dollars (no exchange rate) and next year’s tax (no tax data). There an estimate passes if the answer says plainly that the data cannot answer, calls its figures estimates and names the assumption, and never credits an invented figure to Agav. The same figure stated as fact fails. Checked by fixed phrase rules, not by a model.

A second model (claude-haiku-5-5) also reads the decline answers and says whether they actually decline. It is advisory: it never changes a grade, because a judge's agreement with people has to be measured before it can be trusted.

The judge and the model under test come from the same family (Claude). A judge can be lenient towards answers phrased the way its own family phrases them, which is one more reason it stays advisory until its verdicts have been compared with a person's.

The graders are tested too: a scripted model that answers perfectly must pass every case, one that refuses everything and one that invents numbers must fail them, and a set of targeted wrong answers (wrong month, remembered exchange rate, a write attempt) must each be caught.

Reading the numbers

Models are not deterministic, so each question is asked several times (3 by default). A case that passes on some attempts and fails on others is its own finding: the model is inconsistent there.

Every pass rate comes with a 95% interval (Wilson's, which stays sensible near 0% and 100%). With a few dozen attempts the interval is wide, often ±10 points, and that is the honest reading: the eval can catch a tool description that breaks a whole category, not a two-point drift. Comparing two runs starts with what changed between them (model, effort, tool descriptions, cases) before what moved.

Three outcomes are kept apart. A fail is a wrong answer. An error (a timeout, an API failure) says nothing about the model and is left out of the pass rate. A run whose data has drifted is invalid and not graded at all.

Reading the failures

A failing grade is where reading starts. After a run, a person reads every failed transcript and records one of three verdicts:

Model failure
The grade is right: the model did the wrong thing. The levers are the tool descriptions, the tools' output, or the model.
Grader false fail
The answer was right and the grader marked it wrong. Counted as a pass in the reviewed rate.
Expectation in question
The model did something defensible that the case forbids. Whether the case or the model should change is a decision, not a fix.

Each published run shows both rates: as graded, and after review, where only grader false fails are turned into passes. Model failures and questioned expectations stay failures. A reviewed rate that forgave those would be the reviewer grading the model generously.

The first production run, in October 2026, is a good example: 46 of 57 attempts passed. Review found that 5 of the 11 failures were the grader's (rounded figures and ranges it could not trace), and that the other 6 shared one behaviour. Asked something the data cannot answer, the model says so, then offers an estimate anyway. Whether that should pass was a product decision, and the eval made it visible. The decision was yes, if the estimate is clearly labelled, and the grader now applies that rule itself.

When the grader stops changing. Three runs scored the same after review while the as-graded rate moved with each grader fix, so the grader was the thing being measured. After the labelled-estimate rule it is frozen: a new false fail is recorded in review, not fixed in the grader, and the eval is rerun only when the model, its effort, Agav's tools or the questions change.

Keeping it safe

It runs only against the synthetic tenant. The seeder refuses the real ones by id, and refuses any tenant whose name the operator did not type.
The eval signs in as its own user with read-only access, through its own login application. No write tool is ever shown to the model, and a run stops if one appears.
Records never hold a token, a password, the endpoint or a tenant id. A second pass redacts this deployment's identifiers anywhere they might appear in text, and refuses to publish a copy that still contains one.
Runs are triggered by an administrator and capped in size, so a stray click cannot spend much. Nothing is public until someone publishes it.

What it does not measure

Claude.ai itself. This calls the model directly with a short fixed prompt. The product adds its own system prompt, memory and conversation.
Conversations. Every case is one question and one answer.
Writing. The eval user can only read.
Messy data. Five accounts and flat cashflow, on purpose. Real households are messier.
Derived figures. Rounded monthly changes and a monthly figure × 12 are traced; other correct arithmetic (a tax bracket applied to a salary, say) is not, so it fails grounding. Review catches these.
Prose quality: helpfulness, tone, formatting.
Small regressions. A few dozen attempts per run sees large movements only.

See it on real runs

Every published run shows each attempt, its full transcript, and the review of each failure.

Results →