Agav MCP evals

Does the model use Agav's tools correctly, and say so when it can't answer?

In plain terms

Agav is a dashboard for a household's finances, and people can ask Claude about it through a connector. To check that the AI answers correctly, it is asked 19 questions about a made-up household whose every figure is known in advance, several times each, and every answer is checked against those figures. The results below say how often it was right, where it went wrong, and whether it admits when the data cannot answer.

For the detail: the harness talks to Agav's real MCP connector as a read-only user, and grades four things: whether the model picked the right tools, passed them the right arguments, gave the right number, and declined rather than invented a figure when the data could not answer. Methodology.

Latest published run · 2026-10-12 07:55 UTC

First baseline: 46/57 as graded, 51/57 after review; all 39 attempts with a definite answer passed

80.7%46 of 57 graded attempts passed · 95% CI 69%–89%claude-sonnet-5-5 · effort medium · 3 attempts per question

See every attempt, and what was learned from the failures →

Pass rate across published runs

The shaded band is each run's 95% interval. A run is a few dozen conversations, so a move smaller than the band is noise.

Published runs

First baseline: 46/57 as graded, 51/57 after review; all 39 attempts with a definite answer passed
2026-10-12 07:55 UTC · claude-sonnet-5-5 · medium × 3 · $0.35
81%46/57