How Agav learns

Every bank email you forward is a small puzzle: how much, where, money in or out, and which category. Agav guesses the answer, and sometimes it is wrong.

You might expect the AI model to say “I'm 60% sure” and Agav to ask only when it is unsure. That doesn't work. When we checked, the model said it was about 97% sure almost every time, but it was right only about 8 times in 10. A score that is always high tells you nothing.

Merchant memory is right far more often, because it is repeating something you already told it. So Agav decides whether to ask from who made the guess, not from how sure the guess claims to be:

If Agav never asked about merchant memory, a mistake in memory would never be caught: it would be repeated every month, quietly. So about 1 in 10 of those charges is asked about anyway, at random, like a teacher who checks a few homework sheets even from the best students.

Because the spot checks are random, they also give an honest measure of how often memory is right. Questions people ask only when something looks wrong would make memory look worse than it is.

Each answer you give is a label: the email, next to the right answer. Agav follows one strict rule: only a person's answer counts. It never learns from its own guesses, even confident ones. If it did, a wrong guess would teach it to make the same wrong guess more confidently, and nobody would notice.

Not every forwarded email is a transaction: promotions, one-time passwords, “your statement is ready”. When Agav finds no amount, it posts an alert in Slack with two buttons, Not a transaction and It was a transaction (see Slack). Your answer is a lesson too. These emails are kept for 90 days and then deleted, whether or not anyone answered.

To know whether a new guesser is better than the old one, both have to sit the same test. So about 1 in 5 lessons is set aside as a fixed test and never used for practice. A lesson is always on the same side, so every future version is graded on exactly the same questions.

A household admin can download every lesson from Settings, Training data, as one file. It contains your bank emails' text, so keep it on your own computer. Each download is recorded.

From here on, this page follows the steps that turn your lessons into a guesser of Agav's own. The training steps run on the admin's own computer, from the download above; shadow mode runs inside Agav.

Not every lesson is a good one. Say you renamed a PayNow “Dog toothpaste” and filed it under Pets. You know what it was for, but the bank's email only shows a masked name and an amount: nothing about dogs. Learning from it would teach the guesser that masked PayNows are Pets, which is wrong for most of them.

So, when learning categories, Agav leaves out transfers (PayNow, FAST, GIRO) whose merchant you renamed, and older transfers saved before Agav kept the merchant it first read, because a rename can't be ruled out. They are left out of the test too, so the test can't flatter the guesser. A category with very few lessons isn't guessed at all: the guesser steps aside and today's guess stands.

A small language model reads each email and turns it into 384 numbers. Emails that say similar things get similar numbers, the way nearby houses have nearby addresses. It runs on your computer and inside Agav, so no email text goes to any other company.

Squash those 384 numbers down to two and draw them, and emails from the same bank sit together, because each bank writes its emails from its own template. The next step learns which numbers matter for the category, and looks past the template.

The simplest learner there is, called logistic regression, works out how much each number pushes toward each category: a weighted scorecard. To read a new email, it adds up the points for every category and picks the highest. It learns only from the 4 in 5 lessons that aren't in the test.

It is simple on purpose. With a few hundred lessons, a cleverer learner would memorise them instead of learning from them. What it learns fits in one small file of numbers: no emails, no merchants, only weights.

A scorecard's confidence is usually off: it may say 95% sure and be right only 80% of the time. Calibration adjusts it until “80% sure” means right 8 times in 10.

Think of a weather forecaster. A good one isn't right every day; a good one is honest. Of all the days they said “70% chance of rain”, it rained on about 7 in 10. That is what makes the number useful: you know when to take an umbrella. Agav needs the same honesty to know when to ask you.

On a small test, calibration can overshoot and make things a little worse. The test result is the one to trust, and more lessons make it steadier.

The lessons set aside at the start become the exam. The guesser has never seen them, and every version sits the same one, so versions are compared fairly.

The exam answers one question: if Agav only asked you about guesses it was less than, say, 90% sure of, how many would it ask about, and how often would the rest be right? Raise the bar and Agav asks more, but the guesses it stays quiet about are right more often. Lower it and Agav asks less. Today's way of asking is shown alongside. The bar Agav suggests is the lowest one where the guesses it would stay quiet about are right at least 95% of the time, counted over at least 20 emails. With fewer than 20, the exam says there isn't enough to tell, rather than give a percentage.

You upload the guesser's file in Agav's Settings. Agav shows a short code, the fingerprint, so you can check it got exactly the file you made.

Then it runs in shadow mode: on every new bank email, it makes its guess and writes down how sure it was, but shows nobody. Nothing you see changes. After a few weeks, Settings shows how it would have done on real new emails, next to the checks it must pass before it may be switched on: at least four weeks in shadow, enough answered emails to judge it, a test result at least as good as today's, a suggested bar from the exam (no bar, no switching on), and right at least 95% of the time on the emails it would have stayed quiet about.

Once enough new lessons have built up, train again. Each new version starts in shadow too.

This is the plan, and none of it is built yet.

1. Switch it on. If shadow mode looks good, you press a button. At first that only changes when Agav asks you: it asks when the guesser is unsure and stays quiet when it is sure, apart from the random spot checks. It never changes your categories, and switching back is one click.
2. Ask what a transfer was for. When Agav can't tell what a transfer was for, it always asks, whatever the guesser says. Your answers also let it leave more unhelpful lessons out.
3. Show the scoreboard. Accuracy across all households is published as one total. No household's own numbers are ever shown.
4. Later, your own, online. Each household that wants to can train a guesser of its own, on its own lessons, inside Agav, so your emails never leave it.

Agav guesses, asks when it is unsure (plus spot checks), you answer, it learns again, sits the exam, and tries the new guesser quietly. Switching it on comes next. It learns only from your answers, never from its own guesses.

What this looks like inside Agav: How Agav learns from you.