How Agav learns

Agav reads each forwarded bank email with a language model, which returns the amount, merchant, date, direction and a category, along with its own confidence score. Measured against people's corrections, the model claimed a confidence of about 0.97 on 421 of 450 cases, and those were right 79.8% of the time.

A score that sits near the top almost every time carries no information. A threshold on it either asks about everything or about nothing, so it cannot decide when a person should be asked.

Two things decide a category. The model proposes one; and merchant memory, a person's earlier correction for the same merchant, overrides it when it exists. On the category ladder, memory overrides were right 90.2% of the time and the model 58.3%.

So Agav decides whether to ask from which source decided, never from the stated confidence. A charge the model categorised is always asked about. A memory decision is accepted, apart from the spot checks below. Money coming in is reviewed separately, as money in.

A uniform random 10% of memory-decided charges are asked about anyway, and marked as audit questions.

Two reasons. First, a confident mistake in memory would otherwise repeat every month without ever being seen. Second, the questions people raise unprompted are a biased sample: they are about charges that already looked wrong. A random sample gives an unbiased estimate of how often memory is right.

A label is an example paired with the right answer. Only a person's answer is a label. The model's own guesses never are, including confident ones: training on them would teach the model its own mistakes, more confidently, with nobody to notice.

Forwarding sends everything, not only transactions: promotions, one-time passwords, statement notices. When no amount is found, the email is kept and the household is asked in Slack whether it was a transaction. The answer labels relevance, the data for a third classifier that tells the two apart. Kept emails are deleted after 90 days, answered or not.

About 20% of labelled examples are held out as a test set, chosen by a stable hash of each example's identity. An example never moves between practice and test, so every future model version is graded on exactly the same questions, and versions can be compared fairly.

A household admin can download the household's examples, raw email text included, to train on their own machine. Only an admin who is an actual member of the household can; operator access and emergency access are refused. Each download is audited, and the audit record holds counts only.

From here the page follows the pipeline that turns labels into a model of the household's own, in the order it runs. Training happens on the admin's own computer, from the download above.

A label is only useful when the email carries evidence for it. Some corrections do not. A transfer a person renamed, like a PayNow filed as “Dog toothpaste” under Pets, says nothing about dogs in the email: the bank wrote a masked name and an amount. A classifier trained on it learns that masked PayNows are Pets, and an exam that kept the same labels would flatter it.

So, for the category, rows on a transfer rail (PayNow, FAST, GIRO) are left out of both the practice and the test labels when the merchant was renamed, and when the row was saved before Agav kept the merchant it first read, so a rename cannot be ruled out. Every exclusion is counted by reason. The text the model reads is always what the email said, never an edited title. Direction labels are not affected.

Categories with fewer than 8 practice labels are pooled into one defer class. When it wins, the model abstains and today's decision stands.

A small open embedding model, BGE small for English, quantised to 8 bits, turns each email into 384 numbers. It reads the issuer, the payment rail, the subject, the merchant as first read and the body, in that order, cut to 512 tokens from the end so the body is shortened first. The vector is taken from the classification token and normalised to unit length, so similar text gives nearby vectors. Vectors are cached by a hash of the model and the text, so a re-run embeds only new rows.

The same embedding file runs on the computer and inside Agav, pinned by its content hash; Agav refuses a model trained against any other. No email text is sent to another company to be embedded.

Two feature sets are trained and graded on every run. The first is the embedding plus one-hot issuer and rail. The second also adds the category the language model proposed, which makes the classifier, in effect, a calibrator of the language model: stronger while its prompt is stable, weaker when the prompt changes. A person picks which to package.

Emails cluster by bank template. A two-dimensional projection of the embeddings onto their principal components shows emails from one bank sitting together, because the directions of greatest variance in bank email follow each issuer's template, not the category. That picture is only an illustration: the scorecard weighs all 384 dimensions and learns the ones that separate categories, so it looks past the template.

The learner is a multinomial logistic regression with balanced class weights: one weight per feature per class, plus an intercept. To read an email it adds up the weighted features for each class and turns the totals into probabilities with a softmax. Direction is the same learner with two classes, over labelled rows no per-bank rule already decides.

The regularisation strength is chosen from 0.01, 0.1, 1 and 10 by stratified five-fold cross-validation on the practice labels, scored on log-loss, then the model is refitted on all of them. Below 100 category labels, or 30 per class for direction, a run stops and says the data is insufficient.

It is the simplest learner that could work, on purpose: with a few hundred labels, a bigger model would memorise them. Training is in Python, but what it produces is a plain file of numbers (the classes, the weights, the intercepts and the calibration), never a serialised program. Agav applies it with a dot product, a softmax and the calibration, so Python never runs in production.

A scorecard's probabilities are usually off: it may say 95% and be right 80% of the time. Calibration fixes that. Think of a weather forecaster: of all the days they said 70% chance of rain, it should rain on about 7 in 10. A forecaster can be honest without being right every day, and only an honest number can decide when to ask.

Category uses temperature scaling: one parameter divides the scores before the softmax, softening or sharpening every probability at once. Direction uses Platt scaling, a slope and an offset. Both are fitted on the out-of-fold predictions from the cross-validation, never on the test labels. Isotonic regression is avoided because it overfits a few hundred labels.

The reliability chart puts the test predictions into the same five confidence bands as Agav's inference evaluation and plots each band's stated confidence against how often it was right. An honest model's points sit on the diagonal. The expected calibration error (ECE) is the weighted gap between the two.

On a small exam, calibration can overshoot. It is fitted on the labels learned from, so on few test emails it can make the exam's ECE worse rather than better. The exam number is the one to trust, and more labels make it steadier.

The exam is the fixed test set above: about 20% of labels, never fitted or tuned on, and the same for every version. Accuracy and ECE grade the model's top choice. A few dozen test rows make two separate accuracies too noisy to compare, so the model is compared with today's pipeline on the same rows: both right, only the model, only the pipeline, neither. The results are also split by who decided today, merchant memory or the language model, because the model's value is on merchants memory does not know.

The number that sets everything else is the coverage curve. It uses the probability the model gives the category today's pipeline decided, since the question is whether the stored value is right. For a threshold τ from 0.50 to 0.99, rows below τ would be asked about and the rest would stay quiet. The curve plots the share asked against the accuracy of the quiet rows, with today's pipeline alongside.

The recommended τ is the smallest at which the quiet rows are at least 95% right, over at least 20 rows. If there is no such τ, there is no recommendation. Any ratio over fewer than 20 rows is reported as insufficient rather than as a percentage.

Upload is manual. The admin uploads the weights file in Agav's Settings, which checks it was trained for this household and against the deployed embedding, stores it privately and shows its fingerprint: the first 12 characters of the file's hash, to compare with the one the trainer showed. Every new model, and every retrained version, starts in shadow mode.

In shadow, each new email charge is embedded once, after Agav has made and stored its own decision, and every model in shadow writes down two calibrated probabilities: of its own top choice, and of the category Agav decided. Nothing a person sees changes. A model error is reported and never fails the email.

Settings grades each model's shadow predictions against the final values with the same rules as the weekly evaluation: accuracy, calibration error, the five bands, and no ratio over fewer than 20 rows. Beside them it shows the checks that switching a model on will need, each passed or not, with its numbers:

• on the exam, accuracy at least today's pipeline's on the same rows, and a calibration error no worse than the live model's, when there is one;
• a recommended τ from the exam;
• at least 28 days and 20 labelled predictions in shadow;
• labelled shadow rows at or above τ right at least 95% of the time. These are the charges a live model would stop asking about.

Retraining is manual too. Settings and the trainer show how many labels have arrived since the newest model; at 50 or more, it is worth a retrain.

Not built yet. What comes next:

• Switching a model on. A person promotes a model whose checks all pass, and can roll it back in one click. At first, live means only that it decides when Agav asks: below τ, a category question is asked whatever the source; at or above τ, only the random 1 in 10 spot check asks, for every source, so confident mistakes keep surfacing. It never changes a value; replacing values is a later step with its own gate. Only a category model can be switched on at first, since there is no direction question for one to decide.
• Asking what a transfer was for. A transfer whose purpose is unknown always gets that question, whatever the classifier says, and the first real training run waits for it, so it can also leave out transfers of unknown purpose and transfers between a person's own accounts.
• A public scoreboard: accuracy and calibration across all households, published as one total. No household's own numbers are shown.
• Later, each household that opts in trains its own, online: the same trainer runs as a server job on that household's labels, so raw text never leaves production. Switching on stays a person's click.

Guess, ask when unsure (plus spot checks), a person answers, retrain, sit the exam, run in shadow, and, next, switch on. The model learns only from people's answers, never from its own guesses. Everything up to shadow mode is built.

What a household sees of all this is in Agav's own docs: How Agav learns from you.