Learning on the Job: Agents Can Learn from the Feedback They Already Get

Every business now has access to the same AI models. The lever that separates you from competitors is how well those models learn your specific operation, and our new research shows that learning can come from feedback you already produce.

AI agents are deployed into environments their training never covered. Your policies, your systems, your naming conventions, the exceptions your business has accumulated over decades: none of it was in the training set. Foundation models arrive brilliant and generic, and closing the gap between generic capability and specific competence is where that differentiation happens. An agent that has mastered your specific operation is something competitors cannot download.

The traditional route to that competence has been fine-tuning: collect examples, run a training job, produce a specialised model. It works, but it requires data you may not yet have, specialist skills to run, and repeat investment every time a new model arrives.

There is another resource, available today and growing with every task your agents run: feedback produced by ordinary operation. Tasks succeed or fail, reviewers add corrections, customers respond. That signal is an opportunity, and we just completed a study asking a direct question: is that feedback enough for an agent to learn from? The full results are in our new paper, Learning on the Job, and they come down to two findings.

The experiment, briefly

We ran agents on the banking domain of τ-bench, a benchmark of realistic customer-service tasks in a simulated retail bank. These are hard tasks: even the latest frontier models, searching the knowledge base of a bank's policy documents, succeed on only around a quarter of attempts.

The model's weights stay frozen throughout. In place of retraining, the agent is paired with the Spark external memory: after each task it reflects on the feedback it received and distils it into a lesson that can be reused the next time a similar situation arises.

The feedback we used is the kind organisations already produce: an after-the-fact correction describing what should have been done, the routine output of a QA check or a human review.

The control condition we used is static RAG, a widely deployed technique for giving agents access to knowledge: the baseline agent retrieves over the complete bank policy corpus. It is useful to understand that the learning results reported below apply on top of what RAG already offers.

Finding 1: the feedback you already have is enough

With a large open weights model, learning from corrections lifted success rate to 2.6 times the RAG baseline. We then replicated the effect on a closed-source frontier model, which rose from solving around a quarter of attempts to 40%, a 1.6 times improvement.

For a business, the reading is this: learning on the job is the modern form of fine-tuning. You take a very capable foundation model and make it competent on your specific use case, without touching a model weight, without a training pipeline, and without waiting to accumulate a dataset. The training data is the feedback your operation already generates.

That reframes the value of your review processes. Every QA verdict, reviewer comment, and escalation note is a lesson your agents can keep. The workflows you already run to assure quality can double as the mechanism that compounds it.

There is no training run to fund and no dataset project to wait for. The learning accrues during normal operation, from the first day of deployment.

Finding 2: the learning travels between models

The second experiment took the memory built by one model learning on the job and handed it to a different model from a different provider. The second model improved above its own baseline. We then ran the transfer in the opposite direction, and it worked that way too.

This is the finding with the longest-term consequences. Knowledge captured as external memory is portable: it survives model upgrades and it survives provider changes. One agent's hard-won lessons become something you can hand to any agent you run, including models that don't exist yet.

Which is why owning those memories pays. When the learning lives in a memory layer you own, in a form you can read and audit, it becomes an organisational asset that compounds across every model transition. Each new model starts from everything its predecessors learned. Given how fast the model market moves, we believe that portability is now a crucial capability for enterprises seeking to maximise AI value.

What we built

This is exactly why we built Spark as a shared memory layer that sits outside the model. The study ran on Spark with a general-knowledge domain, the same mechanism that powers our coding product, applied here to a banking use case. The agent connects over MCP, writes what it learns, and retrieves it when it matters, with every insight carrying its provenance and a trust score earned from feedback.

The paper, with the full protocol, the statistics, and the released data, is at arxiv.org/abs/2607.22157. Over the coming weeks I'll go deeper into the individual findings: what different feedback signals are worth, how to measure whether an agent is actually learning, and what cross-model transfer means for teams building on open-weights models.

If your agent workflows are producing feedback today, you already hold the raw material for agents that improve day after day and week after week. Spark gives you the mechanism to capture that data, and turn it into business value.