Learning on the Job, Batteries Included

The curve above shows an agent learning a business's unwritten rules. It answers customer emails at the order desk of a fictional distributor, an experienced reviewer corrects its drafts, and each correction becomes a lesson stored in shared memory. Over a hundred tasks, policy compliance climbs from 20% to 64%, while a control arm answering the same emails without memory stays flat. The gap between the two, drawn with its 95% confidence band, is the memory advantage: 47 percentage points by the end of the run.

We have released the whole setup as an open source repository, learning-on-the-job, and you can reproduce this curve on your own machine this afternoon, for about $3 in tokens.

The repository is the runnable companion to our paper Learning on the Job, and it comes batteries included: the agent, the simulated business, the reviewer, and the measurement are all in the box, connected to the Knowledge Work domain of Spark over MCP, the protocol every major agent platform now speaks. One command runs the experiment end to end.

The order desk at Fenmoor Supplies

Fenmoor Supplies is a fictional B2B distributor. Simulated customers write to its order desk about order changes, returns, delivery exceptions, and credit queries. The desk runs on a set of policies held only by the simulated reviewer: which customers get which exceptions, the threshold above which someone senior has to look, how a reply from the desk is meant to read. They stand in for the tacit knowledge of an experienced team, and the agent is never shown them.

Each task runs the loop an agent-assisted workflow already has. An email arrives; the agent searches memory, looks up the account and order records, and drafts a reply; the reviewer corrects the draft; a reflection step distils the corrections into lessons and writes them back to memory. On an empty memory the agent breaches policies it has no way of knowing. As lessons accumulate, the breaches stop repeating.

The metric is compliance: the share of the policies that applied to a task that the reply got right, measured as a percentage.

Knowledge that retrieval cannot reach

Fenmoor's policies exist in no document. Every company runs on know-how that was never written down, and it surfaces in day-to-day interactions, such as an experienced colleague correcting the work before it goes out.

Retrieval-augmented generation (RAG) can only return what the organisation has already documented, and only after the documentation is written, updated, and re-indexed. The knowledge this scenario tests sits outside that boundary on both counts: nobody wrote it down, and by the time somebody does, the information is months old. Memory captures the lesson at the moment the gap shows itself. The correction happens on one task, and the lesson is available for the next.

What is in the box

The repository serves three purposes, and you can use it for any one of them.

As a tutorial, it shows end to end how an agent uses the Memco Memory MCP in the Knowledge Work domain: searching while working, once for the situation in the request and once for how such replies are written, and writing back lessons that can be reused.

As an example harness, it keeps the scenario as data. Policies, tasks, and the account records behind the lookup tools are YAML, separate from the code, so you can extend the shipped scenario or replace it with your own domain without touching the harness.

As a measurement framework, it lets you put an explicit number on the value that memory brings to your specific use case. Every run is paired: each task is answered twice from the same email, once with memory and once without, and everything is recorded per task in a JSONL file, from what memory returned to which policies the reply breached. The learning curve renders as the run progresses, and runs are seeded with randomised task order, so you can reshuffle and convince yourself the effect survives.

What building it taught us

As we built the harness, the design choices came from watching early runs fail. One failure is worth retelling, for its instructive value. In an early run, the agent breached the same formatting policy in 56 of the 92 episodes where it applied. Reflection wrote a lesson after every one, and the drafting agent searched for exactly that kind of knowledge in nearly every episode. Across roughly ninety searches, those lessons never came back.

The lessons were correct and clearly written, but they were focused on explaining the policy, rather than on what someone might ask when working on a task. The lesson read:

In correspondence concerning a specific order, the first sentence names the order number.

The drafting agent, meanwhile, was asking:

How are replies from this desk written and structured?

That question asks for the knowledge that the lesson holds, but there is no reason a search system would connect the two. One side speaks like a policy manual, the other like a reader, and they have zero lexical overlap and very little semantic alignment.

The fix was to make writing and reading share the same perspective and vocabulary: the two halves of one contract. The drafting prompt and the reflection prompt now build their questions from the same canonical template, and lessons make sense from the reader's perspective: When drafting a reply about a specific order, the first sentence names the order number. The same knowledge, rewritten for the person who needs to find it.

The README's Patterns that transfer section records the insights that the early runs taught us, from scoping lessons to pacing writes against reads. They apply to any agent working with a shared memory, whatever the use case.

Run it, then make it yours

The quickstart is one command. With uv, an LLM API key, and a Memco workspace:

uv sync
cp .env.example .env
# -> Add your API keys in .env
uv run memco-harness run --tasks 50 --seed 42 --paired

The default 50-task paired run takes about 50 minutes and about $3 in tokens, and shows the story: compliance climbs through roughly the first thirty tasks and settles as the hidden policies are learnt. The Knowledge Work domain has a generous free tier for individuals and small teams, so the tokens are the only cost.

When the first run has convinced you, make it yours. The scenario is YAML, the repository includes the generator we used to build the task library, and the reviewer is a self-contained component. If your team already reviews agent output, the harness maps directly onto your real workflow. Encode one of your own unwritten policies, run fifty tasks, and see whether the curve appears. Everything is MIT licensed, so use it any way you need.

And if your curve doesn't rise as quickly as you'd wish, we're happy to help. You know where to find us.