@tduyng/pi-harness-loop0.0.4

@tduyng/pi-harness-loop

The harness fixes itself. It reads what a project's sessions repeated, writes the skill or rule that would have prevented it, installs it, and removes it when the failure rate does not fall.

A correction typed in every session lives in a chat message that dies with the session. A rule written by hand is usually worse, because nothing says which rule earns its place and the front of every request grows either way.

A session pays nothing until the loop acts. The report is a slash command rather than a tool. A tool would put its schema in the system prompt of every session, and that schema is the tax this package exists to reduce. lesson is the one tool, 252 tokens, and it arms only on a request naming a trigger of a lesson still in force.

Measure the rate, not the story

The loop installs a part, runs the project's check, and counts the sessions after the install against the sessions before it. Every session counts, including the ones that did not hit the failure, because a session that did not fail is the evidence.

The loop takes a part back out at the next idle if its failure rate did not fall, without anyone typing a command.

Verdict	Meaning
held	The rate fell to three quarters of what it was, and sessions still call tools at more than half their old rate
flat	The rate stayed within a quarter either way
worse	The rate rose by a quarter or more
avoided	The rate fell but the work did too, so the failure stopped because the agent did less
confounded	A commit touched the failure in the window, so the rate is not the part's
early	Fewer than three sessions have run since the install
out-of-window	The corpus holds no session from before the install

A flat, worse or avoided install is not the end of the finding. The next brief says the earlier form did not carry and shows both rates, and the part's lesson stops matching. A finding whose part was all markdown and still did not carry is marked needs code, and the loop stops proposing prose for it, because the failure lives in tool behaviour and a rule cannot reach it. A finding comes back when the failure grows past the count at the last proposal, and it stops after three.

The loop

The session goes idle. The loop reads the sessions of this project, and a failure that recurred gets a brief with the evidence. One model call turns that brief into the part, the loop checks it against the rules pi's loader applies, and installs it. The loop never takes the session over.

The writer has no tools. It carries the brief, returns files as text, and the loop parses, checks and installs them, so nothing the writer says is executed. A proposal costs one completion, and a part adds to the front of every later request, so both have a ceiling. The scan is cached in .pi/harness/scan.json behind a fingerprint of the session files.

PI_HARNESS_LOOP=off stops all of it.

/harness-loop                 the setup, the findings, the cost, the lessons
/harness-loop propose [id]    run a proposal now
/harness-loop lint            only the problems
/harness-loop revert <id>     put the files back by hand
/harness-loop install <id>    install a part already on disk
/harness-loop clear           take the report off the screen

The next prompt clears a report too, so one already read does not sit above the editor.

What the report covers

The report checks the harness tree against pi's own loader rules: a skill with no description is dropped at load, a frontmatter name that differs from its directory is advertised under one name and looked for under another, a profile with no description cannot be chosen by the model, a skill that runs a script it does not ship sends the agent after a missing file, and an extensions/ package that declares an entry the root manifest never loads is a package nobody gets.

Pi advertises every skill by name and description in the system prompt, so a skill nobody opens is a tax on every request. The report lists the advertised ones with the characters they cost and how many are not advertised at all.

The findings are what recurred in this session: the same failure three times, the same command three times, the same file read three times, a run that hit a failure and changed nothing, a tool call with no result. The recurrences are the same shapes across the sessions of this project.

The lessons are the ones whose cue matches the request in the editor, read from .pi/harness/lessons.jsonl.

Three checks before a part lands

Each one is a false positive that would otherwise get installed.

• Replay: the failure's own command runs again. When it passes, the failure no longer reproduces and nothing is written. The same command runs after an install and its result stays with the install.

• Producer: a failure whose message the tree no longer carries is not a property of the harness. The loop cuts ids, paths and numbers out of the message, searches the tree then the dependencies, and when neither has it and git shows a commit that touched that text after the last session that hit it, the finding is skipped.

• Variant: a new error string that is a smaller variant of an edit failure already tried is not a new finding. The edit tool has one matching mechanic, so the 2-session "oldText must be unique" is the 9-session "oldText must match" with a different error string.

A defect in one run is noise, a defect in every run is the harness. A signature that reduces to <path> or a bare Script failed is not a finding, because it names nothing a part can change. A failure also needs independent evidence: when every session that hit it edited a file and each pair shares one, they are one piece of work, and a rule written from it would encode that task rather than the harness.

The loop stands out of the way when something else drives the session: an active goal, or the day's budget of five proposals, stops it.

What a part may be

The brief asks for a skill, an agent profile or a rule, and the writer puts the files in .pi/harness/<id>/. The part is read back, checked against the same rules the tree is checked against, and installed.

A part under global/ holds in every project, and it is installed only when global is set, because one project's failure does not get to become every project's rule. Without it the part lands in the project: .agents/skills/<name>/, .pi/agents/<name>.md, or the project's context file for a rule.

The prompt says what each form costs. A skill is paid for by its name and description, about 300 characters on every request, and its body costs nothing until opened. A rule is paid for by its length and applies always. The rule wins when the behaviour has to hold every time, and the skill wins when the situation is specific enough to route on.

A part may carry CONFIRM.md: one shell command that exits 0 only when the finding can no longer happen, with no file writes and no network. It runs after the project check, and a failing confirmation puts the part back. Without one the part installs on the project check alone, recorded as unconfirmed.

The check behind an install

The check is the first of check in .pi/harness.json, the test script in package.json, or the test recipe in justfile. The loop installs the part first, so the common case costs one run: when the check passes, the part stays. When it fails, the loop puts the part back and runs the check once more without it, so the verdict is about the difference rather than about a suite that was already red. A rule installs without a check, since no test reads a markdown file.

maxStandingTokens is the second ceiling, 1,500 tokens by default. An install that would cross it stays on disk with its brief, and the message names the current total, so the way through is to revert a part that did not carry rather than to raise the cap. A rule is charged in full, a skill only for its name and description. 0 turns the ceiling off.

/harness-loop revert <install> puts the files back by hand, newest install first when two touched the same file. A file whose content changed after the install is left alone and named, because an edit is not the loop's to overwrite.

The lesson that transfers

When a part passes, the session records the cue, the lesson, how to confirm it worked, the evidence, and a confidence. The model may decline with NO_ENTRY, and a block with no confirm line is dropped, because a rule nobody can check is a rule of thumb.

A request matches a lesson when it carries two triggers, or one trigger of six characters or more that the project rarely uses. One short word is a coincidence, and key is in half the requests of a project that never talks about credentials. A match records a use, so a lesson nobody recalls is visible in the record.

With the rest of the suite

pi-goal owns the turns of a session with a goal, so the loop does not propose while one is active and waits to install while one is, and it uses the same check runner so the two never disagree about what a check said. pi-context prunes what the model sees, not the session, so findings still see the whole branch. pi-guard blocks the shell from reading credentials, and the rule it needs is in the global context, saying what to do instead of what is blocked.

Findings from this machine

• 12 of 19 skills are not advertised at all, because their frontmatter says disable-model-invocation, so they cost nothing. The other 7 make a block of 2,652 characters, about 663 tokens on every request.

• 5 of those 7 were opened in none of 26 sessions scanned, costing 362 tokens per request. tdd was the sixth until the skill stopped being advertised on 2026-10-26, from a proposal and nothing else.

• The shell blocked 14 reads of credential files across 2 sessions, and every retry with another shell reached the same block. The rule from it says what to do instead.

• The edit tool rejected 9 old texts across 4 sessions, and each rejection reported a closest match the model had paraphrased. The rule from it says to copy old text out of the read that preceded the edit.

• 4 reads were refused for being over 350 lines across 3 sessions, each a file a grep would have narrowed. The rule from it says to grep first.

Install

pi install npm:@tduyng/pi-harness-loop

Local checkout: pi install /path/to/dotfiles/pi/agent/extensions/pi-harness-loop

The scans use rg, so it has to be on PATH. Sessions of the project must exist on disk, which they do after one run.

.pi/harness.json, every key optional:

{
  "check": "bun run test",
  "auto": false,
  "autoInstall": false,
  "global": false,
  "maxInstallsPerDay": 3,
  "maxStandingTokens": 1500,
  "writerModel": "opencode-go/mimo-v2.6-flash"
}

writerModel resolves from the project file, then PI_HARNESS_WRITER_MODEL, then the model in subagent.json, which is the model pi already runs chores on. With none of them the session's own model writes it, and a session on an expensive model pays that model for a short chore. auto is read on idle and stops the loop writing a part on its own while the manual command still works. autoInstall is read when the part is ready, so a project that wants every change by hand sets it to false and the part stays on disk.

Limits

• Five proposals a day, three installs a day, six turns per proposal, three proposals per finding.
• It does not score the harness. It measures the rate of the failure a part was written for. Whether the work got better is a different question.
• It does not compare two runs of the same task. The comparison is across the sessions of this project, which are different tasks.
• A rule is markdown in a file no test reads, so /harness-loop lint is the only gate it passes.