@tduyng/pi-autoresearch0.0.4

@tduyng/pi-autoresearch

An experiment loop turned on with one command: measure, read the number, keep or discard. The tools handle git, scope and the noise floor.

An optimization loop run by hand fails in three predictable ways. The benchmark gets edited until it shows a win. A regression gets committed because the number moved. The log is re-read whole every iteration, at more cost than the change.

Nothing at cold start. The extension adds one slash command, and the two tools are declared to the model only while /autoresearch runs. The extension removes them on off, clear, or when max_runs is reached.

Keep or discard stays with the model

No rule can tell whether 1 percent came from the diff, whether the code is now simpler, or whether a clean run was jitter. The tools measure and the model decides. A rule that decided would be wrong in exactly the interesting cases.

The measurement is crash-safe on disk: run_experiment writes a pending measurement before anything else, so a restart can still log it and a crash mid-iteration loses nothing.

The benchmark cannot move under the comparison. The script is hashed, editing it voids the comparison, and /autoresearch rebaseline is the deliberate way to accept a new baseline.

A keep cannot escape its scope. scope globs in the frontmatter bound the files a keep may commit, so an experiment cannot quietly rewrite a file it was not measuring.

The loop

edit -> run_experiment -> read the output -> log_experiment -> repeat

run_experiment snapshots the working tree, runs .auto/measure.sh, parses METRIC name=value lines, runs the optional checks, and returns the output, the metrics, the changed files, the noise advice and any scope violations. It does not touch git history.

log_experiment takes status: keep | discard | crash | checks_failed and the model's reasoning in asi. A keep commits the files the run touched with a conventional subject built from the description. Every other status restores those files.

Mechanism	Effect
Full benchmark output	Phase timings and errors reach the model, so the next hypothesis comes from data
Noise floor from median absolute deviation	Separates jitter from a real win, without overriding the model
Pending measurement	One run between the tools, crash-safe, so a restart can still log it
Scope globs	A keep cannot touch files outside the declared scope
Benchmark hash	Editing the benchmark to move the number voids the comparison
Worktree lock and dirty baseline	One loop per worktree, and files another session left dirty are never committed
Finalize in temporary worktrees	Kept runs become branches without moving HEAD under another session
Compaction digest from disk	Prompt, recent runs, dead ends and ideas survive a reset with no model call
Conventional commits	type(scope): summary, at most 72 characters, one Metric: trailer

The project files

.auto/prompt.md carries the frontmatter and the objective:

---
name: vitest startup
metric: total_ms
unit: ms
direction: lower
command: bash .auto/measure.sh
scope: [vitest.config.ts, test/**]
max_runs: 40
min_gain: 0.5
---

What is optimized, the constraints, and what a good change looks like.

min_gain is the smallest relative improvement the advice calls real, in percent. It never blocks a keep.

File	Purpose
.auto/prompt.md	Frontmatter config, the objective, what has been tried
.auto/measure.sh	The benchmark, printing METRIC name=value
.auto/log.jsonl	Append-only record
.auto/checks.sh	Optional correctness checks, run after a passing benchmark
.auto/ideas.md	Optional backlog
.auto/pending.json	The run waiting for a decision
.auto/lock.json	The session that owns this worktree

The first run adds .auto/ to .git/info/exclude, including from a linked worktree, so none of it lands in a commit.

Commands

Command	Effect
/autoresearch <goal>	Start a session, or resume the one in .auto/
/autoresearch status	Print the digest
/autoresearch off	Stop and remove the tools
/autoresearch rebaseline	Accept a changed benchmark script as a new baseline
/autoresearch finalize	Split kept runs into independent branches from the merge base
/autoresearch clear	Delete the log and stop

finalize groups kept runs by file overlap and creates one branch per group in temporary worktrees. It never checks the current worktree out to another branch, and it verifies the union of the branches against the original before it reports.

Worktrees and other sessions

One loop owns a worktree at a time: /autoresearch takes .auto/lock.json and refuses a second session before it runs anything. Files that were already dirty when the loop started are recorded in the lock and never committed or reverted.

For a second loop, use a separate git worktree, which gets its own .auto/, branch and log. The first run on trunk creates autoresearch/<name>-<timestamp> in a worktree at ~/worktrees/<repo>/autoresearch-<name>-<timestamp> and leaves the shared checkout on trunk. An existing feature branch is used as it is.

Compaction and restarts

While the loop is active, session_before_compact replaces pi's model summary with a deterministic digest built from .auto/: the prompt body, recent runs, kept runs, dead ends, ideas and any pending measurement. No model call, and no history lost to a summary that forgot the metric.

After a restart of the same session the loop re-arms from the session's activation entry and sends one short resume message. A fresh session never auto-activates.

Install

pi install npm:@tduyng/pi-autoresearch

Local checkout: pi install /path/to/dotfiles/pi/agent/extensions/pi-autoresearch

It needs a git repository, since a keep commits, the lock is a file, and finalize uses worktrees. The measure script runs through bash -c.

Run it in an interactive session or through RPC. pi -p exits as soon as the slash command returns, before the loop's first turn, so a print run cannot drive it.

Limits

• A run has a default timeout of 600 seconds, and the measure script's own timeout is the one that applies.
• Auto-resume gives up after 200 turns, and an idle resume after 40.
• Output is trimmed to the last 40 lines, with 128 KB retained and a 20 MB cap on the temp log.
• finalize reports a conflict instead of resolving one. Splitting the kept runs is a bookkeeping job, not a merge.