Self-Evolve Pi Extension

pi-self-evolve is a context-engineering harness for the pi coding agent. It shrinks the agent's context window on long agentic loops. In the author's A/B measurements (n=3 per cell, twice, one agentic benchmark suite), median context dropped ~50% with no solved-rate loss; a 10-task Terminal-Bench 2.1 run (n=1 per task) showed no task regressions. Your results may differ — see the limits section below. Open source, MIT, one-command install: pi install npm:pi-self-evolve.

This is a pi extension. It keeps agent context small on long agent loops. It uses five mechanisms:

• Evidence reducer (shell outputs). It filters shell output. It keeps FAIL, ERROR, and summary lines. Each kept quote is checked word-for-word against the saved log. If the check fails, the original text stays.
• ObservationPack (big tool outputs). It saves outputs over ~10 KiB to disk. It puts a short head/tail excerpt in context. The model recalls the full text later.
• Context compact (full window). When the window is full, it replaces old tool results with one-line markers.
• Action fusion (unverified edits). After an edit without verification, it adds one bounded ask at the turn boundary: run the verify command.
• Telemetry. It writes per-run counters to a JSONL file: provider usage, estimated raw tokens, reduction ratio, mechanism counters.

Install

From npm (recommended):

pi install npm:pi-self-evolve

Or from git:

pi install git:github.com/ralph-abejuela/self-evolve-pi

Or try it for one session without installing:

pi -e npm:pi-self-evolve

Or load a local checkout directly:

pi -e /path/to/self-evolve-pi/self-evolve/index.ts

Usage

• /evolve — show mechanism status and counters
• /evolve pack off — toggle a mechanism (pack, reducer, compact, fusion)
• /evolve reset — reset counters
• Model tools: harness_recall (paged archive reads), harness_commit / harness_track / harness_note / harness_state (BPE slots)
• State lives in .self-evolve/. It is git-ignored.

Extension conflicts (tested against a real install)

This was tested against a pi environment with 15 packages. Two clusters collide. Skills do not collide (prompt-level only, no shared namespace).

1. Output-compaction extensions (RTK, https://github.com/rtk-ai/rtk) — real collision. Do not double-wrap. Both extensions intercept the same tool_result events for the same tools (pwsh/bash). Three failure modes when both are on:

• Double truncation. RTK truncates at its maxChars limit. If RTK runs after this extension, it can clip the se://N handle block. The result is a pointer to nothing.
• Bad evidence. The reducer builds receipts from shell output. If RTK truncated that output first, the receipt silently records lossy text as evidence.
• Load order decides. Package-list order decides which extension wins. The loser's work is discarded without a warning.

Rule: give the shell-output layer to one owner. Either set RTK outputCompaction.enabled: false (this extension is lossless: archive + recall, not truncation), or set ARCHIVED_TOOLS here to exclude the tools RTK owns.

2. Prompt-stack budget. APPEND_SYSTEM.md + two memory recall contexts + this extension's ~80-token guideline sit on the system side. Measure any new injection against the ~23.4k global-extension floor.

Measured results (n=3 per cell, median context tokens = provider input + cacheRead, independent A/B)

The baseline is default pi -ns (global extensions loaded). This is the same floor the extension run starts from. Both arms are measured identically from their own session files. The extension-side estimate (rawEst) is not the headline metric. Provider-reported usage is.

The 5 adversarial tests from harness-test.txt (single-shot)

Test	Baseline ctx	Extension ctx	Δ	Correct
test1-nuance	47,481	48,545	−2.2%	base 2/3, ext 3/3
test2-needle	84,296	86,075	−2.1%	3/3 both
test3-json	47,338	48,402	−2.2%	base 1/3, ext 2/3
test4-contradiction	47,377	48,441	−2.2%	base 0/3, ext 1/3
test5-code	101,857	101,280	+0.6%	3/3 both
Total	328,349	332,743	−1.3%	base 10/15, ext 12/15

An independent second n=3 suite replicated the pattern: totals 348,583 vs 332,364 (+4.7% in the extension's favor), correctness 10/15 vs 13/15, triage 344,781 vs 175,072 (−49.2%).

Read this table this way: single-shot tasks sit within ±5% of parity. The small delta is fixed overhead (tool schema + guideline). The correctness gaps are inside run-to-run noise for this model. Single-shot QA is not where this extension shows gains.

The agentic scenario (run a noisy 20 KB log emitter, fix two failing specs, verify)

	Baseline	Extension	Reduction
median context (suite 1 / suite 2)	363,751 / 344,781	178,063 / 175,072	51.0% / 49.2%
solved	3/3 both suites	3/3 both suites	parity

This is the target regime: long agent loops that re-bill the same large noisy tool output every turn. Here the extension cut context in half at n=3, twice, with no solved-rate loss.

Terminal-Bench 2.1 via Harbor (containerized, verifiable)

The same A/B ran through Harbor on the official terminal-bench/terminal-bench-2-1 dataset. First 10 tasks. Model deepseek-v4.1-flash via opencode-go. n=1 per task. Both arms run in identical task containers, so no environment asymmetry is possible. Adapter: eval/harbor/pi_harbor_agent.py (it subclasses Harbor's built-in pi agent, uploads the extension, and appends --extension in the treatment arm).

Task	Base	Ext	Base input tok	Ext input tok
torch-pipeline-parallelism	0.0	1.0	1,995,491	567,010 (−72%)
regex-chess	hung (no result)	1.0	—	6,431,132
kv-store-grpc	1.0	1.0	59,354	39,639 (−33%)
dna-assembly	0.0	0.0	1,854,678	1,331,504 (−28%)
openssl-selfsigned-cert	1.0	1.0	30,979	48,449
pypi-server	1.0	1.0	60,100	128,161
write-compressor	1.0	1.0	395,176	529,745
torch-tensor-parallelism	1.0	1.0	283,885	509,034
schemelike-metacircular-eval	1.0	1.0	2,126,629	7,118,747 (outlier)
qemu-alpine-ssh	exception (both arms — infra)

Full job results (every trial's trajectory, logs, verifier output) are public on Harbor Hub:

• Baseline arm: https://hub.harborframework.com/jobs/b0a875eb-4b90-4acd-a27c-9b6e1dca83d8
• Extension arm: https://hub.harborframework.com/jobs/cfa8a1d8-6f8b-44d0-a371-0172225fb9ec

Research papers

Mechanism by mechanism:

Mechanism here	Taken from	Paper
Evidence reducer (reducer.ts)	Evidence-preserving reduction: compress logs into a receipt, validate every quoted line word-for-word against the archived source, keep the original on failure	SoL-Pi [^1]
ObservationPack (pack.ts) + harness_recall	ObservationPack: outputs over ~10 KiB move to a disk archive, context keeps a head/tail excerpt plus a handle, paged recall brings slices back	SoL-Pi [^1]
Context compact (bpe.ts commit under pressure)	Online context compact: completed subtasks marked for compaction, window-pressure and cost checks, summarize and reset the active window	SoL-Pi [^1]
Action fusion (fusion.ts)	Action fusion: bundle an edit with its verification step into one turn boundary instead of a second model request	SoL-Pi [^1]
BPE cognitive tools (harness_commit / harness_track / harness_note / harness_state)	Belief, Progress, Experience slots worked through four explicit cognitive actions: commit, track, recall, note	EvoHarness-RL [^2]
L1 active window / L3 disk archive split (store.ts, .self-evolve/l3/)	State-tier management: keep the working set in the live window, push the bulk to a durable lower tier	Prime Agent [^3]

[^1]: SoL-Pi: Recursively Scaling Auto-Research Loops for Efficient Agent Harness — NVIDIA. https://arxiv.org/abs/2609.20519 · code: https://github.com/NVlabs/SoL-Pi [^2]: EvoHarness-RL: Learning Self-Evolving Runtime Harness Policies — Meta. https://arxiv.org/abs/2608.05446 · OpenReview PDF: https://openreview.net/pdf?id=lFlnP9ZJHl [^3]: Prime Agent: A Self-Improving RLM Harness — Prime Intellect. https://arxiv.org/abs/2608.23552 · blog: https://www.primeintellect.ai/blog/prime-agent

Related systems from the same survey, background for the deferred work (harness annealing and distillation are out of scope for this extension):

• SafeEvolve: Harness-Policy Co-Evolution from Agent Experience for Safety Alignment — https://arxiv.org/abs/2609.02786
• Beyond Static Harnesses for Long-Horizon Coding Agents (openJiuwen) — https://arxiv.org/abs/2608.27969
• Harness-Zero: Harness Distillation via Agent-as-Harness — https://arxiv.org/abs/2609.24974
• GLM Infra Agent (Z.ai): recursive self-improvement on its own serving infrastructure — https://z.ai/blog/glm-built-its-inference-infrastructure

What is re-implemented here vs the papers: this extension re-builds SoL-Pi's four mechanisms, EvoHarness-RL's BPE tool interface, and Prime Agent's tiered archive as a single pi extension, and measures them with its own comparison harness instead of EdgeBench or ALFWorld. The training-side ideas (SFT + GRPO from EvoHarness-RL, LoRA distillation from Harness-Zero) stay out of scope.

Potential edge (what the results suggest — not proven at this sample size)

Three possible gains. Read them as suggestions at n=1 per task.

1. No harm. The extension did not lower any score. On the 9 shared tasks: 7 scores matched, 1 got better, 0 got worse. The shared failure (dna-assembly) and the infra exception (qemu) were identical in both arms. This is the floor: the extension is safe to load.

2. Fewer tokens on long tasks. 7 of 8 shared scored tasks used fewer input tokens. One outlier run (schemelike, 7.1M tokens) hid this in the totals. Without it, the extension used −32.6% on the remaining seven. The strongest cell: torch-pipeline. Baseline failed with 2.0M tokens. Extension passed with 0.6M tokens (−72%).

3. Reach on hard tasks. regex-chess was the only task the baseline could not finish (it hung). The extension solved it. The extension-only solve on the hardest task is the capability signal.

Limits, stated plainly:

• n=1 per task. One run can flip by chance. The torch-pipeline flip and the regex-chess solve each have one data point.
• One outlier run dominated the token totals. Median tokens per task are needed before a token claim.
• The robust claim today is the no-harm floor plus the ~50% context reduction on agentic loops (measured at n=3, twice). The capability edge and the token savings are candidate gains. They need more runs.

Reproduce

Adversarial + agentic comparison: node eval/run-compare.mjs from the repo root (env N_RUNS to change cell size; results in results/compare.json).

Harbor A/B:

export OPENCODE_GO_API_KEY=...          # or any provider pi can authenticate
export PI_HARBOR_AUTH_B64=$(base64 -w0 ~/.pi/agent/auth.json)   # container-side auth seed
export PI_HARBOR_MODELS_B64=$(base64 -w0 ~/.pi/agent/models.json)
export PI_HARBOR_EXT_DIR=/path/to/self-evolve-pi/self-evolve
export PYTHONPATH=/path/to/self-evolve-pi/eval/harbor

# baseline arm
harbor run -d terminal-bench/terminal-bench-2-1 \
  -a pi_harbor_agent:PiExtAgent -m opencode-go/deepseek-v4.1-flash \
  -l 10 -n 3 --timeout-multiplier 3 -y -o jobs-baseline

# extension arm (same command; only the variant env changes)
PI_HARBOR_VARIANT=extension harbor run ... -o jobs-ext