pi-tgrep

tgrep for pi: auto-indexes your repo with a trigram index and makes sure every search the LLM performs, whether through the grep tool or through shell commands, runs on tgrep instead of scanning every file.

Why

grep and ripgrep re-scan the whole tree on every query (O(total bytes) each time). On large repos that is the slowest thing an agent does all day. tgrep keeps a trigram index with a persistent, file-watching server and answers the same queries orders of magnitude faster, while staying ripgrep-compatible (same flags, same --json events, same ignore rules).

pi's grep tool is the code-search path of least resistance for a model. pi-tgrep turns that path into the fast one and closes the side doors.

What it does

1. Auto-index: on session_start it finds the enclosing git repo and ensures a tgrep serve daemon is running for it (spawned detached, builds the index if missing, and the file watcher keeps it fresh). The server is shared across sessions and with other tgrep clients. .tgrep/ is added to .git/info/exclude so git status stays clean.
2. Owns the grep tool: registers a tool named grep with the identical schema (pattern, path, glob, ignoreCase, literal, context, limit) and identical output contract. The model calls grep like always, but the results come from the trigram index. Every result carries details.engine for provenance, plus details.fallback (why ripgrep served the query: index still building, no binary, or error) and details.truncated when the match limit kicked in. When the repo index exists, searches under the repo root automatically pass --index-path <index dir>, because tgrep only looks for the index next to the searched path, so a subdirectory search would otherwise silently fall back to scanning every file. A search path that is itself git-ignored (bin/Debug, node_modules/pkg, a single file inside obj/) passes --no-ignore-parent, so it returns the same hits as ripgrep instead of "No matches found"; non-ignored paths are unaffected and still skip their ignored files.
3. Guards the shell: a tool_call hook inspects bash and MCP shell-executing tools (ctx_execute, ctx_execute_file, ctx_batch_execute, bare or namespaced like mcp__context-mode__ctx_execute, extend via PI_TGREP_WATCH_TOOLS), pipeline-aware:
   • commands are split on top-level |; each segment is judged on its primary binary;
   • grep/rg scan segments (pattern + path present) are transparently translated to tgrep search (the CLI's search subcommand — it answers from an existing index or scans directly, while the bare default query mode can hang when no server is running) with quote-preserving re-emission ('foo|bar' stays quoted, an empty "" pattern stays an empty argument, unquoted globs/~/braces like src/*.ts stay unquoted so the shell still expands them, globs translated from --include are always quoted, sudo/env prefixes survive);
   • translated bash commands are stamped onto the tool result (details.engine: "tgrep" plus the rewritten and original commands), so session logs and npm run analyze attribute shell tgrep usage directly; when a translated command fails or prints nothing, a [pi-tgrep] ran … in place of … line is appended to the output the model sees, so an empty result is never mistaken for a block;
   • the index directory is resolved for the directory each grep actually runs in — the session cwd, followed through any cd in the command or script — so cd /other/repo && grep … searches that repo's index, not the session repo's; when a valid index directory exists, translated commands whose positional paths are all relative get --index-path '<index dir>' injected (any absolute positional, or an explicit --index-path, skips the injection); without an index the search scans the files directly (tgrep prints its own notice) — a bare query-mode tgrep, which would rebuild an index beside the searched path or hang, is never emitted; -e/--regexp/ -f/--file pattern arguments are tracked so a pattern never masquerades as a path;
   • the same --no-ignore-parent is added when a translated command's path operand exists in the repo and git check-ignore reports it ignored;
   • rg --files, rg --type-list, rg -h/--help/-V/--version and grep --help/--version print listings or help rather than search, so they run unchanged (tgrep has no file-listing mode);
   • grep/rg stdin post-filters (… | grep -v x) are left verbatim: no tree scan, no block;
   • ctx_execute/ctx_execute_file shell code is scanned exactly like a bash command: newlines separate commands, # comments are ignored, and heredoc bodies are left untouched while a grep on another line is still translated; a grep that reads a heredoc or here-string (<<, <<<) runs unchanged since tgrep search doesn't read stdin;
   • non-shell ctx_execute/ctx_execute_file code (e.g. JavaScript) is scanned for child_process exec/spawn calls (exec, execSync, execFile, execFileSync, spawn, spawnSync): an embedded shell grep inside a backtick template or double-quoted string is translated in place, while single-quoted, escaped, interpolated, or statically unextractable commands are blocked, so execSync('grep -rn …') can no longer bypass the index; member calls only count when the receiver is bound to child_process (so regex.exec(line) is left alone), and an unextractable command only blocks when the code mentions grep;
   • ctx_batch_execute entries are checked individually (the entry's label appears in block reasons);
   • output-side redirects are preserved with fd numbers still glued to their redirect (2>/dev/null, 2>>file, 2>&1), so the fd digit never leaks into tgrep as a search path;
   • ;, && and || split the command and each part is judged on its own; a grep after a shell keyword (do, then, else, if, while, until, !, {) is translated too, so if grep -q …; then keeps its exit status;
   • grep whose operands contain a shell expansion ("$f", $FILES, $(find …), backticks) greps a dynamic list the index can't help with, so it runs unchanged, as does a command follows a cd with a dynamic target (until the next absolute cd);
   • the body of $(…) or backticks is judged like any other command, in the directory it runs in: stdin filters ($(git ls-files | grep -v obj/)) and other binaries ($(which rg)) run unchanged, a tree scan ($(grep -rl X src | grep -v obj/)) is translated in place or, when it can't be, left as written; it is blocked only when an in-place rewrite is unsafe (a backgrounded grep or an unsupported flag inside it, unparseable quoting, or a backtick body with backslash escapes);
   • a grep that is itself backgrounded with & (grep … &, a && grep … &; one inside a backgrounded loop or group is translated) and a grep with a stdin redirect (<) are blocked, while the same constructs on other commands are left alone;
   • every block reason names what triggered it (the construct, or the unsupported flag) and keeps the phrase "bypasses the tgrep index" so npm run analyze can count blocks; BRE patterns the converter can't prove equivalent (back-references such as \1, see Safety notes) and ag/ack/pt (flag semantics diverge from rg/tgrep) run the original command untranslated, exact but slow; under PI_TGREP_BASH_POLICY=block both are still blocked;
   • zgrep, git log --grep, and filenames containing "grep" are never touched.
4. Degrades gracefully: no tgrep binary means it is fully dormant (built-in grep untouched, with a one-time offer to brew install tgrep). Index still building falls back to ripgrep so results are never silently incomplete. Server dead falls back to tgrep's own on-disk index.

Install

# from a local checkout
pi install /path/to/pi-tgrep

# from git
pi install git:github.com/danielmarbach/pi-tgrep

# from npm
pi install npm:@danielmarbach/pi-tgrep

Requires the tgrep binary (brew install tgrep, or cargo from the tgrep repo). If it is missing, pi-tgrep offers to install it via Homebrew once (see PI_TGREP_AUTO_INSTALL).

For development, skip the install and load it directly:

pi -e /path/to/pi-tgrep/extensions

Configuration

All configuration is via environment variables.

Variable	Default	Meaning
PI_TGREP_DISABLED	–	1 disables the extension entirely
PI_TGREP_AUTO_INSTALL	ask	ask | never | always: brew install when tgrep is missing; ask prompts once and remembers (~/.cache/pi-tgrep/auto-install.json)
PI_TGREP_BASH_POLICY	translate	translate (translate scan segments in pipelines, block the rest) | block (block all grep-family shell use) | warn (allow, notify) | off
PI_TGREP_WATCH_TOOLS	–	additional tool names to watch, comma-separated, additive to the default set (bash, ctx_execute, ctx_execute_file, ctx_batch_execute); names match bare or namespace__name
PI_TGREP_SERVE_ARGS	–	extra args passed to tgrep serve (e.g. "--exclude vendor --no-watch"); split on whitespace, quotes are not parsed, so individual args cannot contain spaces
PI_TGREP_INDEX_PATH	tgrep default (<repo>/.tgrep)	index directory override, honored everywhere: serve, status/stop/reindex, the grep tool, and shell injection; relative values resolve against the repo root
PI_TGREP_SCOPE	repo	repo keeps the server after the session ends; session stops servers this session started on shutdown

Commands

Command	Action
/tgrep-status	show server/index status for the current repo
/tgrep-reindex	stop the server, rebuild the index, restart
/tgrep-stop	stop the server for this repo

The footer shows tgrep: N files (or tgrep: indexing…) while active.

Architecture

extensions/index.ts   factory: config → binary check → session lifecycle → hooks → commands
src/config.ts         env-var parsing
src/tgrep-client.ts   binary discovery, status parsing (both output shapes), serve.json
                      discovery + PID liveness, stop (SIGTERM→SIGKILL; tgrep has no stop cmd)
src/server-manager.ts repo-root detection, detached `tgrep serve` spawn, .git/info/exclude
                      maintenance, readiness monitor, per-scope shutdown, reindex
src/git-ignore.ts      `git check-ignore` wrapper deciding when a search path needs --no-ignore-parent
src/grep-tool.ts      the grep override: TypeBox schema identical to the built-in, tgrep
                      --json streaming (ripgrep event schema, parsed 1:1), built-in truncation
                      rules via pi's own helpers, abort handling, rg fallback while indexing
src/bash-policy.ts    shell detection (quote-aware), pipeline segmentation, rg→tgrep
                      pass-through, grep→tgrep flag translator with quote-preserving
                      re-emission, verbatim fallback or guided blocks for the rest

Output parity

The override keeps pi's grep output contract byte-for-byte where it matters: path:line: text matches, path-line- context lines, No matches found, the [N matches limit reached…] and truncation notices, and details fields (matchLimitReached, truncation, linesTruncated). The only additions are metadata: details.engine: "tgrep" | "rg-fallback", details.fallback: { reason: "indexing" | "no-binary" | "error", message? }, and details.truncated.

The grep tool also recovers from common model mistakes, adding only notices and metadata:

• Invalid regex. A pattern tgrep cannot compile (Generate(, Handle(Foo|Bar) is retried once as a literal (-F) before the ripgrep fallback. The result ends with [pi-tgrep] pattern is not a valid regex; searched literally and has details.regexFallback: true.
• Multiple paths in one string. If path does not exist as written but every whitespace-separated token does ("src tests"), all of them are searched (details.searchPaths). Results are shown relative to the working directory.
• Path not found. The error keeps the Path not found: <path> prefix and adds the nearest existing parent directory and up to 20 of its entries.
• tgrep errors. When tgrep fails and the ripgrep fallback fails too, the error carries tgrep's first stderr line (not just tgrep exited with code 2).

Safety notes

• GNU grep's -r is recursive, but tgrep's -r is --replace, so the translator drops -r/-R instead of passing them through (tgrep walks recursively by default). Since grep -r prints file names even for a single file operand while tgrep doesn't, -H is added in that case.
• grep accepts rg's --type/-t, --type-not and --type-add; rg --include=GLOB is translated to -g GLOB as grep --include is; rg -P maps like --pcre2.
• Plain grep (no -E/-P/-F) patterns are POSIX BRE, but tgrep's default engine is a Rust regex (ERE-like), so the pattern is converted before it is passed on: \| \( \) \{ \} \+ \? become | ( ) { } + ?, \< and \> become \b, and the unescaped ( ) { } + ? |, which are literal in BRE, are escaped. A * that starts the pattern, a group or an alternative, and a ^/$ that is not at the edge of the pattern, stay literal. Anything the converter can't prove equivalent runs the original command instead: back-references (\1), \</\> that don't sit next to a required word character, stacked quantifiers, empty alternatives (a\|), brackets containing \, unknown escapes such as \d. -E/egrep need no flag; -F/fgrep maps to -F; -P/--perl-regexp maps to tgrep's -P (PCRE2).
• During a from-scratch index build the tgrep server answers from an empty index, so searches temporarily use ripgrep until Indexing: complete, favoring correctness over speed.

Validation

npm run typecheck   # npx tsc --noEmit -p tsconfig.json
npm test            # node --test over harness.mjs + the *.test.mjs suites

test/differential.test.mjs runs a battery of grep/rg commands twice, as written and as translated against a real index, and compares the sorted output and exit code; it is skipped when tgrep or rg is not installed. Known differences are listed in the battery with a reason.

The test suites shell out to the real tgrep binary (brew install tgrep), so they need it on PATH locally and in CI (see .github/workflows/ci.yml, which runs on macos-latest for that reason).

node scripts/analyze-pi-sessions.mjs (or npm run analyze) mines ~/.pi/agent/sessions/ for the real numbers: indexed vs fallback vs builtin grep traffic, zero-result rate, latency p50/p95, search→read conversion, and how often the shell policy translates or blocks. Translated shell greps are identified by the details.engine: "tgrep" stamp the extension adds to the bash tool result; sessions recorded before the stamp existed count translated greps as plain grep/rg.

See test/e2e.md for headless pi -p scripts and test/tgrep-cli-notes.md for the verified tgrep CLI ground truth this implementation relies on.

License

MIT