ds4-context-engine0.5.3

Non-destructive, provider-independent context management for Pi.

alucard_24 Published 13 days ago MIT ~4.3K downloads/mo

Untested on omp

This is a pi package. No one has reported whether it works on omp yet. If you try it, add your result below.

Reports 0

No reports yet.

Readme

DS4 Context Engine for Pi

DS4 Context Engine is a non-destructive, provider-independent context management layer for Pi. It keeps Pi's native JSONL session as the canonical history and builds a smaller, inspectable, model-aware working context for each model call.

complete Pi JSONL history
          ↓
DS4 planning, retrieval, summaries and policy
          ↓
bounded active context with provenance
          ↓
Pi provider

Project status: The coordinated 0.4.0 release adds storage.scope: "agent" | "project", now defaulting to per-project SQLite projections with shared token calibration in the agent database (opt out with storage.scope: "agent"). The opt-in BPE estimation and bounded auto-tuning from 0.3.10 keep chars-v1 and disabled auto-tuning as their defaults. The bounded compaction controls from 0.3.9 remain in place; canonical history, SQLite schema 16 and runtime contracts are unchanged. Pi remains pinned to 0.84.3. Patch 0.4.1 states the contiguous-span rule for compaction summaries in the summarizer prompt while leaving validation strictness, the eight-bullet repair bound and every provider-facing default unchanged. Patch 0.4.2 adds class-only diagnostics for rejected exact-value spans: the existing custom_fallback warning carries unsupportedSpanClasses as class names and counters, never span text, with validation, repair bounds and provider-facing defaults unchanged. Patch 0.4.3 makes those diagnostics interpretable: every distinct span receives the cheap transformed-form lookups before the bounded near-miss analysis starts, each near-miss candidate may spend only an equal share of what remains, and spans that were not analysed report not-classified-length, not-classified-partial or not-classified-budget instead of no-near-miss. Patch 0.4.4 accepts a backticked exact value whose canonical JSON-escaped rendering is literally present in the evidence — the decoded rendering a summarizer produces when it quotes serialized JSON — while absent values, the reverse escape direction, span composition and single-character variants stay rejected; the diagnostics probe budget rises from 4000 to 24000 evidence-source scans. Patch 0.4.5 grades the summarizer quoting fallback: the prompt requires one contiguous excerpt per backticked span, tells the model to backtick only the fragments that are themselves contiguous or to keep the fact as ordinary text without backticks instead of composing a span, and reserves bullet omission for facts with no support in the evidence; diagnostics split composition into composed-adjacent-present (both parts next to each other in one source, so the span may be a re-rendering of a contiguous region) and composed-two-present-parts (parts found at unrelated positions), with validation strictness, the eight-bullet and 25% repair bounds and every provider-facing default unchanged. Patch 0.4.6 makes an engine/core artifact mismatch visible instead of a missing function: the core exports CORE_VERSION and the engine verifies at session start and at every compaction attempt that the loaded core matches its own version and exposes the entry points it calls, so a core rebuilt under a running Pi — which a /reload does not pick up — reports one actionable line, keeps the DS4 compaction layer inert instead of failing mid-generation, and never blocks Pi; validation strictness, repair bounds and every provider-facing default stay unchanged. Patch 0.4.7 stops the recurrent fail-closed fallback at its source: when the summarizer re-renders a value the evidence holds in another form — JSON-escaped or unescaped, collapsed whitespace, stripped typographic characters, or two parts adjacent in one source — the repair retracts only the backticks and keeps the fact, so those spans no longer consume the eight-bullet bound; invented values, unrelated compositions, one-character deviations, case changes and unanalysed spans keep the existing fail-closed bullet removal, validation strictness is unchanged, and both effects are recorded as counters (unsupported-exact-bullets-pruned, unsupported-exact-spans-unquoted). Patch 0.4.8 ends the recurrent exact-value fallback at its root: an unsupported backticked value no longer costs its bullet or the whole compaction — the repair retracts the quoting of every rejected span, keeps the text as prose and re-validates, so exact-value failures cannot end a compaction while every surviving backticked value is still verified literally or as its canonical JSON-escaped rendering; the eight-bullet and 25% removal bounds no longer apply to this path, structural, transport and budget failures still fall back to Pi, and the class report remains class-only diagnostics. See the 0.4.8 release record, the 0.4.7 release record, the 0.4.6 release record, the 0.4.5 release record, the 0.4.4 release record, the 0.4.3 release record, the 0.4.2 release record, the 0.4.1 release record and the 0.4.0 release record, ADR 069, ADR 068, ADR 067, ADR 066, ADR 065, ADR 064 and model-awareness validation.

Patch 0.4.13: Historical retrieval filters generic Italian/English interaction words and requires a verified match for lexical-only evidence instead of filling spare budget with unrelated rows. /context explain and /context excluded now identify native excluded groups by estimator cost, positions, exclusion reason and unavailable-rescue reason, separately from historical inventory; /context excluded all retains explicit item-level provenance. Recent-tail and input budgets, rescue policy, tool atomicity and provider-facing defaults are unchanged. Genuine oversized exclusions remain visible; no reporting-host or live-provider confirmation is claimed. See the release record.

Patch 0.4.12: DS4 normalizes the twelve required summary sections before validation on every segment, aggregate and direct update — canonicalizing headings, merging duplicates, restoring contract order and marking unreported sections as not reported instead of failing the graph — so a repairable Markdown omission no longer discards an otherwise usable compaction. Exact-value quote retraction now runs independently of unrelated validation errors, unknown headings and free-form text stay quoted notes in Current State, and both repairs are reported as counter-only warning codes with no repair request to the provider. The strict validator, error severities, bullet and removal bounds, transport retry policy and provider-facing defaults are unchanged; no live-provider or real-compaction confirmation is claimed. See the release record.

Patch 0.4.11: Retrieval stops treating generic workflow vocabulary (devi, procedi, allora, continua, fix, please) as a historical query and awards a bounded bonus for each additional distinct topic match, counting matches with word boundaries and accent folding consistent with the FTS index; explicit backticked, qualified, path, flag, error-code and quoted evidence stays searchable, and a bare acknowledgement produces no-query. Tool results now bind to the latest preceding occurrence of their tool-call ID, so a reused ID can no longer merge independent turns or validate an incomplete call. Budgets, recent-tail policy, privacy handling and provider-facing defaults are unchanged; no live-provider or real-session confirmation is claimed. See the release record.

Patch 0.4.10: Missing-terminator compaction streams now use the existing bounded transport retry policy. Provider failures retain only safe category/reason, stage, attempt and optional HTTP-status metadata; planned/completed segment counts and the original proactive trigger remain accurate after Pi fallback. No blanket retry of HTTP or unknown provider errors, validator relaxation, budget increase, or real-provider success is claimed. See the release record.

Patch 0.4.9: Canonical-source mapping now indexes each candidate once. Compaction diagnostics split local preparation into index sync (JSONL/semantic), source mapping, prompt planning and segment planning; output-limit failures identify their generation stage. Synthetic benchmarks do not establish the cause of the 1,131-second preparation reported on another machine; collect the new timings there before attributing it.

Current compaction defaults: compaction.directUpdate=true, compaction.inputBudget="context", compaction.segmentTargetTokens=30000, compaction.maxRequestInputTokens=64000, compaction.maxOperationInputTokens=2000000, compaction.maxConcurrentSegments=2. Every DS4 provider attempt is bounded by the effective request limit, and the operation limit includes retries; inputBudget="summary" remains an explicit throughput-oriented opt-in. Existing compaction/master switches still apply. See latency controls and compatibility. No real-provider speedup is claimed from mock tests. The five optional editing/reading/artifact/job features introduced in 0.3.4 remain default-off.

Why DS4

Long coding sessions accumulate old decisions, repeated context, large tool outputs and project state faster than any model context window can hold them. DS4 separates durable history from the model's current working set.

It provides:

deterministic token budgeting with soft and hard input limits;
preservation of the current request, recent turns and atomic tool call/result groups;
exact and FTS5 historical retrieval with source provenance;
opt-in hybrid semantic retrieval with a deterministic local embedding and lexical fallback;
trust-gated structural project indexing, Git-aware invalidation and bounded source snippets;
hierarchical, validated, non-destructive compaction summaries with overflow-safe multi-request fan-out/fan-in;
persistent pins and append-only durable memory stored canonically in Pi JSONL;
a bounded, metadata-only context_persistence tool with local confirmation for every model-callable write;
opt-in checkpointed project-memory replay across exact trusted Pi project sessions;
content-addressed storage and bounded references for large tool results;
privacy classifications, secret redaction and provider-specific allow rules;
model-specific calibration and adaptive context allocation;
optional verified continuation for eligible OpenAI Responses profiles;
opt-in metadata-only context-quality metrics and deterministic replay comparisons;
checksummed metadata-only learned ranking with shadow mode, canonical classified feedback and static fallback;
a versioned runtime adapter contract, reusable conformance kit and non-Pi callback/JSONL reference adapter;
opt-in exact-prefix local KV reuse for capable local runtime adapters, with volatile handles and full-replay fallback;
an inspectable Context Manifest explaining included and excluded material;
fail-open recovery to Pi's native context path for operational failures.

Release 0.5.3 — stable supplemental placement

Coordinated 0.5.3 release notes: opt in to context.supplementalPlacement: "stable-prefix" for all supplements at the prefix, or "hybrid" for pin/memory/project at the prefix and retrieval near the latest user request. The default remains "latest-user". Native pin indices, privacy metadata and provenance stay aligned; synthetic groups cannot absorb native leading tool exchanges. Cache reuse still depends on unchanged content, retained history and serialization.

Patch 0.5.2 — rebase validates the active handoff

Coordinated 0.5.2 release notes: validate tool exchanges in the exact compaction-aware Pi context transferred to the checkpoint, not the full historical branch. Archived incomplete exchanges remain preserved without blocking the handoff; active incomplete exchanges still block. Rebase diagnostics expose issue counts only, with no tool IDs or bodies.

Patch 0.5.1 — summary recovery after external purge

Coordinated 0.5.1 release notes: recover summary graphs whose sources were removed by another extension. DS4 preserves canonical summary text as explicitly unverified history instead of inheriting dangling source links; original JSONL is never rewritten by the repair. Compaction validation and budgets remain unchanged. The separate large-index FTS rebuild cost is not addressed by this patch.

History Recall / Session Rebase — 0.5.0

Coordinated 0.5.0 release notes: explicit history tools and recoverable manual rebase, enabled by default and configurable from Pi.

DS4 adds explicit context_history_recall, context_history_read and context_history_status tools, plus manual /context rebase --dry-run / /context rebase / /context rebase --recover <operationId>. Both feature switches default on and can be changed through Pi's /context config set historyTools.enabled false and /context config set sessionRebase.enabled false commands. Saved changes apply at the next session start. Rebase never runs automatically. Source JSONL remains intact; migrations 17–18 only add disposable projections. Deterministic checkpoints do not invoke a provider and block if live context exceeds their budget. See configuration, privacy and recovery contracts, implementation handoff and ADR 074.

Architectural guarantees

Pi JSONL remains canonical. DS4 never replaces Pi's session format.
SQLite is disposable. It contains derived projections and can be rebuilt from canonical sources.
Compaction is non-destructive. Raw history is not deleted or rewritten.
Provenance is preserved. Retrieved and summarized context identifies its source.
Tool groups remain atomic. Tool calls are not separated from their results.
Provider state is optional. Continuation handles and cache state are never canonical.
Operational failures fail open. Pi can continue with its native context behavior.
Enabled privacy enforcement fails closed. Restricted content is replaced or the provider payload is rejected instead of being leaked.

Requirements

Pi 0.84.3
Node.js 22.19.0 or newer
SQLite support provided by Node's built-in node:sqlite

Pi is intentionally pinned until extension contract tests validate a newer release.

Installation

Pi packages execute with the user's full permissions. Review and trust the source before installing this or any other extension.

From GitHub

pi install git:github.com/Alucard24/ds4-context-engine

To try it for one run without adding it to settings:

pi -e git:github.com/Alucard24/ds4-context-engine

From npm

Install the latest stable public npm package with:

pi install npm:ds4-context-engine

The three packages (ds4-context-engine, ds4-context-core, and ds4-context-reference-adapter) are released together with the same exact version. Both adapters require the matching core version. See 0.3.5 for this release's changes and compatibility.

To dogfood the beta without replacing a global stable installation, pin the exact version in a disposable project:

pi install -l npm:ds4-context-engine@0.3.0-beta.3

Follow the 0.3 beta dogfooding runbook; use synthetic data and a dedicated session directory.

Local checkout

git clone https://github.com/Alucard24/ds4-context-engine.git
cd ds4-context-engine
npm ci
npm run check
pi install -l .

For extension development without installing the package:

pi -e ./src/extension/index.ts

Restart Pi or run /reload after installing or changing the extension.

Quick start

DS4 starts in managed mode with conservative defaults. No configuration file is required.

After loading the extension, inspect its state:

/context status
/context adapter
/context tokens
/context health

Global configuration is loaded from:

~/.pi/agent/ds4-context.json

A trusted project can override it with:

.pi/ds4-context.json

Use observer mode as a pass-through rollback while retaining diagnostics:

{
  "context": {
    "mode": "observer"
  }
}

Disable the extension's behavior without uninstalling it:

{
  "enabled": false
}

Project configuration and project source indexing are disabled when Pi reports the project as untrusted.

Commands

Inspection

Command	Purpose
`/context` or `/context status`	Runtime, session, planner and subsystem status
`/context adapter`	Runtime contract and per-capability negotiation diagnostics
`/context tokens`	Token budget and active-context composition
`/context manifest`	Latest Context Manifest
`/context explain`	Human-readable planning explanation
`/context included`	Items selected for the latest model call
`/context excluded`	Compact native-group exclusions: tokens, positions, reason and rescue eligibility
`/context excluded all`	Excluded item-level provenance (retained projection after rollup)
`/context summaries`	Hierarchical summary graph diagnostics
`/context retrieved`	Historical retrieval diagnostics
`/context history [search <query>\|status\|read <sourceRef> [startLine] [maxLines]]`	Scoped recall and bounded original-source reads (enabled by default)
`/context rebase [--dry-run\|--recover <operationId>]`	Manual session rebase with source preservation and recoverable activation (enabled by default)
`/context project`	Project index and retrieval status
`/context privacy`	Classification and provider-policy status
`/context model`	Active model profile and calibration
`/context quality`	Metadata-only context-quality scores and sample counts
`/context ranking`	Learned model, promotion gate, aggregate shadow comparison and feedback counts
`/context continuation`	Native continuation decisions and counters
`/context artifacts`	Artifact storage and integrity status
`/context compaction`	Last compaction status
`/context compact-preview`	Preview compaction diagnostics
`/context health`	SQLite and subsystem health checks
`/context rebuild-index`	Rebuild derived state from canonical sources

Pins and durable memory

/context pins
/context pin [--scope session|branch|project] [--classification LEVEL] <content>
/context unpin PIN_ID [reason]

/context memory
/context memory list
/context memory add [--scope session|project] [--key KEY] [--classification LEVEL] <claim>
/context memory supersede MEMORY_ID [--source ID,ID] <new claim>
/context memory invalidate MEMORY_ID [reason]
/context memory expire MEMORY_ID [reason]
/context memory sources
/context memory exclude SESSION_ID [reason]
/context memory include SESSION_ID

Valid privacy classifications are normal, internal, sensitive and local-only.

LLM-callable tools

DS4 registers two model-callable tools by default:

Tool	Purpose
`context_artifact_search`	Search a known DS4 artifact reference with bounded quoted excerpts
`context_persistence`	Inspect Pins, Memory, and project-memory sources; perform explicitly requested persistence mutations

History Recall / Session Rebase is enabled by default. historyTools.enabled controls the additional context_history_recall, context_history_read, and context_history_status tools; sessionRebase.enabled controls the manual rebase command. Both can be disabled through /context config, and rebase never runs automatically or deletes the original session. See History Recall / Session Rebase for scopes, privacy, budgets, recovery, and limitations.

context_persistence read actions return bounded metadata and sanitized find previews. Every write requires a fresh local ctx.ui.confirm() decision. In print/JSON or any other no-UI mode, reads remain available and writes fail closed with confirmation-required. Sessions without a persistent Pi JSONL destination (for example --no-session) fail closed with runtime-unavailable before confirmation. Destructive writes require an exact ID or volatile source reference plus the targetRevision returned by a prior read; fuzzy writes are not supported.

Canonical Pin and Memory changes append Pi custom entries and reconcile disposable SQLite projections. Project-memory source include/exclude is derived local SQLite policy and never appends a fake canonical entry. See docs/CONTEXT_PERSISTENCE_TOOL.md.

Optional anchored editing: /context config set editing.anchored true, then /reload, enables a same-name edit wrapper. Use head[upto]tail in oldText to replace an inclusive range without repeating its intermediate old content. Exact anchors are validated inside Pi's native mutation queue; ordinary calls and diff feedback remain native. All edits in a batch containing anchors must match exactly. Use per-edit literal: true for real marker text. Disabled by default; no inference-time forcer or provider change is involved. See docs/ANCHORED_EDITING.md for semantics and limits.

Other optional agent tools: editing.postEditReport adds bounded old/new line ranges and updated context; reading.adaptive chooses model-window-aware default read limits; artifacts.adaptiveBudget lowers inline/excerpt caps under context pressure; jobs.enabled exposes confirmed, session-owned local bash_job start/status/stop/list operations. All default off and require session reload after configuration changes. Jobs are a separate module, not a replacement for bash. See docs/PORTABLE_AGENT_TOOLS.md for activation, limits, privacy and lifecycle behavior.

Learned-ranking feedback and local training are explicit:

/context ranking feedback useful|irrelevant CANDIDATE_ID [--classification LEVEL]
/context ranking train

See docs/LEARNED_RANKING.md.

Configuration reference

The following example shows the main configuration groups. Omitted values use the defaults in packages/core/src/config/config.ts.

{
  "enabled": true,
  "context": {
    "mode": "managed",
    "targetFillRatio": 0.7,
    "softLimitRatio": 0.8,
    "hardLimitRatio": 0.9,
    "minimumOutputReserve": 8192,
    "preferredOutputReserve": 32768,
    "recentTailTokens": 64000,
    "rescueImmediatePredecessor": true,
    "maxPinnedTokens": 16000,
    "maxMemoryTokens": 8000,
    "maxRetrievedHistoryTokens": 16000,
    "maxProjectTokens": 20000,
    "maxSummaryTokens": 12000
  },
  "editing": {
    "anchored": false,
    "postEditReport": false
  },
  "reading": {
    "adaptive": false
  },
  "jobs": {
    "enabled": false
  },
  "retrieval": {
    "exact": true,
    "fts": true,
    "semantic": false,
    "maxResults": 12,
    "embedding": {
      "mode": "local",
      "provider": "ds4-local",
      "model": "feature-hash-v1",
      "dimensions": 256,
      "remoteProfiles": [],
      "maxSources": 50000,
      "candidatePool": 80,
      "batchSize": 64,
      "queryCacheSize": 64,
      "timeoutMs": 2000
    }
  },
  "project": {
    "enabled": true,
    "maxFiles": 10000,
    "maxFileBytes": 512000,
    "maxTotalBytes": 50000000,
    "snippetLines": 80,
    "snippetOverlapLines": 12,
    "maxResults": 8
  },
  "memory": {
    "enabled": true,
    "crossSession": false,
    "maxProjectSessions": 250,
    "maxPinChars": 4000,
    "maxClaimChars": 2000,
    "maxResults": 12
  },
  "artifacts": {
    "enabled": true,
    "adaptiveBudget": false,
    "maxInlineToolResultChars": 12000,
    "maxArtifactBytes": 100000000,
    "maxSearchBytes": 50000000,
    "excerptChars": 6000,
    "maxSearchMatches": 12,
    "storeLargeOutputs": true
  },
  "compaction": {
    "enabled": true,
    "mode": "hierarchical",
    "validate": true,
    "segmentTargetTokens": 30000,
    "maxRequestInputTokens": 64000,
    "maxOperationInputTokens": 2000000,
    "preserveRecentVerbatim": true,
    "directUpdate": true,
    "inputBudget": "context",
    "maxConcurrentSegments": 2
  },
  "privacy": {
    "enabled": false,
    "defaultClassification": "normal",
    "localProviders": ["faux", "ollama", "llama-cpp", "lmstudio"],
    "remoteDefaultAllowed": ["normal", "internal"],
    "remoteProviders": {
      "openrouter": ["normal"]
    },
    "redactSecrets": true
  },
  "modelAwareness": {
    "enabled": true,
    "calibrationWindow": 24,
    "minimumCalibrationSamples": 3,
    "calibrationRatioLowerBound": 0.5,
    "calibrationRatioUpperBound": 2.0,
    "overrides": {
      "openrouter/vendor/model": {
        "contextWindow": 200000,
        "maxRetrievedHistoryTokens": 12000
      }
    }
  },
  "nativeContinuation": {
    "enabled": false,
    "allowProviderStorage": false,
    "profiles": ["openai/*"],
    "maxStateAgeMs": 1800000,
    "retryManagedReplay": true
  },
  "quality": {
    "enabled": false,
    "maxSamples": 1000
  },
  "ranking": {
    "mode": "off",
    "modelPath": "ds4-context/ranking-model.json",
    "minimumTrainingSamples": 20,
    "maxTrainingSamples": 10000,
    "maxLatencyMs": 10
  },
  "diagnostics": {
    "storeContextManifest": true,
    "storeFullRenderedContext": false,
    "logLevel": "info"
  },
  "storage": {
    "scope": "project",
    "databasePath": "ds4-context/context.db",
    "busyTimeoutMs": 5000,
    "writeRetryTimeoutMs": 30000,
    "projectIndexLeaseMs": 120000
  }
}

Invalid or unknown values are ignored with a warning. Model overrides merge deterministically from * to provider/* to an exact provider/model profile. Routine session open/close, database, rebuild, and project-index summaries are emitted only at debug, so the default info level keeps session changes quiet while preserving actionable warnings. The 0.2 release line freezes this additive surface as ds4-context-config-v1; existing keys, validation, and defaults are pinned by the compatibility golden.

Privacy and provider storage

Privacy enforcement is disabled by default and must be configured for the providers you use. Unknown providers are treated as remote unless explicitly listed as local. local-only content is never permitted by a remote allow rule.

Native continuation is also disabled by default. Enabling it requires both explicit storage consent and an exact or provider-scoped profile:

{
  "nativeContinuation": {
    "enabled": true,
    "allowProviderStorage": true,
    "profiles": ["openai/*"]
  }
}

Eligible OpenAI Responses requests then set store: true. Review the provider's retention policy before enabling this option. DS4 keeps response handles only in volatile memory, verifies exact managed prefixes before reuse and retries once with a full managed replay when recognized continuation state is stale.

Local KV reuse is separately disabled by default through localKvReuse.enabled. It also requires a local runtime adapter with a versioned local-kv-reuse capability and a volatile runtime port. Pi exposes no such handles and remains unsupported even if configuration is enabled.

See docs/PRIVACY.md, docs/NATIVE_CONTINUATION.md, and docs/LOCAL_KV_REUSE.md.

Storage and recovery

By default, derived state is stored below Pi's agent directory:

~/.pi/agent/ds4-context/
├── context.db
├── ranking-model.json
└── artifacts/

The database contains rebuildable indexes, summary metadata, manifests, project projections and calibration data. The optional checksummed learned-ranking model is also derived local state; its classified metadata-only labels remain canonical Pi custom entries. All Pi sessions share this WAL database: writes use bounded busy-aware transaction replay, and a renewable project lease prevents multiple Pi processes from indexing the same project concurrently. busyTimeoutMs controls each SQLite lock wait, while writeRetryTimeoutMs bounds the total replay window. Exhausted lock retries identify only the coordinator operation and categorical SQLite metadata. Diagnostic storage keeps the latest 128 manifests globally and 200 calibration samples per exact profile. Online manifest pruning is bounded to 32 rows and 8 MiB per related write. Manifests above the 256 KiB preferred bound retain complete included provenance and use an explicit deterministic excluded-only rollup; projected payloads above 1 MiB are skipped. Current readers label rollups explicitly; earlier schema-15 readers may parse them but mislabel sampled excluded details, so that historical rendering is not downgrade-supported after rollups are written. Provider usage updates existing scalar columns without rewriting the JSON payload. Canonical memory and pin mutations remain append-only entries in Pi JSONL. Project files remain canonical for project knowledge. Complete tool results remain in Pi JSONL while the artifact store keeps verified, content-addressed copies for bounded retrieval.

To inspect, validate, or rebuild derived state:

/context health
/context storage
/context rebuild-index

Physical size recovery is deliberately offline and interactive:

ds4-context-storage inspect --database <exact-path>
ds4-context-storage compact --database <exact-path>
ds4-context-storage recover --database <exact-path>

Close every Pi process before compact or recover. New runtimes create cooperative client leases and refuse to open SQLite while the maintenance lock exists; the CLI also refuses active or ambiguous clients, validates a standalone backup and candidate, and keeps one fixed pre-compaction backup. See docs/STORAGE_MAINTENANCE.md. Deleting DS4's database must not alter a Pi session or project, although derived indexes and calibration data will be regenerated. When memory.crossSession is enabled for a trusted project, DS4 discovers bounded sibling Pi JSONL files by exact canonical header identity, incrementally replays their explicit project mutations, and excludes missing or unverifiable sources.

Development

npm ci
npm run build:core
npm run build:adapters
npm run typecheck
npm test
npm run check
npm run quality:compare
npm run schema:context-persistence
npm run latency:check -- /path/to/exact/ds4-context-core@0.1.2
npm run pack:check
# Post-publication, with an exact version rather than a dist-tag:
npm run registry:check -- 0.3.0-beta.2
npm pack --dry-run
npm pack --dry-run --workspace ds4-context-core
npm pack --dry-run --workspace ds4-context-reference-adapter

The test suite covers configuration, migrations, canonical JSONL projection, planning, atomic tool groups, retrieval, compaction, project knowledge, artifacts, memory, privacy, model awareness, continuation, local-KV eligibility/replay, runtime-adapter conformance, the portable-core dependency boundary and Pi extension lifecycle behavior. The latency comparison times 50 planner calls per sample to reduce sub-millisecond timer and scheduler noise while preserving the 1.10 p95 rejection threshold. The package check builds all three tarballs, installs them in a clean temporary consumer, reruns compiled reference-adapter conformance and starts the packaged Pi extension with isolated RPC state.

Portable core

ds4-context-core is a compiled ESM package with no runtime SDK dependency. It owns runtime-neutral policy, storage, adapter contracts and projections; agent adapters translate native sessions and lifecycle hooks at the boundary. The root ds4-context-engine package is the Pi adapter. ds4-context-reference-adapter is a separately compiled non-Pi callback/JSONL implementation. Both depend one-way and exactly on matching core.

Repository layout

packages/core/src                 portable policy, adapter kit, planning, retrieval and storage
packages/reference-adapter/src    non-Pi callback/JSONL reference runtime boundary
src/pi-adapter                    Pi JSONL projection, summary completion and provider integration
src/extension       Pi hooks, commands and fail-open orchestration
tests               core contract, unit, integration, golden and benchmark coverage
scripts             package and release-readiness checks
.github/workflows   continuous integration

Documentation

Roadmap

The original M0–M13 roadmap is complete. ds4-context-core contains the compiled runtime-neutral implementation. M14 context-quality metrics, M15 rich symbol indexing, M16 hybrid semantic retrieval, M17 cross-session project memory, M18 learned-ranking shadow evaluation, M19's runtime adapter/conformance kit, and M20 opt-in local KV eligibility/replay are implemented on main. Learned active ranking remains promotion-gated, Pi reports local KV as unsupported, and static ranking/native completion stay authoritative on every failure.

The 0.2.0 roadmap is complete. The stable 0.3 line carries forward the context persistence tool, privacy-safe compaction, bounded persisted manifests, cooperative client leases and recoverable offline maintenance. Version 0.4.0 introduces storage.scope ("agent" | "project", default "project"): one rebuildable SQLite projection per trusted canonical project root, with token calibration shared in the agent database; storage.scope: "agent" restores the previous single-file layout. Version 0.4.1 states the contiguous-span rule for compaction summaries in the summarizer prompt without changing deterministic validation or the repair bounds. Version 0.4.2 adds class-only diagnostics to the same fail-closed path: the custom_fallback warning reports unsupportedSpanClasses as class names and counters only, never span text, without changing validation strictness, repair bounds or defaults. Version 0.3.10 adds opt-in BPE estimation and bounded model-budget auto-tuning while retaining the chars-v1 default. Version 0.3.9 extends the bounded compaction updates, summary input headroom, concurrent segments and phase timings introduced in 0.3.5 with per-request and cumulative operation input limits; 0.3.8 adds indexed FTS key deletion without changing search results. The opt-in anchored editing and portable agent tools from 0.3.4 remain default-off, without backend rewind, forced sampling or operational KV integration. Confirmation, provenance, Pi fallback and canonical/configuration/SQLite/runtime contracts remain unchanged. The 0.2 readiness record remains the compatibility baseline; the lexical planner stays available as the deterministic fallback.

Contributing

Issues and focused pull requests are welcome. Before submitting a change:

preserve Pi JSONL as canonical history;
keep SQLite and artifacts rebuildable;
preserve provenance and atomic tool groups;
retain strict compaction validation and safe fallback behavior;
add or update tests;
run npm run check, npm run pack:check and git diff --check.

Please include reproduction steps for bugs and avoid attaching real session files, credentials or private provider payloads.

License

MIT