Cache verification

This project has two complementary checks. They answer different questions and should not be conflated.

Deterministic payload verification proves that this plugin projects a stable provider payload under a controlled local capture server. It does not measure a provider cache.
Live cache benchmarking records telemetry returned by an already-running provider-backed OpenCode server. It is opt-in, provider-specific, and useful for comparing explicitly controlled arms; it is not a CI check or a general cache guarantee.

Prerequisites

Install dependencies and build the local plugin before either check:

bun install
bun run build

The deterministic harness requires a separately pre-provisioned absolute path to exactly opencode-ai@1.18.2. It deliberately does not install the host binary or dependencies itself.

Offline deterministic payload verification

Run the harness with the pre-provisioned host binary:

OPENCODE_BIN=/absolute/path/to/node_modules/.bin/opencode \
  bun run verify:cache-stability

The script rejects a missing, non-absolute, nonexistent, or non-1.18.2 OPENCODE_BIN. It uses the built local plugin and a local capture provider, disables Bun auto-install, and makes no external provider or dependency resolution network calls.

It asserts the deterministic request shape, including:

exactly four primary/orchestrator model requests;
stable system, tool, and option projections across those requests;
append-only, prefix-stable conversation input and historical transformed user prompts;
exactly one phase reminder in each eligible user projection, with no reminder marker in system instructions;
availability and transcript execution results for read and todowrite.

Passing this check demonstrates payload stability under the fixture. It does not establish that any external model provider accepted or reused a cache prefix.

Opt-in live cache benchmark

The runner talks directly to an existing OpenCode HTTP server. It does not start a server, load credentials, or use the SDK. The required CLI contract is:

bun scripts/benchmark-opencode-cache.ts \
  --server http://127.0.0.1:4096 \
  --provider PROVIDER_ID \
  --model MODEL_ID \
  --runs N \
  --output /safe/output.json \
  --arm ARM_ID \
  --plugin-build BUILD_ID

--server, --provider, --model, --runs, --output, --arm, and --plugin-build are required. Harmless timing/retry controls have defaults: --timeout-ms (120000), --retries (2), and --retry-delay-ms (500). Use repeated --header NAME:VALUE only when the already-running server needs an HTTP header; headers are not persisted. --overwrite permits replacing an existing output file.

Each fresh session targets orchestrator and the requested provider/model. The fixed sequence requires Read(package.json), a completed TodoWrite, a final answer, and a second user follow-up. The runner establishes SSE coverage before prompting, deletes every created session, and records discards rather than resampling. Retry, compaction, routing mismatch, incomplete trace, stream coverage failure, and assistant/provider errors all invalidate a session.

Zen free example

For OpenCode Zen free routing, use provider opencode and model hy3-free. Keep credentials outside the repository, in the environment of an isolated server process. In the isolated OpenCode configuration, make model selection explicit and disable title generation so its separate small-model request does not confound the run:

{
  "preset": "opencode-zen-free",
  "default_agent": "orchestrator",
  "model": "opencode/hy3-free",
  "small_model": "opencode/hy3-free",
  "agent": {
    "orchestrator": { "model": "opencode/hy3-free" },
    "title": { "disable": true }
  }
}

With that isolated server running, a live invocation is:

bun scripts/benchmark-opencode-cache.ts \
  --server http://127.0.0.1:4096 \
  --provider opencode \
  --model hy3-free \
  --runs 12 \
  --arm current \
  --plugin-build "$(git rev-parse HEAD)" \
  --output /tmp/opencode-zen-hy3-current.json

Report interpretation

The JSON report is intentionally redacted. It excludes prompt and response text, request bodies, tool inputs/outputs, workspace paths, headers, nonce values, and raw event properties. It retains normalized routing, tool status, safe assistant/provider error-code classifications, cache token telemetry, timing observations, discard reasons, and comparison metadata.

Request observations preserve cache fields as null when the provider did not report them; null is not a cache miss. totalInput is only defined when it can be reconstructed as input + cacheRead + cacheWrite. Position one in each eligible session is preserved as raw warm-up data but excluded from aggregate cache and timing metrics.

The primary metric block contains session-clustered bootstrap 95% confidence intervals for warm-up-excluded zero-cache miss rate, cache coverage, prompt-loop TTFT p50, and prompt-loop latency p50. Prompt-loop timings are one observation per submitted user turn, not duplicated over provider steps. eligiblePrefixCoverage is only meaningful when the provider reports enough telemetry to calculate it; otherwise treat it as unavailable.

Methodology and limits

Use the same OpenCode host version, workspace fixture, provider configuration, agent/model routing, and runner revision for every arm. Build or package each plugin arm equivalently, run a one-session clean preflight per arm, then run separate baseline and current arms. Compare the resulting redacted reports; do not mix arms in one server or silently replace discarded sessions.

This is not CI. Provider cache semantics, free-route policy, cache TTL, capacity, model revisions, entitlement, and routing can change independently of this repository. A live result is evidence for its recorded environment, not a portable performance claim.

Observed #784 controlled Zen HY3 result

The following is an environment-specific controlled observation, not a guarantee. It used isolated OpenCode 1.18.2 hosts, preset opencode-zen-free, title generation disabled, opencode/hy3-free routing, the fixed tool trace, and 12 independent eligible sessions per arm.

Metric	Baseline `67e2a556`	Current `08e9fa5`
Eligible / discarded sessions	12 / 0	12 / 0
Warm-up-excluded request observations	24	24
Zero-cache miss rate (95% CI)	1.000 [1.000, 1.000]	0.000 [0.000, 0.000]
Cache coverage (95% CI)	0.000 [0.000, 0.000]	0.941345 [0.940887, 0.942179]
Median eligible-prefix coverage	0.000000	0.903893
Cache read / write tokens	0 / 0	397760 / 0
Prompt-loop TTFT p50 ms (95% CI)	2915.24 [2883.83, 2983.40]	2115.54 [2033.55, 2326.98]
Prompt-loop latency p50 ms (95% CI)	3216.77 [3072.35, 3255.93]	2267.81 [2248.48, 3012.78]

Every eligible session completed read and todowrite; observed routing was orchestrator / opencode / hy3-free. Repeat the controlled procedure before drawing conclusions under a different provider account, host, model revision, or cache policy.

Observed OpenAI manual verification

This single-session manual smoke result is not a baseline comparison. It used an isolated host with existing local OpenAI authentication, the current plugin build, and openai/gpt-5.6-terra-fast.

Metric	Observed value
Eligible sessions	1
Cache-read tokens after warm-up	12288, 13824, 14336
Warm-up-excluded cache coverage	93.33%
Eligible-prefix coverage	85.64%
Second-turn prompt-loop TTFT	1983.02 ms
Second-turn prompt-loop latency	2278.46 ms

The required read and todowrite trace completed with no assistant or provider error. Repeat a controlled baseline/current comparison before making a provider-wide performance claim.

cache-verification.md 7.8 KB Permalink History Raw