Browse Source

docs(architecture): add precedence experiment transcript — last-match-wins confirmed live

Verified against a real OpenCode 1.17.20 install: last-match-wins confirmed
(first-match and most-specific refuted by behavioral probes; resolver source
uses findLast). Integer-like and duplicate scope hazards both confirmed real;
no reordering transform or 34-agent migration needed.

Task: canonical-refactor-09
darrenhinde 2 weeks ago
parent
commit
8d64bddc55
1 changed files with 347 additions and 0 deletions
  1. 347 0
      docs/architecture/canonical-refactor/10-PRECEDENCE-EXPERIMENT.md

+ 347 - 0
docs/architecture/canonical-refactor/10-PRECEDENCE-EXPERIMENT.md

@@ -0,0 +1,347 @@
+# 10 — Permission Precedence Experiment (last-match-wins)
+
+> **Status:** Complete — live behavioral verification against a real OpenCode install.
+> **Date:** 2026-07-15 · Task: canonical-refactor-09 · Companion to `00-INDEX.md` §5 (Option A capabilities).
+> **Verdict: CONFIRMED — OpenCode resolves permissions last-match-wins.** Most-specific-wins
+> and first-match-wins are both behaviorally **refuted**. Zero reordering transform is
+> required; the 34 agents need **no migration**. Both order-as-semantics hazards flagged in
+> `00-INDEX` (integer-like scopes, duplicate scopes) were **probed and confirmed real** —
+> the integer-like case is *worse* than the spec assumed.
+
+---
+
+## 1. The question and why it needed a live experiment
+
+`00-INDEX.md` (v2.2) downgraded the last-match-wins claim from "CONFIRMED" to *strongly
+indicated*, because all three prior citations were **this project's own documents**
+(self-citation), and the primary one described a JSON-array permission format that exists in
+no agent. This experiment replaces self-citation with:
+
+1. **Live behavioral probes** against a real OpenCode install (primary evidence), and
+2. **OpenCode's actual resolver source** for the installed version (corroborating evidence).
+
+## 2. Environment
+
+| Item | Value |
+|---|---|
+| OpenCode CLI | `/Users/darrenhinde/.opencode/bin/opencode`, version **1.17.20** (Mach-O arm64 compiled binary) |
+| Host | macOS (Darwin 24.6.0), zsh |
+| Probe project | Fresh temp dir **outside this repo** with `.opencode/agent/probe*.md` primary agents |
+| Invocation | `opencode run --agent <probe> -m <model> "<instruction>"` (non-interactive) |
+| Models | `opencode-go/minimax-m3` and `opencode-go/minimax-m2.7` (probe 3b and 4b re-runs after a model-side timeout; the model choice does not affect the resolver) |
+
+Each probe agent's body was identical boilerplate: *"Run exactly the commands requested, one
+bash tool call each, and report each tool result verbatim… Do not retry, do not substitute
+commands."* Only the `permission:` frontmatter differed. Outputs below are verbatim with ANSI
+escape codes stripped.
+
+---
+
+## 3. Probe 1 — broad-first / specific-after (the pattern all 34 agents use)
+
+**Agent frontmatter** (`.opencode/agent/probe1.md`):
+
+```yaml
+---
+description: Precedence probe 1 - broad deny first, specific allow after
+mode: primary
+tools:
+  bash: true
+permission:
+  bash:
+    "*": "deny"
+    "echo ok*": "allow"
+---
+```
+
+**Command:**
+
+```
+opencode run --agent probe1 -m opencode-go/minimax-m3 "Using the bash tool, run exactly
+this command: echo ok probe1
+Then, as a second separate bash tool call, run exactly: ls
+Report each tool call's raw result verbatim, including any permission or error messages."
+```
+
+**Observed output:**
+
+```
+> probe1 · minimax-m3
+
+I'll run the two commands as separate bash tool calls as requested, starting with the first one.
+$ echo ok probe1
+ok probe1
+
+✗ ls failed
+Error: The user has specified a rule which prevents you from using this specific tool call.
+Here are some of the relevant rules [{"permission":"*","action":"allow","pattern":"*"},
+{"permission":"bash","pattern":"*","action":"deny"},
+{"permission":"bash","pattern":"echo ok*","action":"allow"}]
+```
+
+**Reading:**
+
+- `echo ok probe1` → **executed** (real shell output `ok probe1`). Both `"*": deny` and
+  `"echo ok*": allow` match; the later rule won.
+- `ls` → **denied**. Only `"*": deny` matches (after the baseline).
+- The error message dumps OpenCode's **internal flattened rule list, in order**: a global
+  baseline `{"permission":"*","pattern":"*","action":"allow"}` first, then the agent's rules
+  **in authored key order**. This is direct evidence that (a) authored order is preserved
+  into the rule list for non-integer keys, and (b) resolution walks that ordered list.
+- **First-match-wins is refuted outright:** under first-match, `"*": deny` would have blocked
+  `echo ok probe1`.
+
+Result is consistent with both last-match-wins and most-specific-wins → probe 2 separates them.
+
+## 4. Probe 2 — the distinguishing case (last-match vs most-specific disagree)
+
+A distinguishing case **is** expressible: author the *specific* rule first and a *broad*
+contradicting rule last. The two hypotheses then predict opposite outcomes.
+
+**Agent frontmatter** (`.opencode/agent/probe2.md`):
+
+```yaml
+permission:
+  bash:
+    "echo ok*": "deny"     # specific, authored FIRST
+    "*": "allow"           # broad, authored LAST
+```
+
+| Hypothesis | Prediction for `echo ok probe2` |
+|---|---|
+| Last-match-wins | **ALLOW** (`"*"` is the last matching rule) |
+| Most-specific-wins | **DENY** (`"echo ok*"` is more specific than `"*"`) |
+
+**Command:**
+
+```
+opencode run --agent probe2 -m opencode-go/minimax-m3 "Using the bash tool, run exactly
+this command: echo ok probe2
+Report the tool call's raw result verbatim, including any permission or error messages.
+Do not run anything else."
+```
+
+**Observed output:**
+
+```
+> probe2 · minimax-m3
+
+$ echo ok probe2
+ok probe2
+```
+
+**Reading:** the command **executed**. The specific `deny` authored earlier was overridden by
+the broad `allow` authored later. **Last-match-wins confirmed; most-specific-wins refuted.**
+
+## 5. Probe 3 — integer-like scope (`"8080"`)
+
+`00-INDEX` warned that integer-like scope keys "silently jump to the front under ECMAScript
+integer-key ordering." Probed with a matched pair differing **only** in whether the specific
+key is integer-like.
+
+**Probe 3 frontmatter:**
+
+```yaml
+permission:
+  bash:
+    "*": "deny"
+    "8080": "allow"        # integer-like key
+```
+
+**Command:** run exactly the single token `8080` (expected `command not found` if
+permission-allowed; permission error if denied).
+
+**Observed output (probe 3):**
+
+```
+> probe3 · minimax-m3
+
+I don't have a `bash` tool available in this environment. My available tools are:
+- todowrite
+- task
+- read
+- glob
+- grep
+- edit
+- write
+- webfetch
+- skill
+
+There is no `bash` tool to invoke...
+```
+
+**Probe 3b (control) frontmatter — identical shape, non-integer key:**
+
+```yaml
+permission:
+  bash:
+    "*": "deny"
+    "porthost": "allow"
+```
+
+**Observed output (probe 3b, command `porthost`):**
+
+```
+> probe3b · minimax-m2.7
+
+$ porthost
+zsh:1: command not found: porthost
+```
+
+**Reading:**
+
+- With the **non-integer** key (`porthost`), the last-match `allow` was reachable: the bash
+  tool stayed available and the command was permission-allowed all the way to a real shell
+  execution (`zsh: command not found` = exit 127, i.e. it genuinely ran).
+- With the **integer-like** key (`8080`), the `allow` rule became unreachable — the key was
+  reordered to the front, leaving `"*": deny` as the effective last rule for *every* command —
+  and OpenCode went further: it **removed the bash tool from the agent's toolset entirely**
+  (no rule can ever allow anything, so the tool is dropped). The only diff between 3 and 3b
+  is the key, so the integer-like key is the cause. *(The exact tool-hiding trigger is
+  inferred from behavior, not source-verified; the reordering itself is source-verified below.)*
+- Mechanism demonstrated directly in Node (the resolver flattens config via `Object.entries`,
+  see §7):
+
+```
+$ node -e 'console.log(JSON.stringify(Object.entries({"*":"deny","8080":"allow"})));
+           console.log(JSON.stringify(Object.entries({"*":"deny","porthost":"allow"})))'
+[["8080","allow"],["*","deny"]]
+[["*","deny"],["porthost","allow"]]
+```
+
+**Hazard confirmed — and worse than spec'd:** an integer-like scope doesn't just get wrong
+precedence; combined with a trailing wildcard deny it can silently disable the whole tool.
+The IR parser **MUST reject or warn on integer-like scope keys** at parse time (per `02` Q2).
+
+## 6. Probe 4 — duplicate scopes
+
+**Probe 4 frontmatter (duplicate YAML keys):**
+
+```yaml
+permission:
+  bash:
+    "echo dup*": "deny"
+    "echo dup*": "allow"
+```
+
+**Observed output (command `echo dup test`):**
+
+```
+> probe4 · minimax-m2.7
+
+$ echo dup test
+dup test
+```
+
+**Probe 4b (reversed duplicates + trailing wildcard, consistency check):**
+
+```yaml
+permission:
+  bash:
+    "echo dup*": "allow"
+    "echo dup*": "deny"
+    "*": "allow"
+```
+
+**Observed output (command `echo dup test`):**
+
+```
+> probe4b · minimax-m3
+
+$ echo dup test
+dup test
+```
+
+**Reading:**
+
+- Probe 4 discriminates: if the **first** duplicate survived, the effective ruleset would be
+  `{"echo dup*": deny}` → denied. Observed **allowed** ⇒ the **last occurrence** of a
+  duplicate key silently wins at YAML-parse time. **No load error, no warning** — the `deny`
+  vanished without trace.
+- Probe 4b is consistent: last-occurrence collapse gives `{"echo dup*": deny, "*": allow}`,
+  then last-match-wins makes `"*": allow` final → allowed, as observed. (4b alone is
+  ambiguous — first-occurrence collapse also predicts allow — so probe 4 carries the finding.)
+
+**Hazard confirmed:** duplicate scopes collapse silently. Which occurrence survives is a YAML
+parser detail, not a documented contract. The IR parser **MUST error on duplicate scopes**
+within one capability (per `02` Q2).
+
+## 7. Corroborating evidence — OpenCode's actual resolver source
+
+The installed CLI is a compiled binary, so the resolver was read from the public source for
+the installed version — `github.com/sst/opencode`, tag `v1.17.20`,
+`packages/opencode/src/permission/index.ts` (fetched 2026-07-15):
+
+```typescript
+export function evaluate(permission: string, pattern: string, ...rulesets: PermissionV1.Ruleset[]): PermissionV1.Rule {
+  return (
+    rulesets
+      .flat()
+      .findLast((rule) => Wildcard.match(permission, rule.permission) && Wildcard.match(pattern, rule.pattern)) ?? {
+      action: "ask",
+      permission,
+      pattern: "*",
+    }
+  )
+}
+```
+
+```typescript
+export function fromConfig(permission: ConfigPermissionV1.Info) {
+  const ruleset: PermissionV1.Rule[] = []
+  for (const [key, value] of Object.entries(permission)) {
+    if (typeof value === "string") {
+      ruleset.push({ permission: key, action: value, pattern: "*" })
+      continue
+    }
+    ruleset.push(
+      ...Object.entries(value).map(([pattern, action]) => ({ permission: key, pattern: expand(pattern), action })),
+    )
+  }
+  return ruleset
+}
+```
+
+- `evaluate` is **literally `.findLast()`** over the flattened ordered rulesets —
+  last-match-wins by construction, no specificity ranking anywhere.
+- `fromConfig` flattens the YAML/JSON mapping via **`Object.entries`** — the exact spot where
+  ECMAScript integer-key ordering reorders `"8080"` to the front (probe 3) and where a
+  YAML-parsed object has already collapsed duplicate keys (probe 4).
+- Note the resolver's ultimate fallback (no rule matches at all) is **`ask`**, not `allow` —
+  but in practice a global baseline rule `{"permission":"*","pattern":"*","action":"allow"}`
+  sits first in the ruleset (observed verbatim in probe 1's error dump), so an unmatched
+  bash command resolves `allow` via that baseline. This matches `00-INDEX`'s "absent
+  capability → tool default" framing; the IR should treat "no rule matched" as
+  **tool-default**, not hardcode `allow`.
+
+## 8. Verdict and consequences
+
+**CONFIRMED: last-match-wins.** Evidence chain: probe 1 (refutes first-match) → probe 2
+(refutes most-specific) → source `evaluate()` uses `.findLast()` (mechanism identified) →
+probe 1's error dump shows authored key order preserved in the flattened ruleset.
+
+Consequences for the canonical refactor:
+
+1. **No reordering transform. No migration of the 34 agents.** The authored
+   broad-first/specific-after convention is exactly what OpenCode's resolver honors. The
+   Option A ordered-rule IR serializes to OpenCode key order 1:1, both directions.
+2. **Parse-time validation is mandatory, not optional** (`02` Q2 — both hazards are now
+   observed facts, not theory):
+   - **Reject/warn on integer-like scope keys** (`/^\d+$/` after trimming). Live consequence:
+     rule silently unreachable, potentially entire tool silently removed.
+   - **Error on duplicate scopes** within a capability. Live consequence: one decision
+     silently discarded at YAML load with no warning. The IR's array form
+     (`[{scope, decision}, …]`) can represent duplicates; the OpenCode *serializer* must also
+     refuse to emit them since the map format cannot round-trip them.
+3. **"No rule matched" must map to tool-default in the IR**, not a hardcoded `allow` — the
+   resolver's own fallback is `ask`, with `allow` supplied by a preceding global baseline rule.
+
+## 9. Acceptance criteria coverage
+
+| Criterion | Status |
+|---|---|
+| Broad-first/specific-after agent exercised in a real install; behavior recorded | ✅ Probe 1 (§3) |
+| Second probe distinguishing last-match from most-specific | ✅ Probe 2 (§4) — distinguishing case was expressible |
+| Verdict stated; reordering transform if refuted | ✅ Confirmed (§8); no transform/migration needed |
+| Integer-like ("8080") and duplicate-scope hazards probed or deferred | ✅ Both probed (§5, §6); both confirmed real |
+| Full transcript committed to the spec directory | ✅ This document |