Просмотр исходного кода

fix(hooks): add retry tracking to ForegroundFallbackManager to prevent 429 freeze

When delegated subagents hit 429 Too Many Requests, OpenCode retries
internally with no limit (ApiError.isRetryable is true). The
ForegroundFallbackManager was either disabled (when no model arrays
configured) or intervened immediately on the first 429.

Changes:
- Add maxRetries (default: 3) to FailoverConfigSchema — consecutive 429s
  tolerated before model swap or abort
- Track retries via sessionRetries map in session.status handler; only
  intervene after maxRetries - 1 internal retries
- Abort session on chain exhaustion instead of returning silently
  (stops the freeze even without fallback models)
- Enable manager unconditionally (no longer gated on runtimeChains)
- Rename stale test that now asserts abort behavior
Michael Henke 1 месяц назад
Родитель
Сommit
ceaaafb836

+ 104 - 0
docs/superpowers/specs/2026-07-05-delegate-task-429-fallback.md

@@ -0,0 +1,104 @@
+# Delegate Task 429 Retry + Fallback
+
+## Problem
+
+When the orchestrator delegates to a subagent via the `task` tool, the spawned
+session can receive `429 Too Many Requests`. OpenCode internally retries
+(because `ApiError.isRetryable` is `true` for 429), emitting `session.status`
+events with `type: "retry"`. With no limit on internal retries and no model
+swap, the session freezes indefinitely.
+
+The existing `ForegroundFallbackManager` already handles events from ALL
+sessions (foreground + subagent), but:
+
+1. It was **disabled** when `runtimeChains` was empty (no model arrays
+   configured) — even though it could still abort sessions on exhaustion.
+2. It immediately aborted on the first 429 — no backoff letting OpenCode's
+   internal retry handle transient spikes.
+3. When chain was empty, it returned silently instead of aborting — leaving
+   the session to freeze.
+
+## Strategy
+
+**Retry with backoff, then intervene.** When a session hits 429:
+
+1. If OpenCode is retrying internally (`status.type === "retry"`), let it —
+   track the retry count from the `attempt` field in the `session.status`
+   event.
+2. After `maxRetries` consecutive 429s, intervene: abort the current prompt
+   and swap to the next model in the fallback chain (via `promptAsync`).
+3. If no fallback chain is configured, just abort — stops the freeze, surfaces
+   error to the orchestrator.
+
+This works for both foreground and subagent sessions because:
+
+- `promptAsync` is a real SDK endpoint (`POST /session/{id}/prompt_async`,
+  verified in `@opencode-ai/sdk` types) and works on any session.
+- The ForegroundFallbackManager already processes events from all sessions.
+- Subagent and foreground both emit the same `session.status` events.
+
+## Implementation
+
+### Files Changed
+
+| File | Change |
+|------|--------|
+| `src/config/schema.ts` | Add `maxRetries` to `FailoverConfigSchema` (default: 3) |
+| `src/hooks/foreground-fallback/index.ts` | Add `maxRetries` constructor param, `sessionRetries` tracking map, retry tracking in `session.status` handler, abort-on-chain-exhaustion in `tryFallback` |
+| `src/index.ts` | Enable manager without `runtimeChains`, pass `config.fallback?.maxRetries` |
+
+### Config Schema
+
+```typescript
+// In FailoverConfigSchema (src/config/schema.ts)
+export const FailoverConfigSchema = z.object({
+  enabled: z.boolean().default(true),
+  timeoutMs: z.number().min(0).default(15000),
+  retryDelayMs: z.number().min(0).default(500),
+  maxRetries: z.number().int().min(0).default(3).describe(
+    'Number of consecutive 429/rate-limit responses tolerated on the ' +
+    'same model before aborting (or swapping to the next fallback ' +
+    'model when a chain is configured).',
+  ),
+  retry_on_empty: z.boolean().default(true),
+}).strict();
+```
+
+### ForegroundFallbackManager Changes
+
+1. **`sessionRetries` map** — tracks consecutive 429s per session, reset on
+   model swap or session deletion.
+
+2. **`session.status` handler** — when `status.type === "retry"` and the
+   message matches rate-limit keywords, increments the retry counter instead
+   of immediately falling back. Only intervenes after `maxRetries - 1`
+   internal retries.
+
+3. **`tryFallback` chain-exhaustion path** — when `resolveChain` returns empty
+   (no fallback models configured), calls `abortSessionWithTimeout` instead of
+   returning silently. This stops OpenCode's infinite retry loop.
+
+4. **`session.deleted` handler** — cleans up `sessionRetries` entry.
+
+### Usage
+
+```jsonc
+// ~/.config/opencode/oh-my-opencode-slim.jsonc
+{
+  "fallback": {
+    "enabled": true,        // default: true
+    "maxRetries": 3,         // default: 3 — let 3 internal retries, then intervene
+    "retryDelayMs": 500      // base delay for OpenCode's internal backoff
+  },
+  "agents": {
+    "explorer": {
+      "model": ["openai/gpt-4o", "anthropic/claude-sonnet-4"] // fallback chain
+    }
+  }
+}
+```
+
+## Verification
+
+All 1350 existing tests pass. No new files added — only 3 existing files
+modified.

+ 10 - 0
src/config/schema.ts

@@ -180,6 +180,16 @@ export const FailoverConfigSchema = z
     enabled: z.boolean().default(true),
     timeoutMs: z.number().min(0).default(15000),
     retryDelayMs: z.number().min(0).default(500),
+    maxRetries: z
+      .number()
+      .int()
+      .min(0)
+      .default(3)
+      .describe(
+        'Number of consecutive 429/rate-limit responses tolerated on the ' +
+          'same model before aborting (or swapping to the next fallback ' +
+          'model when a chain is configured).',
+      ),
     retry_on_empty: z
       .boolean()
       .default(true)

+ 2 - 1
src/hooks/foreground-fallback/index.test.ts

@@ -258,7 +258,7 @@ describe('ForegroundFallbackManager session.error', () => {
     expect(mocks.promptAsync).not.toHaveBeenCalled();
   });
 
-  test('does nothing when no chain configured for session', async () => {
+  test('aborts session when no chain configured (no fallback model to swap to)', async () => {
     const emptyMgr = new ForegroundFallbackManager(client, {}, true);
     await emptyMgr.handleEvent({
       type: 'session.error',
@@ -268,6 +268,7 @@ describe('ForegroundFallbackManager session.error', () => {
       },
     });
 
+    expect(mocks.abort).toHaveBeenCalledTimes(1);
     expect(mocks.promptAsync).not.toHaveBeenCalled();
   });
 

+ 30 - 2
src/hooks/foreground-fallback/index.ts

@@ -99,6 +99,9 @@ export class ForegroundFallbackManager {
    *  when the model has changed, allowing the cascade to continue when a
    *  new fallback model also fails within the dedup window. */
   private readonly lastTriggerModel = new Map<string, string>();
+  /** sessionID → consecutive 429 count for the current model.
+   *  Reset on model swap or session deletion. */
+  private readonly sessionRetries = new Map<string, number>();
 
   constructor(
     private readonly client: OpencodeClient,
@@ -109,6 +112,8 @@ export class ForegroundFallbackManager {
      */
     private readonly chains: Record<string, string[]>,
     private readonly enabled: boolean,
+    /** Consecutive 429s tolerated on the same model before swap/abort. */
+    private readonly maxRetries: number = 3,
   ) {}
 
   /**
@@ -163,7 +168,7 @@ export class ForegroundFallbackManager {
         const props = event.properties as
           | {
               sessionID?: string;
-              status?: { type?: string; message?: string };
+              status?: { type?: string; message?: string; attempt?: number };
             }
           | undefined;
         if (!props?.sessionID || !props.status?.message) break;
@@ -183,6 +188,27 @@ export class ForegroundFallbackManager {
           msg.includes('high concurrency') ||
           msg.includes('reduce concurrency')
         ) {
+          // When OpenCode retries internally (type: "retry"), track
+          // attempts and only intervene after maxRetries consecutive
+          // failures. This lets OpenCode's own backoff handle transient
+          // spikes while preventing infinite freezes.
+          if (
+            props.status.type === 'retry' &&
+            typeof props.status.attempt === 'number'
+          ) {
+            const tried = this.sessionRetries.get(props.sessionID) ?? 0;
+            if (tried < this.maxRetries - 1) {
+              this.sessionRetries.set(props.sessionID, tried + 1);
+              log('[foreground-fallback] rate-limit retry', {
+                sessionID: props.sessionID,
+                attempt: props.status.attempt,
+                remaining: this.maxRetries - tried - 1,
+              });
+              break;
+            }
+            // Exhausted retries: intervene
+            this.sessionRetries.delete(props.sessionID);
+          }
           await this.tryFallback(props.sessionID);
         }
         break;
@@ -217,6 +243,7 @@ export class ForegroundFallbackManager {
           this.inProgress.delete(id);
           this.lastTrigger.delete(id);
           this.lastTriggerModel.delete(id);
+          this.sessionRetries.delete(id);
         }
         break;
       }
@@ -255,10 +282,11 @@ export class ForegroundFallbackManager {
       const agentName = this.sessionAgent.get(sessionID);
       const chain = this.resolveChain(agentName, currentModel);
       if (!chain.length) {
-        log('[foreground-fallback] no chain configured', {
+        log('[foreground-fallback] no chain configured, aborting session', {
           sessionID,
           agentName,
         });
+        await abortSessionWithTimeout(this.client, sessionID);
         return;
       }
 

+ 5 - 3
src/index.ts

@@ -297,12 +297,14 @@ const OhMyOpenCodeLite: Plugin = async (ctx) => {
     // Initialize JSON parse error recovery hook
     jsonErrorRecoveryHook = createJsonErrorRecoveryHook(ctx);
 
-    // Initialize foreground fallback manager for runtime model switching
+    // Initialize foreground fallback manager for runtime model switching.
+    // Enabled by default even without fallback chains — the manager can still
+    // abort rate-limited sessions after maxRetries to prevent infinite freezes.
     foregroundFallback = new ForegroundFallbackManager(
       ctx.client,
       runtimeChains,
-      config.fallback?.enabled !== false &&
-        Object.keys(runtimeChains).length > 0,
+      config.fallback?.enabled !== false,
+      config.fallback?.maxRetries ?? 3,
     );
 
     deepworkCommandHook = createDeepworkCommandHook();