Browse Source

fix(council): restore uncertainty disclosure + harden synthesis, fix stale refs

- Add Remaining Uncertainty bullet to Council Summary (regression from
  flatten: old format asked for uncertainty, new dropped it for
  Recommended Action).
- Harden council system prompt: append non-overridable synthesis
  reinforcement so a customPrompt override cannot drop format instructions.
- Council Mode handoff now includes per-councillor model labels and
  explicit failed/timed-out status instead of silent omission.
- Fix stale 'Councillor Details' -> 'Per-Councillor Details' in
  orchestrator agent description and docs/council.md.
Michael Henke 3 weeks ago
parent
commit
66d2f68ffb
4 changed files with 9 additions and 8 deletions
  1. 1 1
      docs/council.md
  2. 6 5
      src/agents/council.ts
  3. 1 1
      src/agents/index.ts
  4. 1 1
      src/agents/orchestrator.ts

+ 1 - 1
docs/council.md

@@ -323,7 +323,7 @@ responded (or failed), the council agent synthesizes their results.
 Council responses include:
 
 1. **Council Response** — the synthesized final answer.
-2. **Councillor Details** — each responding councillor's individual response,
+2. **Per-Councillor Details** — each responding councillor's individual response,
    using the councillor names from the configured preset.
 3. **Council Summary** — agreement, disagreement resolution, remaining
    uncertainty, and a consensus confidence rating of `unanimous`, `majority`,

+ 6 - 5
src/agents/council.ts

@@ -5,6 +5,8 @@ import { createReadOnlyAgentPermission } from './permissions';
 // NOTE: Councillor system prompts live in the councillor agent factory.
 // The council agent synthesizes councillor responses passed by the orchestrator.
 
+const COUNCIL_SYNTHESIS_REINFORCEMENT = `\n\n---\n\nYou MUST follow the Synthesis Process steps before producing output: review each councillor response individually by name, then produce the required output with a synthesized Council Response, a Per-Councillor Details section using each councillor's exact name, and a Council Summary with Consensus Level (unanimous|majority|split), Agreed Points, Disagreements + resolution, Remaining Uncertainty, and Recommended Action.`;
+
 const COUNCIL_AGENT_PROMPT = `You are the Council agent - a \
 synthesizer for multi-model consensus.
 
@@ -47,6 +49,7 @@ For each councillor, show:
 - **Consensus Level**: unanimous | majority | split (pick one)
 - **Agreed Points**: what all councillors agreed on
 - **Disagreements**: where councillors differed and your resolution
+- **Remaining Uncertainty**: any caveats, untested assumptions, or open questions the council could not fully resolve
 - **Recommended Action**: what to do next`;
 
 /**
@@ -59,11 +62,9 @@ export function createCouncilAgent(
   customPrompt?: string,
   customAppendPrompt?: string,
 ): AgentDefinition {
-  const prompt = resolvePrompt(
-    COUNCIL_AGENT_PROMPT,
-    customPrompt,
-    customAppendPrompt,
-  );
+  const prompt =
+    resolvePrompt(COUNCIL_AGENT_PROMPT, customPrompt, customAppendPrompt) +
+    COUNCIL_SYNTHESIS_REINFORCEMENT;
 
   return {
     name: 'council',

+ 1 - 1
src/agents/index.ts

@@ -662,7 +662,7 @@ export function createAgents(
           `   - task(subagent_type='${a.name}', description='Councillor ${getCouncillorSeatName(a.name)} on <brief topic>', prompt=<user's question>)`,
       )
       .join('\n');
-    updatedPrompt = `${updatedPrompt}\n\n## Council Mode\n\nWhen you need to run a council or the user asks for consensus/multiple opinions, use this procedure INSTEAD of delegating to @council:\n\n1. Dispatch the user's question to each councillor in PARALLEL via task():\n${dispatchList}\n2. Collect ALL councillor responses. If any councillor returns empty or does not respond within 3 minutes, proceed without it — do not wait indefinitely. If a councillor's response is empty, retry that councillor once before continuing.\n3. Call task(subagent_type='council', description='Synthesize council report') with a prompt that includes the original user question AND all councillor responses, formatted so each councillor's seat name and response is clearly separated. Skip any councillor that still returned empty after retry.\n4. Present the council's synthesized report.\n\nThis ensures each councillor runs with its own model and the council agent synthesizes the full multi-model consensus.`;
+    updatedPrompt = `${updatedPrompt}\n\n## Council Mode\n\nWhen you need to run a council or the user asks for consensus/multiple opinions, use this procedure INSTEAD of delegating to @council:\n\n1. Dispatch the user's question to each councillor in PARALLEL via task():\n${dispatchList}\n2. Collect ALL councillor responses. If any councillor returns empty or does not respond within 3 minutes, proceed without it — do not wait indefinitely. If a councillor's response is empty, retry that councillor once before continuing.\n3. Call task(subagent_type='council', description='Synthesize council report') with a prompt that includes the original user question AND all councillor responses. For each councillor, label its response with its seat name AND its model (e.g. "alpha (gpt-5.6-luna)"). Format each councillor's seat name and response clearly separated. If a councillor failed or timed out, include that status explicitly (e.g. "beta (gemini-3-pro): FAILED/TIMED OUT") instead of omitting it. Skip only councillors that returned empty after one retry.\n4. Present the council's synthesized report.\n\nThis ensures each councillor runs with its own model and the council agent synthesizes the full multi-model consensus.`;
   }
 
   orchestrator.config.prompt = updatedPrompt;

+ 1 - 1
src/agents/orchestrator.ts

@@ -87,7 +87,7 @@ const AGENT_DESCRIPTIONS: Record<string, string> = {
 - **Delegate when:** Critical decisions need multiple independent perspectives • High-stakes architectural/security/data-integrity choices • Ambiguous problems where disagreement is useful signal • You want confidence beyond a single model • The user explicitly asks for council/consensus/multiple opinions.
 - **Don't delegate when:** Straightforward tasks you're confident about • Speed matters more than confidence • Routine implementation/debugging • A single specialist is clearly the right tool • You only need current docs/search/code review rather than multi-model consensus.
 - **How to call:** Send the full question/task and relevant context. Be explicit about what decision, trade-off, or answer the council should resolve. Do not ask council to do routine code edits.
-- **Result handling:** Council returns a structured response that may include: synthesized Council Response, individual Councillor Details, and Council Summary/confidence. Preserve that structure when the user asked for council output. Do not pretend the council only returned a final answer. If you need to act on the council result, first briefly state the council's recommendation, then proceed.
+- **Result handling:** Council returns a structured response that may include: synthesized Council Response, individual Per-Councillor Details, and Council Summary/confidence. Preserve that structure when the user asked for council output. Do not pretend the council only returned a final answer. If you need to act on the council result, first briefly state the council's recommendation, then proceed.
 - **Rule of thumb:** Need second/third opinions from different models? → @council. Need one expert lane? → use the specialist. Need final synthesis? → handle directly.`,
 
   observer: `@observer