Browse Source

feat(evals): improve delegation testing and behavior validation

Major improvements to eval framework:

Test Suite:
- Add 00-smoke-test.yaml (basic read operation)
- Rename 01-smoke-test.yaml โ†’ 02-delegation-test.yaml (delegation test)
- Add simple-responder test agent for delegation testing
- Add debug test: simple-subagent-call.yaml

Evaluators:
- Add agent-model-evaluator.ts for agent/model validation
- Enhance behavior-evaluator.ts with detailed task tool output
- Improve delegation-evaluator.ts with better evidence tracking

Schema & Execution:
- Add expectedAgent and expectedModel to test schema
- Improve test executor with better logging and error handling

Documentation:
- Update evals/README.md with new features and performance improvements
- Remove outdated MULTI_AGENT_LOGGING_COMPLETE.md

Registry:
- Add simple-responder test agent to registry

Fixes:
- Fix dashboard.sh path resolution

These changes provide better visibility into delegation, improved test
coverage, and clearer validation of agent behavior.
darrenhinde 7 months ago
parent
commit
ab1c8ae801

+ 44 - 0
.opencode/agent/subagents/test/simple-responder.md

@@ -0,0 +1,44 @@
+---
+# OpenCode Agent Configuration
+id: simple-responder
+name: Simple Responder
+description: "Test agent that responds with 'AWESOME TESTING' - for eval framework testing"
+category: test
+type: utility
+version: 1.0.0
+author: opencode
+mode: subagent
+temperature: 0.0
+tools:
+  read: false
+  write: false
+  edit: false
+  grep: false
+  glob: false
+  bash: false
+  task: false
+  patch: false
+---
+
+# Simple Responder - Test Agent
+
+You are a simple test agent designed to validate the eval framework.
+
+## Your ONLY Job
+
+When called, respond with exactly:
+
+```
+AWESOME TESTING DARREN
+```
+
+That's it. No explanations, no tool calls, no additional text. Just those two words.
+
+## Rules
+
+1. **DO NOT** use any tools
+2. **DO NOT** ask questions
+3. **DO NOT** provide explanations
+4. **ONLY** respond with "AWESOME TESTING"
+
+This agent exists purely for testing the eval framework's ability to track subagent calls.

+ 0 - 412
evals/MULTI_AGENT_LOGGING_COMPLETE.md

@@ -1,412 +0,0 @@
-# ๐ŸŽ‰ Multi-Agent Logging System - COMPLETE!
-
-**Status**: โœ… All 3 Days Implemented  
-**Date**: 2025-12-17  
-**Version**: 1.0.0  
-**Implementation Time**: ~4 hours
-
----
-
-## ๐ŸŽฏ Mission Accomplished
-
-We successfully built and integrated a complete multi-agent logging system that provides **full visibility** into delegation hierarchies during eval test execution.
-
-### What We Solved
-
-**THE PROBLEM:**
-- โŒ Zero visibility into subagent execution
-- โŒ Couldn't see what child sessions actually did
-- โŒ Couldn't verify subagent responses
-- โŒ Logs were confusing and hard to follow
-
-**THE SOLUTION:**
-- โœ… Beautiful hierarchical logging with visual indentation
-- โœ… Complete parent and child session tracking
-- โœ… Real-time message capture at all levels
-- โœ… Clear delegation chain visualization
-- โœ… Session duration tracking
-
----
-
-## ๐Ÿ“Š What You See Now
-
-When you run an eval test with `--debug`, you see:
-
-```
-โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”
-โ”‚ ๐ŸŽฏ PARENT: OpenAgent (ses_4d208f96...)                     โ”‚
-โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜
-  ๐Ÿค– Agent: Call the simple-responder subagent...
-  ๐Ÿค– Agent: ## Proposed Plan...
-  
-  ๐Ÿ”ง TOOL: read
-  ๐Ÿ”ง TOOL: bash
-  ๐Ÿ”ง TOOL: write
-  
-  ๐Ÿ”ง TOOL: task
-     โ”œโ”€ subagent: simple-responder
-     โ”œโ”€ prompt: Respond with AWESOME TESTING
-     โ””โ”€ Creating child session...
-  
-  โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”
-  โ”‚ ๐ŸŽฏ CHILD: OpenAgent (ses_4d208a78...)                      โ”‚
-  โ”‚    Parent: ses_4d208f96...                                 โ”‚
-  โ”‚    Depth: 1                                                โ”‚
-  โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜
-    ๐Ÿค– Agent: Load context from .tmp/sessions/...
-    ๐Ÿค– Agent: AWESOME TESTING DARREN
-  โœ… CHILD COMPLETE (15.6s)
-  
-  ๐Ÿค– Agent: ## Summary...
-  
-โœ… PARENT COMPLETE (38.9s)
-```
-
----
-
-## ๐Ÿ—๏ธ What We Built
-
-### Day 1: Core Infrastructure โœ…
-
-**Files Created:**
-- `src/logging/types.ts` - TypeScript interfaces
-- `src/logging/session-tracker.ts` - Hierarchy tracking
-- `src/logging/logger.ts` - Pretty-printing logger
-- `src/logging/formatters.ts` - Visual formatting
-- `src/logging/index.ts` - Module exports
-- `src/logging/__tests__/` - 37 unit tests
-
-**Features:**
-- Session hierarchy tracking (parent โ†’ child โ†’ grandchild)
-- Delegation event recording
-- Visual formatting with boxes and emojis
-- Tree analysis and statistics
-
-### Day 2: Event Stream Integration โœ…
-
-**Files Modified:**
-- `src/sdk/event-stream-handler.ts` - Added logging hooks
-- `src/sdk/test-runner.ts` - Initialize logger in debug mode
-
-**Features:**
-- Hook into session.created events
-- Hook into message.updated events
-- Hook into message.part.updated events
-- Detect task tool calls (delegations)
-- Initialize logger automatically in debug mode
-
-### Day 3: Child Session Capture โœ…
-
-**Enhancements:**
-- Multi-session tracking (parent + all children)
-- Child session detection via timestamp heuristics
-- Message capture from text parts
-- Automatic parent-child linking
-- Message deduplication for cleaner output
-- Session completion tracking with durations
-
----
-
-## ๐ŸŽฏ How to Use It
-
-### Run Any Eval Test with Logging
-
-```bash
-cd evals/framework
-
-# Run with debug mode to enable logging
-npm run eval:sdk -- \
-  --agent=openagent \
-  --pattern="**/debug/simple-subagent-call.yaml" \
-  --debug
-```
-
-### Filter for Clean Output
-
-```bash
-# Show only the hierarchical logging
-npm run eval:sdk -- \
-  --agent=openagent \
-  --pattern="**/debug/*.yaml" \
-  --debug 2>&1 | grep -E "(โ”Œ|โ”‚|โ””|๐ŸŽฏ|๐Ÿ“|๐Ÿค–|๐Ÿ”ง|โœ…)"
-```
-
-### What Gets Logged
-
-**Session Events:**
-- โœ… Session creation (parent and child)
-- โœ… Session hierarchy (depth, parent links)
-- โœ… Session completion (with duration)
-
-**Messages:**
-- โœ… User messages
-- โœ… Assistant messages
-- โœ… Incremental updates (deduplicated)
-
-**Tool Calls:**
-- โœ… read, write, bash, etc.
-- โœ… task tool (delegation) with special formatting
-
-**Delegation:**
-- โœ… Delegation event visualization
-- โœ… Child session linking
-- โœ… Prompt display
-
----
-
-## ๐Ÿ“ˆ Test Results
-
-### Unit Tests
-- โœ… 37/37 tests passing
-- โœ… SessionTracker: 16 tests
-- โœ… MultiAgentLogger: 18 tests
-- โœ… Integration: 3 tests
-
-### Integration Tests
-- โœ… Simple delegation (parent โ†’ child)
-- โœ… Nested delegation (3 levels)
-- โœ… Parallel delegation (2 children)
-- โœ… Real eval test (simple-subagent-call.yaml)
-
-### Performance
-- โœ… Minimal overhead (<1% impact)
-- โœ… Only enabled in debug mode
-- โœ… No impact on production tests
-
----
-
-## ๐Ÿ”ง Technical Implementation
-
-### Architecture
-
-```
-EventStreamHandler (SDK events)
-    โ†“
-handleMultiAgentLogging()
-    โ†“
-MultiAgentLogger
-    โ†“
-SessionTracker (hierarchy)
-    โ†“
-Formatters (visual output)
-    โ†“
-Console (beautiful logs)
-```
-
-### Key Design Decisions
-
-**1. Event-Driven Architecture**
-- Hook into SDK event stream
-- Process events in real-time
-- No polling or delays
-
-**2. Multi-Session Tracking**
-- Track all active sessions simultaneously
-- Use Set for O(1) lookups
-- Timestamp-based child detection
-
-**3. Message Deduplication**
-- Track last logged text per session
-- Only log significant updates
-- Reduces noise from incremental updates
-
-**4. Lazy Loading**
-- Logger only created in debug mode
-- Zero overhead in production
-- Optional feature, not required
-
----
-
-## ๐ŸŽจ Visual Design
-
-### Session Headers
-
-```
-โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”
-โ”‚ ๐ŸŽฏ PARENT: OpenAgent (ses_abc123...)                       โ”‚
-โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜
-```
-
-```
-  โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”
-  โ”‚ ๐ŸŽฏ CHILD: simple-responder (ses_xyz789...)                 โ”‚
-  โ”‚    Parent: ses_abc123...                                   โ”‚
-  โ”‚    Depth: 1                                                โ”‚
-  โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜
-```
-
-### Indentation
-
-- Parent (depth 0): No indentation
-- Child (depth 1): 2 spaces
-- Grandchild (depth 2): 4 spaces
-- And so on...
-
-### Emojis
-
-- ๐ŸŽฏ Session header
-- ๐Ÿ“ User message
-- ๐Ÿค– Assistant message
-- ๐Ÿ”ง Tool call
-- โœ… Session complete
-
----
-
-## ๐Ÿ“ Code Quality
-
-### Standards Applied
-
-- โœ… Modular, functional code patterns
-- โœ… Pure functions with immutability
-- โœ… TypeScript strict mode
-- โœ… Comprehensive error handling
-- โœ… Clear, self-documenting code
-- โœ… 100% test coverage for core logic
-
-### Testing Strategy
-
-- Unit tests for SessionTracker
-- Unit tests for MultiAgentLogger
-- Integration tests for full scenarios
-- Real-world validation with eval tests
-
----
-
-## ๐Ÿš€ Future Enhancements (Optional)
-
-### Potential Improvements
-
-1. **Log Export**
-   - Save logs to JSON file
-   - Enable post-test analysis
-   - Build log viewer dashboard
-
-2. **Advanced Filtering**
-   - Filter by agent type
-   - Filter by depth level
-   - Filter by time range
-
-3. **Performance Metrics**
-   - Track token usage per session
-   - Calculate cost per delegation
-   - Measure latency per tool call
-
-4. **Visualization**
-   - HTML dashboard for logs
-   - Interactive hierarchy viewer
-   - Timeline visualization
-
-5. **Search**
-   - Search across all sessions
-   - Find specific messages
-   - Regex pattern matching
-
----
-
-## ๐Ÿ“š Documentation
-
-### Files Created/Updated
-
-**Core Implementation:**
-- `src/logging/` - Complete logging module (5 files)
-- `src/logging/__tests__/` - Test suite (3 files)
-- `scripts/demo-logging.ts` - Demo script
-
-**Integration:**
-- `src/sdk/event-stream-handler.ts` - Event hooks
-- `src/sdk/test-runner.ts` - Logger initialization
-
-**Documentation:**
-- `src/logging/README.md` - Module documentation
-- `MULTI_AGENT_LOGGING_COMPLETE.md` - This file
-
-**Task Tracking:**
-- `tasks/eval/december/01-multi-agent-logging-system.md` - Original spec
-- `tasks/eval/december/CHECKLIST.md` - Implementation checklist
-- `evals/EVAL_SYSTEM_ANALYSIS.md` - Problem analysis
-- `evals/ACTION_PLAN.md` - Broader roadmap
-
----
-
-## โœ… Success Criteria Met
-
-- [x] Can see subagent name when delegation happens
-- [x] Can see subagent's actual messages
-- [x] Can follow delegation chain visually
-- [x] Can verify subagent responses
-- [x] Logs are easy to read and understand
-- [x] Works with nested delegation (3+ levels)
-- [x] Works with parallel delegation (2+ children)
-- [x] Performance overhead <5%
-- [x] Only enabled in debug mode
-- [x] No impact on production tests
-
----
-
-## ๐ŸŽ“ Lessons Learned
-
-### What Worked Well
-
-1. **Incremental Development**
-   - Day 1: Core infrastructure (standalone)
-   - Day 2: Integration (hook into events)
-   - Day 3: Enhancement (child session capture)
-
-2. **Test-Driven Approach**
-   - 37 unit tests before integration
-   - Caught issues early
-   - Confident in core logic
-
-3. **Event-Driven Architecture**
-   - Clean separation of concerns
-   - Easy to extend
-   - No polling overhead
-
-### Challenges Overcome
-
-1. **SDK Event Structure**
-   - Properties nested in `info` and `part` objects
-   - Required careful event inspection
-   - Solution: Flexible property access
-
-2. **Message Deduplication**
-   - Text parts come through incrementally
-   - Too many duplicate logs
-   - Solution: Track last logged text, filter updates
-
-3. **Child Session Detection**
-   - No explicit parent-child link in events
-   - Had to use timestamp heuristics
-   - Solution: Track delegations, match by timing
-
----
-
-## ๐ŸŽ‰ Final Status
-
-**COMPLETE AND WORKING!**
-
-The multi-agent logging system is fully implemented, tested, and integrated into the eval framework. It provides complete visibility into delegation hierarchies and makes debugging multi-agent scenarios dramatically easier.
-
-### Quick Start
-
-```bash
-# Run any eval test with debug mode
-cd evals/framework
-npm run eval:sdk -- --agent=openagent --pattern="**/debug/*.yaml" --debug
-
-# Look for the beautiful hierarchical output!
-```
-
-### Where to See Logs
-
-- **Console output** during test run
-- Look for box characters: โ”Œโ”€โ””โ”‚
-- Look for emojis: ๐ŸŽฏ๐Ÿ“๐Ÿค–๐Ÿ”งโœ…
-- Indentation shows hierarchy
-
----
-
-**Created**: 2025-12-17  
-**Author**: OpenAgent + Repo Manager  
-**Status**: โœ… COMPLETE  
-**Next Steps**: Use it! Debug multi-agent scenarios with confidence!

+ 30 - 3
evals/README.md

@@ -7,23 +7,50 @@ Test and validate agent behavior with automated evaluations.
 ```bash
 cd evals/framework
 
-# Run golden tests (baseline - 8 tests, ~3-5 min)
+# Run golden tests (baseline - 8 tests, ~2-3 min)
 npm run eval:sdk -- --agent=openagent --pattern="**/golden/*.yaml"
 
 # Run a specific test
 npm run eval:sdk -- --agent=openagent --pattern="**/smoke-test.yaml"
 
-# Run with debug output
+# Run with debug output (includes multi-agent logging)
 npm run eval:sdk -- --agent=openagent --pattern="**/golden/*.yaml" --debug
 ```
 
+## โœจ New Features
+
+### Multi-Agent Logging (Dec 2025)
+Beautiful hierarchical logging shows parent-child delegation chains:
+
+```
+โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”
+โ”‚ ๐ŸŽฏ PARENT: OpenAgent (ses_xxx...)                          โ”‚
+โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜
+  
+  โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”
+  โ”‚ ๐ŸŽฏ CHILD: simple-responder (ses_yyy...)                    โ”‚
+  โ”‚    Parent: ses_xxx...                                      โ”‚
+  โ”‚    Depth: 1                                                โ”‚
+  โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜
+  
+  โœ… CHILD COMPLETE (2.9s)
+โœ… PARENT COMPLETE (20.9s)
+```
+
+Enable with `--debug` flag. See [MULTI_AGENT_LOGGING_COMPLETE.md](MULTI_AGENT_LOGGING_COMPLETE.md) for details.
+
+### Performance Improvements (Dec 2025)
+- **10-20% faster tests** - Grace period reduced from 5s to 2s
+- **Performance metrics** - Automatic collection of tool latencies, inference time
+- **37 unit tests** - Complete test coverage for logging system
+
 ## Golden Tests
 
 8 curated tests that validate core agent behaviors:
 
 | Test | What It Validates |
 |------|-------------------|
-| 01-smoke-test | Basic read operation works |
+| 01-smoke-test | **Agent & subagent delegation** (multi-agent) |
 | 02-context-loading | Agent reads context before answering |
 | 03-read-before-write | Agent inspects before modifying |
 | 04-write-with-approval | Agent asks before writing |

+ 49 - 0
evals/agents/shared/tests/debug/simple-subagent-call.yaml

@@ -0,0 +1,49 @@
+id: debug-simple-subagent-call
+name: "Debug: Simple Subagent Call"
+description: |
+  Minimal test to validate subagent delegation tracking.
+  
+  Tests:
+  1. Parent agent (openagent) delegates to simple-responder
+  2. Simple-responder returns "AWESOME TESTING"
+  3. Eval framework captures both sessions
+  
+  This is a diagnostic test to understand:
+  - How long delegation takes
+  - What logs are generated
+  - Whether child sessions are tracked
+  - Where bottlenecks exist
+category: developer
+
+prompts:
+  - text: |
+      Call the simple-responder subagent and ask it to respond. Use the task tool to delegate.
+  
+  - text: |
+      Yes, proceed with the delegation.
+    delayMs: 2000
+
+approvalStrategy:
+  type: auto-approve
+
+behavior:
+  mustUseTools: [task]
+  minToolCalls: 2
+  maxToolCalls: 5
+
+expectedViolations:
+  - rule: approval-gate
+    shouldViolate: false
+    severity: error
+  - rule: delegation
+    shouldViolate: false
+    severity: error
+
+timeout: 90000
+
+tags:
+  - debug
+  - subagent
+  - delegation
+  - simple
+  - multi-turn

+ 59 - 0
evals/agents/shared/tests/golden/00-smoke-test.yaml

@@ -0,0 +1,59 @@
+id: golden-00-smoke-test
+name: "Golden 00: Minimal Smoke Test - Basic Response"
+description: |
+  Absolute bare minimum smoke test - if this fails, something is fundamentally broken.
+  
+  This is the simplest possible test:
+  - Single prompt
+  - Single response
+  - No tool calls required
+  - No delegation
+  - No file operations
+  
+  Expected completion time: 8-12 seconds (mostly LLM inference)
+  
+  This test validates:
+  - Agent can receive a prompt
+  - Agent can generate a response
+  - Test infrastructure works
+  
+category: developer
+
+prompts:
+  - text: |
+      Hello! Please respond with exactly: "Test infrastructure is working correctly."
+      
+      Do not use any tools. Just respond with that exact text.
+
+approvalStrategy:
+  type: auto-approve
+
+behavior:
+  mustNotUseTools: [bash, write, edit, task, read, glob, grep, list]
+  minToolCalls: 0
+  maxToolCalls: 0
+  requiresApproval: false
+  expectedAgent: "openagent"
+  expectedModel: "opencode/grok-code"
+  expectedResponse:
+    contains:
+      - "Test infrastructure is working correctly"
+    description: "Agent should respond with exact confirmation text"
+
+expectedViolations:
+  - rule: approval-gate
+    shouldViolate: false
+    severity: error
+    description: "No approval needed for simple response"
+  - rule: tool-usage
+    shouldViolate: false
+    severity: error
+    description: "Should not use any tools"
+
+timeout: 30000  # 30 seconds should be plenty for a simple response
+
+tags:
+  - golden
+  - smoke-test
+  - minimal
+  - fast

+ 15 - 14
evals/agents/shared/tests/golden/01-smoke-test.yaml → evals/agents/shared/tests/golden/02-delegation-test.yaml

@@ -1,22 +1,21 @@
-id: golden-01-smoke-test
-name: "Golden 01: Smoke Test - Agent & Subagent Delegation"
+id: golden-02-delegation-test
+name: "Golden 02: Delegation Test - Parent & Child Agent"
 description: |
-  Bare minimum smoke test validating core agent functionality.
+  Tests multi-agent delegation workflow.
   
-  This is the simplest possible test - if this fails, something
-  is fundamentally broken with the agent or test infrastructure.
-  
-  Validates:
-  - Agent can receive and understand a prompt
+  This test validates:
   - Agent can delegate to a subagent using task tool
-  - Subagent responds correctly
+  - Subagent receives context and responds correctly
   - Multi-agent logging captures both sessions
+  - Parent-child communication works
+  
+  Expected completion time: 25-35 seconds (includes LLM inference for both agents)
   
-  This replaces the basic read test as our smoke test because:
-  - Delegation is the core feature of the agent system
-  - Tests both parent and child agent functionality
+  This is more complex than the minimal smoke test because:
+  - Tests delegation (core feature of agent system)
+  - Runs TWO agents (parent + child)
+  - Tests context passing
   - Validates multi-agent logging system
-  - More representative of real-world usage
 category: developer
 
 prompts:
@@ -35,6 +34,8 @@ behavior:
   minToolCalls: 1
   maxToolCalls: 10
   requiresApproval: false  # This is a simple test, no approval ceremony needed
+  expectedAgent: "openagent"  # Validate correct agent is loaded
+  expectedModel: "opencode/grok-code"  # Validate correct model is used
   expectedResponse:
     contains:
       - "AWESOME TESTING"
@@ -58,7 +59,7 @@ timeout: 90000
 
 tags:
   - golden
-  - smoke-test
   - delegation
   - subagent
   - multi-agent
+  - parent-child

+ 215 - 0
evals/framework/src/evaluators/agent-model-evaluator.ts

@@ -0,0 +1,215 @@
+/**
+ * AgentModelEvaluator - Logs agent and model information for test transparency
+ * 
+ * This evaluator provides visibility into which agent and model were used during test execution.
+ * It extracts actual agent metadata from eval-runner.md and logs it prominently.
+ * 
+ * Features:
+ * - Reads eval-runner.md to extract actual agent metadata (id, name, description)
+ * - Shows agent prompt snippet (first 200 chars) to confirm correct agent loaded
+ * - Logs expected vs actual agent/model for comparison
+ * - Always passes (informational only, not validation)
+ * 
+ * Note: This is INFORMATIONAL ONLY. Actual agent/model configuration happens at test setup time
+ * via test-runner.ts setupEvalRunner(). Session data doesn't store agent/model metadata.
+ */
+
+import { BaseEvaluator } from './base-evaluator.js';
+import {
+  TimelineEvent,
+  SessionInfo,
+  EvaluationResult,
+  Evidence,
+  Violation
+} from '../types/index.js';
+import { readFileSync, existsSync } from 'fs';
+import { join } from 'path';
+import { homedir } from 'os';
+
+export interface AgentModelExpectations {
+  /** Expected agent name (e.g., "openagent", "opencoder", "coder-agent") */
+  expectedAgent?: string;
+  /** Expected model (e.g., "opencode/grok-code", "anthropic/claude-3-5-sonnet-20241022") */
+  expectedModel?: string;
+  /** Project path to find eval-runner.md */
+  projectPath?: string;
+}
+
+interface AgentMetadata {
+  id?: string;
+  name?: string;
+  description?: string;
+  category?: string;
+  type?: string;
+  version?: string;
+  mode?: string;
+  promptSnippet?: string;
+}
+
+export class AgentModelEvaluator extends BaseEvaluator {
+  name = 'agent-model';
+  description = 'Logs agent and model information for test transparency';
+
+  private expectations: AgentModelExpectations;
+
+  constructor(expectations: AgentModelExpectations = {}) {
+    super();
+    this.expectations = expectations;
+  }
+
+  /**
+   * Extract agent metadata from eval-runner.md frontmatter
+   */
+  private extractAgentMetadata(projectPath: string): AgentMetadata {
+    const metadata: AgentMetadata = {};
+    
+    try {
+      const evalRunnerPath = join(projectPath, '.opencode', 'agent', 'eval-runner.md');
+      
+      if (!existsSync(evalRunnerPath)) {
+        return metadata;
+      }
+
+      const content = readFileSync(evalRunnerPath, 'utf-8');
+      
+      // Extract frontmatter (between --- markers)
+      const frontmatterMatch = content.match(/^---\n([\s\S]*?)\n---/);
+      if (!frontmatterMatch) {
+        return metadata;
+      }
+
+      const frontmatter = frontmatterMatch[1];
+      
+      // Extract key fields
+      const idMatch = frontmatter.match(/^id:\s*(.+)$/m);
+      const nameMatch = frontmatter.match(/^name:\s*(.+)$/m);
+      const descMatch = frontmatter.match(/^description:\s*["'](.+)["']$/m);
+      const categoryMatch = frontmatter.match(/^category:\s*(.+)$/m);
+      const typeMatch = frontmatter.match(/^type:\s*(.+)$/m);
+      const versionMatch = frontmatter.match(/^version:\s*(.+)$/m);
+      const modeMatch = frontmatter.match(/^mode:\s*(.+)$/m);
+
+      if (idMatch) metadata.id = idMatch[1].trim();
+      if (nameMatch) metadata.name = nameMatch[1].trim();
+      if (descMatch) metadata.description = descMatch[1].trim();
+      if (categoryMatch) metadata.category = categoryMatch[1].trim();
+      if (typeMatch) metadata.type = typeMatch[1].trim();
+      if (versionMatch) metadata.version = versionMatch[1].trim();
+      if (modeMatch) metadata.mode = modeMatch[1].trim();
+
+      // Extract prompt snippet (first 200 chars after frontmatter)
+      const promptStart = content.indexOf('---', 4) + 3; // Skip first ---
+      const promptContent = content.substring(promptStart).trim();
+      const snippet = promptContent.substring(0, 200).replace(/\n/g, ' ').trim();
+      metadata.promptSnippet = snippet + (promptContent.length > 200 ? '...' : '');
+
+    } catch (error) {
+      // Silently fail - this is informational only
+    }
+
+    return metadata;
+  }
+
+  /**
+   * Extract model from timeline events
+   * Model is passed in API calls but not stored in session metadata
+   */
+  private extractModelFromTimeline(timeline: TimelineEvent[]): string | undefined {
+    // Look for session.created or message events that might contain model info
+    // In practice, model isn't in timeline either, but we try
+    return undefined; // Model not available in timeline
+  }
+
+  async evaluate(timeline: TimelineEvent[], sessionInfo: SessionInfo): Promise<EvaluationResult> {
+    const violations: Violation[] = [];
+    const evidence: Evidence[] = [];
+
+    // Extract actual agent metadata from eval-runner.md
+    const projectPath = this.expectations.projectPath || process.cwd();
+    const actualAgent = this.extractAgentMetadata(projectPath);
+    const actualModel = this.extractModelFromTimeline(timeline);
+
+    // Log actual agent information
+    if (actualAgent.id || actualAgent.name) {
+      evidence.push(this.createEvidence(
+        'actual-agent-info',
+        `Actual agent loaded: ${actualAgent.name || actualAgent.id || 'unknown'}`,
+        {
+          id: actualAgent.id,
+          name: actualAgent.name,
+          description: actualAgent.description,
+          category: actualAgent.category,
+          type: actualAgent.type,
+          version: actualAgent.version,
+          mode: actualAgent.mode,
+        }
+      ));
+    }
+
+    // Log agent prompt snippet for verification
+    if (actualAgent.promptSnippet) {
+      evidence.push(this.createEvidence(
+        'agent-prompt-snippet',
+        `Agent prompt snippet: "${actualAgent.promptSnippet}"`,
+        { snippet: actualAgent.promptSnippet }
+      ));
+    }
+
+    // Log expected agent/model if set
+    if (this.expectations.expectedAgent) {
+      evidence.push(this.createEvidence(
+        'expected-agent',
+        `Expected agent: ${this.expectations.expectedAgent}`,
+        { expectedAgent: this.expectations.expectedAgent }
+      ));
+    }
+
+    if (this.expectations.expectedModel) {
+      evidence.push(this.createEvidence(
+        'expected-model',
+        `Expected model: ${this.expectations.expectedModel}`,
+        { expectedModel: this.expectations.expectedModel }
+      ));
+    }
+
+    // Log informational note
+    evidence.push(this.createEvidence(
+      'info-note',
+      'โ„น๏ธ  This evaluator is INFORMATIONAL ONLY - it logs agent/model info but does not validate',
+      {
+        note: 'Agent/model configuration happens at test setup time (test-runner.ts)',
+        actualAgentId: actualAgent.id,
+        expectedAgent: this.expectations.expectedAgent,
+        expectedModel: this.expectations.expectedModel,
+      }
+    ));
+
+    // Always pass - this is informational only
+    const passed = true;
+    const score = 100;
+
+    return {
+      evaluator: this.name,
+      passed,
+      score,
+      violations,
+      evidence,
+      metadata: {
+        actualAgent: {
+          id: actualAgent.id,
+          name: actualAgent.name,
+          description: actualAgent.description,
+          category: actualAgent.category,
+          type: actualAgent.type,
+          version: actualAgent.version,
+          mode: actualAgent.mode,
+          promptSnippet: actualAgent.promptSnippet,
+        },
+        expectedAgent: this.expectations.expectedAgent,
+        expectedModel: this.expectations.expectedModel,
+        mode: 'informational',
+      }
+    };
+  }
+
+}

+ 50 - 3
evals/framework/src/evaluators/behavior-evaluator.ts

@@ -72,8 +72,25 @@ export class BehaviorEvaluator extends BaseEvaluator {
       toolCalls.forEach((tc, i) => {
         const tool = tc.data?.tool || 'unknown';
         const input = tc.data?.state?.input || tc.data?.input || {};
-        const inputStr = JSON.stringify(input).substring(0, 100);
-        console.log(`  ${i + 1}. ${tool}: ${inputStr}${inputStr.length >= 100 ? '...' : ''}`);
+        
+        // Show more details for task tool (delegation)
+        if (tool === 'task') {
+          console.log(`  ${i + 1}. ${tool}:`);
+          if (input.subagent_type) {
+            console.log(`     โ†’ Subagent: ${input.subagent_type}`);
+          }
+          if (input.description) {
+            console.log(`     โ†’ Description: ${input.description}`);
+          }
+          if (input.prompt) {
+            const promptPreview = input.prompt.substring(0, 150);
+            console.log(`     โ†’ Prompt: "${promptPreview}${input.prompt.length > 150 ? '...' : ''}"`);
+          }
+        } else {
+          // Regular tool - show compact format
+          const inputStr = JSON.stringify(input).substring(0, 100);
+          console.log(`  ${i + 1}. ${tool}: ${inputStr}${inputStr.length >= 100 ? '...' : ''}`);
+        }
       });
     }
 
@@ -463,7 +480,37 @@ export class BehaviorEvaluator extends BaseEvaluator {
     // Print summary
     console.log(`\nBehavior Validation Summary:`);
     console.log(`  Checks Passed: ${checks.filter(c => c.passed).length}/${checks.length}`);
-    console.log(`  Violations: ${violations.length}`);
+    
+    // Show which checks passed/failed with reasons
+    if (checks.length > 0) {
+      console.log(`\nCheck Details:`);
+      checks.forEach((check, i) => {
+        const icon = check.passed ? 'โœ“' : 'โœ—';
+        const status = check.passed ? 'PASS' : 'FAIL';
+        console.log(`  ${icon} ${check.name}: ${status}`);
+        
+        // Show reason/evidence
+        if (check.evidence && check.evidence.length > 0) {
+          check.evidence.forEach(ev => {
+            if (ev.description) {
+              console.log(`     โ†’ ${ev.description}`);
+            }
+            // Show key data points
+            if (ev.data) {
+              if (ev.data.expected !== undefined && ev.data.actual !== undefined) {
+                console.log(`     โ†’ Expected: ${JSON.stringify(ev.data.expected)}, Got: ${JSON.stringify(ev.data.actual)}`);
+              } else if (ev.data.toolsUsed !== undefined) {
+                console.log(`     โ†’ Tools used: ${ev.data.toolsUsed.length > 0 ? ev.data.toolsUsed.join(', ') : 'none'}`);
+              } else if (ev.data.count !== undefined) {
+                console.log(`     โ†’ Count: ${ev.data.count}`);
+              }
+            }
+          });
+        }
+      });
+    }
+    
+    console.log(`\n  Violations: ${violations.length}`);
     if (violations.length > 0) {
       console.log(`\nViolations Detected:`);
       violations.forEach((v, i) => {

+ 17 - 0
evals/framework/src/evaluators/delegation-evaluator.ts

@@ -54,6 +54,23 @@ export class DelegationEvaluator extends BaseEvaluator {
     // Get delegation tool calls (task tool)
     const delegationCalls = this.getToolCallsByName(timeline, 'task');
     const didDelegate = delegationCalls.length > 0;
+    
+    // Add evidence for each task tool call
+    if (didDelegate) {
+      delegationCalls.forEach((call, index) => {
+        const input = call.data?.input || {};
+        evidence.push(this.createEvidence(
+          'task-tool-call',
+          `Task tool call #${index + 1}: ${input.description || 'No description'}`,
+          {
+            subagent_type: input.subagent_type,
+            description: input.description,
+            prompt: input.prompt,
+            timestamp: call.timestamp
+          }
+        ));
+      });
+    }
 
     // Determine if delegation was required (multiple criteria)
     const shouldDelegate = 

+ 21 - 0
evals/framework/src/sdk/test-case-schema.ts

@@ -92,6 +92,27 @@ export const BehaviorExpectationSchema = z.object({
    * Agent must use dedicated tools instead of bash
    */
   mustUseDedicatedTools: z.boolean().optional(),
+
+  /**
+   * Expected agent name (for validation)
+   * Example: "openagent", "opencoder", "coder-agent"
+   */
+  expectedAgent: z.string().optional(),
+
+  /**
+   * Expected model (for validation)
+   * Example: "opencode/grok-code", "anthropic/claude-3-5-sonnet-20241022"
+   */
+  expectedModel: z.string().optional(),
+
+  /**
+   * Expected response content (partial matching)
+   */
+  expectedResponse: z.object({
+    contains: z.array(z.string()).optional(),
+    notContains: z.array(z.string()).optional(),
+    description: z.string().optional(),
+  }).optional(),
 });
 
 export type BehaviorExpectation = z.infer<typeof BehaviorExpectationSchema>;

+ 24 - 7
evals/framework/src/sdk/test-executor.ts

@@ -89,6 +89,24 @@ export class TestExecutor {
       this.logger.log(`\n${'='.repeat(60)}`);
       this.logger.log(`Running test: ${testCase.id} - ${testCase.name}`);
       this.logger.log(`${'='.repeat(60)}`);
+      
+      // Show agent/model configuration prominently
+      const modelToUse = testCase.model || this.config.defaultModel;
+      const agentDisplayMap: Record<string, string> = {
+        'openagent': 'OpenAgent',
+        'core/openagent': 'OpenAgent',
+        'opencoder': 'OpenCoder',
+        'core/opencoder': 'OpenCoder',
+        'system-builder': 'System Builder',
+        'meta/system-builder': 'System Builder',
+      };
+      const agentToUse = agentDisplayMap[testCase.agent || 'openagent'] || testCase.agent || 'OpenAgent';
+      
+      this.logger.log(`โ”Œ${'โ”€'.repeat(58)}โ”`);
+      this.logger.log(`โ”‚ ๐Ÿค– Agent: ${agentToUse.padEnd(46)} โ”‚`);
+      this.logger.log(`โ”‚ ๐Ÿง  Model: ${modelToUse.padEnd(46)} โ”‚`);
+      this.logger.log(`โ””${'โ”€'.repeat(58)}โ”˜`);
+      
       this.logger.log(`Approval strategy: ${approvalStrategy.describe()}`);
 
       // Setup event handler
@@ -149,8 +167,8 @@ export class TestExecutor {
       // Note: Agent is already configured via eval-runner.md, no need to inject context
       await this.sendPrompts(testCase, sessionId, errors);
 
-      // Give time for final events to arrive
-      await this.sleep(3000);
+      // Give time for final events to arrive (reduced from 3s to 1.5s)
+      await this.sleep(1500);
 
       // Stop event handler and cleanup
       this.eventHandler.stopListening();
@@ -215,8 +233,7 @@ export class TestExecutor {
     
     const agentToUse = agentDisplayMap[testCase.agent || 'openagent'] || 'OpenAgent';
     
-    this.logger.log(`Agent: ${agentToUse}`);
-    this.logger.log(`Model: ${modelToUse}`);
+    // Agent/Model already logged in execute() method - no need to duplicate
     
     // Check if multi-message test
     if (testCase.prompts && testCase.prompts.length > 0) {
@@ -389,7 +406,7 @@ export class TestExecutor {
    * This prevents race conditions where prompts are sent before the event stream is ready
    */
   private async waitForEventStreamConnection(): Promise<void> {
-    const timeoutMs = 5000;
+    const timeoutMs = 2000; // Reduced from 5000ms - connection is usually immediate
     let eventReceived = false;
 
     const connectionPromise = new Promise<void>((resolve, reject) => {
@@ -443,8 +460,8 @@ export class TestExecutor {
     
     // Grace period after text completion with no tools - wait for tools to potentially start
     // This is shorter because if the agent is going to use tools, it usually starts quickly
-    // Reduced from 5s to 2s based on test analysis - most tools start within 1-2s
-    const noToolsGracePeriod = 2000; // 2 seconds (optimized from 5s)
+    // Reduced from 5s โ†’ 2s โ†’ 1s based on test analysis - most tools start within 500ms-1s
+    const noToolsGracePeriod = 1000; // 1 second (optimized from 2s)
     
     while (Date.now() - startTime < timeoutMs) {
       try {

+ 17 - 10
evals/results/latest.json

@@ -1,16 +1,16 @@
 {
   "meta": {
-    "timestamp": "2025-12-14T23:11:51.536Z",
-    "agent": "unknown",
-    "model": "opencode/big-pickle",
+    "timestamp": "2025-12-18T00:26:11.563Z",
+    "agent": "core/openagent",
+    "model": "opencode/grok-code-fast",
     "framework_version": "0.1.0",
-    "git_commit": "328eb38"
+    "git_commit": "4f778c8"
   },
   "summary": {
     "total": 1,
     "passed": 1,
     "failed": 0,
-    "duration_ms": 34917,
+    "duration_ms": 28002,
     "pass_rate": 1
   },
   "by_category": {
@@ -21,16 +21,23 @@
   },
   "tests": [
     {
-      "id": "golden-02-context-loading-explicit",
+      "id": "golden-02-delegation-test",
       "category": "developer",
       "passed": true,
-      "duration_ms": 34917,
-      "events": 63,
+      "duration_ms": 28002,
+      "events": 62,
       "approvals": 0,
       "violations": {
-        "total": 0,
+        "total": 1,
         "errors": 0,
-        "warnings": 0
+        "warnings": 1,
+        "details": [
+          {
+            "type": "insufficient-read",
+            "severity": "warning",
+            "message": "Read/execution ratio < 1 (0.67)"
+          }
+        ]
       }
     }
   ]

+ 16 - 1
registry.json

@@ -1263,5 +1263,20 @@
   "metadata": {
     "lastUpdated": "2025-12-14",
     "schemaVersion": "1.0.0"
+  },
+  "subagents": {
+    "test": {
+      "simple-responder": {
+        "id": "simple-responder",
+        "name": "Simple Responder",
+        "description": "Test agent that responds with AWESOME TESTING",
+        "category": "test",
+        "type": "utility",
+        "version": "1.0.0",
+        "path": ".opencode/agent/subagents/test/simple-responder.md",
+        "mode": "subagent",
+        "status": "stable"
+      }
+    }
   }
-}
+}

+ 1 - 1
scripts/development/dashboard.sh

@@ -19,7 +19,7 @@ echo -e "${BLUE}๐ŸŒ URL: http://localhost:$PORT${NC}"
 echo ""
 
 # Navigate to results directory
-cd "$(dirname "$0")/../evals/results" || exit 1
+cd "$(dirname "$0")/../../evals/results" || exit 1
 
 # Check if results exist
 if [ ! -f "latest.json" ]; then