GETTING_STARTED.md 10 KB
Getting Started with OpenCode Agent Evaluation
Quick start guide for running and understanding agent tests
Prerequisites
# Install dependencies
cd evals/framework
npm install
npm run build
Running Tests
Quick Start
# Run all tests (uses free model by default)
npm run eval:sdk
# Run specific agent
npm run eval:sdk -- --agent=openagent
npm run eval:sdk -- --agent=opencoder
# Run specific test category
npm run eval:sdk -- --agent=openagent --pattern="context-loading/*.yaml"
# Debug mode (verbose output, keeps sessions)
npm run eval:sdk -- --debug
Batch Execution (Avoid API Limits)
# Run tests in batches of 3 with 10s delays
./scripts/utils/run-tests-batch.sh openagent 3 10
Understanding Test Results
Test Output Example
======================================================================
TEST RESULTS
======================================================================
1. ✅ ctx-simple-coding-standards - Context Loading: Coding Standards
Duration: 22821ms
Events: 18
Approvals: 0
Context Loading: ⊘ Conversational session (not required)
Violations: 0 (0 errors, 0 warnings)
2. ✅ ctx-multi-standards-to-docs - Multi-Turn Standards to Documentation
Duration: 116455ms
Events: 164
Approvals: 0
Context Loading:
✓ Loaded: .opencode/context/core/standards/code.md
✓ Timing: Context loaded 44317ms before execution
Violations: 0 (0 errors, 0 warnings)
======================================================================
SUMMARY: 2/2 tests passed (0 failed)
======================================================================
What Each Field Means
| Field | Meaning |
|---|---|
| Duration | Total test execution time (includes agent thinking + tool execution) |
| Events | Number of events captured from server (messages, tool calls, etc.) |
| Approvals | Tool permission requests handled (not text-based approvals) |
| Context Loading | Whether context files were loaded before execution |
| Violations | Rule violations detected by evaluators |
Test Execution Flow
┌─────────────────────────────────────────────────────────────────┐
│ TEST RUNNER │
├─────────────────────────────────────────────────────────────────┤
│ 1. Clean test_tmp/ directory │
│ 2. Start opencode server (from git root) │
│ 3. For each test: │
│ a. Create session │
│ b. Send prompt(s) with agent selection │
│ c. Capture events via event stream │
│ d. Run evaluators on session data │
│ e. Check behavior expectations │
│ f. Delete session (unless --debug) │
│ 4. Clean test_tmp/ directory │
│ 5. Save results to JSON │
│ 6. Print results │
└─────────────────────────────────────────────────────────────────┘
Agent Differences
Opencoder (Direct Execution)
- Executes tools immediately
- Uses tool permission system only
- No text-based approval workflow
- Tests use single prompts
Example Test:
agent: opencoder
prompt: "List files in current directory"
behavior:
mustUseAnyOf: [[bash], [list]]
OpenAgent (Approval Workflow)
- Outputs "Proposed Plan" first
- Waits for user approval in text
- Then executes tools
- Tests use multi-turn prompts
Example Test:
agent: openagent
prompts:
- text: "List files in current directory"
- text: "approve"
delayMs: 2000
behavior:
mustUseTools: [bash]
Creating New Tests
Simple Test (Single Prompt)
# File: evals/agents/openagent/tests/context-loading/my-test.yaml
id: my-test-001
name: "My Test Name"
description: |
What this test validates
category: developer
agent: openagent
model: anthropic/claude-sonnet-4-5
prompt: "Your test prompt here"
behavior:
mustUseTools: [read]
requiresContext: true
minToolCalls: 1
expectedViolations:
- rule: context-loading
shouldViolate: false
severity: error
approvalStrategy:
type: auto-approve
timeout: 60000
tags:
- context-loading
- simple-test
Complex Test (Multi-Turn)
id: my-complex-test-001
name: "Multi-Turn Test"
description: |
Tests multi-turn conversation with context loading
category: developer
agent: openagent
model: anthropic/claude-sonnet-4-5
prompts:
- text: "What are our coding standards?"
expectContext: true
contextFile: "standards.md"
- text: "approve"
delayMs: 2000
- text: "Create documentation about these standards"
expectContext: true
contextFile: "docs.md"
- text: "approve"
delayMs: 2000
behavior:
mustUseTools: [read, write]
requiresApproval: true
requiresContext: true
minToolCalls: 3
expectedViolations:
- rule: approval-gate
shouldViolate: false
severity: error
- rule: context-loading
shouldViolate: false
severity: error
approvalStrategy:
type: auto-approve
timeout: 300000 # 5 minutes
tags:
- context-loading
- multi-turn
- complex-test
Viewing Results
Dashboard
cd evals/results
./serve.sh
This will:
- Start HTTP server on port 8000
- Open browser automatically
- Load test results dashboard
- Auto-shutdown after 15 seconds
The dashboard caches data in your browser, so it works even after the server shuts down.
JSON Results
# Latest results
cat evals/results/latest.json
# Historical results
ls evals/results/history/2025-11/
File Cleanup
Tests that create files use evals/test_tmp/:
prompt: |
Create a file at evals/test_tmp/test.txt with content "Hello"
The test runner automatically cleans this directory:
- Before tests start - Removes all files except
.gitignoreandREADME.md - After tests complete - Removes all test artifacts
Debugging Tests
Enable Debug Mode
npm run eval:sdk -- --agent=openagent --pattern="my-test.yaml" --debug
Debug mode shows:
- All events captured
- Tool call details with full inputs
- Agent verification steps
- Keeps sessions for inspection (not deleted)
Inspect Sessions
# Sessions are stored here
ls ~/.local/share/opencode/storage/session/
# View session details (in debug mode)
cat ~/.local/share/opencode/storage/session/<session-id>.json
Check Tool Calls
Look for the BEHAVIOR VALIDATION section in output:
============================================================
BEHAVIOR VALIDATION
============================================================
Timeline Events: 28
Tool Calls: 3
Tools Used: read, write
Tool Call Details:
1. read: {"filePath":".opencode/context/core/standards/code.md"}
2. read: {"filePath":".opencode/context/core/standards/docs.md"}
3. write: {"filePath":"evals/test_tmp/output.md"}
[behavior] Files Read (2):
1. .opencode/context/core/standards/code.md
2. .opencode/context/core/standards/docs.md
[behavior] Context Files Read: 2/2
Behavior Validation Summary:
Checks Passed: 4/4
Violations: 0
============================================================
Common Issues
"Agent not set in message"
Cause: SDK might not return the agent field
Impact: Warning only, not an error
Action: Ignore - test still validates correctly
"0 events captured"
Cause: Event stream connection failed
Action: Check server is running, restart test
"Tool X was not used"
Cause: Agent used a different tool
Action: Use mustUseAnyOf for flexibility:
behavior:
mustUseAnyOf: [[bash], [list]] # Either tool is acceptable
"Files created in wrong location"
Cause: Test prompt doesn't specify evals/test_tmp/
Action: Update test prompt to use correct path
"Timeout"
Cause: Test took longer than timeout value
Action: Increase timeout in test YAML:
timeout: 300000 # 5 minutes
Test Categories
| Category | Purpose | Example Tests |
|---|---|---|
| context-loading | Verify context files loaded before execution | ctx-simple-coding-standards |
| developer | Developer workflow tests | create-component, install-dependencies |
| business | Business analysis tests | data-analysis |
| edge-case | Edge cases and error handling | just-do-it, missing-approval |
Model Configuration
Free Tier (Default)
# Uses opencode/grok-code-fast (free)
npm run eval:sdk
Paid Models
# Claude 3.5 Sonnet
npm run eval:sdk -- --model=anthropic/claude-3-5-sonnet-20241022
# GPT-4 Turbo
npm run eval:sdk -- --model=openai/gpt-4-turbo
Per-Test Override
# In test YAML file
model: anthropic/claude-3-5-sonnet-20241022
Next Steps
Read the docs:
- README.md - System overview
- ARCHITECTURE.md - System architecture
- framework/SDK_EVAL_README.md - Complete SDK guide
Explore tests:
evals/agents/openagent/tests/context-loading/- Context loading testsevals/agents/opencoder/tests/developer/- Opencoder tests
Run tests:
npm run eval:sdk -- --agent=openagent --pattern="context-loading/*.yaml"View results:
cd ../results && ./serve.sh
Support
- Issues: Check HOW_TESTS_WORK.md for detailed explanations
- Test Design: See framework/docs/test-design-guide.md
- Agent Rules: See agents/openagent/docs/OPENAGENT_RULES.md
Happy Testing! 🚀