OpenAgentsControl/evals

OpenCode Agent Evaluation Framework

Overview

Comprehensive evaluation framework for testing and validating OpenCode agent behavior against defined standards and rules.

Structure

evals/
├── framework/              # Reusable evaluation framework
│   ├── src/
│   │   ├── collector/     # Session data collection
│   │   ├── evaluators/    # Evaluation logic
│   │   ├── runner/        # Test execution
│   │   ├── reporters/     # Result reporting
│   │   └── types/         # TypeScript types
│   └── package.json
│
├── opencode/              # OpenCode agent evaluations
│   ├── openagent/        # OpenAgent-specific tests
│   ├── opencoder/        # OpenCoder-specific tests
│   └── shared/           # Shared test cases
│
└── results/              # Test results (gitignored)
    └── YYYY-MM-DD/

Quick Start

1. Install Framework

cd evals/framework
npm install
npm run build

2. Run Evaluations

# Evaluate a specific session
npm run eval -- --agent openagent --session ses_xxxxx

# Run all OpenAgent tests
npm run eval -- --agent openagent --all

# Run specific test case
npm run eval -- --agent openagent --test approval-gates

3. View Results

# View latest results
cat evals/results/$(ls -t evals/results | head -1)/openagent/summary.json | jq

# Generate report
npm run report -- --agent openagent --date 2025-11-21

Framework Components

Collector

Reads and parses OpenCode session data from ~/.local/share/opencode/

Components:

• SessionReader - Read session files
• MessageParser - Parse message structure
• TimelineBuilder - Build event timeline

Evaluators

Validate agent behavior against rules

Available Evaluators:

• ApprovalGateEvaluator - Check approval before execution
• ContextLoadingEvaluator - Verify context loading
• DelegationEvaluator - Validate delegation decisions
• ToolUsageEvaluator - Check tool selection
• ModelSelectionEvaluator - Validate model choices

Runner

Execute test cases and collect results

Components:

• TestRunner - Run test suites
• SessionAnalyzer - Analyze historical sessions
• LiveRunner - Run live tests (future)

Reporters

Generate test reports

Formats:

• Console (pretty output)
• JSON (machine-readable)
• Markdown (documentation)
• HTML (dashboard, future)

Agent-Specific Tests

OpenAgent (opencode/openagent/)

Tests for the universal OpenAgent:

• Approval gate enforcement
• Context loading compliance
• Delegation appropriateness
• Tool usage patterns
• Model selection

Test Categories:

• Conversational (simple questions)
• Task (file operations)
• Complex (multi-file features)
• Edge cases (errors, permissions)

OpenCoder (opencode/opencoder/)

Tests for the OpenCoder agent (future)

Shared (opencode/shared/)

Common test cases applicable to all agents

Test Case Format

Test cases are defined in YAML:

test_cases:
  - id: simple-question
    name: "Simple Question (No Execution)"
    description: "Ask a question that requires no execution tools"
    category: conversational
    input: "What does this code do?"
    expected_behavior:
      no_execution_tools: true
      no_approval_required: true
      response_provided: true
    evaluators:
      - approval-gate
      - tool-usage
    pass_threshold: 100

Evaluation Results

Results include:

• Pass/Fail - Did the test pass?
• Score - 0-100 compliance score
• Violations - List of rule violations
• Evidence - Supporting data for violations
• Metadata - Agent, model, timing, costs

Example Result:

{
  "testCaseId": "simple-question",
  "sessionId": "ses_xxxxx",
  "passed": true,
  "score": 100,
  "evaluationResults": [
    {
      "evaluator": "approval-gate",
      "passed": true,
      "score": 100,
      "violations": []
    }
  ],
  "metadata": {
    "agent": "openagent",
    "model": "claude-sonnet-4-20250514",
    "duration": 7205,
    "cost": 0.107
  }
}

Scoring System

• 100 - Perfect compliance
• 75-99 - Minor issues (warnings)
• 50-74 - Moderate issues (violations)
• 0-49 - Major issues (critical violations)

Pass Threshold: 75% (configurable per test)

Development

Adding New Evaluators

1. Create evaluator in framework/src/evaluators/
2. Extend BaseEvaluator
3. Implement evaluate() method
4. Add tests
5. Register in evaluator registry

Adding New Test Cases

1. Create YAML file in opencode/{agent}/test-cases/
2. Define test case structure
3. Specify expected behavior
4. List evaluators to run
5. Set pass threshold

Running Tests

# Framework tests
cd evals/framework
npm test

# Integration tests
npm run test:integration

# Test specific evaluator
npm run test:evaluator -- approval-gate

CI/CD Integration

# .github/workflows/eval.yml
name: Agent Evaluation

on: [push, pull_request]

jobs:
  evaluate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: actions/setup-node@v3
      - run: cd evals/framework && npm install
      - run: npm run eval -- --agent openagent --all
      - run: npm run report -- --format json

Configuration

Framework Config (framework/config.ts)

export const config = {
  projectPath: process.cwd(),
  sessionStoragePath: '~/.local/share/opencode/',
  resultsPath: 'evals/results/',
  passThreshold: 75,
  evaluators: ['approval-gate', 'context-loading', 'delegation', 'tool-usage']
};

Agent Config (opencode/openagent/config.yaml)

agent: openagent
test_cases_path: ./test-cases
sessions_path: ./sessions
evaluators:
  - approval-gate
  - context-loading
  - delegation
  - tool-usage
pass_threshold: 75

Related Documentation

• Framework Architecture
• OpenAgent Tests
• OpenCode Logging System
• Agent Validator Plugin

Contributing

See CONTRIBUTING.md

License

MIT