OpenAgentsControl/evals/VALIDATION_QUICK_REF.md

Test Suite Validation - Quick Reference

🚀 Quick Commands

# Validate specific agent
cd evals/framework && npm run validate:suites openagent

# Validate all agents
cd evals/framework && npm run validate:suites:all

# Setup pre-commit hook
./scripts/validation/setup-pre-commit-hook.sh

# Run tests with validated suite
npm run eval:sdk -- --agent=openagent --suite=core

✅ Validation Layers

Layer	When	Command
IDE	While editing	Automatic (JSON schema)
Pre-commit	Before commit	Automatic (if setup)
Manual	Anytime	npm run validate:suites
CI/CD	On push/PR	Automatic (GitHub Actions)

📋 Required Fields

{
  "name": "Suite Name",
  "description": "What this suite tests",
  "version": "1.0.0",
  "agent": "openagent",
  "totalTests": 3,
  "estimatedRuntime": "3-5 minutes",
  "tests": [
    {
      "id": 1,
      "name": "Test Name",
      "path": "category/test-file.yaml",
      "category": "critical-rules",
      "priority": "critical"
    }
  ]
}

🎯 Valid Enum Values

Agent: openagent | opencoder

Category:

• critical-rules
• workflow-stages
• delegation
• execution-paths
• edge-cases
• integration
• negative
• behavior
• tool-usage

Priority: critical | high | medium | low

🔧 Common Fixes

Missing Field

// ❌ Error: agent: Required
{
  "name": "My Suite",
  "version": "1.0.0"
}

// ✅ Fixed
{
  "name": "My Suite",
  "version": "1.0.0",
  "agent": "openagent"
}

Invalid Version

// ❌ Error: version must match pattern ^\d+\.\d+\.\d+$
"version": "1.0"

// ✅ Fixed
"version": "1.0.0"

Invalid Path

// ❌ Error: path must end with .yaml
"path": "tests/my-test.yml"

// ✅ Fixed
"path": "tests/my-test.yaml"

Missing Test File

// ❌ Error: Required test file not found
"path": "01-critical-rules/wrong-path.yaml"

// ✅ Fixed (check actual file name)
"path": "01-critical-rules/approval-gate/05-approval-before-execution-positive.yaml"

📁 File Locations

evals/
├── agents/
│   └── openagent/
│       └── config/
│           ├── suite-schema.json       # JSON Schema
│           ├── core-tests.json         # Core suite
│           └── suites/                 # Custom suites
│               ├── quick.json
│               └── oss.json
└── framework/
    └── src/sdk/
        ├── suite-validator.ts          # TypeScript validator
        └── validate-suites-cli.ts      # CLI tool

🚨 Troubleshooting

Issue	Solution
ajv-cli not found	cd evals/framework && npm install
Pre-commit not running	./scripts/validation/setup-pre-commit-hook.sh
TypeScript errors	cd evals/framework && npm run build
Validation hangs	Use TypeScript validator: npm run validate:suites

📚 Full Documentation

• Complete Validation Guide
• Suite Configuration README
• JSON Schema