OpenAgentsControl/evals/agents/core/opencoder/tests/contextscout-integration

OpenCoder ContextScout Integration Tests

Purpose: Validate that OpenCoder uses ContextScout proactively when encountering unfamiliar patterns or domain-specific requirements.

Created: 2026-01-09
Status: Ready to Run

---

Test Suite Overview

Test 1: Implicit Pattern Discovery 🔍

File: 01-implicit-pattern-discovery.yaml

Scenario: "Add a new evaluator to the eval framework"

Validates:

• OpenCoder recognizes unfamiliar domain (eval framework)
• OpenCoder delegates to ContextScout WITHOUT being told
• OpenCoder loads discovered context files
• OpenCoder applies discovered patterns to implementation

Expected Behavior:

• ✅ Delegates to ContextScout automatically
• ✅ Loads eval framework context
• ✅ Proposes plan following discovered patterns
• ✅ Requests approval before implementation
• ✅ Implements code matching framework patterns

This test SHOULD FAIL if:

• ❌ OpenCoder doesn't use ContextScout (guesses patterns)
• ❌ OpenCoder implements without loading context
• ❌ OpenCoder uses wrong patterns (not from context)

---

Running Tests

Run All OpenCoder Integration Tests

cd evals/framework
npm run eval:sdk -- --agent=core/opencoder --pattern="contextscout-integration/*.yaml"

Run Individual Test

npm run eval:sdk -- --agent=core/opencoder --pattern="01-implicit-pattern-discovery.yaml"

Run with Debug

npm run eval:sdk -- --agent=core/opencoder --debug --pattern="01-implicit-pattern-discovery.yaml"

---

Success Criteria

OpenCoder is "Using ContextScout Correctly" if:

1. ✅ Proactive: Uses ContextScout without being told
2. ✅ Recognition: Recognizes unfamiliar domains (eval framework, registry, etc.)
3. ✅ Context-First: Loads context BEFORE implementation
4. ✅ Pattern Application: Applies discovered patterns in code
5. ✅ Approval Gate: Still requests approval before implementation

OpenCoder is "Not Using ContextScout" if:

1. ❌ Guessing: Implements without discovering patterns
2. ❌ Wrong Patterns: Uses patterns not from discovered context
3. ❌ Skipping Discovery: Doesn't delegate to ContextScout for unfamiliar domains
4. ❌ Inconsistent: Sometimes uses ContextScout, sometimes doesn't

---

Expected Outcome

Test 1: Implicit Pattern Discovery

✅ PASS

Workflow:
1. User: "Add a new evaluator to the eval framework"
2. OpenCoder: Recognizes "eval framework" as unfamiliar domain
3. OpenCoder: Delegates to ContextScout (WITHOUT being told)
4. ContextScout: Finds eval framework context files
5. OpenCoder: Loads discovered files
6. OpenCoder: Proposes plan following discovered patterns
7. User: Approves
8. OpenCoder: Implements evaluator matching framework patterns

Key Validations:
✅ Used task tool to delegate to ContextScout
✅ Loaded context files before implementation
✅ Proposed plan mentions discovered patterns
✅ Requested approval before implementation
✅ Implementation matches eval framework patterns

---

Debugging Failed Tests

If Test Fails: OpenCoder Didn't Use ContextScout

Problem: OpenCoder implemented without discovering patterns

Check:

1. Did OpenCoder use task tool to delegate to ContextScout?
2. Did OpenCoder load any context files?
3. Did OpenCoder mention "eval framework" patterns in plan?

Possible Causes:

• OpenCoder prompt doesn't emphasize ContextScout usage
• OpenCoder doesn't recognize "eval framework" as unfamiliar
• OpenCoder skips discovery for efficiency

Fix:

• Update OpenCoder prompt to emphasize proactive ContextScout usage
• Add more examples of when to use ContextScout
• Make ContextScout usage more explicit in workflow

---

If Test Fails: OpenCoder Used Wrong Patterns

Problem: OpenCoder implemented but patterns don't match framework

Check:

1. Did OpenCoder load eval framework context?
2. Did OpenCoder read the discovered files?
3. Does implementation match patterns from context files?

Possible Causes:

• OpenCoder loaded context but didn't apply it
• OpenCoder used generic patterns instead of framework-specific
• Context files don't contain clear patterns

Fix:

• Emphasize "apply discovered patterns" in OpenCoder prompt
• Improve context files to include clearer pattern examples
• Add validation step to check pattern matching

---

Related Documentation

• OpenCoder Agent
• ContextScout Agent
• OpenAgent Integration Tests
• ContextScout Tests

---

Key Insight: OpenCoder should proactively use ContextScout when encountering unfamiliar domains or patterns, ensuring implementations match project standards and framework patterns.