| 123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354 |
- id: neg-wrong-context-001
- name: Wrong Context File Loaded (Negative Test)
- description: |
- NEGATIVE TEST: Validates that loading the WRONG context file is detected.
-
- This test asks for a CODE task but expects the agent to load code.md.
- If the agent loads docs.md instead, it should be flagged as a violation.
-
- The evaluator should:
- 1. Classify the task as 'code' (based on "Create a function")
- 2. Expect code.md to be loaded
- 3. Detect if wrong context file (docs.md, tests.md, etc.) is loaded
- 4. Flag as violation if wrong context file is used
- category: edge-case
- agent: openagent
- prompts:
- - text: |
- Create a TypeScript function called 'multiply' that takes two numbers and returns their product.
- Save it to evals/test_tmp/multiply.ts
-
- IMPORTANT: This is a CODE task - you should load the code.md context file.
-
- - text: |
- Yes, proceed with the plan.
- delayMs: 3000
- behavior:
- mustUseTools: [read, write]
- requiresContext: true
- minToolCalls: 2
- expectedViolations:
- - rule: context-loading
- shouldViolate: false # Should NOT violate if correct context loaded
- severity: error
- description: Must load code.md for code tasks, not docs.md or tests.md
-
- - rule: execution-balance
- shouldViolate: false
- severity: warning
- description: Should read context before writing code
- approvalStrategy:
- type: auto-approve
- timeout: 120000
- tags:
- - context-loading
- - context-file-mapping
- - code-task
- - critical
|
