| 1234567891011121314151617181920212223242526272829303132333435363738394041424344454647484950515253545556575859606162636465666768697071727374 |
- id: contextscout-bad-request
- name: "ContextScout: Bad Request Handling - Vague or Invalid Queries"
- description: |
- Tests that ContextScout handles bad requests gracefully:
- - Vague queries with no clear intent
- - Requests for non-existent topics
- - Ambiguous requests that need clarification
-
- This validates:
- - ContextScout doesn't fabricate context that doesn't exist
- - ContextScout reports when no relevant files found
- - ContextScout suggests alternatives or clarifications
- - ContextScout still uses tools to verify (not assumptions)
-
- Expected Behavior:
- - MUST use tools to search (even if nothing found)
- - MUST report honestly when no relevant files exist
- - SHOULD suggest related topics or clarifications
- - MUST NOT fabricate file paths
- - SHOULD explain what was searched and why nothing matched
-
- This test SHOULD FAIL if:
- - ContextScout fabricates files that don't exist
- - ContextScout doesn't search (assumes nothing exists)
- - ContextScout returns generic answer without verification
- - ContextScout provides unhelpful response
- category: edge-case
- agent: subagents/core/contextscout
- prompts:
- - text: |
- Search for context files related to: quantum blockchain AI optimization
-
- Task type: unknown
-
- Return:
- - Exact file paths with line ranges
- - Priority order (critical, high, medium)
- - Key findings from each file
- approvalStrategy:
- type: auto-approve
- behavior:
- forbiddenTools:
- - task
- - bash
- - write
- - edit
- minToolCalls: 1
- maxToolCalls: 10
- expectedViolations:
- - rule: approval-gate
- shouldViolate: false
- severity: error
- timeout: 60000
- tags:
- - contextscout
- - error-handling
- - bad-request
- - edge-case
- - important
- # Expected outcome:
- # - Uses glob to search for related terms
- # - Finds no relevant files (topic doesn't exist in repo)
- # - Reports honestly: "No context files found for this topic"
- # - Suggests: "Did you mean: [related topics]?" or "Available topics: [list]"
- # - Does NOT fabricate paths
- # - Does NOT return generic answer without searching
|
