Przeglądaj źródła

ci(security): run deterministic eval tests in affected-area CI (Task 04)

The build-check job validated build + suite structure but never ran the
tests, and its summary referenced a `test:ci` script that did not exist.
Make affected-area CI meaningful:

- Add `test:ci` to evals/framework running the offline, deterministic
  Vitest allowlist (7 files / 112 tests) — no model, network, or paid
  execution.
- Add a required "Run deterministic tests" step to build-check; a failure
  fails the job and the existing PR summary gate (which keys on
  build-check.result) fails closed.
- Correct the summary note to state test:ci runs the deterministic
  allowlist and that model/network agent suites are excluded on PRs.

The broader deterministic suite has 3 pre-existing failing files
(framework-confidence, logger, test-runner; confirmed failing before this
work) which are intentionally out of the allowlist and tracked separately.

Validated locally: build, validate:suites:all, and test:ci all exit 0.
darrenhinde 2 tygodni temu
rodzic
commit
fccc464d45
2 zmienionych plików z 9 dodań i 3 usunięć
  1. 8 3
      .github/workflows/pr-checks.yml
  2. 1 0
      evals/framework/package.json

+ 8 - 3
.github/workflows/pr-checks.yml

@@ -214,7 +214,11 @@ jobs:
       - name: Validate test suites
         working-directory: evals/framework
         run: npm run validate:suites:all
-      
+
+      - name: Run deterministic tests
+        working-directory: evals/framework
+        run: npm run test:ci
+
       - name: Summary
         if: success()
         run: |
@@ -222,9 +226,10 @@ jobs:
           echo "" >> $GITHUB_STEP_SUMMARY
           echo "- ✅ TypeScript compilation successful" >> $GITHUB_STEP_SUMMARY
           echo "- ✅ Test suite validation passed" >> $GITHUB_STEP_SUMMARY
+          echo "- ✅ Deterministic tests passed (\`npm run test:ci\`)" >> $GITHUB_STEP_SUMMARY
           echo "" >> $GITHUB_STEP_SUMMARY
-          echo "**Note:** Full agent tests are not run on PRs to save time and costs." >> $GITHUB_STEP_SUMMARY
-          echo "Maintainers can run \`npm run test:ci\` locally if needed." >> $GITHUB_STEP_SUMMARY
+          echo "**Note:** \`test:ci\` runs the offline, deterministic Vitest allowlist only." >> $GITHUB_STEP_SUMMARY
+          echo "Model- or network-dependent agent tests (\`eval:sdk\`, integration suites) are not run on PRs." >> $GITHUB_STEP_SUMMARY
       
       - name: Failure summary
         if: failure()

+ 1 - 0
evals/framework/package.json

@@ -10,6 +10,7 @@
     "build:watch": "tsc --watch",
     "test": "vitest",
     "test:watch": "vitest --watch",
+    "test:ci": "vitest run src/__tests__/error-handling.test.ts src/evaluators/__tests__/evaluator-reliability.test.ts src/evaluators/__tests__/context-loading-evaluator.test.ts src/evaluators/__tests__/tool-usage-enhanced.test.ts src/sdk/__tests__/yaml-loader.test.ts src/sdk/__tests__/suite-validator.test.ts src/sdk/__tests__/validate-suites-cli.test.ts",
     "test:integration": "vitest --config vitest.integration.config.ts",
     "lint": "eslint src/**/*.ts",
     "lint:fix": "eslint src/**/*.ts --fix",