feat(e2e): implement fan-out matrix validation and credential scoping (#6660)

* feat(e2e): implement fan-out matrix validation and credential scoping

Signed-off-by: Alexander Chernov <alexander@chernov.it>

* test(e2e): split vault family into per-provider legs

The grouped leg ran vault, openbao and conjur together, so a broken conjur
addon (issue #6657) timed out the whole leg and the name did not match what
ran. Give each in-cluster backend its own leg for isolation and to match the
per-provider goal.

Signed-off-by: Alexander Chernov <alexander@chernov.it>

* test(e2e): enable the full provider fan-out matrix

Turn on the remaining legs (gcp, azure, scaleway, delinea, secretserver,
infisical, generator, flux, argocd) now that the mechanism is validated. Each
cloud leg is scoped to its own secret group; infisical, flux and argocd run
against in-cluster addons only.

Signed-off-by: Alexander Chernov <alexander@chernov.it>

* docs(e2e): document the e2e CI fan-out

Explain the matrix, per-leg credential scoping, the e2e-required gate, the
trusted/fork/managed run paths, and how to add or enable a provider.

Signed-off-by: Alexander Chernov <alexander@chernov.it>

* fix(e2e): build chart deps in HelmServer before packaging

The charts/ subchart tarballs are gitignored, so on a fresh checkout the ESO
chart's charts/ is empty and 'helm package' fails with 'missing in charts/
directory: bitwarden-sdk-server'. The provider and argocd suites hide this
because an earlier HelmChart addon runs 'helm dependency update' on the chart
path first; a suite that only uses HelmServer (flux) has nothing to populate
charts/. Build the deps from the committed Chart.lock in Setup so the addon is
self-contained.

Signed-off-by: Alexander Chernov <alexander@chernov.it>

* test(e2e): give the generator leg aws credentials

The ecr and sts generators mint tokens against real AWS, so the generator leg
needs the aws secret group in addition to grafana. Without it the ecr spec
failed once the leg ran in isolation.

Signed-off-by: Alexander Chernov <alexander@chernov.it>

* test(e2e): isolate the grafana generator into its own leg

The grafana generator talks to a live external Grafana Cloud instance, so its
readiness is flaky in a way the in-cluster generators are not. Split it out with
a label filter: the generator leg runs everything except grafana (scoped to aws
for the ecr and sts generators), and a dedicated grafana leg runs only grafana
(scoped to grafana). This keeps the external dependency from masking the rest of
the generator suite.

Signed-off-by: Alexander Chernov <alexander@chernov.it>

---------

Signed-off-by: Alexander Chernov <alexander@chernov.it>

.github/workflows/e2e-reusable.yml

@@ -104,6 +104,37 @@ env:
   VERSION: "e2e"
 
 jobs:
+  # Turn e2e/matrix.yaml into the test job's strategy matrix. Validating here
+  # (check-matrix.sh) fails the run early if a provider was added to the suite
+  # without a covering leg, rather than letting it go silently untested.
+  prepare-matrix:
+    runs-on: ubuntu-latest
+    permissions:
+      contents: read
+    outputs:
+      matrix: ${{ steps.set.outputs.matrix }}
+    steps:
+      - uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
+        with:
+          egress-policy: audit
+
+      - name: Checkout
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
+        with:
+          ref: ${{ env.TARGET_SHA || github.sha }}
+          persist-credentials: false
+
+      - name: Validate and build the e2e matrix
+        id: set
+        # This job has no secrets in scope. matrix.py reads only matrix.yaml and
+        # the workflow text, so the plan below proves per-leg credential scoping
+        # without ever touching a secret value.
+        run: |
+          ./e2e/matrix.py check
+          ./e2e/matrix.py plan
+          matrix="$(./e2e/matrix.py json)"
+          echo "matrix=${matrix}" >> "$GITHUB_OUTPUT"
+
   build:
     runs-on: ubuntu-latest
     permissions:
@@ -176,16 +207,30 @@ jobs:
           retention-days: 1
 
   test:
-    needs: build
+    needs: [build, prepare-matrix]
+    # One leg per enabled area in e2e/matrix.yaml. Each leg builds its own kind
+    # cluster and runs a single suite under one label filter, so a flaky addon
+    # in one provider cannot fail the others. fail-fast is off so one red leg
+    # does not cancel the rest.
+    strategy:
+      fail-fast: false
+      matrix: ${{ fromJSON(needs.prepare-matrix.outputs.matrix) }}
+    name: test (${{ matrix.name }})
     runs-on: ubuntu-latest
     permissions:
       id-token: write # for oidc auth with aws/gcp/azure
       contents: read  # for checkout
     env:
-      # Role ARN (an identifier, not a credential) lives at job level so the AWS
-      # step can be skipped when it is absent, e.g. fork runs without secrets.
-      # The actual provider credentials stay scoped to the Run e2e step below.
-      AWS_OIDC_ROLE_ARN: ${{ secrets.AWS_OIDC_ROLE_ARN }}
+      # AWS_OIDC_ROLE_ARN is an identifier, not a credential, but it is still
+      # injected only for legs whose secret_groups include "aws", so the
+      # Configure AWS step (and AWS auth) is skipped on every other leg. The
+      # per-provider credentials are scoped the same way, per leg, in the Run
+      # e2e step below: a vault or core-smoke leg receives no cloud secrets.
+      AWS_OIDC_ROLE_ARN: ${{ contains(matrix.secret_groups, 'aws') && secrets.AWS_OIDC_ROLE_ARN || '' }}
+      # Selects the suite binary and label filter for this leg. run.sh forwards
+      # both into the e2e pod; entrypoint.sh runs ginkgo with them.
+      TEST_SUITES: ${{ matrix.suite }}
+      GINKGO_LABELS: ${{ matrix.labels }}
     steps:
       - uses: step-security/harden-runner@bf7454d06d71f1098171f2acdf0cd4708d7b5920 # v2.20.0
         with:
@@ -219,43 +264,47 @@ jobs:
           path: e2e/image-artifacts
 
       - name: Run e2e
+        # Each provider's secrets are injected only when this leg's
+        # secret_groups (from e2e/matrix.yaml) lists that group; otherwise the
+        # value is empty. So a leg receives exactly the credentials it needs and
+        # nothing else, instead of every leg seeing every secret.
         env:
-          GCP_SERVICE_ACCOUNT_KEY: ${{ secrets.GCP_SERVICE_ACCOUNT_KEY }}
-          GCP_FED_REGION: ${{ secrets.GCP_FED_REGION }}
-          GCP_GSA_NAME: ${{ secrets.GCP_GSA_NAME }}
-          GCP_KSA_NAME: ${{ secrets.GCP_KSA_NAME }}
-          GCP_FED_PROJECT_ID: ${{ secrets.GCP_FED_PROJECT_ID }}
-          AWS_SA_NAME: ${{ secrets.AWS_SA_NAME }}
-          AWS_SA_NAMESPACE: ${{ secrets.AWS_SA_NAMESPACE }}
-          TFC_AZURE_CLIENT_ID: ${{ secrets.TFC_AZURE_CLIENT_ID }}
-          TFC_AZURE_CLIENT_SECRET: ${{ secrets.TFC_AZURE_CLIENT_SECRET }}
-          TFC_AZURE_TENANT_ID: ${{ secrets.TFC_AZURE_TENANT_ID }}
-          TFC_AZURE_SUBSCRIPTION_ID: ${{ secrets.TFC_AZURE_SUBSCRIPTION_ID }}
-          TFC_VAULT_URL: ${{ secrets.TFC_VAULT_URL }}
-          SCALEWAY_API_URL: ${{ secrets.SCALEWAY_API_URL }}
-          SCALEWAY_REGION: ${{ secrets.SCALEWAY_REGION }}
-          SCALEWAY_PROJECT_ID: ${{ secrets.SCALEWAY_PROJECT_ID }}
-          SCALEWAY_ACCESS_KEY: ${{ secrets.SCALEWAY_ACCESS_KEY }}
-          SCALEWAY_SECRET_KEY: ${{ secrets.SCALEWAY_SECRET_KEY }}
-          DELINEA_TLD: ${{ secrets.DELINEA_TLD }}
-          DELINEA_URL_TEMPLATE: ${{ secrets.DELINEA_URL_TEMPLATE }}
-          DELINEA_TENANT: ${{ secrets.DELINEA_TENANT }}
-          DELINEA_CLIENT_ID: ${{ secrets.DELINEA_CLIENT_ID }}
-          DELINEA_CLIENT_SECRET: ${{ secrets.DELINEA_CLIENT_SECRET }}
-          SECRETSERVER_USERNAME: ${{ secrets.SECRETSERVER_USERNAME }}
-          SECRETSERVER_PASSWORD: ${{ secrets.SECRETSERVER_PASSWORD }}
-          SECRETSERVER_URL: ${{ secrets.SECRETSERVER_URL }}
-          GRAFANA_URL: ${{ secrets.GRAFANA_URL }}
-          GRAFANA_TOKEN: ${{ secrets.GRAFANA_TOKEN }}
-          AKEYLESS_ACCESS_ID: ${{ secrets.AKEYLESS_ACCESS_ID }}
-          AKEYLESS_ACCESS_TYPE: ${{ secrets.AKEYLESS_ACCESS_TYPE }}
-          AKEYLESS_ACCESS_TYPE_PARAM: ${{ secrets.AKEYLESS_ACCESS_TYPE_PARAM }}
-          GITLAB_TOKEN: ${{ secrets.GITLAB_TOKEN }}
-          GITLAB_PROJECT_ID: ${{ secrets.GITLAB_PROJECT_ID }}
-          GITLAB_ENVIRONMENT: ${{ secrets.GITLAB_ENVIRONMENT }}
-          ORACLE_USER_OCID: ${{ secrets.ORACLE_USER_OCID }}
-          ORACLE_TENANCY_OCID: ${{ secrets.ORACLE_TENANCY_OCID }}
-          ORACLE_REGION: ${{ secrets.ORACLE_REGION }}
-          ORACLE_FINGERPRINT: ${{ secrets.ORACLE_FINGERPRINT }}
-          ORACLE_KEY: ${{ secrets.ORACLE_KEY }}
+          GCP_SERVICE_ACCOUNT_KEY: ${{ contains(matrix.secret_groups, 'gcp') && secrets.GCP_SERVICE_ACCOUNT_KEY || '' }}
+          GCP_FED_REGION: ${{ contains(matrix.secret_groups, 'gcp') && secrets.GCP_FED_REGION || '' }}
+          GCP_GSA_NAME: ${{ contains(matrix.secret_groups, 'gcp') && secrets.GCP_GSA_NAME || '' }}
+          GCP_KSA_NAME: ${{ contains(matrix.secret_groups, 'gcp') && secrets.GCP_KSA_NAME || '' }}
+          GCP_FED_PROJECT_ID: ${{ contains(matrix.secret_groups, 'gcp') && secrets.GCP_FED_PROJECT_ID || '' }}
+          AWS_SA_NAME: ${{ contains(matrix.secret_groups, 'aws') && secrets.AWS_SA_NAME || '' }}
+          AWS_SA_NAMESPACE: ${{ contains(matrix.secret_groups, 'aws') && secrets.AWS_SA_NAMESPACE || '' }}
+          TFC_AZURE_CLIENT_ID: ${{ contains(matrix.secret_groups, 'azure') && secrets.TFC_AZURE_CLIENT_ID || '' }}
+          TFC_AZURE_CLIENT_SECRET: ${{ contains(matrix.secret_groups, 'azure') && secrets.TFC_AZURE_CLIENT_SECRET || '' }}
+          TFC_AZURE_TENANT_ID: ${{ contains(matrix.secret_groups, 'azure') && secrets.TFC_AZURE_TENANT_ID || '' }}
+          TFC_AZURE_SUBSCRIPTION_ID: ${{ contains(matrix.secret_groups, 'azure') && secrets.TFC_AZURE_SUBSCRIPTION_ID || '' }}
+          TFC_VAULT_URL: ${{ contains(matrix.secret_groups, 'azure') && secrets.TFC_VAULT_URL || '' }}
+          SCALEWAY_API_URL: ${{ contains(matrix.secret_groups, 'scaleway') && secrets.SCALEWAY_API_URL || '' }}
+          SCALEWAY_REGION: ${{ contains(matrix.secret_groups, 'scaleway') && secrets.SCALEWAY_REGION || '' }}
+          SCALEWAY_PROJECT_ID: ${{ contains(matrix.secret_groups, 'scaleway') && secrets.SCALEWAY_PROJECT_ID || '' }}
+          SCALEWAY_ACCESS_KEY: ${{ contains(matrix.secret_groups, 'scaleway') && secrets.SCALEWAY_ACCESS_KEY || '' }}
+          SCALEWAY_SECRET_KEY: ${{ contains(matrix.secret_groups, 'scaleway') && secrets.SCALEWAY_SECRET_KEY || '' }}
+          DELINEA_TLD: ${{ contains(matrix.secret_groups, 'delinea') && secrets.DELINEA_TLD || '' }}
+          DELINEA_URL_TEMPLATE: ${{ contains(matrix.secret_groups, 'delinea') && secrets.DELINEA_URL_TEMPLATE || '' }}
+          DELINEA_TENANT: ${{ contains(matrix.secret_groups, 'delinea') && secrets.DELINEA_TENANT || '' }}
+          DELINEA_CLIENT_ID: ${{ contains(matrix.secret_groups, 'delinea') && secrets.DELINEA_CLIENT_ID || '' }}
+          DELINEA_CLIENT_SECRET: ${{ contains(matrix.secret_groups, 'delinea') && secrets.DELINEA_CLIENT_SECRET || '' }}
+          SECRETSERVER_USERNAME: ${{ contains(matrix.secret_groups, 'secretserver') && secrets.SECRETSERVER_USERNAME || '' }}
+          SECRETSERVER_PASSWORD: ${{ contains(matrix.secret_groups, 'secretserver') && secrets.SECRETSERVER_PASSWORD || '' }}
+          SECRETSERVER_URL: ${{ contains(matrix.secret_groups, 'secretserver') && secrets.SECRETSERVER_URL || '' }}
+          GRAFANA_URL: ${{ contains(matrix.secret_groups, 'grafana') && secrets.GRAFANA_URL || '' }}
+          GRAFANA_TOKEN: ${{ contains(matrix.secret_groups, 'grafana') && secrets.GRAFANA_TOKEN || '' }}
+          AKEYLESS_ACCESS_ID: ${{ contains(matrix.secret_groups, 'akeyless') && secrets.AKEYLESS_ACCESS_ID || '' }}
+          AKEYLESS_ACCESS_TYPE: ${{ contains(matrix.secret_groups, 'akeyless') && secrets.AKEYLESS_ACCESS_TYPE || '' }}
+          AKEYLESS_ACCESS_TYPE_PARAM: ${{ contains(matrix.secret_groups, 'akeyless') && secrets.AKEYLESS_ACCESS_TYPE_PARAM || '' }}
+          GITLAB_TOKEN: ${{ contains(matrix.secret_groups, 'gitlab') && secrets.GITLAB_TOKEN || '' }}
+          GITLAB_PROJECT_ID: ${{ contains(matrix.secret_groups, 'gitlab') && secrets.GITLAB_PROJECT_ID || '' }}
+          GITLAB_ENVIRONMENT: ${{ contains(matrix.secret_groups, 'gitlab') && secrets.GITLAB_ENVIRONMENT || '' }}
+          ORACLE_USER_OCID: ${{ contains(matrix.secret_groups, 'oracle') && secrets.ORACLE_USER_OCID || '' }}
+          ORACLE_TENANCY_OCID: ${{ contains(matrix.secret_groups, 'oracle') && secrets.ORACLE_TENANCY_OCID || '' }}
+          ORACLE_REGION: ${{ contains(matrix.secret_groups, 'oracle') && secrets.ORACLE_REGION || '' }}
+          ORACLE_FINGERPRINT: ${{ contains(matrix.secret_groups, 'oracle') && secrets.ORACLE_FINGERPRINT || '' }}
+          ORACLE_KEY: ${{ contains(matrix.secret_groups, 'oracle') && secrets.ORACLE_KEY || '' }}
         run: make -C e2e test.run

.github/workflows/e2e.yml

@@ -221,3 +221,35 @@ jobs:
         issue-number: ${{ github.event.client_payload.pull_request.number }}
         body: |
             [Bot] - :x: [e2e for ${{ env.TARGET_SHA }} failed](https://github.com/external-secrets/external-secrets/actions/runs/${{ github.run_id }})
+
+  # Single stable status to point branch protection at. The individual matrix
+  # legs (and their names) change as e2e/matrix.yaml grows, but this gate does
+  # not. It passes when the e2e path that ran for this event succeeded; the
+  # other path is skipped and does not count against it, and a genuinely failed
+  # leg propagates up through its caller job and fails this gate.
+  e2e-required:
+    if: always()
+    needs: [integration-trusted, integration-fork]
+    runs-on: ubuntu-latest
+    permissions:
+      contents: read
+    steps:
+    - uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
+      with:
+        egress-policy: audit
+
+    - name: Require the e2e path to have passed
+      env:
+        TRUSTED: ${{ needs.integration-trusted.result }}
+        FORK: ${{ needs.integration-fork.result }}
+      run: |
+        echo "integration-trusted=${TRUSTED} integration-fork=${FORK}"
+        for r in "${TRUSTED}" "${FORK}"; do
+          case "$r" in
+            failure|cancelled)
+              echo "::error::an e2e path did not pass (${r})"
+              exit 1
+              ;;
+          esac
+        done
+        echo "e2e ok (success or skipped on both paths)"

.gitignore

@@ -32,6 +32,10 @@ e2e/suites/flux/flux.test
 e2e/suites/provider/provider.test
 e2e/suites/generator/generator.test
 
+# python bytecode cache (e.g. from e2e/matrix.py)
+__pycache__/
+*.pyc
+
 # tf ignores
 # Local .terraform directories
 **/.terraform/*

e2e/Makefile

@@ -67,6 +67,12 @@ test.managed: e2e-image ## Run e2e tests against current kube context
 	./run.sh
 
 
+matrix.check: ## Validate matrix.yaml (coverage, secret scoping consistency)
+	./matrix.py check
+
+matrix.plan: ## Show the credential env vars each enabled e2e leg receives
+	./matrix.py plan
+
 e2e-bin: install-ginkgo
 	   CGO_ENABLED=0 GOOS=linux GOARCH=amd64 ginkgo build ./suites/...
 

e2e/README.md

@@ -0,0 +1,174 @@
+# End-to-end (e2e) tests and CI
+
+This document explains how the e2e suite is built and run in CI after the
+fan-out change: what the moving parts are, how a run flows, how credentials are
+scoped per provider, and how to add or enable a provider.
+
+## Goals
+
+- Run the core controller behaviour and each provider as **separate CI legs**,
+  so a flaky or broken addon in one provider fails only its own leg.
+- Give CI a **single, stable required status** even though the set of legs
+  changes over time.
+- Give each leg **only the credentials it needs**, so a leg that tests one
+  provider never has another provider's secrets in its environment.
+
+## The pieces
+
+| File | Role |
+| ---- | ---- |
+| `e2e/suites/<suite>/` | Ginkgo suites, compiled to `<suite>.test` binaries: `provider`, `generator`, `flux`, `argocd`. |
+| `e2e/suites/provider/cases/import.go` | Blank-imports every provider case into the single `provider.test` binary. Providers are told apart at run time by Ginkgo label. |
+| `e2e/matrix.yaml` | Source of truth for the fan-out: one `area` (leg) per provider, with its suite, label filter, secret groups, and trigger paths. |
+| `e2e/matrix.py` | Validates the matrix (`check`), emits the CI matrix JSON (`json`), and prints the per-leg credential plan (`plan`). |
+| `e2e/run.sh` | Host-side launcher. Runs `kubectl run` to start the e2e pod, forwarding `TEST_SUITES`, `GINKGO_LABELS`, and the (scoped) credentials as pod env. |
+| `e2e/entrypoint.sh` | In-pod entry (image `CMD`). Loops over `TEST_SUITES` and runs `ginkgo -label-filter="$GINKGO_LABELS"` against each `<suite>.test`. |
+| `.github/workflows/e2e.yml` | Non-managed e2e. Fans out into per-provider legs. Owns the `e2e-required` gate. |
+| `.github/workflows/e2e-reusable.yml` | The reusable build + matrix-test pipeline that `e2e.yml` calls. |
+| `.github/workflows/e2e-managed.yml` | Managed e2e (real cloud IRSA / workload-identity), run on demand via `/ok-to-test-managed`. Already per-provider. |
+
+## How a run flows
+
+```mermaid
+flowchart TD
+    T[pull_request or /ok-to-test] --> P[prepare-matrix: matrix.py check + plan + json]
+    T --> B[build: compile controller + e2e images once, upload tarball]
+    P --> M{fan out over enabled areas}
+    B --> M
+    M --> L1[test core-smoke]
+    M --> L2[test vault]
+    M --> L3[test aws]
+    M --> Ln[test ...]
+    L1 --> R[e2e-required]
+    L2 --> R
+    L3 --> R
+    Ln --> R
+    R --> G[single stable green/red status]
+```
+
+1. **prepare-matrix** runs `matrix.py check` (fail early if the matrix is
+   inconsistent), prints the credential `plan`, and emits the enabled-areas
+   matrix as JSON. This job has no secrets in scope.
+2. **build** compiles the controller and e2e images once and uploads them as a
+   tarball artifact. The test legs load that tarball; they need no Go toolchain.
+3. **test** is a `fail-fast: false` matrix over the enabled areas. Each leg gets
+   its own runner and its own kind cluster, loads the shared image tarball, and
+   runs one suite under one label filter (`TEST_SUITES` + `GINKGO_LABELS`).
+4. **e2e-required** aggregates the result into one status (see below).
+
+## The matrix (`e2e/matrix.yaml`)
+
+Each `area` is one leg:
+
+```yaml
+- name: aws                       # leg id, shown as "test (aws)"
+  suite: provider                 # which suite binary (TEST_SUITES)
+  labels: "aws && !managed"       # Ginkgo -label-filter (GINKGO_LABELS)
+  providers: [aws]                # for the coverage check only
+  secret_groups: [aws]            # which credential groups this leg receives
+  needs_secrets: true             # mirror of "secret_groups is non-empty"
+  paths:                          # phase 2 seed (affected-only), unused today
+    - "providers/v1/aws/**"
+    - "e2e/suites/provider/cases/aws/**"
+  enabled: true                   # whether the phase 1 matrix runs it now
+```
+
+Notes:
+
+- **One suite per leg.** A label filter never has to span binaries that lack
+  the labels. Provider legs use `suite: provider`; `generator`, `flux`, and
+  `argocd` are their own suites.
+- **`!managed` everywhere.** This workflow runs only the non-managed specs; the
+  managed IRSA / workload-identity specs run in `e2e-managed.yml`.
+- **`enabled`** lets the matrix grow gradually. Disabled areas still count for
+  coverage and document the intended full matrix; flip to `true` to run them.
+
+## Credential scoping (no secret spread)
+
+A leg receives a provider's secrets only when its `secret_groups` lists that
+group. In `e2e-reusable.yml` every secret is gated:
+
+```yaml
+GCP_SERVICE_ACCOUNT_KEY: ${{ contains(matrix.secret_groups, 'gcp') && secrets.GCP_SERVICE_ACCOUNT_KEY || '' }}
+```
+
+So a `vault` or `core-smoke` leg (`secret_groups: []`) gets empty strings for
+every cloud credential, and the `aws` leg gets only the `aws` group. The
+group -> variable mapping lives in `e2e-reusable.yml`.
+
+Scoping is proven **without reading any secret**: `matrix.py plan` runs in
+`prepare-matrix` (which has no secrets in scope) and derives each leg's
+credential list from `matrix.yaml` plus the group mapping parsed out of the
+workflow text. It never references the `secrets` context, so nothing depends on
+GitHub's log masking. Example:
+
+```
+core-smoke: groups=[] -> (none: in-cluster only)
+vault:      groups=[] -> (none: in-cluster only)
+aws:        groups=['aws'] -> AWS_OIDC_ROLE_ARN, AWS_SA_NAME, AWS_SA_NAMESPACE
+```
+
+Which providers actually need external credentials: `fake`, `kubernetes`,
+`template`, `vault`, `openbao`, `conjur`, and `infisical` run against in-cluster
+addons and need none. The rest hit real APIs and are scoped to their group.
+
+The `generator` suite is split across two legs by label. The `generator` leg
+runs every generator except grafana (`!managed && !grafana`) and is scoped to
+`aws`, because the ecr and sts generators mint tokens against real AWS. The
+`grafana` leg (`grafana && !managed`, scoped to `grafana`) is isolated on its
+own because the grafana generator depends on a live external Grafana Cloud
+instance; keeping it separate means its external flakiness is attributable and
+never masks the other generators.
+
+## The `e2e-required` gate
+
+The individual leg names change as the matrix grows, which makes them a poor
+target for branch protection. `e2e-required` (in `e2e.yml`) is one job that
+`needs` the trusted and fork callers and reports a single status:
+
+- passes when the e2e path that ran for this event succeeded,
+- treats the other (skipped) path as a non-failure,
+- fails if any leg failed or was cancelled (a failed leg propagates up through
+  its caller job).
+
+Point branch protection at `e2e-required` and it stays stable regardless of how
+many legs exist.
+
+## Trusted vs fork runs
+
+- **Same-repo PR** (`integration-trusted`): runs automatically with secrets.
+- **Fork PR**: a maintainer comments `/ok-to-test sha=<40-char-sha>` (or submits
+  a review whose body contains `/ok-to-test`, which pins the reviewed commit).
+  `guard-fork` rejects a bare command without a pinned SHA; `integration-fork`
+  then runs. Note that the fork path runs the workflow from `main`, not from the
+  PR branch.
+- **Managed** (`e2e-managed.yml`): `/ok-to-test-managed`, one job per cloud
+  provider, `GINKGO_LABELS="<provider> && managed"`.
+
+## Local usage
+
+```bash
+# validate the matrix (coverage, secret-scoping consistency, wiring)
+make -C e2e matrix.check
+
+# show which credentials each enabled leg will receive (reads no secrets)
+make -C e2e matrix.plan
+
+# run a single provider locally (overrides the Makefile defaults)
+make -C e2e test.run TEST_SUITES=provider GINKGO_LABELS="vault && !managed"
+```
+
+## Adding or enabling a provider
+
+1. Add the provider case under `e2e/suites/provider/cases/<name>/` and blank
+   import it in `import.go`.
+2. Add an `area` for it in `matrix.yaml` (its label, `providers: [<name>]`, and
+   `paths`).
+3. If it needs external credentials, add its secret group to `secret_groups`
+   and wire that group's env vars in `e2e-reusable.yml`.
+4. Set `enabled: true` when you want CI to run it.
+
+`matrix.py check` (run in `prepare-matrix`) enforces steps 1-3: it fails the
+build if a provider is compiled into the suite but not covered by an area, if
+`needs_secrets` disagrees with `secret_groups`, or if an area names a secret
+group that the workflow does not wire.

e2e/framework/addon/helmserver.go

@@ -48,10 +48,21 @@ func (s *HelmServer) Setup(config *Config) error {
 		return err
 	}
 
+	// The chart's charts/ subchart tarballs are gitignored, so a fresh
+	// checkout packages an empty charts/ and `helm package` fails with
+	// "missing in charts/ directory". Build deps from the committed
+	// Chart.lock first so this addon is self-contained: a suite that uses
+	// only HelmServer (e.g. flux) has nothing else to populate charts/.
+	cmd := exec.Command("helm", "dependency", "build", s.ChartDir)
+	out, err := cmd.CombinedOutput()
+	if err != nil {
+		return fmt.Errorf("unable to build helm chart dependencies: %w %s", err, string(out))
+	}
+
 	// nolint:gosec
-	cmd := exec.Command("helm", "package", s.ChartDir, "--version", s.ChartRevision)
+	cmd = exec.Command("helm", "package", s.ChartDir, "--version", s.ChartRevision)
 	cmd.Dir = s.serveDir
-	out, err := cmd.CombinedOutput()
+	out, err = cmd.CombinedOutput()
 	if err != nil {
 		return fmt.Errorf("unable to package helm chart: %w %s", err, string(out))
 	}

e2e/matrix.py

@@ -0,0 +1,159 @@
+#!/usr/bin/env python3
+"""Validate and render the e2e fan-out matrix defined in e2e/matrix.yaml.
+
+Subcommands:
+  check   Fail early if the matrix is inconsistent: a provider compiled into
+          the suite (suites/provider/cases/import.go) is not covered by any
+          area, needs_secrets disagrees with secret_groups, or an area names a
+          secret group that the reusable workflow does not wire up.
+  json    Print the GitHub Actions matrix (enabled areas only) as compact JSON
+          for the workflow's strategy.matrix.
+  plan    Print, per enabled leg, exactly which credential env vars it will
+          receive. Derived from each area's secret_groups and the group -> var
+          mapping parsed out of e2e-reusable.yml. This reads NO secret values
+          (it never touches the secrets context), so it proves the scoping
+          without any risk of leaking a value, masked or not.
+
+Paths are resolved relative to this file, so the working directory does not
+matter. YAML is read with PyYAML when present, else via yq (mikefarah), so no
+new runtime dependency is required in CI.
+"""
+
+import json
+import re
+import subprocess
+import sys
+from pathlib import Path
+
+HERE = Path(__file__).resolve().parent
+MATRIX = HERE / "matrix.yaml"
+IMPORT = HERE / "suites/provider/cases/import.go"
+WORKFLOW = HERE.parent / ".github/workflows/e2e-reusable.yml"
+
+
+def load_yaml(path: Path):
+    """Load a YAML file as a dict. Prefer PyYAML; fall back to yq -> JSON."""
+    try:
+        import yaml  # type: ignore
+        return yaml.safe_load(path.read_text())
+    except ModuleNotFoundError:
+        out = subprocess.run(
+            ["yq", "-o=json", str(path)],
+            check=True, capture_output=True, text=True,
+        ).stdout
+        return json.loads(out)
+
+
+def imported_providers() -> list[str]:
+    """Provider names compiled into the suite: the segment after cases/ in
+    each blank import of import.go (cases/aws/secretsmanager -> aws)."""
+    text = IMPORT.read_text()
+    return sorted({m.group(1) for m in re.finditer(r"cases/([a-z0-9]+)", text)})
+
+
+def group_to_vars() -> dict[str, list[str]]:
+    """Map each secret group to the env vars the reusable workflow gates on it,
+    parsed from lines like:
+        FOO: ${{ contains(matrix.secret_groups, 'aws') && secrets.BAR || '' }}
+    Reads only the workflow text, never any secret value."""
+    pat = re.compile(
+        r"^\s*([A-Z0-9_]+):\s*\$\{\{\s*"
+        r"contains\(matrix\.secret_groups,\s*'([a-z0-9]+)'\)",
+        re.MULTILINE,
+    )
+    mapping: dict[str, list[str]] = {}
+    for var, group in pat.findall(WORKFLOW.read_text()):
+        mapping.setdefault(group, []).append(var)
+    for group in mapping:
+        mapping[group].sort()
+    return mapping
+
+
+def cmd_check(matrix: dict) -> int:
+    areas = matrix["areas"]
+    errors: list[str] = []
+
+    # 1. Every imported provider is covered by some area.
+    covered = {p for a in areas for p in (a.get("providers") or [])}
+    missing = [p for p in imported_providers() if p not in covered]
+    if missing:
+        errors.append(
+            "providers imported into the e2e suite but not covered by any "
+            "area (add each to an area's providers list and a leg):\n  - "
+            + "\n  - ".join(missing)
+        )
+
+    # 2. needs_secrets must mirror "secret_groups is non-empty".
+    for a in areas:
+        has_groups = bool(a.get("secret_groups"))
+        if bool(a.get("needs_secrets")) != has_groups:
+            errors.append(
+                f"area {a['name']!r}: needs_secrets={a.get('needs_secrets')} "
+                f"disagrees with secret_groups={a.get('secret_groups')}"
+            )
+
+    # 3. Every secret group an area uses is actually wired in the workflow.
+    wired = set(group_to_vars())
+    for a in areas:
+        for group in a.get("secret_groups") or []:
+            if group not in wired:
+                errors.append(
+                    f"area {a['name']!r}: secret group {group!r} is not wired "
+                    f"in {WORKFLOW.name} (no env var gates on it)"
+                )
+
+    if errors:
+        print("ERROR: matrix.yaml is inconsistent:", file=sys.stderr)
+        for e in errors:
+            print(f"- {e}", file=sys.stderr)
+        return 1
+
+    enabled = sum(1 for a in areas if a.get("enabled"))
+    print(
+        f"matrix.yaml ok: {len(imported_providers())} providers covered, "
+        f"{enabled} leg(s) enabled"
+    )
+    return 0
+
+
+def cmd_json(matrix: dict) -> int:
+    include = [
+        {
+            "name": a["name"],
+            "suite": a["suite"],
+            "labels": a["labels"],
+            "secret_groups": a.get("secret_groups") or [],
+        }
+        for a in matrix["areas"]
+        if a.get("enabled")
+    ]
+    print(json.dumps({"include": include}, separators=(",", ":")))
+    return 0
+
+
+def cmd_plan(matrix: dict) -> int:
+    """Show the credential env vars each enabled leg will receive. No secret
+    values are read; the list comes from matrix.yaml + the workflow mapping."""
+    mapping = group_to_vars()
+    print("Per-leg credential scoping (from matrix.yaml + e2e-reusable.yml):")
+    for a in matrix["areas"]:
+        if not a.get("enabled"):
+            continue
+        groups = a.get("secret_groups") or []
+        env_vars = sorted({v for g in groups for v in mapping.get(g, [])})
+        shown = ", ".join(env_vars) if env_vars else "(none: in-cluster only)"
+        print(f"  {a['name']}: groups={groups or '[]'} -> {shown}")
+    return 0
+
+
+def main() -> int:
+    cmd = sys.argv[1] if len(sys.argv) > 1 else "check"
+    if cmd not in ("check", "json", "plan"):
+        print(f"usage: {sys.argv[0]} [check|json|plan]", file=sys.stderr)
+        return 2
+    matrix = load_yaml(MATRIX)
+    return {"check": cmd_check, "json": cmd_json, "plan": cmd_plan}[cmd](matrix)
+
+
+if __name__ == "__main__":
+    sys.exit(main())

e2e/matrix.yaml

@@ -0,0 +1,236 @@
+# Source of truth for the non-managed e2e fan-out (e2e.yml). Each area is one CI
+# leg: a suite binary run under a Ginkgo label filter. The build job compiles
+# the images once; every enabled area runs as its own matrix leg on its own kind
+# cluster, so a flaky addon in one provider cannot fail the others.
+#
+# Fields:
+#   name          leg id, shown as "test (<name>)" in the checks list.
+#   suite         which suite binary to run (TEST_SUITES). One suite per leg so a
+#                 label filter never has to span binaries that lack the labels.
+#   labels        Ginkgo -label-filter (GINKGO_LABELS). This workflow is
+#                 non-managed only; the managed IRSA/workload-identity specs run
+#                 in e2e-managed.yml, so every filter ends with "&& !managed".
+#   providers     provider case names this leg covers. Used only by
+#                 check-matrix.sh to prove every provider compiled into the suite
+#                 (suites/provider/cases/import.go) is tested somewhere. Not read
+#                 by CI at run time.
+#   secret_groups external credential groups this leg receives. The reusable
+#                 workflow injects a provider's secrets only when its group is
+#                 listed here, so a vault leg never sees AWS creds. Empty means
+#                 the leg runs against in-cluster addons only (fork-safe). Group
+#                 names map to secret sets in e2e-reusable.yml.
+#   needs_secrets convenience mirror of "secret_groups is non-empty". Documents
+#                 intent; not read by CI.
+#   paths         globs whose change should trigger this leg. Seed for the
+#                 phase 2 affected-only resolver; ignored by the static phase 1
+#                 matrix, which runs every enabled area.
+#   enabled       whether phase 1 CI runs this leg now. Disabled areas still
+#                 count for coverage and document the target matrix; flip to true
+#                 to expand the fan-out.
+#
+# All legs are enabled. Set enabled: false on an area to skip it (e.g. while a
+# provider's e2e is being fixed) without deleting its definition.
+
+areas:
+  # In-cluster only, no external credentials. The required floor: this leg runs
+  # on every PR (including forks) and is the stable branch-protection gate.
+  - name: core-smoke
+    suite: provider
+    labels: "(fake || kubernetes || template) && !managed"
+    providers: [fake, kubernetes, template]
+    secret_groups: []
+    needs_secrets: false
+    paths:
+      - "apis/**"
+      - "pkg/**"
+      - "runtime/**"
+      - "cmd/**"
+      - "deploy/**"
+      - "e2e/framework/**"
+      - "e2e/suites/provider/cases/fake/**"
+      - "e2e/suites/provider/cases/kubernetes/**"
+      - "e2e/suites/provider/cases/template/**"
+      - "e2e/suites/provider/cases/common/**"
+    enabled: true
+
+  # Self-hosted secret backends, installed as in-cluster addons (no cloud
+  # creds). One leg per provider so a broken addon (e.g. conjur) fails only its
+  # own leg, not the others, and the leg name matches what it runs.
+  - name: vault
+    suite: provider
+    labels: "vault && !managed"
+    providers: [vault]
+    secret_groups: []
+    needs_secrets: false
+    paths:
+      - "providers/v1/vault/**"
+      - "e2e/suites/provider/cases/vault/**"
+    enabled: true
+
+  - name: openbao
+    suite: provider
+    labels: "openbao && !managed"
+    providers: [openbao]
+    secret_groups: []
+    needs_secrets: false
+    paths:
+      - "providers/v1/openbao/**"
+      - "e2e/suites/provider/cases/openbao/**"
+    enabled: true
+
+  - name: conjur
+    suite: provider
+    labels: "conjur && !managed"
+    providers: [conjur]
+    secret_groups: []
+    needs_secrets: false
+    paths:
+      - "providers/v1/conjur/**"
+      - "e2e/suites/provider/cases/conjur/**"
+    enabled: true
+
+  # Cloud provider, non-managed specs. Needs AWS OIDC credentials.
+  - name: aws
+    suite: provider
+    labels: "aws && !managed"
+    providers: [aws]
+    secret_groups: [aws]
+    needs_secrets: true
+    paths:
+      - "providers/v1/aws/**"
+      - "e2e/suites/provider/cases/aws/**"
+    enabled: true
+
+  # ---- Cloud / external-API providers (each scoped to its own secret group). ----
+
+  - name: gcp
+    suite: provider
+    labels: "gcp && !managed"
+    providers: [gcp]
+    secret_groups: [gcp]
+    needs_secrets: true
+    paths:
+      - "providers/v1/gcp/**"
+      - "e2e/suites/provider/cases/gcp/**"
+    enabled: true
+
+  - name: azure
+    suite: provider
+    labels: "azure && !managed"
+    providers: [azure]
+    secret_groups: [azure]
+    needs_secrets: true
+    paths:
+      - "providers/v1/azure/**"
+      - "e2e/suites/provider/cases/azure/**"
+    enabled: true
+
+  - name: scaleway
+    suite: provider
+    labels: "scaleway && !managed"
+    providers: [scaleway]
+    secret_groups: [scaleway]
+    needs_secrets: true
+    paths:
+      - "providers/v1/scaleway/**"
+      - "e2e/suites/provider/cases/scaleway/**"
+    enabled: true
+
+  - name: delinea
+    suite: provider
+    labels: "delinea && !managed"
+    providers: [delinea]
+    secret_groups: [delinea]
+    needs_secrets: true
+    paths:
+      - "providers/v1/delinea/**"
+      - "e2e/suites/provider/cases/delinea/**"
+    enabled: true
+
+  - name: secretserver
+    suite: provider
+    labels: "secretserver && !managed"
+    providers: [secretserver]
+    secret_groups: [secretserver]
+    needs_secrets: true
+    paths:
+      - "providers/v1/secretserver/**"
+      - "e2e/suites/provider/cases/secretserver/**"
+    enabled: true
+
+  # Infisical runs against an in-cluster Infisical addon (addon.NewInfisical),
+  # not an external tenant, so it needs no repo secrets despite being a SaaS.
+  - name: infisical
+    suite: provider
+    labels: "infisical && !managed"
+    providers: [infisical]
+    secret_groups: []
+    needs_secrets: false
+    paths:
+      - "providers/v1/infisical/**"
+      - "e2e/suites/provider/cases/infisical/**"
+    enabled: true
+
+  # Non-provider suites. No provider labels, so providers is empty; they run the
+  # whole non-managed suite. flux and argocd exercise ESO under GitOps tools and
+  # need no creds.
+  #
+  # The generator suite is split in two: the grafana generator talks to a live
+  # external Grafana Cloud instance (its own leg below), while every other
+  # generator runs in-cluster. This leg runs all generators except grafana; the
+  # ecr and sts generators mint tokens against real AWS, so it needs the aws
+  # group.
+  - name: generator
+    suite: generator
+    labels: "!managed && !grafana"
+    providers: []
+    secret_groups: [aws]
+    needs_secrets: true
+    paths:
+      - "apis/**"
+      - "pkg/**"
+      - "runtime/**"
+      - "generators/**"
+      - "e2e/suites/generator/**"
+    enabled: true
+
+  # The grafana generator, isolated. It depends on a live external Grafana Cloud
+  # instance (GRAFANA_URL / GRAFANA_TOKEN); the suite even wakes a sleepy
+  # instance in BeforeEach. Kept as its own leg so its external flakiness is
+  # attributable and never masks the other generators.
+  - name: grafana
+    suite: generator
+    labels: "grafana && !managed"
+    providers: []
+    secret_groups: [grafana]
+    needs_secrets: true
+    paths:
+      - "generators/v1/grafana/**"
+      - "e2e/suites/generator/grafana.go"
+    enabled: true
+
+  - name: flux
+    suite: flux
+    labels: "!managed"
+    providers: []
+    secret_groups: []
+    needs_secrets: false
+    paths:
+      - "apis/**"
+      - "pkg/**"
+      - "runtime/**"
+      - "e2e/suites/flux/**"
+    enabled: true
+
+  - name: argocd
+    suite: argocd
+    labels: "!managed"
+    providers: []
+    secret_groups: []
+    needs_secrets: false
+    paths:
+      - "apis/**"
+      - "pkg/**"
+      - "runtime/**"
+      - "e2e/suites/argocd/**"
+    enabled: true