test(e2e): isolate the grafana generator into its own leg

The grafana generator talks to a live external Grafana Cloud instance, so its
readiness is flaky in a way the in-cluster generators are not. Split it out with
a label filter: the generator leg runs everything except grafana (scoped to aws
for the ecr and sts generators), and a dedicated grafana leg runs only grafana
(scoped to grafana). This keeps the external dependency from masking the rest of
the generator suite.

Signed-off-by: Alexander Chernov <alexander@chernov.it>

e2e/README.md

@@ -112,6 +112,14 @@ Which providers actually need external credentials: `fake`, `kubernetes`,
 `template`, `vault`, `openbao`, `conjur`, and `infisical` run against in-cluster
 addons and need none. The rest hit real APIs and are scoped to their group.
 
+The `generator` suite is split across two legs by label. The `generator` leg
+runs every generator except grafana (`!managed && !grafana`) and is scoped to
+`aws`, because the ecr and sts generators mint tokens against real AWS. The
+`grafana` leg (`grafana && !managed`, scoped to `grafana`) is isolated on its
+own because the grafana generator depends on a live external Grafana Cloud
+instance; keeping it separate means its external flakiness is attributable and
+never masks the other generators.
+
 ## The `e2e-required` gate
 
 The individual leg names change as the matrix grows, which makes them a poor

e2e/matrix.yaml

@@ -173,13 +173,18 @@ areas:
 
   # Non-provider suites. No provider labels, so providers is empty; they run the
   # whole non-managed suite. flux and argocd exercise ESO under GitOps tools and
-  # need no creds. The generator suite needs grafana (grafana generator) and aws
-  # (the ecr and sts generators mint tokens against real AWS).
+  # need no creds.
+  #
+  # The generator suite is split in two: the grafana generator talks to a live
+  # external Grafana Cloud instance (its own leg below), while every other
+  # generator runs in-cluster. This leg runs all generators except grafana; the
+  # ecr and sts generators mint tokens against real AWS, so it needs the aws
+  # group.
   - name: generator
     suite: generator
-    labels: "!managed"
+    labels: "!managed && !grafana"
     providers: []
-    secret_groups: [grafana, aws]
+    secret_groups: [aws]
     needs_secrets: true
     paths:
       - "apis/**"
@@ -189,6 +194,21 @@ areas:
       - "e2e/suites/generator/**"
     enabled: true
 
+  # The grafana generator, isolated. It depends on a live external Grafana Cloud
+  # instance (GRAFANA_URL / GRAFANA_TOKEN); the suite even wakes a sleepy
+  # instance in BeforeEach. Kept as its own leg so its external flakiness is
+  # attributable and never masks the other generators.
+  - name: grafana
+    suite: generator
+    labels: "grafana && !managed"
+    providers: []
+    secret_groups: [grafana]
+    needs_secrets: true
+    paths:
+      - "generators/v1/grafana/**"
+      - "e2e/suites/generator/grafana.go"
+    enabled: true
+
   - name: flux
     suite: flux
     labels: "!managed"