doc: revise and enhance Google Secret Manager authentication (#4430)

* doc: revise and enhance Google Secret Manager authentication

Signed-off-by: Kilian Kluge <dev@kluge.ai>

* fix(doc): add link to WIF issue on GitHub

Signed-off-by: Kilian Kluge <dev@kluge.ai>

---------

Signed-off-by: Kilian Kluge <dev@kluge.ai>
Co-authored-by: Gergely Brautigam <182850+Skarlso@users.noreply.github.com>

docs/provider/google-secrets-manager.md

@@ -1,118 +1,250 @@
-## Google Cloud Secret Manager
-
-External Secrets Operator integrates with [GCP Secret Manager](https://cloud.google.com/secret-manager) for secret management.
+External Secrets Operator integrates with the [Google Cloud Secret Manager](https://cloud.google.com/secret-manager).
 
 ## Authentication
 
-### Workload Identity
+### Workload Identity Federation
+
+Through [Workload Identity Federation](https://cloud.google.com/kubernetes-engine/docs/concepts/workload-identity) (WIF), [Google Kubernetes Engine](https://cloud.google.com/kubernetes-engine) (GKE) workloads can authenticate with Google Cloud Platform (GCP) services like Secret Manager without using static, long-lived credentials.
+
+Authenticating through WIF is the recommended approach when using the External Secrets Operator (ESO) on GKE clusters. ESO supports three options:
+
+- **Using a Kubernetes service account as a GCP IAM principal**: The `SecretStore` (or `ClusterSecretStore`) references a [Kubernetes service account](https://kubernetes.io/docs/concepts/security/service-accounts) that is authorized to access Secret Manager secrets.
+- **Linking a Kubernetes service account to a GCP service account:** The `SecretStore` (or `ClusterSecretStore`) references a Kubernetes service account, which is linked to a [GCP service account](https://cloud.google.com/iam/docs/service-accounts) that is authorized to access Secret Manager secrets. This requires that the Kubernetes service account is annotated correctly and granted the `iam.workloadIdentityUser` role on the GCP service account.
+- **Authorizing the Core Controller Pod:** The ESO Core Controller Pod's service account is authorized to access Secret Manager secrets. No authentication is required for `SecretStore` and `ClusterSecretStore` instances.
+
+In the following, we will describe each of these options in detail.
 
-Your Google Kubernetes Engine (GKE) applications can consume GCP services like Secrets Manager without using static, long-lived authentication tokens. This is our recommended approach of handling credentials in GCP. ESO offers two options for integrating with GKE workload identity: **pod-based workload identity** and **using service accounts directly**. Before using either way you need to create a service account - this is covered below.
+#### Prerequisites
 
-#### Creating Workload Identity Service Accounts
+* Ensure that [Workload Identity Federation is enabled](https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity) for the GKE cluster.
 
-You can find the documentation for Workload Identity [here](https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity). We will walk you through how to navigate it here.
+_Note that while Google Cloud WIF [is available for AKS, EKS, and self-hosted Kubernetes clusters](https://cloud.google.com/iam/docs/workload-identity-federation-with-kubernetes), ESO currently supports WIF authentication only for GKE ([Issue #1038](https://github.com/external-secrets/external-secrets/issues/1038))._
 
-Search [the document](https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity) for this editable values and change them to your values:
-_Note: If you have installed ESO, a serviceaccount has already been created. You can either patch the existing `external-secrets` SA or create a new one that fits your needs._
+#### Using a Kubernetes service account as a GCP IAM principal
 
-- `CLUSTER_NAME`: The name of your cluster
-- `PROJECT_ID`: Your project ID (not your Project number nor your Project name)
-- `K8S_NAMESPACE`: For us following these steps here it will be `es`, but this will be the namespace where you deployed the external-secrets operator
-- `KSA_NAME`: external-secrets (if you are not creating a new one to attach to the deployment)
-- `GSA_NAME`: external-secrets for simplicity, or something else if you have to follow different naming conventions for cloud resources
-- `ROLE_NAME`: should be `roles/secretmanager.secretAccessor` - so you make the pod only be able to access secrets on Secret Manager
+The `SecretStore` (or `ClusterSecretStore`) references a Kubernetes service account that is authorized to access Secret Manager secrets.
 
-#### Using Service Accounts directly
+To demonstrate this approach, we'll create a `SecretStore` in the `demo` namespace.
 
-Let's assume you have created a service account correctly and attached a appropriate workload identity. It should roughly look like this:
+First, create a Kubernetes service account in the `demo` namespace:
 
 ```yaml
 apiVersion: v1
 kind: ServiceAccount
 metadata:
-  name: external-secrets
-  namespace: es
-  annotations:
-    iam.gke.io/gcp-service-account: example-team-a@my-project.iam.gserviceaccount.com
+  name: demo-secrets-sa
+  namespace: demo
 ```
 
-You can reference this particular ServiceAccount in a `SecretStore` or `ClusterSecretStore`. It's important that you also set the `projectID`, `clusterLocation` and `clusterName`. The Namespace on the `serviceAccountRef` is ignored when using a `SecretStore` resource. This is needed to isolate the namespaces properly.
+To grant a Kubernetes service account access to Secret Manager secret(s), you need to know four values:
 
-*When filling `clusterLocation` parameter keep in mind if it is Regional or Zonal cluster.*
+* `PROJECT_ID`: Your GCP project ID, which you can find under "Project Info" on your console dashboard. Note that this might be different from your project's _name_.
+* `PROJECT_NUMBER`: Your GCP project number, which you can find under "Project Info" on your console dashboard or through `gcloud projects describe $PROJECT_ID --format="value(projectNumber)"`.
+* `K8S_SA`: The name of the Kubernetes service account you created. (In our example, `demo-secrets-sa`.)
+* `K8S_NAMESPACE`: The namespace where you created the Kubernetes service account (In our example, `demo`.)
 
-```yaml
-{% include 'gcpsm-wi-secret-store.yaml' %}
+For example, the following CLI call grants the Kubernetes service account access to a secret `demo-secret`:
+
+```shell
+gcloud secrets add-iam-policy-binding demo-secret \
+  --project=$PROJECT_ID \
+  --role="roles/secretmanager.secretAccessor"
+  --member="principal://iam.googleapis.com/projects/${PROJECT_NUMBER}/locations/global/workloadIdentityPools/${PROJECT_ID}.svc.id.goog/subject/ns/${K8S_NAMESPACE}/sa/${K8S_SA}"
 ```
 
-*You need to give the Google service account the `roles/iam.serviceAccountTokenCreator` role so it can generate a service account token for you (not necessary in the Pod-based Workload Identity bellow)*
+You can also grant the Kubernetes service account access to _all_ secrets in a GCP project:
 
-#### Using Pod-based Workload Identity
+```shell
+gcloud project add-iam-policy-binding $PROJECT_ID \
+  --role="roles/secretmanager.secretAccessor"
+  --member="principal://iam.googleapis.com/projects/${PROJECT_NUMBER}/locations/global/workloadIdentityPools/${PROJECT_ID}.svc.id.goog/subject/ns/${K8S_NAMESPACE}/sa/${K8S_SA}"
+```
 
-You can attach a Workload Identity directly to the ESO pod. ESO then has access to all the APIs defined in the attached service account policy. You attach the workload identity by (1) creating a service account with a attached workload identity (described above) and (2) using this particular service account in the pod's `serviceAccountName` field.
+Note that this allows anyone who can create `ExternalSecret` resources referencing a `SecretStore` instance using this service account access to all secrets in the project.
 
-For this example we will assume that you installed ESO using helm and that you named the chart installation `external-secrets` and the namespace where it lives `es` like:
+_For more information about WIF and Secret Manager permissions, refer to:_
 
-```sh
-helm install external-secrets external-secrets/external-secrets --namespace es
+* _[Authenticate to Google Cloud APIs from GKE workloads](https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity) in the GKE documentation._
+* _[Access control with IAM](https://cloud.google.com/secret-manager/docs/access-control) in the Secret Manager documentation._
+
+To create a `SecretStore` that references a service account, in addition to the four values above, you need to know:
+
+* `CLUSTER_NAME`: The name of the GKE cluster.
+* `CLUSTER_LOCATION`: The location of the GKE cluster. For a regional cluster, this is the region. For a zonal cluster, this is the zone.
+
+You can optionally verify these values through the CLI:
+
+```shell
+gcloud container clusters describe $CLUSTER_NAME \
+  --project=$PROJECT_ID --location=$CLUSTER_LOCATION
 ```
 
-Then most of the resources would have this name, the important one here being the k8s service account attached to the external-secrets operator deployment:
+If the three values are correct, this returns information about your cluster.
+
+Finally, create the `SecretStore`:
 
 ```yaml
-# ...
-      containers:
-      - image: ghcr.io/external-secrets/external-secrets:vVERSION
-        name: external-secrets
-        ports:
-        - containerPort: 8080
-          protocol: TCP
-      restartPolicy: Always
-      schedulerName: default-scheduler
-      serviceAccount: external-secrets
-      serviceAccountName: external-secrets # <--- here
+{% include 'gcpsm-wif-iam-secret-store.yaml' %}
 ```
 
-The pod now has the identity. Now you need to configure the `SecretStore`.
-You just need to set the `projectID`, all other fields can be omitted.
+In the case of a `ClusterSecretStore`, you additionally have to define the service account's `namespace` under `auth.workloadIdentity.serviceAccountRef`.
+
+#### Linking a Kubernetes service account to a GCP service account
+
+The `SecretStore` (or `ClusterSecretStore`) references a Kubernetes service account, which is linked to a GCP service account that is authorized to access Secret Manager secrets.
+
+To demonstrate this approach, we'll create a `SecretStore` in the `demo` namespace.
+
+To set up the Kubernetes service account, you need to know or choose the following values:
+
+* `PROJECT_ID`: Your GCP project ID, which you can find under "Project Info" on your console dashboard. Note that this might be different from your project's _name_.
+* `GCP_SA`: The name of the GCP service account you are going to create and use (e.g., `external-secrets`).
+
+First, create the Kubernetes service account with an annotation that references the GCP service account:
 
 ```yaml
-{% include 'gcpsm-pod-wi-secret-store.yaml' %}
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: demo-secrets-sa
+  namespace: demo
+  annotations:
+    iam.gke.io/gcp-service-account: [GCP_SA]@[PROJECT_ID].iam.gserviceaccount.com
+```
+
+Next, create the GCP service account:
+
+```shell
+gcloud iam service-accounts create $GCP_SA \
+  --project=$PROJECT_ID
 ```
 
-### GCP Service Account authentication
+To finalize the link between the GCP service account and the Kubernetes service account, you need two additional values:
 
-You can use [GCP Service Account](https://cloud.google.com/iam/docs/service-accounts) to authenticate with GCP. These are static, long-lived credentials. A GCP Service Account is a JSON file that needs to be stored in a `Kind=Secret`. ESO will use that Secret to authenticate with GCP. See here how you [manage GCP Service Accounts](https://cloud.google.com/iam/docs/creating-managing-service-accounts).
-After creating a GCP Service account go to `IAM & Admin` web UI, click `ADD ANOTHER ROLE` button, add `Secret Manager Secret Accessor` role to this service account.
-The `Secret Manager Secret Accessor` role is required to access secrets.
+* `K8S_SA`: The name of the Kubernetes service account you created. (In our example, `demo-secrets-sa`.)
+* `K8S_NAMESPACE`: The namespace where you created the Kubernetes service account (In our example, `demo`.)
 
-```yaml
-{% include 'gcpsm-credentials-secret.yaml' %}
+Grant the Kubernetes service account the `iam.workloadIdentityUser` role on the GCP service account:
+
+```shell
+gcloud iam service-accounts add-iam-policy-binding \
+  ${GCP_SA}@${PROJECT_ID}.iam.gserviceaccount.com \
+  --role="roles/iam.workloadIdentityUser" \
+  --member "serviceAccount:${PROJECT_ID}.svc.id.goog[${K8S_NAMESPACE}/${K8S_SA}]"
+```
+
+Next, grant the GCP service account access to a secret in the Secret Manager.
+For example, the following CLI call grants it access to a secret `demo-secret`:
+
+```shell
+gcloud secrets add-iam-policy-binding demo-secret \
+  --project=$PROJECT_ID \
+  --role="roles/secretmanager.secretAccessor"
+  --member "serviceAccount:${GCP_SA}@${PROJECT_ID}.iam.gserviceaccount.com"
+```
+
+You can also grant the GCP service account access to _all_ secrets in a GCP project:
+
+```shell
+gcloud project add-iam-policy-binding $PROJECT_ID \
+  --role="roles/secretmanager.secretAccessor"
+  --member "serviceAccount:${GCP_SA}@${PROJECT_ID}.iam.gserviceaccount.com"
 ```
 
+Note that this allows anyone who can create `ExternalSecret` resources referencing a `SecretStore` instance using this service account access to all secrets in the project.
+
+_For more information about WIF and Secret Manager permissions, refer to:_
+
+* _[Authenticate to Google Cloud APIs from GKE workloads](https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity) in the GKE documentation._
+* _[Access control with IAM](https://cloud.google.com/secret-manager/docs/access-control) in the Secret Manager documentation._
+
+To create a `SecretStore` that references the Kubernetes service account, you need to know:
+
+* `CLUSTER_NAME`: The name of the GKE cluster.
+* `CLUSTER_LOCATION`: The location of the GKE cluster. For a regional cluster, this is the region. For a zonal cluster, this is the zone.
+
+You can optionally verify the information through the CLI:
+
+```shell
+gcloud container clusters describe $CLUSTER_NAME \
+  --project=$PROJECT_ID --location=$CLUSTER_LOCATION
+```
 
-#### Update secret store
-Be sure the `gcpsm` provider is listed in the `Kind=SecretStore`
+If the three values are correct, this returns information about your cluster.
+
+Finally, create the `SecretStore`:
 
 ```yaml
-{% include 'gcpsm-secret-store.yaml' %}
+{% include 'gcpsm-wif-sa-secret-store.yaml' %}
+```
+
+In the case of a `ClusterSecretStore`, you additionally have to define the service account's `namespace` under `auth.workloadIdentity.serviceAccountRef`.
+
+#### Authorizing the Core Controller Pod
+
+Instead of managing authentication at the `SecretStore` and `ClusterSecretStore` level, you can give the [Core Controller](/api/components/) Pod's service account access to Secret Manager secrets using one of the two WIF approaches described in the previous sections.
+
+To demonstrate this approach, we'll assume you installed ESO using Helm into the `external-secrets` namespace, with `external-secrets` as the release name:
+
+```shell
+helm repo add external-secrets https://charts.external-secrets.io
+helm install external-secrets external-secrets/external-secrets \
+  --namespace external-secrets --create-namespace
 ```
 
-**NOTE:** In case of a `ClusterSecretStore`, Be sure to provide `namespace` for `SecretAccessKeyRef` with the namespace of the secret that we just created.
+This creates a Kubernetes service account `external-secrets` in the `external-secrets` namespace, which is used by the Core Controller Pod.
 
-#### Creating external secret
+To verify this (or to determine the service account's name in a different setup), you can run:
 
-To create a kubernetes secret from the GCP Secret Manager secret a `Kind=ExternalSecret` is needed.
+```shell
+kubectl get pods --namespace external-secrets \
+  --selector app.kubernetes.io/name=external-secrets \
+  --output jsonpath='{.items[0].spec.serviceAccountName}'
+```
+
+Use WIF to grant this Kubernetes service account access to the Secret Manager secrets.
+You can use either of the approaches described in the previous two sections.
+
+_For details and further information on WIF and Secret Manager permissions, refer to:_
+
+* _[Authenticate to Google Cloud APIs from GKE workloads](https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity) in the GKE documentation._
+* _[Access control with IAM](https://cloud.google.com/secret-manager/docs/access-control) in the Secret Manager documentation._
+
+Once the Core Controller Pod can access the Secret Manager secret(s) through WIF via its Kubernetes service account, you can create `SecretStore` or `ClusterSecretStore` instances that only specify the GCP project ID, omitting the `auth` section entirely:
 
 ```yaml
-{% include 'gcpsm-external-secret.yaml' %}
+{% include 'gcpsm-wif-core-controller-secret-store.yaml' %}
 ```
 
-The operator will fetch the GCP Secret Manager secret and inject it as a `Kind=Secret`
+### Authenticating with a GCP service account
+
+The `SecretStore` (or `ClusterSecretStore`) use a long-lived, static [GCP service account key](https://cloud.google.com/iam/docs/service-account-creds#key-types) to authenticate with GCP.
+This approach can be used on any Kubernetes cluster.
+
+To demonstrate this approach, we'll create a `SecretStore` in the `demo` namespace.
+
+First, create a GCP service account and grant it the `secretmanager.secretAccessor` role on the Secret Manager secret(s) you want to access.
+
+_For details and further information on managing service account permissions and Secret Manager roles, refer to:_
+
+* _[Attach service accounts to resources](https://cloud.google.com/iam/docs/attach-service-accounts) in the IAM documentation._
+* _[Access control with IAM](https://cloud.google.com/secret-manager/docs/access-control) in the Secret Manager documentation._
+
+Then, create a service account key pair using one of the methods described on the page [Create and delete service account keys](https://cloud.google.com/iam/docs/keys-create-delete) in the Google Cloud IAM documentation and store the JSON file with the private key in a Kubernetes `Secret`:
+
+```yaml
+{% include 'gcpsm-sa-credentials-secret.yaml' %}
 ```
-kubectl get secret secret-to-be-created -n <namespace> -o jsonpath='{.data.dev-secret-test}' | base64 -d
+
+Finally, reference this secret in the `SecretStore` manifest:
+
+```yaml
+{% include 'gcpsm-sa-secret-store.yaml' %}
 ```
 
-### PushSecret owning an existing Google Secret Manager Secret
+In the case of a `ClusterSecretStore`, you additionally have to specify the service account's `namespace` under `auth.secretRef.secretAccessKeySecretRef`.
+
+## Using PushSecret with an existing Google Secret Manager secret
 
 There are some use cases where you want to use PushSecret for an existing Google Secret Manager Secret that already has labels defined. For example when the creation of the secret is managed by another controller like Kubernetes Config Connector (KCC) and the updating of the secret is managed by ESO.
 
@@ -155,9 +287,9 @@ spec:
 {% endraw %}
 ```
 
-### Secret Replication and Encryption Configuration
+## Secret Replication and Encryption Configuration
 
-#### Location and Replication
+### Location and Replication
 
 By default, secrets are automatically replicated across multiple regions. You can specify a single location for your secrets by setting the `location` field:
 
@@ -173,7 +305,7 @@ spec:
       location: us-east1  # Specify a single location
 ```
 
-#### Customer-Managed Encryption Keys (CMEK)
+### Customer-Managed Encryption Keys (CMEK)
 
 You can use your own encryption keys to encrypt secrets at rest. To use Customer-Managed Encryption Keys (CMEK), you need to:
 
@@ -214,11 +346,11 @@ spec:
       location: us-east1  # Required when using CMEK
 ```
 
-### Migration Guide: PushSecret Metadata Format (v0.11.x to v0.12.0)
+## Migration Guide: PushSecret Metadata Format (v0.11.x to v0.12.0)
 
 In version 0.12.0, the metadata format for PushSecrets has been standardized to use a structured format. If you're upgrading from v0.11.x, you'll need to update your PushSecret specifications.
 
-#### Old Format (v0.11.x)
+### Old Format (v0.11.x)
 ```yaml
 apiVersion: external-secrets.io/v1alpha1
 kind: PushSecret
@@ -238,7 +370,7 @@ spec:
           - "topic2"
 ```
 
-#### New Format (v0.12.0+)
+### New Format (v0.12.0+)
 ```yaml
 apiVersion: external-secrets.io/v1alpha1
 kind: PushSecret

docs/snippets/gcpsm-credentials-secret.yaml

@@ -1,9 +1,8 @@
 apiVersion: v1
 kind: Secret
 metadata:
-  name: gcpsm-secret
-  labels:
-    type: gcpsm
+  name: gcp-sa-secret
+  namespace: demo
 type: Opaque
 stringData:
   secret-access-credentials: |-

docs/snippets/gcpsm-sa-secret-store.yaml

@@ -0,0 +1,14 @@
+apiVersion: external-secrets.io/v1beta1
+kind: SecretStore
+metadata:
+  name: demo-store
+  namespace: demo
+spec:
+  provider:
+    gcpsm:
+      auth:
+        secretRef:
+          secretAccessKeySecretRef:
+            name: gcp-sa-secret
+            key: secret-access-credentials
+      projectID: [PROJECT_ID]

docs/snippets/gcpsm-secret-store.yaml

@@ -1,14 +0,0 @@
-apiVersion: external-secrets.io/v1beta1
-kind: SecretStore
-metadata:
-  name: gcp-store
-  namespace: example
-spec:
-  provider:
-    gcpsm:                                  # gcpsm provider
-      auth:
-        secretRef:
-          secretAccessKeySecretRef:
-            name: gcpsm-secret              # secret name containing SA key
-            key: secret-access-credentials  # key name containing SA key
-      projectID: alphabet-123               # name of Google Cloud project

docs/snippets/gcpsm-wi-secret-store.yaml

@@ -1,20 +0,0 @@
-apiVersion: external-secrets.io/v1beta1
-kind: ClusterSecretStore
-metadata:
-  name: gcp-store
-spec:
-  provider:
-    gcpsm:
-      projectID: alphabet-123
-      auth:
-        workloadIdentity:
-          # name of the cluster Location, region or zone
-          clusterLocation: europe-central2
-          # name of the GKE cluster
-          clusterName: alpha-cluster-42
-          # projectID of the cluster (if omitted defaults to spec.provider.gcpsm.projectID)
-          clusterProjectID: my-cluster-project
-          # reference the sa from above
-          serviceAccountRef:
-            name: team-a
-            namespace: team-a

docs/snippets/gcpsm-pod-wi-secret-store.yaml

@@ -1,8 +1,9 @@
 apiVersion: external-secrets.io/v1beta1
 kind: SecretStore
 metadata:
-  name: gcp-store
+  name: demo-store
+  namespace: demo
 spec:
   provider:
     gcpsm:
-      projectID: alphabet-123
+      projectID: [PROJECT_ID]

docs/snippets/gcpsm-wif-iam-secret-store.yaml

@@ -0,0 +1,15 @@
+apiVersion: external-secrets.io/v1beta1
+kind: SecretStore
+metadata:
+  name: demo-store
+  namespace: demo
+spec:
+  provider:
+    gcpsm:
+      projectID: [PROJECT_ID]
+      auth:
+        workloadIdentity:
+          clusterLocation: [CLUSTER_LOCATION]
+          clusterName: [CLUSTER_NAME]
+          serviceAccountRef:
+            name: demo-secrets-sa

docs/snippets/gcpsm-wif-sa-secret-store.yaml

@@ -0,0 +1,15 @@
+apiVersion: external-secrets.io/v1beta1
+kind: SecretStore
+metadata:
+  name: demo-store
+  namespace: demo
+spec:
+  provider:
+    gcpsm:
+      projectID: [PROJECT_ID]
+      auth:
+        workloadIdentity:
+          clusterLocation: [CLUSTER_LOCATION]
+          clusterName: [CLUSTER_NAME]
+          serviceAccountRef:
+            name: demo-secrets-sa