docs(webhook): add debugging guide and fix stale secret-label claim (#6746)

* docs(webhook): add debugging guide and fix stale secret-label claim

Debugging the webhook provider currently has no documentation, so the
recommendation given on issue #5576, putting a logging proxy between the
operator and the target service, exists only in that thread. Write it down,
along with why no built-in request or response dump is offered: templated
urls, headers and bodies routinely carry credentials from the referenced
secrets, so dumping them would put those credentials in the operator log.

Also correct the claim that only generators require their source secrets to
be labeled. That was true when the generator landed in #3121, but #3753 made
store secrets require the label too and the page was never updated, so users
following it hit an opaque lookup failure.

The three common misconfigurations documented here are the ones that produce
errors pointing away from their cause: the missing label, the two-level shape
of template values, and store validation probing a templated url before the
templating engine runs.

Refs: #5576
Signed-off-by: Alexander Chernov <alexander@chernov.it>

* docs(webhook): correct the debug proxy example and widen the label rule

Review feedback on the debugging guide, plus one bug it turned up.

The sidecar example mounted a debug-proxy-config volume that was never
defined, so pasting it produced a pod that would not schedule. Add the
volume and the backing ConfigMap, and note that the key has to end in
.conf: the stock nginx config only includes /etc/nginx/conf.d/*.conf, and
with any other name the sidecar comes up serving the default nginx page
instead of proxying, which looks like the proxy silently doing nothing.

The prose claimed the proxy logs every request line and header. It logs
the method, uri, query, upstream status and Authorization, so say that,
and say how to add other headers and that bodies are not captured.

nginx sends no SNI unless proxy_ssl_server_name is on, which breaks any
endpoint that selects a certificate by server name, so turn it on. It
also does not verify the upstream certificate by default, which matters
more here: the guide is for debugging, and inserting a proxy that trusts
anything makes caProvider and certificate errors disappear rather than
reproduce. Warn about it rather than quietly shipping a proxy that is
laxer than the operator.

The label rule named only provider.webhook.secrets while claiming it
covered generators as well. Both the provider and the generator set
EnforceLabels unconditionally, and getStoreSecret is the single point
that checks it, so the rule also reaches the auth.ntlm username and
password refs. Say all three paths.

That last point exposed a real error: the NTLM example's Secret carries
no label, so following it verbatim fails with exactly the message this
page documents. It predates this branch, but the new section made the
page contradict its own example, so fix it here.

The troubleshooting command only described a namespaced SecretStore on a
page that also documents ClusterSecretStore. Add the cluster-scoped form.

Refs: #5576
Signed-off-by: Alexander Chernov <alexander@chernov.it>

---------

Signed-off-by: Alexander Chernov <alexander@chernov.it>

docs/provider/webhook.md

@@ -152,6 +152,8 @@ kind: Secret
 metadata:
   name: webhook-credentials
   namespace: externalsecrets
+  labels:
+    external-secrets.io/type: webhook # Also required for auth.ntlm secrets
 data:
   username: dGVzdA== # "test"
   password: dGVzdA== # "test"
@@ -213,4 +215,156 @@ spec:
 ```
 
 ### Webhook as generators
-You can also leverage webhooks as generators, following the same syntax. The only difference is that the webhook generator needs its source secrets to be labeled, as opposed to webhook secretstores. Please see the [generator-webhook](../api/generator/webhook.md) documentation for more information.
+You can also leverage webhooks as generators, following the same syntax. Please see the
+[generator-webhook](../api/generator/webhook.md) documentation for more information.
+
+Note that source secrets must be labeled for both secretstores and generators, see
+[Referenced secrets must be labeled](#referenced-secrets-must-be-labeled) below.
+
+### Debugging
+
+#### Start with the store status and events
+
+Most webhook problems surface on the `SecretStore` itself rather than in the logs:
+
+```sh
+kubectl describe secretstore <name> -n <namespace>
+# or, for a cluster-scoped store
+kubectl describe clustersecretstore <name>
+```
+
+A `Ready` condition of `False` with reason `InvalidProviderConfig` means the client could
+not be created or that store validation failed. The accompanying event carries the
+underlying error, which is usually more specific than the condition message.
+
+For per-secret failures, check the `ExternalSecret` instead:
+
+```sh
+kubectl describe externalsecret <name> -n <namespace>
+```
+
+#### Increase the operator log level
+
+Run the controller with `--loglevel debug` to get the reconcile decisions for each object.
+This logs what the operator did and why, but it deliberately does not log HTTP request or
+response contents, see below.
+
+#### Why there is no built-in request or response dump
+
+The webhook provider renders `url`, `headers` and `body` through the templating engine, and
+those templates typically contain credentials pulled from the referenced secrets. Dumping
+requests or responses would therefore write those credentials to the operator log, where
+anyone with log access could read them. That is why no trace or wire-dump option is
+provided, and why one is unlikely to be added.
+
+!!! warning
+      Setting `GODEBUG=http2debug=2` on the operator does produce HTTP/2 frame dumps
+      including authorization headers and full bodies, entirely unredacted. It only covers
+      HTTP/2, it is not a supported debugging path, and it must not be enabled against a
+      production instance.
+
+#### Inspecting traffic with a logging proxy
+
+The supported way to see the traffic is to put a proxy between the operator and the target
+service, and read the proxy's logs. Point the webhook `url` at the proxy over plain HTTP so
+the request is readable there, and let the proxy terminate TLS towards the real endpoint.
+Running it as a sidecar keeps the plaintext hop inside the pod.
+
+Add the sidecar and its config volume to the external-secrets deployment. Only the fields
+relevant to the proxy are shown; keep the rest of the pod spec as it is.
+
+```yaml
+# spec.template.spec.containers
+- name: debug-proxy
+  image: nginx:alpine
+  ports:
+    - containerPort: 8080
+  volumeMounts:
+    - name: debug-proxy-config
+      mountPath: /etc/nginx/conf.d
+# spec.template.spec.volumes
+- name: debug-proxy-config
+  configMap:
+    name: debug-proxy-config
+```
+
+The key must end in `.conf`, because the stock nginx config only includes
+`/etc/nginx/conf.d/*.conf`. With any other key the sidecar starts and serves the default
+nginx page instead of proxying.
+
+```yaml
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: debug-proxy-config
+data:
+  debug-proxy.conf: |
+    log_format dump escape=none '$request_method $uri $args -> $upstream_status'
+                                ' authorization="$http_authorization"';
+
+    server {
+      listen 8080;
+      access_log /dev/stdout dump;
+      location / {
+        proxy_pass https://secrets.example.com;
+        proxy_set_header Host secrets.example.com;
+        # nginx sends no SNI by default, which breaks endpoints that select a
+        # certificate by server name.
+        proxy_ssl_server_name on;
+      }
+    }
+```
+
+That logs the request method, path and query, the upstream status, and the `Authorization`
+header. Add more `$http_<header>` variables to the `log_format` for other headers. Request
+and response bodies are not captured; use `mirror` or a dedicated capture proxy if you need
+them.
+
+Then set `url: "http://localhost:8080/..."` on the store while debugging. Remember that the
+proxy log now contains the same credentials the operator refuses to log, so treat it as
+sensitive and remove the sidecar when you are done.
+
+!!! warning
+      nginx does not verify the upstream certificate unless `proxy_ssl_verify on` and
+      `proxy_ssl_trusted_certificate` are set, so this proxy is deliberately more permissive
+      than the operator. Do not use it to diagnose TLS trust problems: a `caProvider` or
+      certificate error will appear to go away as soon as the proxy is in the path.
+
+#### Referenced secrets must be labeled
+
+Every Secret the webhook reads must carry the label `external-secrets.io/type: webhook`.
+That covers `spec.provider.webhook.secrets` on a `SecretStore` or `ClusterSecretStore`,
+`spec.secrets` on a `Webhook` generator, and the `usernameSecret` and `passwordSecret` of
+`auth.ntlm`, which resolve through the same code path. Without it the lookup fails with:
+
+```
+secret does not contain needed label 'external-secrets.io/type: webhook'. Update secret label to use it with webhook
+```
+
+#### Template values are two levels deep
+
+Secrets listed under `secrets` are exposed as `.<name>.<keyInSecret>`, not `.<name>`. For:
+
+```yaml
+secrets:
+  - name: creds
+    secretRef:
+      name: webhook-credentials
+```
+
+the values are `{{ .creds.username }}` and `{{ .creds.password }}`. Referring to
+`{{ .creds }}` renders a Go map rather than a value. Note also that every key of the
+referenced secret is exposed; a `key` field on the `secretRef` does not narrow it.
+
+#### A templated `url` is validated before templating
+
+Store validation performs a reachability check against `spec.provider.webhook.url` as
+written, before the templating engine runs. A url whose host comes from a template
+therefore fails validation with a message such as:
+
+```
+error accessing external store: dial tcp :443: connect: connection refused
+```
+
+even when the templated request itself would succeed. Keep the host literal and template
+only the path or query string.