Troubleshooting

Read the state first

kubectl -n flux-system get resourcesetinputprovider socle -o yaml   # what OpenTofu deposited: your configuration, normalised
kubectl -n flux-system get ocirepository socle                      # the pulled tag and digest, and SourceVerified
kubectl -n flux-system get resourceset                              # socle-root and one per module, with Ready
kubectl -n flux-system describe resourceset <module>                # why one is not Ready
kubectl -n <namespace> get helmrelease                              # the module's chart

A green tofu apply is not convergence: it proves the objects were deposited. socle-root is Ready only when every module is.

Symptoms

The plan refuses kube

What you see:

kube = { external_dsn = { enabled = true } }
→ kube: unknown module(s) external_dsn. Catalog: argocd, crossplane, external_dns, gateway_api, …

kube = { argocd = { domian = "argocd.acme.example" } }
→ kube: unknown attribute. Allowed per module: {"argocd":["admin_enabled","domain",…],…}

kube = { hello = { replicas = "three" } }
→ kube: an attribute has the wrong type. Each value must have the type of its catalog default: …

kube = { metrics_server = { enabled = true } }        # on gcp
→ kube: a module is not offered on gcp. Cloud-bound modules: {"gateway_api":["aws","azure","gcp","scaleway"],"metrics_server":["aws"],"velero":["aws","gcp"]}

Why: kube is validated against the catalog at plan. Modules add their own rules: external_dns without domain_filters, kyverno_policies without kyverno, keda.services or velero without crossplane, a secret-bearing path in values.

Fix: use what the message allows; the schema is in Inputs, the rules in variables.tf.

The plan warns that Crossplane has no identity

What you see: on AWS, kube.crossplane.enabled is set without aws.crossplane: the AWS provider runs with no identity, and every CloudAccess will fail.; on GCP, kube.crossplane.enabled is set without gcp.crossplane: the Google Cloud providers run as a principal with no grant, and every binding a module declares will fail.

Why: Crossplane converges, but every module role (AWS) or binding (GCP) then fails at the cloud’s API.

Fix: aws.crossplane = { allowed_services = [ … ] } with the services your modules use, or gcp.crossplane = { allowed_roles = [ … ] } with their roles (GCP prerequisites); or keep Crossplane off.

The plan refuses a cluster with no node

What you see: bootstrap_node_count must be a whole number of at least 1: a cluster with no node runs nothing, the socle included.

Why: the foundations’ bootstrap_node_count is 0 (on AWS; GKE Autopilot has no node group to size).

Fix: set bootstrap_node_count to at least 1.

Calls to AWS hang without an error

What you see: a pod using Pod Identity (Crossplane’s AWS providers first) waits on 169.254.170.23 and logs nothing.

Why: the Pod Identity Agent is missing, or the pod was admitted before its association existed.

Fix: set eks_addons.pod_identity_agent = true and check aws eks describe-addon --cluster-name <cluster> --addon-name eks-pod-identity-agent; delete a pod admitted too early so it is recreated.

A module’s binding never gets Ready on GCP

What you see: the module’s cloud step waits; its ProjectIAMMember or BucketIAMMember stays not Ready, its conditions carrying the error IAM returned.

Why: the role the module needs is not in gcp.crossplane.allowed_roles: Crossplane may grant nothing else.

Fix: add the role from Crossplane’s allowed roles and apply.

A Crossplane provider stays Healthy=False while its pods are Ready

What you see: Healthy=False, Deployment does not have minimum availability, on running pods; every module with cloud access waits.

Why: the package manager’s view of the Deployment is stale (once in five e2e runs).

Fix: it clears when Crossplane re-evaluates the revision. To read it:

kubectl get providers.pkg.crossplane.io
kubectl get providerrevisions.pkg.crossplane.io
kubectl -n crossplane-system logs deploy/crossplane

Turning a module off takes five minutes

What you see: helm-controller logs running 'upgrade' action with timeout of 5m0s, then uninstalled Helm release for deleted resource.

Why: the module was turned off during the Helm upgrade a values change drives; the deletion waits for it.

Fix: before turning a module off, wait for its release to be idle: Ready and Released true, no Reconciling.

The OCIRepository refuses the signature

What you see: SourceVerified false.

Why: the signature does not match cosign_identity, whose subject is publish-artifact.yaml on its ref.

Fix:

• a branch build: set cosign_identity to that branch (Upgrade the socle);
• a hand-written identity: anchored regexes, every dot escaped, each backslash doubled in HCL: "^https://github\\.com/…$";
• a re-signed mirror: copy the original signature with the artifact.

MANIFEST_UNKNOWN on the OCIRepository

What you see: MANIFEST_UNKNOWN; the cluster stops pulling.

Why: the tag was deleted: branch tags with their branch and after seven days, alphas after thirty (Artifacts). feat-x and feat/x share their tags.

Fix: move socle_version to a release, or to a newer build.

When one apply is not enough

• Replacing the cluster. A change that replaces the cluster leaves the helm provider unable to refresh. Fix: apply the foundations alone first (-target=module.socle.module.foundations, on both roots), then a full apply.
• Destroying with the API unreachable. The releases need the API to go. Fix: drop the bootstrap from the state first (tofu state rm module.socle.module.socle, on both roots) (Uninstall).
• Destroying on GCP. The first tofu destroy refuses while gcp.deletion_protection is on: set it to false and apply. Once it is off, the first destroy still fails on the certificate map, the regional certificate and the proxy-only subnetwork, which GKE’s load balancers release after the cluster. Fix: run tofu destroy again a few minutes later.
• Changing gcp.gateway_certificate.domains. The internal Gateway moves to the new certificate only once Flux reconciles it, so deleting the old one fails “in use”. Fix: wait for the Gateway to name the new certificate, then apply again (GCP root).