Abstract

GitOps treats a Git repository as the authoritative source of truth for infrastructure and application state, but the model only holds if cluster state remains continuously reconciled against that source. In practice, configuration drift - divergence between the declared desired state and the actual running state - accumulates through manual kubectl edits, admission controller mutations, and operator side-effects. Robust reconciliation loops detect and correct this drift automatically, turning GitOps from a deployment convention into a live enforcement mechanism.

Sources of Drift in Production Clusters

Drift enters clusters through several vectors that are easy to underestimate. Direct kubectl edit or kubectl patch commands made under incident pressure are the most common. Cluster autoscaler, VPA, and HPA controllers mutate resource specs at runtime, producing annotations and replica counts that diverge from the committed manifests. External secrets operators and cert-manager inject certificates and credentials that were never part of the original YAML, creating fields that reconcilers must learn to ignore through server-side apply field managers.

Flux and Argo CD handle this differently. Argo CD computes a diff between the live object and the rendered Helm or Kustomize output, flagging fields that appear in the live cluster but not in the desired state. Flux uses server-side apply exclusively, which means the field-manager ownership model determines what is considered drift - fields owned by other controllers are left alone.

Structuring the Reconciliation Loop

A well-structured reconciliation loop has three distinct phases: observe, diff, and remediate. The observe phase reads the live cluster state, typically via the API server’s informer cache rather than direct API calls, to avoid rate-limiting at scale. The diff phase compares normalized desired state against normalized live state, stripping managed fields that are not owned by the GitOps controller. The remediate phase applies the delta using server-side apply with a force flag when conflicts arise.

Reconciliation intervals matter. Argo CD’s default sync interval of three minutes is acceptable for most workloads but too slow for compliance-sensitive environments. Flux allows per-resource interval configuration down to thirty seconds. Setting intervals too aggressively increases API server load; profiling the reconciler’s API request rate against cluster size is a practical prerequisite before tightening intervals below one minute.

Alerting on Persistent Drift

Not all drift should trigger automated remediation. A drift event that persists beyond one reconciliation cycle - meaning the controller observed it, attempted to correct it, and it reappeared - is a signal worth alerting on. Both Argo CD and Flux expose Prometheus metrics: argocd_app_info with sync_status=OutOfSync and gotk_reconcile_condition with type=Ready and status=False are useful alert targets. Pairing these with a hold-for duration in Alertmanager prevents transient API latency from generating noise.

Drift dashboards built in Grafana, sourced from these metrics over time, reveal patterns - particular namespaces or resource types that drift repeatedly point to process gaps, not just tooling gaps. Closing those process gaps is the durable fix.

Handling Unreconcilable Resources

Some resources are intentionally excluded from reconciliation: HPA-managed replica counts, resources owned by operators, and objects with the argocd.argoproj.io/managed-by annotation set to a different app. Argo CD’s resource exclusion configuration and Flux’s ignore field in HelmRelease and Kustomization resources make these exclusions explicit and auditable. Keeping the exclusion list short and reviewed in code is a useful discipline; a long exclusion list usually means the GitOps boundary is poorly defined rather than that reconciliation is unnecessary.