ACE Journal

Progressive Delivery Using Argo Rollouts

Who this is for: platform engineers and SREs already running services on Kubernetes who want to replace manual deploy gates, the “watch the dashboard for ten minutes, then click promote” ritual, with automated metric analysis. If you are still deploying with kubectl apply of a Deployment and approving releases by eye, this is the upgrade path. You do not need a service mesh to start, but you do need some way to read service-level metrics.

The deploy gate that nobody wants to be holding

Most teams that have run production Kubernetes for a while have the same scar. A deploy goes out at 2pm, the rolling update completes cleanly, the pods are all Ready, and then twenty minutes later the error rate climbs because the new build has a subtle regression that only shows up under real traffic. The Deployment controller did its job perfectly. It is just that “all pods healthy” and “the release is actually good” are different questions, and the stock Deployment only knows how to answer the first one.

The usual patch for this is a human gate. Ship to a small slice, watch Grafana, and promote by hand if the graphs look fine. This works, sort of, until you notice that it does not scale, it is inconsistent between whoever is on call, and it tends to evaporate exactly when you need it most, late at night, during an incident, when the person watching is tired.

Progressive delivery moves that judgment into the system. Instead of a person deciding “the canary looks healthy, proceed,” you encode what healthy means as a metric query with a threshold, and a controller checks it on every release the same way every time. Argo Rollouts is the Kubernetes-native implementation of that idea. It is a CNCF project under the Argo umbrella, the same family as Argo CD, and it extends rather than replaces the deployment primitives you already know.

Mechanics: the canary strategy and traffic splitting

A Rollout is a drop-in replacement for a Deployment. It carries the same pod template, the same selector, the same replica count, plus a strategy block that describes how to move from the old version to the new one. The canary strategy is a list of steps. Each step is either a setWeight (send this percentage of traffic to the new version), a pause (stop here, optionally for a fixed duration), or an analysis hook.

There are two ways the controller can split traffic. The naive way is replica-count proportioning: if you want roughly 20 percent on the canary, run one canary pod for every four stable pods and let the Service load-balance across them. This is approximate and gets coarse at low replica counts. The precise way is to delegate traffic shifting to an ingress controller or service mesh, Istio, NGINX, or an AWS ALB, so the canary weight is enforced at the proxy regardless of how many pods are behind it. The mesh route is what you want for real percentage control, because “5 percent of traffic” should mean 5 percent of requests, not “one pod out of twenty that happens to be getting whatever the load balancer feels like.”

A typical progression climbs in widening steps with a gate between each:

graph LR
    A[0%<br/>stable] --> B[5%<br/>analysis]
    B --> C[20%<br/>analysis]
    C --> D[50%<br/>analysis]
    D --> E[100%<br/>promoted]
    B -. abort .-> A
    C -. abort .-> A
    D -. abort .-> A

The steps widen on purpose. The first step is small because it is the riskiest moment: the new version has never seen production traffic. Once it has survived 5 percent for a measurement window, surviving 20 percent is more likely, and so on. Each arrow back to stable is an abort path, and which gate trips the abort is decided by analysis, not by a clock.

A worked example: the Rollout spec

Here is a canary Rollout for a service called checkout. It steps through 5, 20, and 50 percent, runs an analysis at each weight, and uses an NGINX-based traffic router. Note the analysis block under strategy.canary referencing a template by name rather than inlining the metric queries.

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: checkout
  namespace: storefront
spec:
  replicas: 8
  selector:
    matchLabels:
      app: checkout
  template:
    metadata:
      labels:
        app: checkout
    spec:
      containers:
        - name: checkout
          image: registry.internal/storefront/checkout:1.8.2
          ports:
            - containerPort: 8080
  strategy:
    canary:
      canaryService: checkout-canary
      stableService: checkout-stable
      trafficRouting:
        nginx:
          stableIngress: checkout
      analysis:
        templates:
          - templateName: success-rate-and-latency
        startingStep: 1
        args:
          - name: service-name
            value: checkout
      steps:
        - setWeight: 5
        - pause: { duration: 5m }
        - setWeight: 20
        - pause: { duration: 5m }
        - setWeight: 50
        - pause: { duration: 10m }

Two things are worth calling out. First, startingStep: 1 tells the controller to begin running the referenced analysis once the rollout reaches the second step (steps are zero-indexed), so analysis runs alongside every weight increase from 5 percent onward. Second, the analysis is referenced by templateName, and the actual queries live elsewhere. That separation is the second of the three ideas this article is built around, and it is worth doing from day one.

Keep analysis templates in a shared GitOps directory

The AnalysisTemplate object holds the metric queries and the pass/fail criteria, fully decoupled from any single Rollout. The temptation under release pressure is to inline a quick PromQL query into the rollout spec for one service, ship it, and move on. Do that across a dozen services and you have a dozen slightly different, individually untested definitions of “healthy,” each one a place where a typo in a query silently means “this gate always passes.”

The better pattern is a single directory in the GitOps repo, something like platform/analysis-templates/, holding the canonical templates, referenced by name from each rollout. A change to the success-rate definition then lands in one reviewed pull request and applies everywhere. The templates become a small, versioned, shared library of what your organization means by a good release.

Here is the template the checkout rollout references. It checks an HTTP success rate and a p99 latency, both from Prometheus, and it carries the initialDelay that the next section is about.

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: success-rate-and-latency
  namespace: storefront
spec:
  args:
    - name: service-name
  metrics:
    - name: success-rate
      initialDelay: 120s
      interval: 60s
      count: 5
      successCondition: result[0] >= 0.99
      failureLimit: 1
      provider:
        prometheus:
          address: http://prometheus.monitoring.svc:9090
          query: |
            sum(rate(
              http_requests_total{service="", code!~"5.."}[2m]
            ))
            /
            sum(rate(
              http_requests_total{service=""}[2m]
            ))
    - name: p99-latency
      initialDelay: 120s
      interval: 60s
      count: 5
      successCondition: result[0] <= 0.4
      failureLimit: 1
      provider:
        prometheus:
          address: http://prometheus.monitoring.svc:9090
          query: |
            histogram_quantile(0.99,
              sum(rate(
                http_request_duration_seconds_bucket{service=""}[2m]
              )) by (le)
            )

The success-rate metric passes when at least 99 percent of requests are non-5xx, and the latency metric passes when p99 stays at or under 400ms. count: 5 with interval: 60s means each runs five measurements a minute apart, and failureLimit: 1 means a single bad measurement is tolerated before the whole analysis fails. Tuning failureLimit is how you trade sensitivity against noise: zero is twitchy, too high and a real regression rides through.

Analysis and automated rollback: the part that earns its keep

This is the substantive difference from a vanilla rolling update, and it is the third of the three core ideas. When an analysis run fails, an error rate over the threshold, a p99 that blows past the latency budget, or simply not enough data inside the measurement window, the controller scales the stable version back to 100 percent, holds the canary at zero, and marks the rollout Degraded. No page, no human, no dashboard-watching. Failure is self-correcting.

It helps to think of the rollout as a small state machine:

stateDiagram-v2
    [*] --> Progressing: new revision applied
    Progressing --> Paused: reached a pause step
    Paused --> Progressing: analysis passed / manual promote
    Progressing --> Degraded: analysis failed
    Degraded --> Progressing: new revision / retry
    Progressing --> Healthy: final step promoted
    Healthy --> [*]
    Healthy --> Progressing: next deploy

Progressing is the active state while traffic shifts and analysis runs. Paused is a deliberate stop, either a timed pause step or an indefinite one awaiting a human promote. Degraded is the abort state after a failed analysis, with the stable version serving all traffic again. Healthy is a fully promoted, settled release. The whole value proposition lives in that Progressing -> Degraded edge happening automatically.

The initialDelay gotcha

Now the trap that the TL;DR warned about, and the reason every example above carries initialDelay: 120s. When the controller shifts traffic to a fresh canary and immediately asks Prometheus “what is the success rate,” the rate query is computing over a window, rate(...[2m]) above, that has barely started to fill. For the first 60 to 120 seconds after traffic shifts, the aggregation is still stabilizing. The query may return an empty result, a NaN, or a wildly noisy value computed from a handful of samples.

To the analysis engine, an empty or NaN result is not “wait, there is no data yet.” It is a failed measurement, because it does not satisfy the success condition. So the analysis trips, the controller marks the rollout Degraded, and an engineer spends the next half hour debugging a rollback that had nothing to do with the code. The canary was fine. The query was just early.

initialDelay fixes this by telling the analysis run to wait, here two minutes, before taking its first measurement, giving the metric window time to fill with real canary traffic. The right value is roughly the length of your rate window plus the scrape and aggregation lag of your metrics provider. Datadog’s monitor evaluation, CloudWatch’s metric resolution, and Prometheus’s scrape interval each add their own delay, so tune initialDelay to whichever backend you query. Teams new to Rollouts almost always skip this and then learn it the hard way. If you take one operational lesson from this article, take this one.

For a Datadog-backed analysis the same idea applies. The provider block looks different but the initialDelay discipline is identical:

    - name: error-rate
      initialDelay: 120s
      interval: 60s
      count: 5
      successCondition: result <= 0.01
      failureLimit: 1
      provider:
        datadog:
          interval: 5m
          query: |
            sum:checkout.requests.error{service:checkout}.as_count() /
            sum:checkout.requests.total{service:checkout}.as_count()

Watching and integrating: CI and notifications

Day to day, you watch a rollout with the kubectl plugin. The --watch flag streams state as the controller works through the steps:

$ kubectl argo rollouts get rollout checkout --watch

Name:            checkout
Namespace:       storefront
Status:          ॥ Paused
Message:         CanaryPauseStep
Strategy:        Canary
  Step:          3/6
  SetWeight:     20
  ActualWeight:  20
Images:          registry.internal/storefront/checkout:1.8.1 (stable)
                 registry.internal/storefront/checkout:1.8.2 (canary)
Replicas:
  Desired:       8
  Current:       9
  Updated:       2
  Ready:         9
  Available:     9

NAME                                  KIND         STATUS        AGE   INFO
⟳ checkout                            Rollout      ॥ Paused      9m
├──# revision:2
│  └──⧉ checkout-7c9f8b5d4            ReplicaSet   ✔ Healthy     4m    canary
│     ├──□ checkout-7c9f8b5d4-x2kqp   Pod          ✔ Running     4m    ready:1/1
│     └──□ checkout-7c9f8b5d4-mz7rt   Pod          ✔ Running     4m    ready:1/1
└──# revision:1
   └──⧉ checkout-5d6c7f8b9            ReplicaSet   ✔ Healthy     2h    stable

Note Current: 9 against Desired: 8. That extra pod is both ReplicaSets running at once during the canary, which is a cost worth remembering and which the “where it breaks” section returns to.

For automation, Argo Rollouts ships a notifications controller that fires on lifecycle events. A common setup posts to a deploy channel whenever a rollout pauses at a step, so engineers get the context without opening a terminal:

apiVersion: v1
kind: ConfigMap
metadata:
  name: argo-rollouts-notification-configmap
  namespace: argo-rollouts
data:
  trigger.on-paused: |
    - send: [rollout-paused]
      when: rollout.status.phase == 'Paused'
  template.rollout-paused: |
    slack:
      attachments: |
        [{
          "title": " paused at ",
          "color": "#f4c20d"
        }]

The end-to-end picture ties this together. Code merges, CI builds an image and pushes it to the registry, Argo CD syncs the new manifest, the Rollouts controller takes over the traffic-shifting dance, and the metrics the canary generates feed straight back into the controller’s own decision to advance or abort:

flowchart LR
    Dev[commit] --> CI[CI build]
    CI --> Reg[(image registry)]
    Reg --> CD[Argo CD sync]
    CD --> RC[Rollouts controller]
    RC --> Mesh[ingress / mesh]
    Mesh --> Users((users))
    Users --> Metrics[(Prometheus /<br/>Datadog)]
    Metrics -. analysis feedback .-> RC
    RC -.->|advance or abort| Mesh

That dotted feedback edge from metrics back to the controller is the whole point. In a vanilla rolling update there is no such edge. The deploy completes and the system stops paying attention. Here, the system keeps reading reality and acts on it.

For teams running Argo CD, the integration is tight enough that the Argo CD Application status reflects whether the rollout is progressing, degraded, or paused, so the single pane of glass stays useful during a release instead of sending you to a separate tool.

Argo Rollouts is not the only way to do this, and it is worth being clear about where it sits.

Against vanilla Kubernetes rolling updates, the difference is the feedback loop. A Deployment’s maxSurge and maxUnavailable control how fast pods get replaced, but the controller’s only health signal is the pod readiness probe. It has no notion of error rate or latency and no way to pause for analysis. Rolling updates are the right tool when “the pods came up” really is a sufficient release signal. They stop being enough the moment correctness depends on production traffic.

Flagger is the closest peer, a progressive-delivery operator from the Flux ecosystem. It covers much of the same ground, canary and blue-green driven by metric analysis, and integrates with the same meshes. The practical split tends to follow your GitOps choice: teams on Argo CD usually reach for Argo Rollouts because the controllers share a project, a UI, and operational conventions, while teams on Flux reach for Flagger for the same reason. Both are solid. This is more an ecosystem-fit decision than a capability gap.

Spinnaker comes at the problem from a different era and altitude. It is a full multi-cloud continuous-delivery platform with deployment pipelines, manual judgment stages, and automated canary analysis through Kayenta. It does more than Rollouts, and it is correspondingly heavier to run and operate. If you want a Kubernetes-native controller that lives in your cluster and is configured by the same YAML and GitOps flow as everything else, Rollouts is the lighter fit. If you are orchestrating delivery across many clouds and platforms with rich pipeline logic, Spinnaker’s scope may justify its weight.

Where it breaks

Progressive delivery is not free, and a few situations need care or are genuinely poor fits.

Stateful services. Canary assumes you can run two versions side by side and route traffic between them. For stateless request handlers that is fine. For a service backed by a schema migration, two versions reading and writing the same database at once, that is a correctness problem the traffic split cannot solve. You need backward-compatible, expand-and-contract migrations so both versions tolerate both schemas, and Rollouts does nothing to give you that. The traffic mechanism is sound; the data layer is on you.

Metric-provider outages. The controller’s decisions are only as available as the metrics backend. If Prometheus is down or scraping is broken, analysis queries fail or return nothing, which the engine reads as failure, and your rollout aborts even though the canary was healthy. The same dependency that makes automated analysis powerful makes it a single point of failure, so the health of your metrics pipeline is now release-critical infrastructure.

No traffic-shaping ingress. Without a mesh or a supported ingress controller, you fall back to replica-count proportioning, which is coarse and imprecise at low replica counts. You can run Rollouts this way, but precise weights like “5 percent” become aspirational. If you are serious about small first steps, budget for the mesh.

Low-traffic services. Analysis is statistics, and statistics need samples. A service handling a few requests a minute will not generate enough data inside a reasonable window for a success-rate query to mean anything, so you get flaky, noisy results and false aborts. For low-traffic services, lengthen windows, loosen failureLimit, or accept that progressive delivery may simply be the wrong tool and a careful manual gate is more honest.

The cost of two ReplicaSets. During a canary both versions run at once, as the Current: 9 line in the watch output showed. For most services that overhead is trivial. For a large, resource-hungry fleet, the extra capacity during every release is a real, if temporary, cost worth sizing before you assume canary everywhere is free.

Further reading

Start with the Argo Rollouts canary specification [1] and the analysis and AnalysisTemplate reference [2], which together cover the two objects this article is built on. The traffic-management overview [3] explains the mesh and ingress integrations, and the kubectl plugin docs [4] cover the get --watch workflow. For the notifications setup, see the Argo Rollouts notifications guide [5]. To go deeper on writing good gate queries, the Prometheus querying basics [6] and the histogram_quantile documentation [7] are the right starting points for PromQL. For positioning, compare against Flagger’s progressive-delivery docs [8] and read the CNCF’s framing of progressive delivery as a practice [9]. The Argo project blog [10] is the best source for what is current in the controller itself.

Written with AI assistance, editorially reviewed.

References

[1] Argo Rollouts, “Canary Deployment Strategy.” https://argo-rollouts.readthedocs.io/en/stable/features/canary/

[2] Argo Rollouts, “Analysis and Progressive Delivery.” https://argo-rollouts.readthedocs.io/en/stable/features/analysis/

[3] Argo Rollouts, “Traffic Management.” https://argo-rollouts.readthedocs.io/en/stable/features/traffic-management/

[4] Argo Rollouts, “Kubectl Plugin.” https://argo-rollouts.readthedocs.io/en/stable/features/kubectl-plugin/

[5] Argo Rollouts, “Notifications.” https://argo-rollouts.readthedocs.io/en/stable/features/notifications/

[6] Prometheus, “Querying Basics.” https://prometheus.io/docs/prometheus/latest/querying/basics/

[7] Prometheus, “Querying Functions: histogram_quantile.” https://prometheus.io/docs/prometheus/latest/querying/functions/#histogram_quantile

[8] Flagger, “Progressive Delivery.” https://docs.flagger.app/

[9] CNCF, “Argo Project.” https://www.cncf.io/projects/argo/

[10] Argo Project, “Blog.” https://blog.argoproj.io/

[11] Argo Rollouts, “Specification.” https://argo-rollouts.readthedocs.io/en/stable/features/specification/

[12] Kubernetes, “Performing a Rolling Update.” https://kubernetes.io/docs/tutorials/kubernetes-basics/update/update-intro/