ACE Journal

KEDA event-driven autoscaling beyond HPA

Who this is for: platform engineers and SREs running Kubernetes workloads that consume from queues, topics, or other external event sources. You should already be comfortable with Deployments, Services, and basic HPA. You do not need to know KEDA’s internals, but you should know what a consumer group is if you work with Kafka, or what queue depth means if you work with RabbitMQ or SQS.

The queue that HPA cannot see

Picture a microservice that processes orders from a Kafka topic. Traffic is bursty: at midday the consumer group falls behind; at 3am the topic is empty. CPU utilization follows consumption lag with a delay, but only loosely. When the topic is empty, the consumers spin at near-zero CPU doing nothing. HPA will not scale them to zero because HPA has no concept of “no work is queued.” When a surge hits, lag builds for 30 seconds or more before CPU rises enough to trigger a scale-out.

This is the structural gap HPA cannot close: it reacts to resource utilization, not to the presence or depth of work. For stateless, queue-backed consumers that pattern means you are either over-provisioned at idle or under-provisioned during the ramp-up window of a burst.

KEDA was built specifically to fill this gap. It reads the trigger source directly (Kafka consumer group lag, RabbitMQ queue depth, Prometheus metric values, SQS approximate message count), converts those values to a target replica count, and hands that count to Kubernetes to execute. It also handles the one transition HPA never could: scaling a Deployment to exactly zero replicas when the trigger shows no work.

KEDA architecture: operator, metrics adapter, scalers

KEDA installs three components into the cluster, shown in Figure 1.

graph TB
    subgraph cluster["Kubernetes cluster"]
        direction TB
        op["KEDA operator"]
        ma["KEDA metrics adapter\n(custom metrics API)"]
        hpa["HPA (managed by KEDA)"]
        dep["Deployment / ReplicaSet"]

        op -->|creates + updates| hpa
        op -->|reads trigger, patches replicas 0↔1| dep
        hpa -->|reads custom metric| ma
        ma -->|scrapes| scaler["Scaler (Kafka / RabbitMQ / Prometheus / ...)"]
        hpa -->|drives| dep
    end

    ext[("External source\n(broker, DB, queue)")] --> scaler

Figure 1. KEDA component relationships inside a cluster. The operator owns the ScaledObject lifecycle; the metrics adapter surfaces external trigger values to HPA as a custom metric; the operator alone handles the 0-to-1 and 1-to-0 transitions by patching the Deployment directly.

The operator watches ScaledObject and ScaledJob custom resources. When it sees a ScaledObject it creates a corresponding HPA, configures it to use a custom metric sourced from the KEDA metrics adapter, and then monitors the trigger value continuously. When the trigger drops to zero, the operator patches the Deployment’s replica count to 0 directly, bypassing HPA (because HPA has a hard floor of 1). When the trigger rises above the activation threshold, the operator patches replicas to 1, and then HPA takes over to drive from 1 up to maxReplicaCount.

The metrics adapter implements the Kubernetes custom metrics API. HPA calls this API the same way it calls core metrics for CPU: it asks “what is the current value of this metric for this object?” The adapter forwards the question to the appropriate scaler, which queries the actual external system.

Scalers are the library of connectors. As of early 2026, KEDA ships with more than 60 built-in scalers, covering Apache Kafka, RabbitMQ, AWS SQS and Kinesis, Azure Service Bus and Storage Queues, GCP Pub/Sub, Redis, NATS, Prometheus, Datadog, Cron, HTTP, and many more. Each scaler knows how to authenticate with its external system and translate the raw metric (lag, depth, count) into a normalized value that the metrics adapter can serve.

A worked example: Kafka consumer scaling

The central configuration object is the ScaledObject. Here is one for a service called order-processor that consumes from a Kafka topic called orders, scaling based on consumer group lag. Figure 2 shows where this ScaledObject fits in the end-to-end flow.

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: order-processor-scaler
  namespace: commerce
spec:
  scaleTargetRef:
    name: order-processor
  minReplicaCount: 0
  maxReplicaCount: 20
  pollingInterval: 15
  cooldownPeriod: 60
  triggers:
    - type: kafka
      metadata:
        bootstrapServers: kafka.commerce.svc:9092
        consumerGroup: order-processor-group
        topic: orders
        lagThreshold: "50"
        activationLagThreshold: "5"

One constraint worth calling out upfront: maxReplicaCount should match the Kafka topic’s partition count. Kafka assigns at most one partition per consumer in a group, so replicas beyond the partition count sit idle. This connects directly to the over-scaling failure mode covered in the “Where it breaks” section below.

A few fields worth unpacking:

For Kafka specifically, KEDA needs credentials to query the consumer group lag. A common pattern is to supply a TriggerAuthentication object that references a Kubernetes Secret, keeping credentials out of the ScaledObject spec:

apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
  name: kafka-auth
  namespace: commerce
spec:
  secretTargetRef:
    - parameter: sasl
      name: kafka-credentials
      key: sasl-mechanism
    - parameter: username
      name: kafka-credentials
      key: username
    - parameter: password
      name: kafka-credentials
      key: password

Reference this from the ScaledObject trigger by adding authenticationRef: { name: kafka-auth } under the trigger spec.

flowchart LR
    Kafka[("Kafka topic\n'orders'")] --> KEDA["KEDA scaler\n(reads lag every 15s)"]
    KEDA --> MA["Metrics adapter\n(custom metrics API)"]
    MA --> HPA["HPA\n(drives 1..20 replicas)"]
    KEDA -->|"patch replicas 0↔1"| Dep["order-processor\nDeployment"]
    HPA --> Dep
    Dep --> Pods["consumer pods"]
    Pods --> Kafka

Figure 2. End-to-end scaling path for the Kafka consumer example. The KEDA scaler reads consumer group lag directly from Kafka; the metrics adapter surfaces that value to HPA; the operator handles the 0-to-1 and 1-to-0 edges that HPA cannot.

The hard part: cold starts, polling lag, and activation thresholds

Scale-to-zero is KEDA’s most distinctive feature and also its sharpest edge in production.

Cold-start latency is the primary operational risk. When minReplicaCount: 0 is set and the trigger fires, KEDA patches the Deployment to 1 replica and then waits for that pod to become Ready. For a small JVM or Python service pulling a compact image from a local registry, that can be under 10 seconds. For a service with a large image, slow init code, or an image that is not cached on the target node, it can be 30-90 seconds. During that entire window, messages accumulate in the topic and nothing is consuming them. For workloads where any latency spike is unacceptable, set minReplicaCount: 1. For workloads that only run during business hours, a cron scaler in combination is the pragmatic middle ground:

triggers:
  - type: kafka
    metadata:
      bootstrapServers: kafka.commerce.svc:9092
      consumerGroup: order-processor-group
      topic: orders
      lagThreshold: "50"
      activationLagThreshold: "5"
  - type: cron
    metadata:
      timezone: America/New_York
      start: 7 * * * *
      end: 20 * * * *
      desiredReplicas: "1"

With this configuration, the cron scaler keeps one replica alive from 7am to 8pm Eastern; outside those hours the Kafka scaler alone controls the count, and it can drop to zero when the topic is empty.

Polling interval and reaction time interact more than they appear to. A pollingInterval of 15 means the cluster will react to a new burst in at most 15 seconds, plus pod startup time. For most queue consumers that is acceptable. If you set it to 60 to reduce broker query load, a sudden burst that arrives just after a poll will not be seen for up to 60 seconds. For latency-sensitive workloads, err toward shorter intervals and accept the broker traffic.

The activation threshold vs the scaling threshold is a distinction teams often miss until they see the consumer fleet wake from zero on a handful of test messages. lagThreshold drives the target replica count calculation once the fleet is running. activationLagThreshold is a separate gate that controls when the 0-to-1 transition fires. The two should be set independently based on what constitutes “real work” versus “scaling pressure.”

Cooldown and oscillation can cause visible lag spikes in poorly tuned configurations. If cooldownPeriod is too short and the consumer drains the topic quickly, KEDA will scale to zero, see new messages arrive within seconds, and scale back up, incurring another cold start. Setting cooldownPeriod to a value longer than the typical quiet gap between bursts avoids this cycle.

Integration and day-to-day operations

KEDA installs alongside any existing HPA setup without conflict, because it creates separate HPA objects for each ScaledObject. If you already have a manually created HPA for a Deployment, you can migrate by deleting the manual HPA and creating a ScaledObject that references the same Deployment. KEDA will recreate the HPA under its own management.

For monitoring, the KEDA operator exposes Prometheus metrics at its metrics endpoint. Key ones to track: keda_scaler_metrics_value (the current trigger value), keda_scaler_active (whether the ScaledObject is currently active), and keda_scaler_errors_total (a count of scaler query failures, which is the early warning for authentication or connectivity problems with the external source).

For observability of scaling events themselves, KEDA emits Kubernetes events on the ScaledObject. A quick way to watch scaling decisions in real time:

kubectl describe scaledobject order-processor-scaler -n commerce
kubectl get events -n commerce --field-selector reason=KEDAScalerFailed

The kubectl describe output includes the current metric value, the target, and the computed desired replica count, which is the fastest way to diagnose why a ScaledObject is or is not scaling as expected.

KEDA also supports ScaledJob, a separate resource type for batch workloads. Rather than maintaining a pool of long-running consumers, ScaledJob creates a new Kubernetes Job for each unit of work up to a maxReplicaCount parallelism limit. This is the right model for tasks that are expensive to interrupt midway, such as video transcoding, report generation, or ML inference jobs, where the work unit is discrete and completion is meaningful. Teams moving from cron-triggered batch jobs to event-driven ScaledJobs typically see lower processing latency and lower idle compute cost.

Figure 3 shows the positioning of KEDA relative to the most common alternatives.

graph TD
    Q["Scaling trigger"]
    Q --> CPU["CPU/memory only\n→ vanilla HPA"]
    Q --> ExtMet["External metrics\n(queue, Kafka, Prometheus)"]
    ExtMet --> KEDA["KEDA\n(60+ scalers, scale-to-zero,\nHPA-managed)"]
    ExtMet --> Knative["Knative Serving\n(HTTP autoscaling,\nKPA/HPA, scale-to-zero)"]
    Q --> HTTP["HTTP request count\n(no external queue)"]
    HTTP --> Knative
    HTTP --> KEDA2["KEDA HTTP add-on\n(external scaler)"]

Figure 3. Where KEDA sits relative to alternatives, organized by trigger type. KEDA covers the broadest range of external event sources; Knative Serving is the better fit when the primary trigger is HTTP request volume and the workload is a stateless HTTP handler.

Vanilla HPA is the right answer when CPU and memory are sufficient scaling signals, which they often are for synchronous, request-driven services. HPA is simpler, has no additional operator to run, and carries no external dependency risk. KEDA’s value only appears when you need to scale on something HPA cannot see natively.

Knative Serving solves a similar scale-to-zero problem but from the HTTP direction. It ships its own pod autoscaler (KPA) that counts in-flight requests and scales accordingly, and it also falls back to HPA when needed. Knative is the better fit for HTTP workloads where the traffic source is the trigger. KEDA is the better fit when the trigger is a queue, a broker, or a database. For teams already running Knative, KEDA’s Kafka and RabbitMQ scalers can complement it by driving the non-HTTP portions of the same system.

Cluster Autoscaler operates at the node level rather than the pod level. KEDA and Cluster Autoscaler work together naturally: KEDA scales pods up based on queue depth, and when the cluster runs out of node capacity, Cluster Autoscaler adds nodes to accommodate the new pods. For scale-to-zero workloads in particular, the combination means the cluster can scale down to a minimal node count overnight and back up in the morning without manual intervention.

Where it breaks

KEDA is not the right tool in every situation, and several failure modes emerge reliably in production.

Broker authentication drift is the most common operational failure. The scaler credentials live in a Secret referenced by a TriggerAuthentication object. When those credentials rotate and the Secret is updated, KEDA picks up the new values on the next poll. If the rotation happens in the broker before the Secret is updated, or if the Secret update is missed, the scaler begins failing silently: the keda_scaler_errors_total counter climbs, KEDA uses a stale metric value, and the Deployment stays at whatever replica count it was at before the failure. The correct alert is keda_scaler_errors_total > 0 for more than two polling intervals.

High-cardinality topics with shared consumer groups can cause KEDA to over-scale. If a consumer group subscribes to a topic with many partitions and the lag is computed as the sum across all partitions, a brief processing pause can produce a very large lag number that drives KEDA to the maxReplicaCount even though adding more pods than there are partitions accomplishes nothing (Kafka assigns at most one partition per consumer in a group). For Kafka scalers, set maxReplicaCount to the partition count of the topic to cap this.

Scale-to-zero in test environments causes surprise failures when developers apply production-patterned ScaledObjects to low-traffic or synthetic test workloads. A test producer that sends one message every five minutes will trigger constant scale-up and scale-down cycles. Either set minReplicaCount: 1 in non-production environments or accept that the cold-start latency will affect every test run.

Metrics adapter availability is release-critical. KEDA’s metrics adapter is now a dependency of the HPA objects it manages. If the adapter pod crashes or is evicted, HPA can no longer read its custom metrics. Depending on HPA’s behavior when metrics are unavailable, this may leave the Deployment frozen at its current replica count until the adapter recovers. Run the metrics adapter with appropriate Pod Disruption Budgets and node affinity to reduce the chance of eviction during node churn.

ScaledJob parallelism and resource exhaustion can interact badly. ScaledJob creates one Job per work item up to maxReplicaCount. If the jobs are resource-heavy and maxReplicaCount is set without accounting for cluster capacity, a large queue depth can trigger a burst of job creation that exhausts available CPU or memory, causing jobs to enter Pending rather than running. Always set resource requests on ScaledJob pod templates and validate that maxReplicaCount * resources fits within realistic cluster capacity.

Polling interval granularity limits real-time reaction. KEDA is a polling-based system, not a push-based one. The minimum pollingInterval is 1 second, but in practice polling too frequently increases load on the external source (Kafka admin API, RabbitMQ management API) and on the KEDA operator itself. For workloads where a 15-30 second reaction time is too slow, KEDA may not be the right fit. HTTP-triggered autoscaling via Knative or a custom external scaler may be more appropriate.

Further reading

Start with the KEDA ScaledObject specification reference [1], which covers every field in the spec including the stabilization window passthrough to HPA. The full scaler catalog [2] documents the configuration options and authentication patterns for each of the 60+ supported triggers. For the scale-to-zero mechanics and the operator’s interaction with HPA, the KEDA FAQ [3] is the most direct source. The TriggerAuthentication reference [4] covers every credential injection pattern. For ScaledJob specifically, the ScaledJob specification [5] documents the parallelism model and history retention fields.

On the Kubernetes side, the HPA walkthrough [6] and the custom metrics API documentation [7] explain the plumbing KEDA sits on top of. For Kafka consumer group lag concepts, the Kafka consumer group documentation [8] and the consumer group offset management guide [9] are the essential grounding. For integrating KEDA with Cluster Autoscaler, the Cluster Autoscaler FAQ [10] covers the interaction between pod-level and node-level scaling. The KEDA CNCF project page [11] is the canonical entry point for governance, roadmap, and release notes. For a broader view of event-driven architecture in Kubernetes, the CloudEvents specification [12], which several KEDA scalers optionally support, is worth knowing.

Written with AI assistance, editorially reviewed.

References

[1] KEDA, “ScaledObject Specification.” https://keda.sh/docs/latest/reference/scaledobject-spec/

[2] KEDA, “Scalers.” https://keda.sh/docs/latest/scalers/

[3] KEDA, “FAQ.” https://keda.sh/docs/latest/faq/

[4] KEDA, “TriggerAuthentication.” https://keda.sh/docs/latest/concepts/authentication/

[5] KEDA, “ScaledJob Specification.” https://keda.sh/docs/latest/reference/scaledjob-spec/

[6] Kubernetes, “Horizontal Pod Autoscaling.” https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/

[7] Kubernetes, “Custom Metrics API.” https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/#support-for-custom-metrics

[8] Apache Kafka, “Consumer Group.” https://kafka.apache.org/documentation/#intro_consumers

[9] Confluent, “Kafka Consumer Group Offset Management.” https://docs.confluent.io/platform/current/clients/consumer.html

[10] Kubernetes, “Cluster Autoscaler FAQ.” https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md

[11] CNCF, “KEDA Project.” https://www.cncf.io/projects/keda/

[12] CloudEvents, “CloudEvents Specification.” https://cloudevents.io/