HPA vs VPA: When Running Both Wrong Actively Fights Itself

If you've ever watched a deployment's pod count and memory requests both change every few minutes for no obvious reason, there's a good chance you're running the Horizontal Pod Autoscaler (HPA) and the Vertical Pod Autoscaler (VPA) on the same workload, configured in a way that makes them fight each other instead of cooperate.

Both autoscalers answer "does this workload need more resources," but they answer it in different dimensions, and when they're both watching the same metric on the same pods, they can create a feedback loop that never settles.

1. What each one actually does

The HPA scales the number of pod replicas up or down based on observed metrics — CPU, memory, or custom metrics — compared against a target value. More load, more pods. It changes the replicas field on a Deployment or StatefulSet.

The VPA scales the resource requests and limits on individual pod containers, based on historical usage. It doesn't touch replica count; it rewrites what each pod asks the scheduler for. In Auto or Recreate mode, applying a new recommendation means evicting and recreating the pod with new resource requests, because Kubernetes doesn't support in-place resizing of a running pod's requests on most clusters.

On paper they're complementary: HPA handles "how many," VPA handles "how big." The conflict shows up when both are keyed off the same metric, typically CPU utilization, on the same set of pods.

2. The worked example: how the conflict actually plays out

Say you have a Deployment with an HPA targeting 50% CPU utilization, and a VPA in Auto mode also watching CPU:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: api-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api
  minReplicas: 3
  maxReplicas: 20
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 50

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: api-vpa
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api
  updatePolicy:
    updateMode: "Auto"
  resourcePolicy:
    containerPolicies:
      - containerName: api
        minAllowed:
          cpu: 100m
        maxAllowed:
          cpu: 2
        controlledResources: ["cpu", "memory"]

Here's the sequence that plays out under a sustained traffic increase:

• CPU utilization on existing pods climbs above 50%. The HPA reacts first, since it typically evaluates every 15 seconds, and scales replicas up.
• More replicas spread the same total load thinner. Per-pod CPU usage drops.
• VPA, watching the same CPU usage history (now diluted across more pods), sees lower average usage per pod and recommends a smaller CPU request on the next recommendation cycle (VPA recommender runs roughly every 1-2 minutes by default).
• Pods get recreated with smaller CPU requests. Because HPA's averageUtilization metric is computed as usage / request, shrinking the request raises the computed utilization percentage even though actual usage didn't change.
• The HPA sees utilization spike again (because the denominator shrank), and scales out further.
• More replicas dilute usage again, VPA shrinks requests again.

You get a slow oscillation: replica count climbs, CPU requests shrink, utilization appears to spike again, replica count climbs more. In the worst case the Deployment creeps toward maxReplicas on pods that are each requesting a trivial sliver of CPU, which also wrecks bin-packing on the nodes and can trigger unnecessary cluster-autoscaler scale-out.

3. Why this isn't just a tuning problem

The root cause is that both controllers are closing a feedback loop on the same signal, and each one's action changes the signal the other one reads. HPA's utilization metric has the VPA-controlled request in its denominator; VPA's recommendation has the HPA-controlled replica count as a hidden input (more replicas → different usage distribution → different recommendation). Tightening either controller's thresholds slows the oscillation; it doesn't remove it.

4. The fix: split the metrics, not just the tuning

The documented, supported pattern is to make HPA and VPA watch different resources on the same workload, so neither one's output is the other's input.

The most common split: HPA scales on a custom or external metric (requests per second, queue depth, a Prometheus-sourced metric via the custom metrics adapter) instead of CPU, while VPA manages CPU and memory requests uncontested:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: api-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api
  minReplicas: 3
  maxReplicas: 20
  metrics:
    - type: Pods
      pods:
        metric:
          name: http_requests_per_second
        target:
          type: AverageValue
          averageValue: "100"

Now HPA scales replica count based on request throughput per pod, a signal VPA has no influence over, and VPA continues adjusting CPU/memory requests based on actual observed usage, a signal HPA no longer reads. The two loops stop feeding each other.

If you genuinely can't move off CPU-based HPA — no metrics pipeline for a custom metric yet — the safer fallback is to restrict VPA to memory only and leave CPU requests static, set deliberately high enough to give HPA a stable denominator:

resourcePolicy:
  containerPolicies:
    - containerName: api
      controlledResources: ["memory"]

This is a real tradeoff — you lose VPA's CPU right-sizing — but it's a working, stable configuration rather than a slow oscillation.

5. Using VPA in recommendation-only mode alongside HPA

A third option, and often the pragmatic default for production, is running VPA in Off mode so it never actually mutates pods, just emits recommendations you review manually or feed into capacity planning:

updatePolicy:
  updateMode: "Off"

You still get the VerticalPodAutoscalerCheckpoint and recommendation objects (readable via kubectl describe vpa api-vpa) showing target, lower-bound, and upper-bound CPU/memory recommendations, without any pod churn and without touching the HPA's denominator. This is the lowest-risk way to run both tools on one workload while you build out a proper custom-metrics pipeline.

6. Checking for the oscillation in a running cluster

Watch replica count and pod resource requests together over a 30-60 minute window:

kubectl get hpa api-hpa -w
kubectl get events --field-selector reason=EvictedByVPA --sort-by='.lastTimestamp'

If you see VPA evictions clustering right before or after HPA scaling events in the same window, repeatedly, that's the signature of the feedback loop, not coincidence.

Common pitfalls

• Running both in Auto mode on CPU with default settings. This is the exact setup that oscillates. It often looks fine in low-traffic staging and only shows up under real production load.
• Setting minReplicas too low to mask the symptom. It doesn't fix the loop, it just changes where it oscillates.
• Forgetting that VPA Auto/Recreate mode evicts pods. On a workload without multiple replicas or a PodDisruptionBudget, this causes real availability gaps, independent of the HPA conflict.
• Not setting maxAllowed/minAllowed on the VPA. Without bounds, a VPA recommendation during a traffic spike or a weird deploy can request far more or less CPU than intended.
• Assuming the newer in-place VPA resize feature removes all of this. In-place resizing (where supported) avoids pod eviction, but the metric feedback loop between HPA utilization and VPA-controlled requests is unchanged — it's still the same resource, still the same denominator problem.

Troubleshooting

• HPA utilization reads as unstable/spiky despite steady traffic: check whether a VPA is managing the same resource. Compare the pod's resources.requests.cpu over time — if it's changing, that's your denominator moving.
• VPA recommendations seem to chase replica count: confirm the VPA recommender has enough history (it needs sustained observation windows, not just a few minutes) and that HPA scaling events aren't happening faster than VPA's recommendation cycle, which compounds the noise.
• Pods churn constantly with no clear cause: check kubectl describe vpa for the updateMode, and check for both an HPA and a VPA targeting the same scaleTargetRef/targetRef — this is easy to miss when they're defined in separate YAML files by different teams.

Where to go next

Once HPA and VPA are watching disjoint signals, look at the Kubernetes in-place pod resize feature (where your cluster version supports it) to reduce VPA's eviction cost, and consider KEDA if your real scaling signal is an external queue or event source rather than anything Kubernetes-native metrics can see cleanly.

Mumuni Mohammed

Experienced DevOps Engineer proficient in Python, Docker, Kubernetes, CI/CD, and cloud technologies. Specialized in automating processes and building scalable infrastructure for seamless development and deployment.