Edit

Troubleshoot memory saturation in AKS clusters

Summary

This article helps you troubleshoot memory saturation in AKS clusters. Learn how to identify saturated nodes, find high-memory pods and processes, and reduce memory pressure.

Prerequisites

Symptoms

The following table outlines the common symptoms of memory saturation.

Symptom Description
Unschedulable pods You can't schedule more pods if the node is close to its set memory limit.
Pod eviction If a node is running out of memory, the kubelet can evict pods. Although the control plane tries to reschedule the evicted pods on other nodes that have resources, there's no guarantee that other nodes have enough memory to run these pods.
Node not ready Memory saturation can cause kubelet and containerd to become unresponsive, eventually causing node readiness problems.
Out-of-memory (OOM) kill An OOM problem occurs if the pod eviction can't prevent a node problem. For more information, see Troubleshoot OOMkilled in AKS clusters.

Troubleshooting checklist

To reduce memory saturation, use effective monitoring tools and apply best practices.

Step 1: Identify nodes with memory saturation

Use either of the following methods to identify nodes with memory saturation:

  • In a web browser, use the Container Insights feature of AKS in the Azure portal.

  • In a console, use the Kubernetes command-line tool (kubectl).

Container Insights is a feature within AKS that monitors container workload performance. For more information, see Enable Container insights for Azure Kubernetes Service (AKS) cluster.

  1. In the Azure portal, search for and select Kubernetes services.

  2. In the list of Kubernetes services, select the name of your cluster.

  3. In the navigation pane of your cluster, find the Monitoring heading, and then select Insights.

  4. Set the appropriate Time Range value.

  5. Select the Nodes tab.

  6. In the Metric list, select Memory working set (computed from Allocatable).

  7. In the percentiles selector, set the sample to Max, and then select the Max % column label two times. This action sorts the table nodes by the maximum percentage of memory used, from highest to lowest.

    Screenshot of the Nodes view in Container Insights for an AKS cluster in the Azure portal.

    The Azure portal screenshot shows a table of nodes. The table column values include Name, Status, Max % (the percentage of memory capacity that's used), Max (memory usage), Containers, UpTime, Controller, and Trend Max % (1 bar = 15m). The nodes have an expand/collapse arrow icon next to their names.

    There are four rows in the table that represent four nodes in an AKS agent pool virtual machine (VM) scale set. The statuses are all Ok, the maximum percentage of memory used is from 64 to 58 percent, the maximum memory used is from 2.6 GB to 2.86 GB, the number of containers used is 20 to 24, and the uptime spans 6 to 15 days. No controllers are listed. :::image-end:::

  8. Because the first node has the highest memory usage, select that node to investigate the memory usage of the pods that are running on the node.

    Screenshot of container memory details for a selected AKS node in the Nodes view of Container Insights.

    The Azure portal screenshot shows a table of nodes. The first node is expanded to display an Other processes heading and a sublist of processes that are running within the first node. As for the nodes themselves, the table column values for the processes include Name, Status, Max % (the percentage of memory capacity that's used), Max (memory usage), Containers, UpTime, Controller, and Trend Max % (1 bar = 15m). The processes also have an expand/collapse arrow icon next to their names.

    Nine processes are listed under the node. The statuses are all Ok, the maximum percentage of memory used for the processes ranges from 16 to 0.3 percent, the maximum memory used is from 0.7 mc to 22 mc, the number of containers used is 1 to 3, and the uptime is 3 to 4 days. Unlike for the node, the processes all have a corresponding controller listed. In this screenshot, the controller names are prefixes of the process names, and they're hyperlinked. :::image-end:::

    Note

    The percentage of memory usage for pods is based on the memory request specified for the container. It doesn't represent the percentage of memory usage for the node. Look at the actual memory usage values rather than the percentage when assessing pod-level consumption.

After you identify the pods that use high memory, you can identify the applications that run on the pod or identify processes that might be consuming excess memory.

Step 2: Identify process-level memory usage

For advanced process-level memory analysis, use Inspektor Gadget to monitor real-time memory usage at the process level within pods:

  1. Install Inspektor Gadget by following the instructions in the documentation.

  2. Run the top_process gadget to identify processes that use large amounts of memory. Use --fields to select certain columns and --filter to filter events based on specific field values. For example, filter by the pod names of previously identified pods with high memory consumption. You can also use the following commands:

    • Identify top 10 memory-consuming processes across the cluster:

      kubectl gadget run top_process --sort -memoryRelative --max-entries 10

    • Identify top memory-consuming processes on a specific node:

      kubectl gadget run top_process --sort -memoryRelative --filter k8s.node==<node-name>

    • Identify top memory-consuming processes in a specific namespace:

      kubectl gadget run top_process --sort -memoryRelative --filter k8s.namespace==<namespace>

    • Identify top memory-consuming processes in a specific pod:

      kubectl gadget run top_process --sort -memoryRelative --filter k8s.podName==<pod-name>

    The output of the Inspektor Gadget top_process command resembles the following output:

    
   K8S.NODE                          K8S.NAMESPACE          K8S.PODNAME            PID       COMM   MEMORYVIRTUAL    MEMORYRSS   MEMORYRELATIVE     
   aks-agentpool-3…901-vmss000001    default                memory-stress        21676     stress          944 MB       943 MB              5.6     
   aks-agentpool-3…901-vmss000001    default                memory-stress        21678     stress          944 MB       943 MB              5.6      
   aks-agentpool-3…901-vmss000001    default                memory-stress        21677     stress          944 MB       872 MB              5.2     
   aks-agentpool-3…901-vmss000001    default                memory-stress        21679     stress          944 MB       796 MB              4.8

Use this output to identify the processes that consume the most memory on the node. The output can include the node name, namespace, pod name, container name, process ID (PID), command name (COMM), CPU, and memory usage. For more details, see the documentation.

Step 3: Determine the cause of memory pressure

After you identify which pods and processes consume the most memory, determine the nature of the problem before you act. The appropriate remediation depends on whether you're dealing with an OOM kill event, stable high memory usage, or growing memory over time.

Confirm an OOM kill

Exit code 137 indicates that a container received a SIGKILL signal, but this code can have more than one cause. To determine whether an OOM kill occurred, inspect the pod's last termination state:

kubectl describe pod <pod-name> -n <namespace>

The following output shows a container that was terminated because it exceeded its memory limit:

Last State:     Terminated
  Reason:       OOMKilled
  Exit Code:    137
  Started:      Mon, 18 Aug 2025 10:14:32 +0000
  Finished:     Mon, 18 Aug 2025 10:16:08 +0000

The following output shows a container that exited with code 137, but wasn't explicitly reported as OOMKilled:

Last State:     Terminated
  Reason:       Error
  Exit Code:    137

In the Last State section of the output:

  • Reason: OOMKilled — the container exceeded its cgroup memory limit and the Linux OOM killer terminated it. This condition can occur even when the node itself isn't under memory pressure, if the container exceeds its own configured limit.
  • Reason: Error with exit code 137 — the container received a SIGKILL from another source, such as a liveness probe failure, kubelet-initiated eviction, or manual deletion.

Note

The Linux OOM killer uses heuristics to select which process to terminate when memory is exhausted. The result is non-deterministic and the terminated process might not be the highest memory consumer. To influence kubelet eviction decisions under node memory pressure, configure Quality of Service (QoS) classes by setting appropriate resource requests and limits, and use PriorityClass for critical workloads.

Identify the memory usage pattern

The remediation for memory saturation depends on whether memory usage is stable or growing over time.

  • Stable high usage — Memory is consistently high but not increasing. The workload legitimately requires that amount of memory, or requests and limits are misconfigured. Right-size requests and limits to match actual observed usage.

  • Growing memory usage — Memory increases steadily over time and doesn't return to a baseline after load decreases. This pattern typically indicates a memory leak or accumulation problem in the application. Growing memory usage can come from application heap growth, native or off-heap allocations, excessive thread stacks, memory-mapped files, in-memory caches, or memory-backed volumes such as emptyDir with medium: Memory. If the container exceeds its memory limit, the kernel OOM subsystem can terminate a process in the container. Scaling solutions can temporarily reduce pressure but don't resolve the root cause. Application-level investigation is required.

Use the Trend column in Container Insights or the time-series memory chart for a specific pod to identify which pattern applies to your workload.

Step 4: Review best practices to avoid memory saturation

Review the following table to learn how to implement best practices for avoiding memory saturation.

Best practice Description
Use memory requests and limits Kubernetes provides options to specify the minimum memory size (request) and the maximum memory size (limit) for a container. By configuring limits on pods, you can avoid memory pressure on the node. Make sure that the aggregate memory requests for pods that are running don't exceed the node's allocatable memory. Setting limits much higher than requests, or not setting limits, can increase the risk of memory overcommitment at runtime. The Kubernetes scheduler places pods based on resource requests, not actual usage. Additionally, while the kubelet is evicting pods, it prioritizes pods in which the memory usage exceeds their defined requests. Set the memory request close to the actual usage.
Configure QoS classes and pod priority A pod's QoS class is determined by the CPU and memory requests and limits of its containers. Pods receive the Guaranteed class only when every container has CPU and memory requests and limits set, and each request equals its corresponding limit. Pods receive the BestEffort class only when no container has CPU or memory requests or limits set. Configure PriorityClass for critical workloads to influence scheduling and kubelet eviction decisions under resource pressure.
Consider scaling solutions (HPA, VPA, KEDA) Scaling solutions can relieve memory pressure in specific scenarios. Use HPA when memory usage increases proportionally with load and distributing requests across additional replicas reduces per-pod consumption. Use VPA when containers consistently use more or less memory than their configured requests, as it recommends or updates resource requests based on observed usage. Use KEDA when memory pressure correlates with external event load or queue depth. Scaling solutions provide temporary mitigation for memory leaks but don't resolve the underlying cause.
Use dedicated node pools, node affinity, taints and tolerations, and pod anti-affinity For scenarios in which memory usage is unbounded by design, isolate the workload on a dedicated node pool. Use node affinity or node selectors to place the workload on those nodes, and use taints and tolerations to prevent unrelated workloads from scheduling there. Use pod anti-affinity when you need to spread high-memory replicas across nodes so that they don't concentrate memory pressure on a single node.
Choose higher SKU VMs VMs that have more random-access memory (RAM) are better suited to handle high memory usage. To use this option, you must create a new node pool, cordon the nodes (make them unschedulable), and drain the existing node pool.
Isolate system and user workloads Run your applications on a user node pool. This configuration makes sure that you can isolate the Kubernetes-specific pods to the system node pool and maintain the cluster performance.

More information

Third-party information disclaimer

The third-party products that this article discusses are manufactured by companies that are independent of Microsoft. Microsoft makes no warranty, implied or otherwise, about the performance or reliability of these products.

Third-party contact disclaimer

Microsoft provides third-party contact information to help you find additional information about this topic. This contact information may change without notice. Microsoft does not guarantee the accuracy of third-party contact information.

  • Last updated on