AKS Node configuration mismatch

Hi David Poor,

Thank you for sharing the detailed symptoms and command outputs.

Based on the information provided, this looks like the AKS resource provider/control-plane metadata for the system node pool is not aligned with the underlying node/VM state. Since the underlying VMs are still running and kubectl get nodes shows the nodes as Ready, this appears to be a control-plane/node pool metadata issue rather than an immediate node health issue.

Microsoft documentation recommends validating both the AKS cluster provisioning state and the node pool provisioning state, and also checking the underlying VM/VMSS or VM resources, activity logs, quota/capacity, policies, permissions, and locks when a node pool is stuck in Failed state. The AKS failed-state troubleshooting guidance also notes that reconciliation can be triggered by running an Azure Resource Manager update against the AKS cluster resource.

Please try the following validation/reconciliation steps first:

# Confirm AKS cluster state
az aks show </span>
 --resource-group <resource-group> </span>
 --name <cluster-name> </span>
 --query "{provisioningState:provisioningState, powerState:powerState.code, kubernetesVersion:kubernetesVersion}" </span>
 -o table
 
# Get AKS cluster resource ID
AKS_ID=$(az aks show </span>
 --resource-group <resource-group> </span>
 --name <cluster-name> </span>
 --query id -o tsv)
 
# Trigger AKS managed cluster reconciliation
az resource update --ids $AKS_ID

After the reconciliation completes, validate whether the node pool is visible again:

az aks nodepool list </span>
 --resource-group <resource-group> </span>
 --cluster-name <cluster-name> </span>
 -o table
 
az aks nodepool show </span>
 --resource-group <resource-group> </span>
 --cluster-name <cluster-name> </span>
 --name system </span>
 -o json

Also review the Azure Activity Log around the time of the failed upgrade/update operation to identify the exact failure code and operation that left the node pool in the failed state:

az monitor activity-log list </span>
 --resource-group <resource-group> </span>
 --status Failed </span>
 --max-events 50 </span>
 -o table

Regarding the QuotaExceeded - unable to create minimum required surge nodes message: AKS upgrades might require temporary surge capacity. Microsoft documentation notes that quota/capacity limitations can block create, scale, or upgrade operations, and for allocation/capacity-related upgrade failures, using maxUnavailable instead of surge can help avoid creating additional buffer nodes during upgrade. This should only be tried after the node pool metadata is restored and the node pool is visible again.

Note: Please avoid directly deleting, resizing, or modifying the underlying VMs outside the AKS API. AKS manages the lifecycle of agent nodes, and direct customisation of the underlying IaaS resources is not supported because such changes may not persist through AKS update, scale, upgrade, or reboot operations.

If the system node pool still does not appear in az aks nodepool list after cluster reconciliation, this will likely require Microsoft Support/AKS backend investigation because the customer-side AKS API no longer has a valid node pool object to operate on.

Referenced documents:

https://learn.microsoft.com/en-us/troubleshoot/azure/azure-kubernetes/availability-performance/cluster-node-virtual-machine-failed-state

https://learn.microsoft.com/en-us/troubleshoot/azure/azure-kubernetes/create-upgrade-delete/aks-at-scale-troubleshoot-guide

https://learn.microsoft.com/en-us/troubleshoot/azure/azure-kubernetes/availability-performance/node-not-ready-basic-troubleshooting

Please review above shared steps and let us know if you have any further queries.

Hello @Prasad Chaganti thanks for the breakdown, I have gone through the process, and still reconciliation fails after steps indidcated, and nodepool still doesn't show. Myself and @David Poor were trying to create a support request for this issue, but were forced to only access this Q&A. How do we raise the ticket, cause we think we are at that point of backend intervention as we speak.