Kubernetes is the leading managed container orchestration platform preferred by customers deploying microservices-based architectures in the cloud. Azure Kubernetes Service offers Kubernetes as a managed service, where the container orchestration platform is handled by Azure, enabling customers to focus on the developments of applications. Then how does AKS meet Kubernetes storage demands?

Containers are stateless, which means data is not stored locally—these containers depend on attached persistent volumes to handle the data lifecycle. This blog will walk through the steps required for provisioning persistent volumes and configuring them to be used by containers in Azure Kubernetes Service clusters.

What is Persistent Storage?

Persistent Storage is a mechanism to store data outside the lifecycle of a container or application, ensuring that critical information (like databases, user uploads, or configuration files) persists even if the container restarts, crashes, or is replaced. It’s essential for stateful applications (e.g., databases, CMS) that require data durability.

By default, the storage associated with pods is deleted when the pod lifecycle ends. For stateful applications, however, storage is expected to be persistent so that the data can remain available every time the pods gets recreated in the cluster. Container persistent storage in a Kubernetes cluster is provisioned using the PersistentVolume subsystem, which provides PersistentVolume and PersistentVolumeClaim API resources.

The PersistentVolumeClaim requests for a specific storage class, say Azure disks or Azure Files, and the underlying storage resource gets provisioned. This PersistentVolumeClaim is referenced in the pod definition so that the provisioned storage is mounted to the pod. In this way the PersistentVolume is linked to the PersistentVolumeClaim whenever the provisioned storage is mounted to the pod requesting the resource.

Note: The following steps will assume that the AKS cluster is already provisioned and that the administrator has access to execute the commands listed in this blog.

Provision Persistent Storage Using Azure Disks

There are four major steps in creating and attaching persistent storage using Azure disks in Azure Kubernetes Service.

1. Define/Create of storage class.
2. Configuration of persistent volume claim (PVC) that references of storage class.
3. Create the persistent volume claim and provision the volume.
4. Reference the PVC in the Pod specification to attach the Azure Disk ie., Attaching the provisioned volume to the pod by referencing the specific pod in the pod definition file.’

Let’s explore this process in detail with sample configuration files.

How AKS Uses Azure Disk Storage Classes

Azure Kubernetes Service (AKS) simplifies persistent storage management by offering two pre-configured storage classes for Azure Disks: default and managed-premium. These classes let teams provision storage tailored to workload requirements while abstracting backend complexity. Here’s how they work and where they shine—or fall short.

• Built-in Storage Classes
• default (Standard HDD)

1. default (Standard HDD)
   • Backend: Relies on cost-effective Azure Standard HDD storage.
   • Use Case: Ideal for non-critical workloads like backups, logs, or dev/test environments where high throughput isn’t a priority.
2. managed-premium (Premium SSD)
   • Backend: Leverages Azure Premium SSD for low latency and high IOPS.
   • Use Case: Suited for production workloads (e.g., databases, transactional apps) demanding consistent performance and faster data access.

Limitations of Built-in Classes

While convenient, the pre-built storage classes come with constraints:

No Post-Provisioning Resizing: Once a volume is created, you can’t expand its size—a hurdle for growing datasets or scaling applications.

Fixed Configuration: Default settings (e.g., performance tiers, redundancy) may not align with specialized needs.

Default: This storage class uses the standard Azure storage which leverages HDDs in the backend.

Creating Custom Storage Classes

In this section we’ll show you how users can create custom storage classes to suit their specific requirements.

1. Make Ready of StorageClass YAML

Connect to the AKS cluster from your management tool of choice (here i used powershell/CMD to connect the cluster using kubectl). For this demonstration we will be using Azure CL. From the Azure CLI, save the following manifest as storage-class.yaml

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: managed-standard-custom
provisioner: disk.csi.azure.com
reclaimPolicy: retain
allowVolumeExpansion: true
parameters:
  storageaccounttype: Premium_LRS
  kind: Managed

This file can be used to create a storage class that uses Premium_LRS managed disks. The reclaimPolicy is set to retain so that the persistent volume is not deleted even if the pod to which it is attached gets deleted. With the parameter allowVolumeExpansion set to true, the volume can now be resized even after provisioning.

2. Apply the yaml file using the following command

$ kubectl apply -f storage-class.yaml

Upon successful execution, you will get a message that the storage class has been created.

Create Persistent Volume Claim and Persistent Volume

The next step is to create a persistent volume claim (PVC), which uses the storage class defined in the above to provision an Azure disk as a persistent volume.

1. Create custom-pvc.yaml file in the Azure CLI window

Create the PVC creation YAML as shown below,

The PVC will request for an Azure disk of 10 GB with accessmode as ReadWriteOnce. That means only one node can mount the volume as read-write.

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: custom-pvc
  namespace: default
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  storageClassName: managed-standard-custom
  volumeMode: Filesystem

2. To create the volume itself, run the following command:

kubectl apply -f custom-pvc.yaml

On successful execution you will see a message that the persistent volume claim has been created.

After creation, the provisioned volume can also be seen from the Azure portal. Browse to the resource group where the AKS nodes are created to find the newly provisioned disk listed there. 

As highlighted in yellow, you can able to view  Persistent Volume has created with name of  “pvc-ae14b4f9-68d5-4aed-baf3-6addfe4e8b3d”

Once PVC is create successfully (status is bound) then the Persistent volume will create automatically and mapped to the PVC (refer above and below image)

And you can also verify disk with name of Persistent Volume by searching in the Azure portal as shown in below, where our mount data will be stored.

You can also run the following command to list the pvc status. It will be listed as “bound” to the created persistent volume:

You can also get all the PVC in your cluster, you can use kubectl get pvc

Attach a Persistent Volume to AKS

After creating the persistent volume claim and the Azure disk, it can be attached to a new pod by referencing the name of the persistent volume claim in the deployment or pod yaml.

1. Create file Nginx.yaml with the following content

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      restartPolicy: Always
      containers:
        - name:  nginx
          image: crdgtlshared02.azurecr.io/nginx:latest
          volumeMounts:
            - name: volume
              mountPath: /tmp/
      volumes:
        - name: volume
          persistentVolumeClaim:
            claimName: custom-pvc
      nodeSelector:
        kubernetes.azure.com/agentpool: userlnxpool
        kubernetes.azure.com/mode: user
        kubernetes.io/os: linux

The volume is mounted at /tmp/ as specified by the mountpath parameter.

Note that the image for the container is taken from my Azure container registry. Alternatively, it can also be downloaded from the Azure container registry.

2. Next, create the pod using the following command:

kubectl apply -f Nginx.yaml

When successfully executed, we can able to view the deployment is created successfully as shown below.

We also ensure in Azure portal by viewing pod will be created as shown below images under default namespace.

3. Verify that the persistent volume is attached to the pod 

kubectl describe pod nginx

The PVC will request for an Azure disk of 10 GB with access mode as ReadWriteOnce. That means only one node can mount the volume as read-write.

You can see that the volume is listed and and uses the claim name “custom-pv.”

Check Volume mount in your pod

To connect to the running Azure container instance, use the following command and run the df command to view the volumes:

kubectl exec -it <pod-name> -n <namespace> -- /bin/bash
kubectl exec -it nginx-7466c47dd6-h9tvz -n default -- /bin/bash

/dev/sdb : >> This indicates the device file associated with the storage. In this case, /dev/sdb refers to the second SCSI disk on the system.]

1K-blocks: 10218772 >>This shows the total size of the filesystem in 1-kilobyte blocks. Here, it amounts to approximately 10,218,772 KB, or roughly 9.75 GB.

Use%: 1% >> This shows the percentage of the filesystem’s capacity that is currently used. In this case, only 1% of the space is utilized.

Mounted on: /tmp >>This specifies the mount point, which is the directory where the filesystem is attached to the system. Here, the device /dev/sdb is mounted on /tmp, the standard directory for temporary files in Unix-like systems.

Managing autoscaling in Azure Kubernetes Service (AKS) is not always straightforward. Recently, we ran into a problem where node pool scale down was not happening in both regions, and workloads were consuming resources beyond the defined cluster threshold. This article walks through the scenario, investigation, and resolution.

Scenario: Node Pool Not Scaling Down

The Context: A Planned Node Migration

Our production AKS cluster was running a node pool named mslnxpool02 (D-series VMs) on Kubernetes version 1.31.9. Following a performance review, Microsoft recommended migrating our workloads to E-series VMs for better resource utilization and cost-efficiency aligned with our application’s profile.

1. To execute this migration, our team proceeded with a blue-green approach:
2. Created a new node pool with E-series VMs.
3. Used identical node labels on both the old (D-series) and new (E-series) node pools.

Relied on our Deployments, which used these labels as node Selectors (from starting we having this node selectors so only applied same label to new node pool), to automatically schedule new pods onto the E-series nodes as we cordoned and drained the D-series nodes.

The strategy was to incrementally scale up the E-series pool while simultaneously scaling down the D-series pool, allowing the cluster auto scaler to seamlessly re-schedule pods and reduce node count.

The Problem: Scale-Down Stalls, Incurring Cost

The initial phase of scaling up the new node pool and migrating pods worked as expected. However, we soon noticed a critical issue: the D-series node pool failed to scale down.

Despite workloads being successfully migrated and cluster resource usage dropping significantly, several nodes in the mslnxpool02 pool remained active. Even after waiting for an extended period to account for the cluster auto scaler’s cooldown delays, the node count remained stubbornly high. This resulted in unnecessary cloud costs and a cluster full of underutilized nodes.

The question was: Why was the scale-down stuck?

The following sections detail our investigation and the root cause we uncovered.

Step 1: Investigating the Node health

In my cluster, I had a node pool mslnxpool02 with multiple nodes running Kubernetes v1.31.9. Despite low workloads, the cluster auto scaler wasn’t scaling down the pool. Checking the nodes showed that they were all in a Ready state:

#check with powershell
kubectl get nodes | findstr lnxpool02

At first glance, the nodes looked healthy, but the auto scaler wasn’t freeing up any of them.

Step 2: Describe the Each Node for cause

To understand why the node couldn’t be scaled down, I began with a detailed inspection of one of the stuck nodes: aks-mslnxpool02-33931444-vmss000017.

kubectl describe node aks-lnxpool02-33931444-vmss000017

This command provides a comprehensive overview of the node’s status, capacity, and all the pods running on it. Here’s a breakdown of what I looked for and what I found:

1. The node was Ready.
2. Multiple workloads were running on it, including system pods (e.g., kube-proxy, csi-azure*, ama-logs, myapplication pods).
3. One workload (my service) (customerfeed-service) had large CPU and memory requests as shown below:

CPU Requests: 6100m of 7820m (78%)
CPU Limits: 13 cores (166%)
Memory Requests: 2688Mi
Memory Limits: 10Gi

 

This explained part of the problem: auto scaler cannot evict pods that request a large percentage of node resources unless they can be rescheduled elsewhere.

Now we found the cause, the scaling down was not happened due to the high utilization of some application pods, which not allowing Node pool to scale down.

Step 3: Isolating the Node with Cordon Command

After identifying that the node was blocked from scaling down due to the customerfeed-service pod, the next step was to manually intervene and signal to the cluster that this node was a candidate for removal. This is where the kubectl cordon command comes in.

The primary goal of cordoning a node is to isolate it from receiving any new workloads. Think of it as putting up a “Do Not Enter” sign for the Kubernetes scheduler. This is a crucial, non-disruptive first step in any node maintenance or decommissioning process.

In simple , To prepare the node for scale-down, I manually cordoned it so no new pods would schedule on it:

kubectl cordon aks-mslnxpool02-33931444-vmss000017

Now the node appeared as Ready,SchedulingDisabled

kubectl get nodes | findstr mslnxpool02

aks-lnxpool02-33931444-vmss000017 Ready,SchedulingDisabled <none> 47d v1.31.9

This change told the cluster auto scaler that the node was un-schedulable.

In my case, i had done cordon for all the node pool and  you can confirm in the UI as well like below

Key Takeaways Highlights:

• cordon is Non-Disruptive: It’s a safe first step that doesn’t affect running pods.

• It’s a Signal: It tells both the Kubernetes scheduler and the Cluster Autoscaler to avoid this node.

• It’s Not a Solution by Itself: Cordoning prepares the node but doesn’t solve the underlying resource fragmentation problem. It’s often the prelude to a drain operation.

Step 4: Allocation adjustment or restart the effected pod

After cordoning the node pool, we needed to address the affected pods that were preventing the autoscaler from acting. Simply cordoning ensures no new pods are scheduled on the node, but the existing high-utilization pod continues running and blocking scale down.

To resolve this, we took two possible actions:

Option A:  Restart the Affected Pod (Quick Fix)

If adjusting manifests immediately isn’t feasible (for example, in production during peak hours), a faster approach is to restart the affected pod so it can be rescheduled to a healthier node pool(as old node pool is restricted to schedule so it will map to new node pool):

1. Delete the Pod

 
kubectl delete pod <pod-name> -n <namespace>

2. Kubernetes Rescheduling

Kubernetes automatically recreates the pod on another available node pool.

This helps free up utilization on the cordoned pool, enabling the Cluster Auto scaler to scale it down.

⚠️ Note: This is a temporary workaround. Without fixing the resource requests in the deployment, the issue may reappear.

Option B:  Allocation Adjustment (Preferred)

1. Review Pod Resource Requests and Limits

Check the resources.requests and resources.limits configuration in the deployment manifest. In our case, the application pod had very high actual CPU utilization, but its configuration did not reflect realistic usage, keeping the node pinned.

In our case, the application pod had very high CPU requests, which forced the scheduler to keep nodes active.

Example snippet:

resources:
  requests:
    cpu: "2000m"   # 2 cores requested
    memory: "4Gi"
  limits:
    cpu: "3000m"
    memory: "6Gi"

2. Adjust Resource Requests

Increase the CPU or memory requests (based on observed utilization) to match actual needs. This ensures Kubernetes schedules pods more accurately and avoids overcommitting nodes.

Example snippet:

resources:
  requests:
    cpu: "6000m"   # increased to 4 core
    memory: "4Gi"
  limits:
    cpu: "4000m"
    memory: "6Gi"

This alignment gives the auto scaler a true view of resource usage, enabling it to correctly evaluate which nodes can safely scale down.

Conclusion

Node pool scale-down issues in AKS often come down to workload placement and resource requests. In this scenario, a single oversized pod was preventing the auto scaler from acting. By analyzing node workloads, cordoning, and draining, I was able to resolve the problem and allow auto scaler to scale down efficiently.

Managing Kubernetes resources efficiently is essential for scalable cloud-native applications. The kubectl patch command lets you quickly update running Kubernetes resources — such as labels, container images, or replicas — without redeploying entire configurations. In this article, you’ll learn how to use kubectl patch within Azure Kubernetes Service (AKS), complete with practical examples.

What is the kubectl patch Command?

Patch is a command line option for updating Kubernetes API objects. You can use it to update a running configuration. You do this by supplying it with the section to update, instead of a completely new configuration, as you would with kubectl apply.

The command works by “patching” changes onto the current resource configuration. Unlike a kubectl replace operation, the patch operation only modifies specific fields in the resource configuration.

What can we do using Kubectl patch command?

With kubectl patch, you can quickly fix issues with updating the name, image, label, replicas, affinity/tolerations, environment variables, configaps, secrets, volumes, ports, etc. kubectl patch supports YAML and JSON formats. Using either format, you can drill into the specific field of the resource and change it to your desired value. Kubectl supports three different patching strategies: Strategic Merge (the default) and JSON Merge. You utilize these formats through the patching strategy.

“kubectl patch supports three strategies: Strategic Merge Patch (default), JSON Merge Patch, and JSON Patch. Each has different use cases depending on the complexity of your update.”

Sample deployment YAML file

Let’s say we have the following Nginx deployment YAML file, we originally had only one label (environment: prod), and we want to add a new label to specify the app we are running (app: nginx). We will use the –patch command to enter the new labels:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
        - name: nginx
          image: nginx:1.14.2
          ports:
            - containerPort: 80

Example 1: Patch the label to the deployment

Method 1: Patch using Inline JSON 

In our case, we would like to add environment for the deployment as prod, for this we would run the following command to add the new labels:

kubectl patch deployment nginx-deployment --patch '{\"metadata\": {\"labels\": {\"environment\": \"prod\"}}}'

For the patch command to work, you must correctly include the map and lists in the original YAML file, so Kubernetes knows exactly where to go to add/update the new value properly.

We are going to discuss very simple example of using the kubectl patch command to update fields in your Kubernetes deployment file.

If you want to patch from the YAML file, you can create file with the required update and save as .YAML as shown in below and execute the file using kubectl.

Method 2: Patch using YAML file

“This patch file only contains the section you want to update, not the full deployment manifest.

#Save the file using below manifest (patch.yaml)
metadata:
  labels:
    environment: prod

#execute the command to apply the new changes in the existing pod
kubectl patch deployment nginx-deployment --patch-file ./thi/patch.yaml

You can run the following to check the YAML for the new updates:

#View the YAML in the notpad
kubectl edit deployment nginx-deployment
#View the YAML in the console itself
kubectl get deployment nginx-deployment -o yaml

In return, we get the following YAML file:

Example 2: Patch the label to the Pod level

In this same scenario, if we wanted to update the labels within the Pod template, you would have to drill into the spec field instead of the metadata.

Usually, you would drill into the metadata and spec fields for kubectl patch, but this would also work with other YAML fields, such as status or even a new custom field you created through a CRD (custom resource definition).

In the following example, we are attempting to add a new label environment: prod to the existing labels within the pod template, as the following:

#replace : Modifies the value of an existing field.
#add: Adds/replace a new field or value to an array.
#remove: Removes a field or an element from an array.
#copy: Copies the value of one field to another
#test: Verifies that a specific field has a certain value. The patch operation will fail if the condition is not met
#move: Moves a value from one location to another within the object
kubectl patch deployment nginx-deployment --type='json' --patch='[{"op": "add", "path": "/spec/template/metadata/labels/environment", "value":"prod"}]

The above cmd will be used for both adding new or updating the existing fields. To check if the deployment got the new labels, you can run the following:

kubectl get deployment nginx-deployment --show-labels

Example 3: Update Pod label and container image

We will attempt to update the image of the ingress from older version to new version and update the label value together . This will require us to drill further into the list (the – is a list and will use [ ] to tap into it from the kubectl patch command) instead of a map (uses { }):

kubectl patch deployment nginx-deployment --patch '{\"spec\": {\"template\": {\"metadata\": {\"labels\": {\"environment\": \"production\"}}, \"spec\": {\"containers\": [{\"name\": \"nginx\", \"image\": \"nginx:1.14.2\"}]}}}}'

Conclusion

The kubectl patch command is a versatile tool for managing Kubernetes resources, particularly in dynamic environments like Azure Kubernetes Service. It allows for targeted updates with minimal effort, making it ideal for quick fixes and operational adjustments. By mastering the kubectl patch command, AKS administrators can maintain agility and efficiency in managing their clusters.

Whether you are adding labels, scaling deployments, or updating container images, kubectl patch ensures you can implement changes quickly and confidently. Embrace this command to streamline your Kubernetes workflows and keep your AKS environment optimized.

“Next time you’re troubleshooting or making small updates in AKS, try kubectl patch instead of redeploying. What’s your favorite kubectl command? Drop it in the comments 👇”

Frequently Asked Questions (FAQ)

1. When should I use kubectl patch instead of kubectl apply?

Use kubectl patch when you want to make quick, targeted changes to a Kubernetes resource without redeploying the entire manifest. For example, updating a label, changing an image tag, or adding an environment variable can be done faster with kubectl patch. If you need to manage large or version-controlled manifests, kubectl apply is still the better choice.

2. Can I rollback a kubectl patch?

No, kubectl patch itself does not provide a rollback mechanism. Once applied, the changes are immediately reflected in the resource. To revert, you must either:

• Manually patch again with the previous values, or

• Redeploy the original manifest using kubectl apply -f <file.yaml>.
  Using GitOps or storing manifests in source control is recommended to track and revert changes easily.

3. What are common errors with kubectl patch?

Some common errors include:

• Incorrect JSON/YAML syntax – forgetting quotes, brackets, or commas.

• Missing path values – targeting the wrong field (e.g., metadata vs spec.template.metadata).

• Unsupported patch type – not specifying --type=json when using JSON Patch operations.

• Read-only fields – trying to patch fields like status or metadata.uid, which cannot be modified.

4. Does kubectl patch work for all Kubernetes resources?

Yes, kubectl patch works for most standard Kubernetes resources like Deployments, Pods, Services, ConfigMaps, and Secrets. However, some Custom Resource Definitions (CRDs) may not fully support all patch types depending on their schema.

5. Is kubectl patch safe to use in production?

Yes, but with caution. Since it makes live changes to resources, there is no automatic rollback. It is best used for small, urgent fixes in production. For long-term changes, update your manifests and use kubectl apply to keep your deployments consistent.