Create and manage persistent volumes (PVs) with Azure Blob storage in Azure Kubernetes Service (AKS)

If multiple pods need concurrent access to the same storage volume, you can use Azure Blob storage to connect using blobfuse or Network File System (NFS).

This article shows you how to dynamically and statically create Azure Blob storage containers for use by multiple pods in an Azure Kubernetes Service (AKS) cluster.

Prerequisites

• The Azure Blob storage CSI driver enabled on your AKS cluster.

• A storage account that's NFS v3 enabled so you can mount a persistent volume using the NFS protocol. You can't enable NFS v3 on an existing storage account. For more information, see Create an NFS v3 account.

• To support an Azure Data Lake Storage account when using blobfuse mount, complete the following tasks:

  • To create a Data Lake Storage account using the driver in dynamic provisioning, specify isHnsEnabled: "true" in the storage class parameters.
  • To enable blobfuse access to a Data Lake Storage account in static provisioning, specify the mount option --use-adls=true in the persistent volume.
  • If you're going to enable a storage account with Hierarchical Namespace, existing persistent volumes should be remounted with --use-adls=true mount option.

• By default, the blobfuse cache is located in the /mnt directory. If the VM SKU provides a temporary disk, the /mnt directory is mounted on the temporary disk. However, if the VM SKU doesn't provide a temporary disk, the /mnt directory is mounted on the OS disk, you can set --tmp-path= mount option to specify a different cache directory.

Use built-in storage classes to create dynamic PVs with Azure Blob storage

A storage class is used to define how an Azure Blob storage container is created. A storage account is automatically created in the node resource group for use with the storage class to hold the Azure Blob storage container. When you use storage CSI drivers on AKS, there are two extra built-in StorageClasses that use the Azure Blob storage CSI driver.

The reclaim policy on both storage classes ensures that the underlying Azure Blob storage is deleted when the respective PV is deleted. The storage classes also configure the container to be expandable by default, as the set allowVolumeExpansion parameter is set to true.

You can select one of the following Azure storage redundancy SKUs for the skuname parameter in the storage class definition:

• Standard_LRS: Standard locally redundant storage
• Premium_LRS: Premium locally redundant storage
• Standard_ZRS: Standard zone redundant storage
• Premium_ZRS: Premium zone redundant storage
• Standard_GRS: Standard geo-redundant storage
• Standard_RAGRS: Standard read-access geo-redundant storage

Create custom storage classes for dynamic PVs with Azure Blob storage

The default storage classes are suitable for most scenarios. In some cases, you might want to have your own storage class customized with your own parameters. In this section, we provide two examples: one using NFS protocol and one using blobfuse.

Custom storage class example using NFS protocol

The manifest in this example mounts a blob storage container using the NFS protocol. You can use it to add the tags parameter.

1. Create a file named blob-nfs-sc.yaml and paste in the following example manifest:

   apiVersion: storage.k8s.io/v1
   kind: StorageClass
   metadata:
     name: blob-nfs
   provisioner: blob.csi.azure.com
   parameters:
     protocol: nfs
     tags: environment=Development
   volumeBindingMode: Immediate
   allowVolumeExpansion: true
   

2. Create the storage class using the kubectl apply command:

   kubectl apply -f blob-nfs-sc.yaml
   

   Your output should resemble the following example output:

   storageclass.storage.k8s.io/blob-nfs created
   

Custom storage class example using blobfuse

The manifest in this example uses blobfuse and mounts a Blob storage container. You can use it to update the skuName parameter.

1. Create a file named blobfuse-sc.yaml and paste in the following example manifest:

   apiVersion: storage.k8s.io/v1
   kind: StorageClass
   metadata:
     name: blob-fuse
   provisioner: blob.csi.azure.com
   parameters:
     skuName: Standard_GRS  # available values: Standard_LRS, Premium_LRS, Standard_GRS, Standard_RAGRS
   reclaimPolicy: Delete
   volumeBindingMode: Immediate
   allowVolumeExpansion: true
   mountOptions:
     - -o allow_other
     - --file-cache-timeout-in-seconds=120
     - --use-attr-cache=true
     - --cancel-list-on-mount-seconds=10  # prevent billing charges on mounting
     - -o attr_timeout=120
     - -o entry_timeout=120
     - -o negative_timeout=120
     - --log-level=LOG_WARNING  # LOG_WARNING, LOG_INFO, LOG_DEBUG
     - --cache-size-mb=1000  # Default will be 80% of available memory, eviction will happen beyond that.
   

2. Create the storage class using the kubectl apply command:

   kubectl apply -f blobfuse-sc.yaml
   

   Your output should resemble the following example output:

   storageclass.storage.k8s.io/blob-fuse created
   

Storage class parameters for dynamic PVs with Azure Blob storage

Use the following parameter groups to define a custom storage class for your persistent volume claims (PVCs) with Azure Blob storage:

• Storage account: skuName, location, resourceGroup, subscriptionID, storageAccount, networkEndpointType, accessTier, allowBlobPublicAccess, allowSharedKeyAccess, requireInfraEncryption, publicNetworkAccess, tags, and matchTags.
• Container and endpoint: protocol, containerName, containerNamePrefix, server, storageEndpointSuffix, useDataPlaneAPI, softDeleteBlobs, softDeleteContainers, and enableBlobVersioning.
• BlobFuse: storeAccountKey, getLatestAccountKey, secretName, secretNamespace, and isHnsEnabled.
• NFS: mountPermissions and fsGroupChangePolicy.
• Virtual network: vnetResourceGroup, vnetName, subnetName, and vnetLinkName.

Configure storage account parameters for dynamic PVs with Azure Blob storage

Use the following parameters to configure the Azure storage account for a dynamically provisioned PV:

Configure container and endpoint parameters for dynamic PVs with Azure Blob storage

Use the following parameters to configure the container, mount protocol, storage endpoint, and tags for a dynamically provisioned PV:

Configure BlobFuse parameters for dynamic PVs with Azure Blob storage

The following parameters apply only when you use BlobFuse for a dynamically provisioned PV:

Configure NFS parameters for dynamic PVs with Azure Blob storage

The following parameter applies only when you use NFS for a dynamically provisioned PV:

Configure virtual network parameters for dynamic PVs with Azure Blob storage

Use the following parameters when the driver configures virtual network access for a dynamically provisioned PV:

Configure private endpoints for dynamic PVs with Azure Blob storage

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: blob-fuse
provisioner: blob.csi.azure.com
parameters:
  skuName: Premium_LRS  # available values: Standard_LRS, Premium_LRS, Standard_GRS, Standard_RAGRS, Standard_ZRS, Premium_ZRS
  protocol: fuse2
  networkEndpointType: privateEndpoint
reclaimPolicy: Delete
volumeBindingMode: Immediate
allowVolumeExpansion: true
mountOptions:
  - -o allow_other
  - --file-cache-timeout-in-seconds=120
  - --use-attr-cache=true
  - --cancel-list-on-mount-seconds=10  # prevent billing charges on mounting
  - -o attr_timeout=120
  - -o entry_timeout=120
  - -o negative_timeout=120
  - --log-level=LOG_WARNING  # LOG_WARNING, LOG_INFO, LOG_DEBUG
  - --cache-size-mb=1000  # Default will be 80% of available memory, eviction will happen beyond that.

Create a PVC for dynamic provisioning

A PVC uses the storage class object to dynamically provision an Azure Blob storage. You can use the example YAML manifest in this section to create a PVC that's 5 GB in size with ReadWriteMany access. For more information on access modes, see Kubernetes PV access modes.

1. Create a file named blob-nfs-pvc.yaml and paste in the following YAML manifest:

   apiVersion: v1
   kind: PersistentVolumeClaim
   metadata:
     name: azure-blob-storage
   spec:
     accessModes:
     - ReadWriteMany
     storageClassName: azureblob-nfs-premium
     resources:
       requests:
         storage: 5Gi
   

2. Create the PVC using the kubectl create command:

   kubectl create -f blob-nfs-pvc.yaml
   

3. View the status of the PVC with the kubectl get command:

   kubectl get pvc azure-blob-storage
   

   Your output should resemble the following example output, which shows that the PVC is in a Bound state:

   NAME                 STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                AGE
   azure-blob-storage   Bound    pvc-aaaaaaaa-0000-1111-2222-bbbbbbbbbbbb   5Gi        RWX            azureblob-nfs-premium       92m
   

Mount a dynamically provisioned blob storage volume in a pod

The following YAML creates a pod that uses the persistent volume claim azure-blob-storage to mount the Azure Blob storage at the /mnt/blob path.

1. Create a file named blob-nfs-pv and paste the following YAML manifest. Make sure the claimName matches the PVC you created earlier (azure-blob-storage).

   kind: Pod
   apiVersion: v1
   metadata:
     name: mypod
   spec:
     containers:
     - name: mypod
       image: mcr.microsoft.com/oss/nginx/nginx:1.17.3-alpine
       resources:
         requests:
           cpu: 100m
           memory: 128Mi
         limits:
           cpu: 250m
           memory: 256Mi
       volumeMounts:
       - mountPath: "/mnt/blob"
         name: volume
         readOnly: false
     volumes:
       - name: volume
         persistentVolumeClaim:
           claimName: azure-blob-storage
   

2. Create the pod using the kubectl apply command:

   kubectl apply -f blob-nfs-pv.yaml
   

3. Once the pod is successfully running, create a new file named test.txt using the following command:

   kubectl exec mypod -- touch /mnt/blob/test.txt
   

4. Validate the disk is correctly mounted using the following command to list the files in the mounted directory:

   kubectl exec mypod -- ls /mnt/blob
   

   Your output should resemble the following example output, which shows the test.txt file you created in the mounted Azure Blob storage:

   test.txt
   

Use a StatefulSet to manage the lifecycle of a volume with Azure Blob storage

To have a storage volume persist for your workload, you can use a StatefulSet. This state makes it easier to match existing volumes to new pods that replace any that have failed. The following examples demonstrate how to set up a StatefulSet for Blob storage using the NFS protocol or Blobfuse.

Create a static PV with Azure Blob storage

The following sections provide instructions for creating a static PV with Azure Blob storage. A static PV is a persistent volume that an administrator creates manually. This PV is available for use by pods in the cluster. To use a static PV, you create a PVC that references the PV, and then create a pod that references the PVC.

CSI volume parameters for static PVs with Azure Blob storage

The following table lists parameters you can use in the CSI volume source for a static PV with Azure Blob storage:

Create a Blob storage container

When you create an Azure Blob storage resource for use with AKS, you can create the resource in the node resource group. This approach allows the AKS cluster to access and manage the blob storage resource.

1. Get the node resource group name of your AKS cluster using the az aks show command with the --query nodeResourceGroup parameter.

   az aks show --resource-group myResourceGroup --name myAKSCluster --query nodeResourceGroup -o tsv
   

   The output of the command resembles the following example:

   MC_myResourceGroup_myAKSCluster_eastus
   

2. If you create the storage account in the node resource group, use the node resource group name returned in the previous step (for example, MC_myResourceGroup_myAKSCluster_eastus). Then, follow the steps in Manage blob storage to authorize access and create a container in that storage account.

Mount volume

In this section, you mount the persistent volume using the NFS protocol or Blobfuse.

Use the persistent volume

The following YAML creates a pod that uses the PV or PVC named pvc-blob created earlier to mount the Azure Blob storage at the /mnt/blob path.

1. Create a file named nginx-pod-blob.yaml and paste in the following YAML manifest. Make sure the claimName matches the PVC created earlier (pvc-blob).

   kind: Pod
   apiVersion: v1
   metadata:
     name: nginx-blob
   spec:
     nodeSelector:
       "kubernetes.io/os": linux
     containers:
       - image: mcr.microsoft.com/oss/nginx/nginx:1.17.3-alpine
         name: nginx-blob
         volumeMounts:
           - name: blob01
             mountPath: "/mnt/blob"
             readOnly: false
     volumes:
       - name: blob01
         persistentVolumeClaim:
           claimName: pvc-blob
   

2. Create the pod and mount the PVC using the kubectl create command:

   kubectl create -f nginx-pod-blob.yaml
   

3. Create an interactive shell session with the pod to verify the Blob storage is mounted correctly using the following kubectl exec command:

   kubectl exec -it nginx-blob -- df -h
   

   Your output should resemble the following example output, which shows the Blob storage is mounted at the /mnt/blob path:

   Filesystem      Size  Used Avail Use% Mounted on
   ...
   blobfuse         14G   41M   13G   1% /mnt/blob
   ...
   

Related content

• Use Azure tags in Azure Kubernetes Service (AKS)
• Known issues with Network File System (NFS) 3.0 protocol support for Azure Blob Storage