Use Azure Kubernetes Fleet Manager resource placement to deploy cluster-scoped resources across multiple clusters

1. Set the following environment variables for your subscription ID, resource group, and Kubernetes Fleet resource:

   export SUBSCRIPTION_ID=<subscription-id>
   export GROUP=<resource-group-name>
   export FLEET=<fleet-name>
   

2. Set the default Azure subscription by using the [az account set][az-account-set] command:

   az account set \
       --subscription ${SUBSCRIPTION_ID}
   

3. Get the kubeconfig file of the Kubernetes Fleet hub cluster by using the [az fleet get-credentials][az-fleet-get-credentials] command:

   az fleet get-credentials \
       --resource-group ${GROUP} \
       --name ${FLEET}
   

   Your output should look similar to the following.

   Merged "hub" as current context in /home/fleet/.kube/config
   

   Note

   If you receive an error of type InvalidHubOperation with the message indicating the fleet is hubless, add a hub cluster. For further information, see upgrade hub cluster type.

4. Create a namespace on the hub cluster.

   kubectl create namespace test-app
   

5. Save the following YAML as test-workload.yaml.

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: nginx-deployment
     namespace: test-app
   spec:
     selector:
       matchLabels:
         app: nginx
     replicas: 2
     template:
       metadata:
         labels:
           app: nginx
       spec:
         containers:
         - name: nginx
           image: mcr.microsoft.com/azurelinux/base/nginx:1.28@sha256:3352a36cbcab4708883a3e77b64f64159e11e1aab358c010e1c4e465dbfb4f57
           ports:
           - containerPort: 80
   ---
   apiVersion: v1
   kind: Service
   metadata:
     name: nginx-service
     namespace: test-app
   spec:
     selector:
       app: nginx
     ports:
     - protocol: TCP
       port: 80
       targetPort: 80
     type: LoadBalancer
   

6. Stage the test workload onto the Fleet Manager hub cluster by using kubectl.

   kubectl apply -f test-workload.yaml
   

7. Save the following YAML as crp-distribute-workload.yaml.

   apiVersion: placement.kubernetes-fleet.io/v1
   kind: ClusterResourcePlacement
   metadata:
     name: distribute-test-app
   spec:
     resourceSelectors:
       - group: ""
         kind: Namespace
         version: v1          
         name: test-app
     policy:
       placementType: PickAll
   

8. Apply the placement manifest to the Fleet Manager hub cluster.

   kubectl apply -f crp-distribute-workload.yaml
   

   Note

   The resource rollout begins immediately the manifest is applied. An implicit RollingUpdate strategy is used. To learn how to have additional control over the rollout, see defining a rollout strategy for resource placement.

9. Check the progress of the resource placement.

kubectl get clusterresourceplacement distribute-test-app

Your output should look similar to the following example:

NAME                   GEN   SCHEDULED   SCHEDULEDGEN   APPLIED   APPLIEDGEN   AGE
distribute-test-app    2     True        2              True      2            10s

1. View the details of the placement.

kubectl describe clusterresourceplacement distribute-test-app

Your output should look similar to the following example:

Name:         distribute-test-app
Namespace:    
Labels:       <none>
Annotations:  <none>
API Version:  placement.kubernetes-fleet.io/v1
Kind:         ClusterResourcePlacement
Metadata:
  Creation Timestamp:  2024-04-01T18:55:31Z
  Finalizers:
    kubernetes-fleet.io/crp-cleanup
    kubernetes-fleet.io/scheduler-cleanup
  Generation:        2
  Resource Version:  6949
  UID:               815b1d81-61ae-4fb1-a2b1-06794be3f986
Spec:
  Policy:
    Placement Type:  PickAll
  Resource Selectors:
    Group:                 
    Kind:                  Namespace
    Name:                  test-app
    Version:               v1
  Revision History Limit:  10
  Strategy:
    Type:  RollingUpdate
Status:
  Conditions:
    Last Transition Time:   2024-04-01T18:55:31Z
    Message:                found all the clusters needed as specified by the scheduling policy
    Observed Generation:    2
    Reason:                 SchedulingPolicyFulfilled
    Status:                 True
    Type:                   ClusterResourcePlacementScheduled
    Last Transition Time:   2024-04-01T18:55:36Z
    Message:                All 3 cluster(s) are synchronized to the latest resources on the hub cluster
    Observed Generation:    2
    Reason:                 SynchronizeSucceeded
    Status:                 True
    Type:                   ClusterResourcePlacementSynchronized
    Last Transition Time:   2024-04-01T18:55:36Z
    Message:                Successfully applied resources to 3 member clusters
    Observed Generation:    2
    Reason:                 ApplySucceeded
    Status:                 True
    Type:                   ClusterResourcePlacementApplied
  Observed Resource Index:  0
  Placement Statuses:
    Cluster Name:  membercluster1
    Conditions:
      Last Transition Time:  2024-04-01T18:55:31Z
      Message:               Successfully scheduled resources for placement in membercluster1 (affinity score: 0, topology spread score: 0): picked by scheduling policy
      Observed Generation:   2
      Reason:                ScheduleSucceeded
      Status:                True
      Type:                  ResourceScheduled
      Last Transition Time:  2024-04-01T18:55:36Z
      Message:               Successfully Synchronized work(s) for placement
      Observed Generation:   2
      Reason:                WorkSynchronizeSucceeded
      Status:                True
      Type:                  WorkSynchronized
      Last Transition Time:  2024-04-01T18:55:36Z
      Message:               Successfully applied resources
      Observed Generation:   2
      Reason:                ApplySucceeded
      Status:                True
      Type:                  ResourceApplied
    Cluster Name:            membercluster2
    Conditions:
      Last Transition Time:  2024-04-01T18:55:31Z
      Message:               Successfully scheduled resources for placement in membercluster2 (affinity score: 0, topology spread score: 0): picked by scheduling policy
      Observed Generation:   2
      Reason:                ScheduleSucceeded
      Status:                True
      Type:                  ResourceScheduled
      Last Transition Time:  2024-04-01T18:55:36Z
      Message:               Successfully Synchronized work(s) for placement
      Observed Generation:   2
      Reason:                WorkSynchronizeSucceeded
      Status:                True
      Type:                  WorkSynchronized
      Last Transition Time:  2024-04-01T18:55:36Z
      Message:               Successfully applied resources
      Observed Generation:   2
      Reason:                ApplySucceeded
      Status:                True
      Type:                  ResourceApplied
    Cluster Name:            membercluster3
    Conditions:
      Last Transition Time:  2024-04-01T18:55:31Z
      Message:               Successfully scheduled resources for placement in membercluster3 (affinity score: 0, topology spread score: 0): picked by scheduling policy
      Observed Generation:   2
      Reason:                ScheduleSucceeded
      Status:                True
      Type:                  ResourceScheduled
      Last Transition Time:  2024-04-01T18:55:36Z
      Message:               Successfully Synchronized work(s) for placement
      Observed Generation:   2
      Reason:                WorkSynchronizeSucceeded
      Status:                True
      Type:                  WorkSynchronized
      Last Transition Time:  2024-04-01T18:55:36Z
      Message:               Successfully applied resources
      Observed Generation:   2
      Reason:                ApplySucceeded
      Status:                True
      Type:                  ResourceApplied
  Selected Resources:
    Kind:     Namespace
    Name:     my-namespace
    Version:  v1
Events:
  Type    Reason                     Age   From                                   Message
  ----    ------                     ----  ----                                   -------
  Normal  PlacementScheduleSuccess   108s  cluster-resource-placement-controller  Successfully scheduled the placement
  Normal  PlacementSyncSuccess       103s  cluster-resource-placement-controller  Successfully synchronized the placement
  Normal  PlacementRolloutCompleted  103s  cluster-resource-placement-controller  Resources have been applied to the selected clusters

1. In the Azure portal, go to your Fleet Manager.

2. On the service menu, under Fleet Resources, select Namespaces > + Create.

   Note

   If you don't see Fleet Resources, your Fleet Manager doesn't have a hub cluster. For further information on how to add one, see upgrade hub cluster type.

3. In the menu, select Namespace, enter a Name, and then select Create.

   

   After a few moments, the page refreshes and the namespace appears in the list of namespaces on the Fleet Manager hub cluster. It's now ready to host any resources you want to distribute across member clusters.

4. At the top of the namespace list, select + Create > Apply a YAML, and use the following samples.

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: nginx-deployment
     namespace: test-app
   spec:
     selector:
       matchLabels:
         app: nginx
     replicas: 2
     template:
       metadata:
         labels:
           app: nginx
       spec:
         containers:
         - name: nginx
           image: mcr.microsoft.com/azurelinux/base/nginx:1.28@sha256:3352a36cbcab4708883a3e77b64f64159e11e1aab358c010e1c4e465dbfb4f57
           ports:
           - containerPort: 80
   

   apiVersion: v1
   kind: Service
   metadata:
     name: nginx-service
     namespace: test-app
   spec:
     selector:
       app: nginx
     ports:
     - protocol: TCP
       port: 80
       targetPort: 80
     type: LoadBalancer
   

   Copy and paste the samples and apply them one at a time as shown in the following image.

   

   The namespace and its workloads are now ready to be distributed to member clusters. The Deployment and Service aren't scheduled on the Fleet Manager hub cluster.

5. On the service menu, under Fleet Resources, select Resource placements > + Create.

6. On the Basics tab, configure the following options:

   • In Placement details, enter a name for the placement.

   • Under Resource details, leave Scope as Cluster-scoped and enter the Group Version Kind (GVK) and name of the resource to distribute. In this sample, use the following values.

     group: ""
     kind: Namespace
     version: v1          
     name: test-app
     

   • In Member cluster selection, for Placement type choose All clusters.

   • For Rollout, leave the Rolling update strategy selected.

   

7. Select Next to review the resulting ClusterResourcePlacement manifest. You can modify the manifest if required, validate by using the Validate (dry run) option, or select Review + create to proceed to final confirmation.

   

8. Select Create to start the distribution of the namespace immediately.

   

   Note

   The resource rollout begins immediately when you apply the resource placement by using the selected RollingUpdate strategy. To learn how to have additional controls over the rollout, see defining a rollout strategy for resource placement.

9. The page refreshes and the Resource placements list displays the newly created placement. It shows the number of clusters selected and whether the fleet scheduler can fulfill the placement policy.

   

10. View the rollout status of the placement by selecting the placement name in the list. Select individual clusters to view the resource rollout on that cluster.

    

On the selected Kubernetes clusters, you find the namespace, Deployment, and Service deployed and operational.

kubectl delete clusterresourceplacement distribute-test-app

1. On the service menu, under Fleet Resources, select Resource placements.

2. In the resource placements list, select the placement and select Delete.

3. On Delete, verify that you chose the correct placement. When you're ready, select Confirm delete > Delete.

   

4. The page refreshes and the placement no longer appears in the list. The resources are removed from member clusters.