Configure data flow endpoints

To get started with data flows, first create data flow endpoints. A data flow endpoint is the connection point for the data flow. You can use an endpoint as a source, a destination, or both. Some endpoint types can be used as both sources and destinations, while others are for destinations only.

Use the following table to choose the endpoint type to configure:

Endpoint type	Description	Can be used as a source	Can be used as a destination	Data flow graphs support
MQTT	For bi-directional messaging with MQTT brokers, including the one built-in to Azure IoT Operations and Event Grid.	Yes	Yes	Source and destination
Kafka	For bi-directional messaging with Kafka brokers, including Azure Event Hubs.	Yes	Yes	Source and destination
OpenTelemetry	For sending metrics and logs to OpenTelemetry collectors and observability platforms like Grafana and Azure Monitor.	No	Yes	Destination only
Data Lake	For uploading data to Azure Data Lake Gen2 storage accounts.	No	Yes	Not supported
Microsoft Fabric OneLake	For uploading data to Microsoft Fabric OneLake lakehouses.	No	Yes	Not supported
Azure Data Explorer	For uploading data to Azure Data Explorer databases.	No	Yes	Not supported
Local storage	For sending data to a locally available persistent volume, optionally configurable with Azure Container Storage enabled by Azure Arc.	No	Yes	Not supported

Important

Data flow graphs limitation: Data flow graphs (WASM) currently only support MQTT, Kafka, and OpenTelemetry endpoints. OpenTelemetry endpoints can only be used as destinations in data flow graphs. Other endpoint types aren't supported for data flow graphs.

Important

Storage endpoints require a schema for serialization. To use data flow with Microsoft Fabric OneLake, Azure Data Lake Storage, Azure Data Explorer, or Local Storage, you must specify a schema reference.

To generate the schema from a sample data file, use the Schema Gen Helper.

Set your environment variables

The Azure CLI examples in this article use environment variables so that you can set each value once and then copy and paste the commands as-is. If you're using the Azure IoT Operations Codespaces environment from the quickstart, these variables are already set for you and you can skip this step. Otherwise, set the following environment variables in your shell before you run the commands.

The following scripts set the most commonly used environment variables:

Environment variable	Description
`SUBSCRIPTION_ID`	The ID of the subscription that contains your Azure IoT Operations instance.
`RESOURCE_GROUP`	The name of the resource group that contains your Azure IoT Operations instance.
`AIO_INSTANCE_NAME`	The name of your Azure IoT Operations instance. To list your instances, run `az iot ops list -o table`.
`CLUSTER_NAME`	The name of the Azure Arc-enabled Kubernetes cluster that hosts your instance.
`LOCATION`	The Azure region to use for new resources, for example `eastus`.

Bash
PowerShell

SUBSCRIPTION_ID=<subscription-id>
RESOURCE_GROUP=<resource-group-name>
AIO_INSTANCE_NAME=<instance-name>
CLUSTER_NAME=<cluster-name>
LOCATION=<region>

$SUBSCRIPTION_ID = "<subscription-id>"
$RESOURCE_GROUP = "<resource-group-name>"
$AIO_INSTANCE_NAME = "<instance-name>"
$CLUSTER_NAME = "<cluster-name>"
$LOCATION = "<region>"

You only need to set the variables that this article uses. This article might use additional environment variables for resource names that you choose. The article explains how to set them where they're introduced.

This article also uses the following environment variables for resource names that you choose: DATAFLOW (the name of the data flow), PROFILE (the name of the data flow profile). Set each one to a value that you want before you run the related commands.

Data flows must use local MQTT broker endpoint

When you create a data flow, you specify the source and destination endpoints. The data flow moves data from the source to the destination. You can use the same endpoint for multiple data flows, and you can use the same endpoint as both the source and destination.

However, you can't use custom endpoints as both the source and destination. The built-in MQTT broker in Azure IoT Operations must be at least one endpoint (either the source, the destination, or both). To avoid deployment failures, use the default MQTT data flow endpoint as either the source or destination for every data flow.

Each data flow must have either the source or destination configured with an MQTT endpoint that has the host aio-broker. You don't have to use the default endpoint specifically. You can create other data flow endpoints pointing to the local MQTT broker as long as the host is aio-broker. However, using the default endpoint is the recommended approach.

The following table shows the supported scenarios:

Scenario	Supported
Default endpoint as source	Yes
Default endpoint as destination	Yes
Custom endpoint as source	Yes, if destination is default endpoint or an MQTT endpoint with host `aio-broker`
Custom endpoint as destination	Yes, if source is default endpoint or an MQTT endpoint with host `aio-broker`
Custom endpoint as source and destination	No, unless one of them is an MQTT endpoint with host `aio-broker`

For information about how the local MQTT broker buffers data when a destination endpoint is unavailable, see Configure data buffering and disk persistence for data flows.

Reuse endpoints

Think of each data flow endpoint as a bundle of configuration settings: where the data should come from or go to (the host value), how to authenticate, and other settings like TLS configuration or batching preference. You create an endpoint once and reuse it in multiple data flows that share the same settings.

To make it easier to reuse endpoints, the MQTT or Kafka topic filter isn't part of the endpoint configuration. Instead, you specify the topic filter in the data flow configuration. This means you can use the same endpoint for multiple data flows that use different topic filters.

For example, you can use the default MQTT broker data flow endpoint. You can use it for both the source and destination with different topic filters:

Use the az iot ops dataflow apply command to create or change a data flow.

az iot ops dataflow apply --resource-group $RESOURCE_GROUP --instance $AIO_INSTANCE_NAME --profile $PROFILE --name $DATAFLOW --config-file config.json

The --config-file parameter is the path and file name of a JSON configuration file containing the resource properties.

In this example, assume a configuration file named data-flow.json with the following content stored in the user's home directory:

{
  "mode": "Enabled",
  "operations": [
    {
      "operationType": "Source",
      "sourceSettings": {
        "endpointRef": "default",
        "dataSources": [
          "example/topic/1"
        ]
      }
    },
    {
      "operationType": "Destination",
      "destinationSettings": {
        "endpointRef": "default",
        "dataDestination": "example/topic/2"
      }
    }
  ]
}

resource dataflow 'Microsoft.IoTOperations/instances/dataflowProfiles/dataflows@2026-03-01' = {
  parent: <DEFAULT_PROFILE_RESOURCE>
  name: 'broker-to-broker'
  extendedLocation: {
    name: <CUSTOM_LOCATION_RESOURCE>.id
    type: 'CustomLocation'
  }
  properties: {
    mode: 'Enabled'
    operations: [
      {
        operationType: 'Source'
        sourceSettings: {
          endpointRef: 'default'
          dataSources: [
            'example/topic/1'
          ]
        }
      }
      {
        operationType: 'Destination'
        destinationSettings: {
          endpointRef: 'default'
          dataDestination: 'example/topic/2'
        }
      }
    ]
  }
}

Important

The use of Kubernetes deployment manifests isn't supported in production environments and should only be used for debugging and testing.

apiVersion: connectivity.iotoperations.azure.com/v1
kind: Dataflow
metadata:
  name: broker-to-broker
  namespace: azure-iot-operations
spec:
  profileRef: default
  operations:
    - operationType: Source
      sourceSettings:
        endpointRef: default
        dataSources:
        - example/topic/1
    - operationType: Destination
      destinationSettings:
        endpointRef: default
        dataDestination: example/topic/2

Similarly, you can create multiple data flows that use the same MQTT endpoint for other endpoints and topics. For example, you can use the same MQTT endpoint for a data flow that sends data to an Event Hubs endpoint.

Use the az iot ops dataflow apply command to create or change a data flow.

az iot ops dataflow apply --resource-group $RESOURCE_GROUP --instance $AIO_INSTANCE_NAME --profile $PROFILE --name $DATAFLOW --config-file config.json

The --config-file parameter is the path and file name of a JSON configuration file containing the resource properties.

In this example, assume a configuration file named data-flow.json with the following content stored in the user's home directory:

{
  "mode": "Enabled",
  "operations": [
    {
      "operationType": "Source",
      "sourceSettings": {
        "endpointRef": "default",
        "dataSources": [
          "example/topic/3"
        ]
      }
    },
    {
      "operationType": "Destination",
      "destinationSettings": {
        // The endpoint needs to be created before you can reference it here
        "endpointRef": "example-event-hub-endpoint",
        "dataDestination": "example/topic/4"
      }
    }
  ]
}

resource dataflow 'Microsoft.IoTOperations/instances/dataflowProfiles/dataflows@2026-03-01' = {
  parent: <DEFAULT_PROFILE_RESOURCE>
  name: 'broker-to-eh'
  extendedLocation: {
    name: <CUSTOM_LOCATION_RESOURCE>.id
    type: 'CustomLocation'
  }
  properties: {
    mode: 'Enabled'
    operations: [
      {
        operationType: 'Source'
        sourceSettings: {
          endpointRef: 'default'
          dataSources: [
            'example/topic/3'
          ]
        }
      }
      {
        operationType: 'Destination'
        destinationSettings: {
          // The endpoint needs to be created before you can reference it here
          endpointRef: 'example-event-hub-endpoint'
          dataDestination: 'example/topic/4'
        }
      }
    ]
  }
}

Important

The use of Kubernetes deployment manifests isn't supported in production environments and should only be used for debugging and testing.

apiVersion: connectivity.iotoperations.azure.com/v1
kind: Dataflow
metadata:
  name: broker-to-eh
  namespace: azure-iot-operations
spec:
  profileRef: default
  operations:
    - operationType: Source
      sourceSettings:
        endpointRef: default
        dataSources:
        - example/topic/3
    - operationType: Destination
      destinationSettings:
        # The endpoint needs to be created before you can reference it here
        endpointRef: example-event-hub-endpoint
        dataDestination: example/topic/4

Similar to the MQTT example, you can create multiple data flows that use the same Kafka endpoint for different topics, or the same Data Lake endpoint for different tables.

Next steps

Tip

To route dataflow traffic to cloud destinations through Private Link instead of public endpoints, see Configure dataflow destinations with private endpoints.

Create a data flow endpoint:

Feedback

Was this page helpful?

Last updated on 2026-07-31

Configure data flow endpoints

Set your environment variables

Data flows must use local MQTT broker endpoint

Reuse endpoints

Next steps

Feedback

Additional resources