Throttle data in data flow graphs

A throttle transform limits how often messages on a topic are forwarded. Instead of dropping messages based on their content, the throttle transform drops messages based on when it processes them, forwarding at most one message per topic pattern within each configured interval. Use it to protect downstream systems from bursty or high-frequency sources without changing message content. For an overview of data flow graphs and how transforms compose in a pipeline, see Data flow graphs overview.

Prerequisites

An instance of Azure IoT Operations deployed in a Kubernetes cluster. For more information, see Deploy Azure IoT Operations.

A default registry endpoint named default that points to mcr.microsoft.com is automatically created during deployment. The built-in transforms use this endpoint.

Scaling limitation for stateful graphs

Important

Data flow graphs that contain a throttle transform are stateful—each instance tracks its own last-forwarded time per topic pattern. When the data flow profile has an instance count greater than one, messages are distributed across instances through shared subscriptions. Because each instance maintains its own throttle state and the instances don't share state with each other, the configured rate limit is enforced independently per instance instead of across the whole pipeline.

To ensure the configured rate limit is enforced across all messages, set the data flow profile instance count to 1 for any data flow graph that uses a throttle transform.

How the throttle transform works

The throttle transform evaluates each incoming message's topic against an ordered list of per-topic rules:

First match wins. The transform evaluates rules in order. The first rule whose topic pattern matches the message's topic determines whether the transform forwards the message. The transform doesn't check later rules, even if they would also match.
Unmatched topics pass through. If no rule matches the message's topic, the transform forwards the message without any throttling.
Forwarding is time-based, not count-based. For a matched rule, the transform forwards the first message it processes, then drops every subsequent message on that pattern until the configured interval (1000 / maxMessagesPerSecond milliseconds, rounded up) elapses since the last forwarded message. This bounds the maximum rate but doesn't allow bursts to make up for earlier drops.
Shared state per pattern, not per topic. When a rule's topic pattern uses a wildcard, all concrete topics that match it share the same throttle state within a transform instance. For example, a single sensors/+ rule limits the combined rate across sensors/temperature and sensors/humidity, not each one independently.
0 drops everything. Setting maxMessagesPerSecond to 0 for a rule drops every message that matches that rule's pattern.
Timing is based on processing time, not message content. The transform uses its monotonic clock when it processes each message. It doesn't read a timestamp field from the message payload.

Note

The throttle transform only decides whether to forward or drop a message. It never modifies message content.

Configure per-topic throttle rules

Define throttle rules in the throttle configuration key (not rules) as a JSON object with a perTopicThrottles array.

In the throttle transform configuration, add one or more throttle rules. For each rule, specify:

Setting	Description
Topic	The MQTT topic pattern to match. Supports the `+` (single-level) and `#` (multi-level) wildcards.
Max messages per second	The maximum forwarding rate for topics that match this pattern. Set to `0` to drop every matching message.

The CLI applies the whole graph from one config file. Add this configuration to the throttle transform node's configuration in your graph.json file, and apply it by using az iot ops dataflowgraph apply. The configuration is a JSON object:

{
  "perTopicThrottles": [
    {
      "topic": "sensors/temperature",
      "maxMessagesPerSecond": 10
    }
  ]
}

Add this configuration in the value field as an escaped string, under the throttle key:

"configuration": [
  {
    "key": "throttle",
    "value": "{\"perTopicThrottles\":[{\"topic\":\"sensors/temperature\",\"maxMessagesPerSecond\":10}]}"
  }
]

Tip

To generate the escaped string, save the rules to a file like rules.json, run jq -c . rules.json, and paste the single-line output into the value field.

configuration: [
  {
    key: 'throttle'
    value: '{"perTopicThrottles":[{"topic":"sensors/temperature","maxMessagesPerSecond":10}]}'
  }
]

Important

The use of Kubernetes deployment manifests isn't supported in production environments and should only be used for debugging and testing.

configuration:
  - key: throttle
    value: |
      {
        "perTopicThrottles": [
          {
            "topic": "sensors/temperature",
            "maxMessagesPerSecond": 10
          }
        ]
      }

With this rule, the transform forwards the first sensors/temperature message it processes, then drops any further messages on that topic that it processes less than 100 milliseconds later (1000 / 10). Topics other than sensors/temperature aren't affected. Each entry in perTopicThrottles has these properties:

Property	Required	Description
`topic`	Yes	MQTT topic pattern to match. Supports the `+` (single-level) and `#` (multi-level, trailing-only) wildcards.
`maxMessagesPerSecond`	Yes	Maximum forwarding rate for topics that match this pattern, in messages per second. Fractional values are allowed. For example, `0.1` allows one message every 10 seconds. Must be zero or a positive, finite number. Set to `0` to drop every message that matches the pattern. Timing has 1-millisecond precision, so values greater than `1000` have the same effective limit as `1000`.

Important

Each topic pattern in perTopicThrottles must be unique. The transform rejects configuring the same pattern string more than once during initialization.

Use multiple per-topic rules

Because rules are evaluated in order and the first match wins, list more specific patterns before more general ones if you want them to have their own rate limit:

Add two rules, in this order:

Order	Topic	Max messages per second
1	`sensors/temperature`	`10`
2	`sensors/#`	`1`

{
  "perTopicThrottles": [
    {
      "topic": "sensors/temperature",
      "maxMessagesPerSecond": 10
    },
    {
      "topic": "sensors/#",
      "maxMessagesPerSecond": 1
    }
  ]
}

configuration: [
  {
    key: 'throttle'
    value: '{"perTopicThrottles":[{"topic":"sensors/temperature","maxMessagesPerSecond":10},{"topic":"sensors/#","maxMessagesPerSecond":1}]}'
  }
]

Important

The use of Kubernetes deployment manifests isn't supported in production environments and should only be used for debugging and testing.

configuration:
  - key: throttle
    value: |
      {
        "perTopicThrottles": [
          { "topic": "sensors/temperature", "maxMessagesPerSecond": 10 },
          { "topic": "sensors/#", "maxMessagesPerSecond": 1 }
        ]
      }

Messages on sensors/temperature match the first rule and are limited to 10 messages per second. Messages on any other sensors/* topic (for example, sensors/humidity) match the second rule and share a combined limit of 1 message per second.

Important

Order matters. If the sensors/# rule were listed first, it would match sensors/temperature messages too, and the more specific rule would never be reached.

Use wildcards to throttle groups of topics

The topic pattern supports the same wildcards as MQTT topic filters:

Wildcard	Matches	Example
`+`	Exactly one topic level	`sensors/+/status` matches `sensors/line1/status` but not `sensors/line1/sub/status`
`#`	Zero or more remaining topic levels, and must be the last segment	`sensors/#` matches `sensors`, `sensors/temperature`, and `sensors/line1/temperature`

All concrete topics that match the same wildcard rule share one throttle state.

Add a rule with topic sensors/+ and max messages per second 1.

{
  "perTopicThrottles": [
    {
      "topic": "sensors/+",
      "maxMessagesPerSecond": 1
    }
  ]
}

configuration: [
  {
    key: 'throttle'
    value: '{"perTopicThrottles":[{"topic":"sensors/+","maxMessagesPerSecond":1}]}'
  }
]

Important

The use of Kubernetes deployment manifests isn't supported in production environments and should only be used for debugging and testing.

configuration:
  - key: throttle
    value: |
      {
        "perTopicThrottles": [
          { "topic": "sensors/+", "maxMessagesPerSecond": 1 }
        ]
      }

With this rule, a message on sensors/temperature and a message on sensors/humidity processed 500 milliseconds apart aren't both forwarded—the second one is dropped, because it counts against the same shared 1-second interval as the first, regardless of which concrete topic it's on.

Note

A bare # pattern matches every topic and applies a single, combined rate limit within each transform instance.

Drop all messages for a topic

Set maxMessagesPerSecond to 0 to drop every message that matches a pattern, without removing the rule or the topic from your pipeline:

Add a rule with topic debug/# and max messages per second 0.

{
  "perTopicThrottles": [
    {
      "topic": "debug/#",
      "maxMessagesPerSecond": 0
    }
  ]
}

configuration: [
  {
    key: 'throttle'
    value: '{"perTopicThrottles":[{"topic":"debug/#","maxMessagesPerSecond":0}]}'
  }
]

Important

The use of Kubernetes deployment manifests isn't supported in production environments and should only be used for debugging and testing.

configuration:
  - key: throttle
    value: |
      {
        "perTopicThrottles": [
          { "topic": "debug/#", "maxMessagesPerSecond": 0 }
        ]
      }

Deploy a data flow graph with throttle

In the Operations experience, create a data flow graph with a throttle transform:

Add a source that reads from your MQTT topic.
Add a throttle transform. Add one or more per-topic rules, ordered from most to least specific.
Add a destination that sends to your output topic.

The Azure CLI applies a data flow graph from a single JSON config file. Create a graph.json file with the graph properties. In the graph.json file, each transform's configuration is stored in the value field as an escaped JSON string. For the readable form of the throttle configuration, see Configure per-topic throttle rules. Apply it by using az iot ops dataflowgraph apply.

{
  "mode": "Enabled",
  "nodes": [
    {
      "nodeType": "Source",
      "name": "sensors",
      "sourceSettings": {
        "endpointRef": "default",
        "dataSources": [
          "sensors/#"
        ]
      }
    },
    {
      "nodeType": "Graph",
      "name": "throttle",
      "graphSettings": {
        "registryEndpointRef": "default",
        "artifact": "azureiotoperations/graph-dataflow-throttle:1.0.0",
        "configuration": [
          {
            "key": "throttle",
            "value": "{\"perTopicThrottles\":[{\"topic\":\"sensors/temperature\",\"maxMessagesPerSecond\":10},{\"topic\":\"sensors/#\",\"maxMessagesPerSecond\":1}]}"
          }
        ]
      }
    },
    {
      "nodeType": "Destination",
      "name": "output",
      "destinationSettings": {
        "endpointRef": "default",
        "dataDestination": "telemetry/throttled"
      }
    }
  ],
  "nodeConnections": [
    {
      "from": {
        "name": "sensors"
      },
      "to": {
        "name": "throttle"
      }
    },
    {
      "from": {
        "name": "throttle"
      },
      "to": {
        "name": "output"
      }
    }
  ]
}

resource dataflowGraph 'Microsoft.IoTOperations/instances/dataflowProfiles/dataflowGraphs@2026-03-01' = {
  name: 'sensors-throttle'
  parent: dataflowProfile
  properties: {
    mode: 'Enabled'
    nodes: [
      {
        nodeType: 'Source'
        name: 'sensors'
        sourceSettings: {
          endpointRef: 'default'
          dataSources: [ 'sensors/#' ]
        }
      }
      {
        nodeType: 'Graph'
        name: 'throttle'
        graphSettings: {
          registryEndpointRef: 'default'
          artifact: 'azureiotoperations/graph-dataflow-throttle:1.0.0'
          configuration: [
            {
              key: 'throttle'
              value: '{"perTopicThrottles":[{"topic":"sensors/temperature","maxMessagesPerSecond":10},{"topic":"sensors/#","maxMessagesPerSecond":1}]}'
            }
          ]
        }
      }
      {
        nodeType: 'Destination'
        name: 'output'
        destinationSettings: {
          endpointRef: 'default'
          dataDestination: 'telemetry/throttled'
        }
      }
    ]
    nodeConnections: [
      { from: { name: 'sensors' }, to: { name: 'throttle' } }
      { from: { name: 'throttle' }, to: { name: 'output' } }
    ]
  }
}

Important

The use of Kubernetes deployment manifests isn't supported in production environments and should only be used for debugging and testing.

apiVersion: connectivity.iotoperations.azure.com/v1
kind: DataflowGraph
metadata:
  name: sensors-throttle
  namespace: azure-iot-operations
spec:
  profileRef: default
  nodes:
    - nodeType: Source
      name: sensors
      sourceSettings:
        endpointRef: default
        dataSources:
          - sensors/#
    - nodeType: Graph
      name: throttle
      graphSettings:
        registryEndpointRef: default
        artifact: azureiotoperations/graph-dataflow-throttle:1.0.0
        configuration:
          - key: throttle
            value: |
              {
                "perTopicThrottles": [
                  { "topic": "sensors/temperature", "maxMessagesPerSecond": 10 },
                  { "topic": "sensors/#", "maxMessagesPerSecond": 1 }
                ]
              }
    - nodeType: Destination
      name: output
      destinationSettings:
        endpointRef: default
        dataDestination: telemetry/throttled
  nodeConnections:
    - from: { name: sensors }
      to: { name: throttle }
    - from: { name: throttle }
      to: { name: output }

Limitations

Doesn't modify messages. The throttle transform only forwards or drops messages; it doesn't change message content.
First match wins. Only the first rule whose topic pattern matches the message's topic is applied. List more specific patterns before more general ones.
Shared state per pattern. A wildcard rule's rate limit is shared across every concrete topic it matches - it isn't a separate limit per topic.
No burst allowance. The transform enforces a minimum time between forwarded messages per matched pattern. It doesn't accumulate unused capacity from earlier, slower periods.
Millisecond precision. The minimum throttle interval is 1 millisecond, so values greater than 1000 for maxMessagesPerSecond don't increase the effective forwarding rate beyond 1,000 messages per second.
Timing is based on processing time, not message content. The transform uses the time it processes each message, not the time the broker received it or a timestamp field in the payload.
State is local and in memory. Each data flow profile instance maintains its own throttle state. Restarting or reconfiguring the transform resets that state. If a profile has multiple instances, each instance enforces the configured rate independently.
Duplicate topic patterns aren't allowed. Configuring the same topic string more than once in perTopicThrottles fails when the transform initializes.

Next steps

Feedback

Was this page helpful?

Last updated on 2026-07-30

Throttle data in data flow graphs

Prerequisites

Scaling limitation for stateful graphs

How the throttle transform works

Configure per-topic throttle rules

Use multiple per-topic rules

Use wildcards to throttle groups of topics

Drop all messages for a topic

Deploy a data flow graph with throttle

Limitations

Next steps

Feedback

Additional resources