Expressions reference for data flows and data flow graphs

This reference applies to both data flows and data flow graphs. Both use the same expression language for map, filter, and enrichment transforms. Data flow graphs also support branch and window (accumulate) transforms, which are noted where applicable.

Positional variables

Each rule's inputs array determines the variables available in the expression. The first input becomes $1, the second becomes $2, and so on.

If only one input is specified and no expression is provided, the value at that input is copied directly to the output.

Operators

Expressions support the following operators, listed from highest to lowest precedence:

The + operator concatenates strings when at least one operand is a string. Use parentheses to override default precedence.

Examples:

Built-in functions

Unit conversion functions

These functions accept a single numeric value and return a float.

Scaling and rounding functions

Math functions

These functions come from the built-in math library.

String functions

Date and time functions

Date and time functions are available only in data flow graphs. Except for parse_timestamp, functions that accept a time stamp require an RFC 3339 string. The string must include an explicit offset from Coordinated Universal Time (UTC), such as 2026-06-17T12:00:00Z or 2026-06-17T12:00:00-08:00. The Returns column lists the conceptual result type, which typeof reports in lowercase as "float", "int", or "string".

now() reads the system clock of the Kubernetes node that hosts the data flow workload each time the function runs. Calls in the same expression can return different values. In an accumulation rule, now() returns the time when the window is processed, not when an input message arrives. To reduce clock skew across cluster nodes, synchronize each node clock with a reliable time source, such as a Network Time Protocol server.

Calendar and clock functions, such as day_of_week, hour_of_day, and year_of, use the offset in the input time stamp. They don't first convert the value to UTC. The following examples show this behavior.

Format and parse time stamps

The format_timestamp and parse_timestamp functions use strftime patterns. The following table lists common specifiers.

For all supported specifiers, see the Chrono strftime documentation.

format_timestamp preserves the input time stamp's offset. For example, format_timestamp("2026-06-09T14:00:00+02:00", "%Y-%m-%d %H:%M %:z") returns "2026-06-09 14:00 +02:00".

parse_timestamp handles inputs in the following order:

1. A date and time with a numeric offset. The function honors the offset and normalizes the result to UTC.
2. A date and time without an offset. The function assumes UTC.
3. A date without a time. The function assumes midnight UTC.

The following examples show how parse_timestamp handles each type of input.

Convert Unix epoch values

Use from_epoch and to_epoch to convert between Unix epoch values and RFC 3339 time stamps.

from_epoch accepts negative values for dates before January 1, 1970. Its output has millisecond precision, so converting epoch microseconds to a time stamp discards precision below one millisecond. to_epoch accounts for the time stamp's UTC offset and rounds values before 1970 down to the next lower integer for the selected unit.

Date and time functions can be composed. For example, duration_between($1, now(), "days") calculates the age in days of an RFC 3339 time stamp supplied as the first rule input. To read a time stamp's hour after converting it to UTC, use hour_of_day(from_epoch(to_epoch($1, "micros"), "micros")).

Conditional and collection functions

Aggregation functions (window transforms only)

These functions are available only in accumulation rules within window transforms. Each takes a single positional variable.

For details on using aggregation functions, see Aggregate data over time.

Conditional logic

Use the if function to branch logic within an expression:

Use () (the empty value) in comparisons to detect missing fields.

Metadata fields

Read from and write to message metadata by using the $metadata. prefix in the inputs or output fields of a rule. Metadata references go in the field path, not in the expression itself.

Metadata properties

• Topic: Works for both MQTT and Kafka. It contains the string where the message was published. Example: $metadata.topic.
• User property: In MQTT, this refers to the free-form key/value pairs an MQTT message can carry. For example, if the MQTT message was published with a user property with key "priority" and value "high", then the $metadata.user_property.priority reference holds the value "high". User property keys can be arbitrary strings and may require escaping: $metadata.user_property."weird key" uses the key "weird key" (with a space).
• System property: This term is used for every property that is not a user property. Currently, only a single system property is supported: $metadata.system_property.content_type, which reads the content type property of the MQTT message (if set).
• Header: This is the Kafka equivalent of the MQTT user property. Kafka can use any binary value for a key, but data flows support only UTF-8 string keys. Example: $metadata.header.priority. This functionality is similar to user properties.

Read from metadata

To reference the source topic and a user property in an expression, list them as inputs:

Expression: $1 + "/" + $2

In the following example, the MQTT topic property is mapped to the origin_topic field in the output:

If the user property priority is present in the MQTT message, the following example demonstrates how to map it to an output field:

Write to metadata

To set a user property on the output message, use $metadata.user_property.<key> as the output field.

Setting a metadata field to an empty value (()) removes it. For user properties, duplicate keys are allowed.

You can also map metadata properties to an output header or user property. In the following example, the MQTT topic is mapped to the origin_topic field in the output's user property:

If the incoming payload contains a priority field, the following example demonstrates how to map it to an MQTT user property:

The same example for Kafka:

Metadata fields are supported in map, filter, and branch rules. They aren't available in window (accumulate) rules.

Last known value

Use the ? $last suffix on an input to tell the runtime to remember the most recent value for that field. If the field is missing in the current message, the last known value is used instead.

The ? $last directive is case-insensitive and supports flexible whitespace.

Last known value is supported in map, filter, and branch rules. It isn't available in window (accumulate) rules.

Default values

Use the ?? <default> suffix on an input to provide a fallback value when the field is missing. Supported default types: integer, float, boolean, string, and null.

Combine last known value and default

You can combine ? $last and ?? <default>. The runtime checks the current message first, then the last known value, then the default. If you use ?? <default> without ? $last, the runtime checks the current message and then the default directly.

Default values are supported in map, filter, and branch rules. They aren't available in window (accumulate) rules.

Data types

JSON objects and arrays are preserved as-is when fields are copied without an expression, but they can't be used as inputs to expression evaluation.

Feature support by transform type

The same expression language works across transforms, but the result of an expression means something different in each one:

¹ Available in data flow graphs only. Not supported in data flow builtInTransformation inputs.

Dot notation and escaping

Dot notation is widely used to reference nested fields. A standard dot-notation path looks like Person.Address.Street.Number.

In a data flow, a path described by dot notation might include strings and some special characters without needing escaping, such as Person.Date of Birth.

In other cases, escaping is necessary, for example: nsu=http://opcfoundation.org/UA/Plc/Applications;s=RandomSignedInt32. This path, among other special characters, contains dots within the field name. Without escaping, the field name would serve as a separator in the dot notation itself.

While a data flow parses a path, it treats only two characters as special:

• Dots (.) act as field separators.
• Double quotation marks, when placed at the beginning or the end of a segment, start an escaped section where dots aren't treated as field separators.

Any other characters are treated as part of the field name. This flexibility is useful in formats like JSON, where field names can be arbitrary strings.

The path definition must also adhere to the rules of the configuration format. When a character with special meaning is included in the path, proper quoting is required. For example, field names that start with a colon (like :Person:.:name:) or that begin with a number followed by text (like 100 celsius.hot) need quoting in the configuration to be interpreted correctly as strings.

Escaping

The primary function of escaping in a dot-notated path is to accommodate the use of dots that are part of field names rather than separators. For example, the path Payload."Tag.10".Value consists of three segments: Payload, Tag.10, and Value. The double quotation marks around Tag.10 prevent the dot from acting as a separator.

Escaping rules in dot notation

• Escape each segment separately: If multiple segments contain dots, those segments must be enclosed in double quotation marks. Other segments can also be quoted, but it doesn't affect the path interpretation. For example: Payload."Tag.10".Measurements."Vibration.$12".Value

• Quote a segment only when it needs escaping. Add double quotation marks around a segment only when it contains a dot (or another character that would otherwise be misread as a separator). Don't quote an ordinary field name.

  For example, for a field named TagName inside Records:

  Do this	Not this
  Records.TagName	Records."TagName"

  The extra quotation marks don't help, and in some tools, such as the operations experience data flow editor, they become part of the field name. The mapping then looks for a field named "TagName" (quotes included), which doesn't exist, so nothing matches. You can spot this in the exported data flow definition, where the name appears with escaped quotes, like \"TagName\". To fix it, remove the quotation marks so the name is just TagName.

• Proper use of double quotation marks: Double quotation marks must open and close an escaped segment. Any quotation marks in the middle of the segment are considered part of the field name. For example, the path Payload.He said: "Hello", and waved defines two fields: Payload and He said: "Hello", and waved. When a dot appears under these circumstances, it continues to serve as a separator. For example, the path Payload.He said: "No. It is done" is split into the segments Payload, He said: "No, and It is done" (starting with a space).

Segmentation algorithm

• If the first character of a segment is a quotation mark, the parser searches for the next quotation mark. The string enclosed between these quotation marks is considered a single segment.
• If the segment doesn't start with a quotation mark, the parser identifies segments by searching for the next dot or the end of the path.

Wildcards

Use a wildcard (*) in input and output paths to match multiple fields at once. This is useful when the output closely resembles the input, or when you need to apply the same transformation across many fields without listing each one.

Copy all fields

To pass every field through unchanged:

The * matches each field path in the input and places it at the same path in the output. The portion of the path that * matches is called the captured segment. In the output, the captured segment replaces the *.

Flatten nested fields

To move fields out of a nested object to the root level, put the prefix in the input and * in the output:

Given this input:

{
  "Sensors": { "Temperature": 72.5, "Pressure": 14.7 },
  "Metadata": { "LineId": "Line-3", "Shift": "A" }
}

The output flattens both objects:

{
  "Temperature": 72.5,
  "Pressure": 14.7,
  "LineId": "Line-3",
  "Shift": "A"
}

Restructure fields

To move fields under a new parent, put * in the input and add a prefix in the output:

This wraps all top-level fields inside a Telemetry object.

Wildcard placement rules

• Only one * is allowed per input or output path.
• The * must match a complete segment (not a partial segment like Sensor*).
• The * can appear at the beginning (*.Value), middle (Sensors.*.Reading), or end (Sensors.*) of a path.

Multi-input wildcards

When a rule has multiple inputs with wildcards, the * must capture the same segment across all inputs. The runtime resolves the * from the first input, then looks for matching paths in the other inputs.

For example, to average the max and min readings for each sensor:

Given this input:

{
  "Temperature": { "Max": 85.3, "Min": 62.1 },
  "Pressure": { "Max": 15.2, "Min": 14.1 }
}

The * captures Temperature first, so the rule looks for both Temperature.Max and Temperature.Min. Then it captures Pressure and looks for Pressure.Max and Pressure.Min. The output is:

{
  "Averaged": { "Temperature": 73.7, "Pressure": 14.65 }
}

If any input can't resolve for a captured segment (for example, *.Mid.Avg when the field is nested differently), that segment is skipped. Make sure the paths in all inputs reflect the actual structure of the data.

Override a wildcard for specific fields

You can combine a wildcard rule with specific rules. Specific rules take precedence when they have a lower coverage (fewer segments matched by *). This is called specialization.

The first rule applies to all fields. The second rule overrides it for Pressure only, because Pressure.Max is more specific than *.Max (coverage 0 vs. coverage 1).

To exclude a field entirely, use an empty output:

An empty output drops the field from the result. This overrides any wildcard rule that would otherwise include it.

Multiple rules on the same inputs

If two rules have the same or higher coverage, both apply. This lets you compute multiple derived values from the same inputs:

Both rules execute for each captured segment, producing two output fields per sensor.

Wildcards in contextualization datasets

You can use wildcards with $context references to copy all fields from a dataset:

This copies every field from the assetMeta dataset into the Asset section of the output.

Contextualization datasets

Contextualization datasets let mappings integrate extra data from external databases. Use the $context(datasetName) prefix to reference fields from a dataset. For example, $context(position).BaseSalary reads the BaseSalary field from a dataset named position.

For details on configuring contextualization datasets, see Enrich data by using data flows and Enrich with external data in data flow graphs.

Related content

If you arrived here looking for the syntax used in a specific transform, these articles show expressions in context:

For data flows, see Map data by using data flows and Filter data in a data flow.