OpenTelemetry table reference for Zerobus Ingest

This page provides reference information for the OpenTelemetry (OTLP) table schemas and data mapping used by Zerobus Ingest OTLP.

Table schema

When OTLP data arrives, Zerobus Ingest converts each record from the nested OTLP resource/scope/record hierarchy into a flat, denormalized row. Resource attributes and instrumentation scope information are embedded directly in each row, making the data immediately queryable without joins.

All attribute fields (attributes, resource.attributes, instrumentation_scope.attributes, body for logs, metadata for metrics) are stored as VARIANT columns. VARIANT is a semi-structured type in Delta Lake that stores JSON data while preserving the original types.

Each record is augmented with Databricks-specific fields:

Field Description Source
record_id A system-generated ID for unique identification and time-ordered sorting. Generated based on time
time Timestamp in microseconds from the Unix epoch. Timestamp (in microseconds) derived from start_time_unix_nano (spans) or time_unix_nano (logs, metrics)
date Date partition column, for efficient time-range filtering. Derived from time
service_name Top-level column for efficient filtering by service name, as defined in the OTel semantic convention. Extracted from resource.attributes["service.name"]

Numeric precision and signed types

Due to some Delta type limitations, please be mindful of the following:

  • Integer metric values lose precision above 2^53. Gauge, Sum, and Exemplar data points that report their value as as_int, a 64-bit integer, are converted to DOUBLE, since Zerobus Ingest stores all metric values in a single floating-point column. DOUBLE only represents integers exactly up to 2^53, about 9 quadrillion, so larger as_int values lose exact precision once converted. This mainly matters for very large cumulative counters, such as lifetime byte or request totals.
  • Unsigned OTLP fields are stored as signed. INT and BIGINT are always signed in Delta, but OTLP defines some numeric fields as unsigned (uint32, uint64, fixed64) or as very large fixed-width counters. Fields such as *_time_unix_nano, flags, dropped_attributes_count, etc., are stored as-is in a signed INT or BIGINT column. A value above the signed maximum (i32::MAX for INT, i64::MAX for BIGINT) wraps to a negative number instead of erroring.

Schema mapping

Zerobus Ingest maps OTLP data to Delta table columns as described below.

Denormalization

In the OTLP protocol, telemetry data is nested like so.

ResourceSpans (or ResourceLogs, ResourceMetrics)
  └── Resource (attributes, schema_url)
       └── ScopeSpans (or ScopeLogs, ScopeMetrics)
            └── InstrumentationScope (name, version, attributes)
                 └── Span (or LogRecord, Metric)

Zerobus Ingest flattens this hierarchy so that each row contains the full context:

  • resource: A struct containing the resource attributes (as VARIANT) and dropped_attributes_count.
  • resource_schema_url: The schema URL from the enclosing ResourceSpans, ResourceLogs, or ResourceMetrics.
  • instrumentation_scope: A struct containing the scope name, version, attributes (as VARIANT), and dropped_attributes_count.
  • span_schema_url / log_schema_url / metric_schema_url: The schema URL from the enclosing ScopeSpans, ScopeLogs, or ScopeMetrics.

ID encoding

trace_id, span_id, and parent_span_id are stored as lowercase hex-encoded strings:

  • trace_id: 32-character hex string (16 bytes)
  • span_id: 16-character hex string (8 bytes)

Enum encoding

Enum values (kind, status.code, aggregation_temporality, severity_number) are stored as their string names as defined in the OTLP specification. For example: SPAN_KIND_SERVER, STATUS_CODE_OK, AGGREGATION_TEMPORALITY_DELTA.

Next steps

  • Last updated on