IoT analytics with Azure Data Explorer and Azure IoT Hub

This solution idea describes how Azure Data Explorer provides near real-time analytics for fast-flowing, high-volume streaming data from Internet of Things (IoT) devices and sensors. This data flow is part of an overall IoT solution that integrates operational and analytical workloads with Azure Cosmos DB and Azure Data Explorer.

Jupyter is a trademark of its respective company. No endorsement is implied by the use of this mark. Apache® and Apache Kafka® are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries. No endorsement by The Apache Software Foundation is implied by the use of these marks.

Architecture

Download a Visio file of this architecture.

Data flow

The following data flow corresponds to the previous diagram:

1. Azure Event Hubs, Azure IoT Hub, or Apache Kafka ingests a wide range of fast-flowing streaming data, such as logs, business events, and user activities.

2. Azure Functions or Azure Stream Analytics processes the data in near real time.

3. Azure Cosmos DB stores streamed messages in JSON format to serve a real-time operational application.

4. Azure Data Explorer ingests data for analytics by using its connectors for Event Hubs, IoT Hub, or Kafka for low latency and high throughput.

   Alternatively, you can ingest blobs from your Azure Blob Storage or Azure Data Lake Storage account into Azure Data Explorer by using an Azure Event Grid data connection.

   You can also continuously export data to Azure Storage in compressed, partitioned Apache Parquet format and seamlessly query the data with Azure Data Explorer. For more information, see Continuous data export overview.

5. To serve both the operational and analytical use cases, route data either to Azure Data Explorer and Azure Cosmos DB in parallel, or from Azure Cosmos DB to Azure Data Explorer.

   Azure Cosmos DB transactions can trigger Azure Functions via change feed. Azure Functions streams data to Event Hubs for ingestion into Azure Data Explorer. Alternatively, Azure Functions can invoke Azure Digital Twins through its API, which then streams data to Event Hubs for ingestion into Azure Data Explorer.

6. The following interfaces get insights from data stored in Azure Data Explorer:

   • Custom analytics apps that blend data from Azure Digital Twins and Azure Data Explorer APIs

   • Near real-time analytics dashboards that use Azure Data Explorer dashboards, Power BI, or Grafana

   • Alerts and notifications from the Azure Data Explorer connector for Azure Logic Apps

   • The Azure Data Explorer web UI, Kusto.Explorer, and Jupyter notebooks

7. Azure Data Explorer integrates with Azure Databricks and Azure Machine Learning to provide machine learning services. You can also build machine learning models by using other tools and services, and export them to Azure Data Explorer for scoring data.

Components

This solution idea uses the following Azure components.

Azure Data Explorer

• Anomaly detection and forecasting is a built-in analytics feature in Azure Data Explorer. It detects outliers and predicts future values to support proactive monitoring and decision-making. In this architecture, it identifies unusual patterns in IoT data and forecasts expected behavior over time.

• Anomaly diagnosis for root analysis is a KQL capability that helps identify the root causes of anomalies. It analyzes contributing dimensions and metrics to streamline troubleshooting. In this architecture, it isolates the source of anomalies that are detected in device data.

• Azure Data Explorer is a fully managed, high-performance analytics service. It processes large volumes of streaming data from applications, websites, and IoT devices in near real time. In this architecture, it serves as the central analytics engine for the ingestion, querying, and visualization of IoT data.

• Azure Data Explorer dashboards are a visualization feature in the Azure Data Explorer web UI. You can use Azure Data Explorer dashboards to export Kusto queries into interactive dashboards for real-time data exploration. In this architecture, they display insights from IoT data streams and anomaly detection results.

• Azure Data Explorer web UI is a browser-based interface for Azure Data Explorer clusters. It supports users who write, run, and share KQL commands and queries. In this architecture, it provides a workspace for analysts to query and explore IoT data.

• Time series analysis is a built-in capability in Azure Data Explorer. It helps users explore temporal patterns, trends, and seasonality in time-based data. In this architecture, it reveals long-term trends and cyclical behavior in IoT sensor readings.

Other Azure components

• Azure Cosmos DB is a fully managed, fast NoSQL database with open APIs for any scale. In this architecture, it stores operational data from IoT devices for scalable, low-latency access.

• Azure Digital Twins is a platform for modeling physical environments as digital representations. In this architecture, it maintains digital models of IoT-connected assets to support spatial analysis and contextual insights.

• IoT Hub enables bidirectional communication between IoT devices and the Azure cloud. In this architecture, it serves as the central messaging hub for device data and command-and-control operations.

• Event Hubs is a fully managed, real-time data ingestion service. In this architecture, it ingests data from IoT devices and streams it into the analytics pipeline.

• Kafka on HDInsight is an enterprise-grade, cost-effective service for Apache Kafka on Azure. In this architecture, it provides an alternative streaming backbone for IoT data ingestion and distribution.

Scenario details

This solution uses Azure Data Explorer to get near real-time IoT data analytics on fast-flowing, high-volume streaming data from a wide range of IoT devices.

Potential use cases

• Fleet management, for predictive maintenance of vehicle parts. This solution is ideal for the automotive and transportation industry.

• Facilities management, for energy and environment optimization.

• Combining real-time road conditions with weather data for safer autonomous driving.

Contributors

Microsoft maintains this article. The following contributors wrote this article.

Principal author:

• Shlomo Sagir | Senior Content Developer

Other contributors:

• Sreedhar Pelluru | Senior Content Developer

To see nonpublic LinkedIn profiles, sign in to LinkedIn.

Next steps

• What is Azure Data Explorer?
• Visualize data with Azure Data Explorer dashboards

Related resource

• Analytics architecture design