Azure Data Factory

How to fix "Status code 400, "{"error":{"code":"InvalidSubscriptionId","message":"The provided subscription identifier 'resourceGroups' is malformed or invalid."}}""

While testing connectivity in Azure Data Factory (ADF), the dataset test connection succeeds, but the data flow test connection fails with the following error: Status code: 400 { "error": { "code":…

Access to pandey is denied Network type is OnPremise IP: fe80::603a:ba7:ae5e:9ba0%13

Azure Data Factory UserErrorSourceQueryTimeout caused by Linked Service Azure Database for Postgresql

Hello, Recently, I have encountered the following error for Lookup or Script activities when the activity takes longer than 15 minutes to complete: Operation on target Lookup1 failed: Failure happened on 'Source' side.…

AzureDataLakeStore does not support MSI authentication in data flow.

We are using MSI in Azure data but when we make a change to this pipeline even if not related (triggers) we are always seeing the same issue: AzureDataLakeStore does not support MSI authentication in data flow. Would it be possible to make a change to…

Best Approach to Move Selected Azure Resources Between Dev and Prod Subscriptions

Hi Everyone, I currently have a Dev Azure subscription containing the following resources: Azure Data Factory with 100 pipelines Azure SQL Database with: 150 tables 100 stored procedures 20 views 10 functions 10 Logic Apps ADLS Gen2 storage account Azure…

ADF when publishing, the arm templates generation failed with Corporate Public network Disabled errror- how to resolve?

ADF settings show networking with Public Endpoint ADF is using Managed Virtual networks with private endpoints to reach out of other AZ resources When publishing the arm templates generation failed with this error: Generating ARM templates …

Copy only new data via a copy data pipeline from PostgreSQL to Azure PostgreSQL

Hello community: I am trying to copy a number of tables from PostgreSQL to Azure PostgresSQL using Azure DataFactory. To do this, I create a pipeline and configure the input and output, the source table and the copied table are exactly the same. As I…

Incremental ETL From Mysql to Azure

Hello Professionals, I am having a totally new experience with AZURE as I have never worked on it , but due to some conditions I have to use AZURE , I am having some issues In making a incremental Data Flow that should extract data from 4 MySQL Server…

Precision loss when migrating data from SQL Server to Snowflake using ADF via CSV staging (but not with Direct Copy)

I’m migrating SQL Server tables to Snowflake using Azure Data Factory. Direct Copy from SQL Server to Snowflake works, but I can’t use it because of the 100 MB row size limit. That forces me to take an intermediate staging approach. Approach 1: Direct…

ADF Data Flow - toTimestamp() returning NULL on blob source filter

I am trying to read files from Azure Blob Storage using a Data Flow, and only read files modified since a date stored in a database. I am trying to use the following setting "Filter by last modified" to limit the time range: The source data…

how connect to a source system - SAP database?

how connect to a source SAP database?This project involves the migration of the existing data warehouse environment from Qlik to Microsoft Azure. The initiative aims to modernize the analytics platform by leveraging Azure’s scalable storage, compute, and…

How to break ForEach loop in ADF

Hi everyone, I have a ForEach activity containing a Copy Data task. The Problem: By default, if one iteration fails, the ForEach continues to trigger all other iterations. In my case, if the first one fails, the following ones are guaranteed to fail as…

Azure Data Factory many activities timeout

My ADF instances pipelines is on Queued status for a long time, and then timed out on the Script, Web and Azure Function activities. Is there some issues with ADF service?

Why an activity was not triggered on the Failure of previous Web activity in ADF

My web activity, Get ChangeOrderRequest, is in a sequential ForEach Prime Contract activity and can fail with the message, "Item not found". I have a Set exit message variable activity to be executed upon failure of the Get ChangeOrderRequest…

Timestamp being logged incorrectly some of the time.

I have an ADF pipeline that starts by writing utcnow to a string pipeline variable, then performs operations with assorted casts to different form factors with formatdatetime, including several script blocks that log to a snowflake log table, all based…

I could not connect to ADF. it is asking me to pay-as-you-go. previously with *******@gmail.com

Hi, I could not connect to ADF. it is asking me to pay-as-you-go. previously with ****@gmail.com , connected with other user . How could I use free . It is showing my one subscription and all. Even I closed that subscription also. I can not open other…

ADF pipeline succeeds but data is wrong - common patterns and how to catch them

Wanted to start a discussion around a category of ADF issues that I see repeatedly in production but rarely documented well. These are cases where the pipeline runs successfully with no errors, but the data loaded is incorrect. Examples include column…

Xero 2.0 ADF connector not supporting the same tables as Xero 1.0 connector

The Xero 2.0 connector in ADF no longer supports the ProjectTasks and Time tables while the Xero 1.0 connector does. This is also mentioned in the documentation: …

Deleted DataFactory Trigger still executing daily for months after being disabled and deleted

We used to execute an SSIS package 2x daily on a Data Factory trigger that started an Integration Runtime, ran the package, and stopped the runtime. This was all supposedly disabled over a year ago. Today I accidentally found from the logs that the…

Join: Microsoft Azure Q&A Champions Program

The Microsoft Azure Q&A Champions Program recognizes and empowers a global community of internal and external Azure experts who help customers succeed by providing high-quality, trusted answers on Microsoft Q&A. The program scales expert led…