livy session

Hi @Sarvasuddi Saikumar (NON EA SC ALT)

The LIVY_JOB_STATE_DEAD error indicates that the Livy/Spark session was terminated unexpectedly during execution.

Along with checking the driver and executor logs from Monitoring Hub, could you also please help share the below details for further investigation in private message:

• Spark Application ID

Livy Session ID

Pipeline Run ID / Notebook Run ID

Approximate failure timestamp (UTC)

Spark pool name and node size

Whether auto-pause/auto-scale is enabled

Notebook duration before failure

Whether issue happens consistently or intermittently

Driver/executor log screenshots or error snippets

Whether the issue occurs with a specific notebook or all notebooks

Additionally, please check:

if the Spark pool was auto-paused/restarted during execution

if there were any recent changes to Spark runtime version/configuration

whether the pool had sufficient available nodes/resources during execution

In many cases, this issue is caused by:

driver OOM (Out Of Memory)

executor crashes

Spark pool resource exhaustion

session timeout

transient backend interruption

dependency/package conflicts

Once the above details are available, we can help narrow down the exact root cause further.

Hi Sarvasuddi Saikumar ,

This LIVY_JOB_STATE_DEAD error usually means the Livy session got terminated unexpectedly during execution. In most cases, this points to a Spark-side issue (for example in Azure Synapse Analytics or Microsoft Fabric), rather than anything related to a database engine.

A good starting point would be to check the Spark driver and executor logs from the Monitoring hub, these typically reveal whether the failure is due to memory pressure, timeout, or connectivity issues.

You may also want to review session behavior and workload patterns. Long-running or idle sessions can sometimes hit timeout limits, and heavy workloads or insufficient cluster sizing can cause the driver or executors to fail. If you’re running multiple notebooks or jobs in parallel, reducing concurrency can also help stabilize the sessions.

In some scenarios, this can be transient (for example, temporary control plane or service-side interruptions), so a retry can help confirm if it’s intermittent.

If the issue continues, sharing a bit more detail would help narrow it down further, such as the runtime duration before failure, Spark pool configuration, concurrency level, and any errors seen in the driver/executor logs.