Cannot enumerate blobs in Databricks-managed storage account (11.68 TiB) - 403 AuthorizationPermissionDenied

Hi @David Nagy ,

sorry you’re running into this. A 403 AuthorizationPermissionDenied when trying to list/enumerate blobs usually points to one of two broad categories:

1. the security principal you’re using doesn’t have the right “data-plane” permission for listing/deleting objects, or
2. storage access is being blocked by network/firewall/private endpoint settings (so the request never successfully authenticates for the operation).

Are Databricks-managed workspace storage accounts restricted from customer blob access?

The provided documentation doesn’t state that Databricks-managed workspace storage is “customer-blocked” by design. What it does say is that access problems (including 403s) are typically caused by RBAC permissions and/or network/firewall/private endpoint configuration.

So the practical answer is: the account isn’t inherently “unlistable”; rather, your identity must have the correct permissions and network path must allow access.

What role/permission is needed to enumerate and clean blobs?

From the provided 403 authorization guidance:

• For read operations, use Storage Blob Data Reader
• For write/delete operations, use Storage Blob Data Contributor
• For management operations, use Storage Blob Data Owner

Also, role assignment scope matters: assign roles at the storage account, container, or queue level as appropriate (not only at broader scopes if Databricks/workspace storage uses more specific access boundaries).

Important detail

Your current setup uses Storage Blob Data Reader at the storage account scope. That role supports reading, but depending on what your cleanup requires, you may also need permissions that enable:

• listing (enumeration)
• and/or deleting

If you’re trying to delete old data (not just read), the “Contributor” level is typically what the docs call out as the role for write/delete-capable data operations.

Could this be network-related instead of RBAC?

Yes. The provided troubleshooting guidance repeatedly calls out network/firewall causes for 403s, including:

• Storage firewall enabled while the Databricks workspace (or your client where you run CLI/SDK) isn’t allowlisted
• Private endpoint enabled but required subnet/network allowlisting is missing
• Missing/incorrect private DNS zone linkage (for blob/dfs)
• Databricks VNet not properly whitelisted when firewalls are enabled

If you’re testing from outside Databricks (CLI/Portal/SDK from a workstation), note that those requests also have to satisfy storage firewall rules—the docs explicitly mention firewall/IP subnet allowlists for storage.

Alternative ways to identify old data to delete

The provided docs focus on access methods (ABFS/DBFS mounts, Spark DataFrame/RDD/Hive access) and general connectivity/auth troubleshooting, but they don’t include a Databricks-managed-data-specific “inventory/delete API” for workspace storage.

That said, a common approach aligned with the docs is:

• Use Databricks-native access patterns like ABFS / dbutils.fs.ls (when permissions allow) to identify which paths/blobs correspond to old artifacts/logs/ephemeral data.
• Then perform deletion using a principal that has the right write/delete role.

If list/enumeration remains blocked due to 403, you typically won’t be able to build an inventory that way.

Practical next checks for the customer

1. Confirm whether the cleanup requires delete (or only read/list).
   • If delete is required, ensure you have Storage Blob Data Contributor (or equivalent) rather than only Storage Blob Data Reader.
2. Verify network access:
   • If the storage account firewall/private endpoint is enabled, ensure the correct Databricks subnet/VNet and any private DNS entries are configured.
   • The docs also call out telnet testing and whitelisting for networking-related 403 causes.
3. Confirm you’re using the same security principal for testing:

The docs note that job/service identities may differ from interactive users, and missing permissions at the storage layer leads to 403.

Follow-up questions (to pinpoint root cause)

1. Are you attempting cleanup from inside Databricks or from an external machine (CLI/Portal/SDK)?
2. Does the cleanup require deletion, or is it only identification/listing?
3. Is the storage account configured with firewalls and/or a private endpoint? If yes, was the required private DNS setup done?
4. Which identity is actually used when listing blobs (Databricks workspace managed identity / service principal / your signed-in user)?
5. What exact operation triggers the 403 (e.g., “list containers”, “list blobs in container”, “HEAD blob”, “delete blob”)?
6. Are there any customer-managed keys / encryption scope constraints on the containers that might block access with 403?

Reference links

• Troubleshoot 403 errors in Azure Blob Storage (permission vs network causes; Reader/Contributor/Owner mapping): https://learn.microsoft.com/troubleshoot/azure/azure-storage/blobs/authentication/storage-troubleshoot-403-errors
• Connect to Azure Data Lake Storage and Blob Storage (Databricks access patterns with abfss): https://learn.microsoft.com/azure/databricks/connect/storage/azure-storage

If you answer the follow-up questions, we can help narrow down whether you’re missing delete/list permissions vs running into network/private endpoint/firewall blocking.

Hope this helps. If you have any follow-up questions, please let me know. I would be happy to help.

Hi @David Nagy ,
I hope you had a chance to review the information shared earlier, and I hope this information has been helpful! If you still have questions, please let us know what is needed in the comments so the question can be answered.

HI @SAI JAGADEESH KUDIPUDI and @Sina Salam

 We ran all workspace purge actions (workspace storage, revision history, cluster logs) and deleted all workspace models, but storage is still ~11.86 TB with no visible drop. We also cannot inspect the remaining data: internal dbfs:/databricks/mlflow-tracking/... paths are protected (No operations allowed on this path), and artifact list APIs return no files/bytes. Question: what is the supported way to identify and reclaim this residual storage in Databricks-managed workspace storage, and is backend intervention from Databricks support required?

We are able to see the root folder, (attached image) but if we would like to explore what is inside then

json
{ "shellProps": { "sessionId": "<session-id>", "extName": "Microsoft_Azure_Storage", "contentName": "BlobsBlade", "resourceId": "/subscriptions/<subscription-id>/resourceGroups/<databricks-managed-rg>/providers/Microsoft.Storage/storageAccounts/<managed-storage-account>", "code": 403 }, "error": { "message": "DenyAssignmentAuthorizationFailed", "code": 403, "details": { "viewModel": { "causes": [ { "details": "The client '<user-email>' with object id '<user-object-id>' has permission to perform action 'Microsoft.Storage/storageAccounts/listKeys/action' on scope '/subscriptions/<subscription-id>/resourceGroups/<databricks-managed-rg>/providers/Microsoft.Storage/storageAccounts/<managed-storage-account>'; however, the access is denied because of the deny assignment with name 'System deny assignment created by Azure Databricks /subscriptions/<subscription-id>/resourceGroups/<workspace-rg>/providers/Microsoft.Databricks/workspaces/<workspace-name>' and Id '<deny-assignment-id>' at scope '/subscriptions/<subscription-id>/resourceGroups/<databricks-managed-rg>'." } ] } } } }

We are changing the storage from GRS to LRS also.