Some issues with Purview's under-the-hood implementation.

Mofei Zhuang 160 Reputation points Microsoft Employee
2026-06-17T03:55:13.85+00:00
  1. What is the current write limit and the maximum concurrent write limit for a single Purview Account? Under what specific circumstances will an HTTP 408 error be triggered?
  2. I would like to know about the underlying architecture of Purview. Is the Index layer currently implemented using Azure AI Search (formerly Azure Search)? And is Azure Cosmos DB used for the data storage layer?
  3. Based on my testing, if an Entity contains a large number of relationships, querying it is still significantly slower than querying an entity without relationships, even when ignoreRelationship=true is specified. What is the technical reason behind this behavior?
Microsoft Security | Microsoft Purview
0 comments

Answer accepted by question author

Pilladi Padma Sai Manisha 11,620 Reputation points Microsoft External Staff Moderator
2026-06-17T04:12:10.1266667+00:00

Hi @Mofei Zhuang
Thanks for your questions.

  1. Write limits and HTTP 408 errors

There isn't a publicly documented per-account write throughput limit or maximum concurrent write limit for a single Microsoft Purview account. The effective throughput depends on several factors, such as the API being used, request size, entity complexity (attributes, classifications, relationships), and the current service load.

An HTTP 408 (Request Timeout) generally indicates that the service could not complete processing within the allowed time window. This may occur when:

  • Processing very large payloads or batches
  • Entities contain a large number of relationships or complex metadata structures
  • Multiple concurrent write operations create contention
  • The service is under temporary load or throttling conditions

For sustained high-volume workloads, Microsoft recommends implementing retry logic with exponential backoff.

  1. Purview internal architecture

Microsoft doesn't publicly disclose the underlying implementation details or guarantee which Azure services are used internally (for example, whether specific indexing or storage layers are backed by Azure AI Search or Azure Cosmos DB). Since these implementation details may change over time, they shouldn't be relied upon for application design or performance expectations.

  1. Why entities with many relationships are slower to query

The behavior you're observing is expected.

Even when ignoreRelationship=true is specified, the service may still need to process relationship metadata internally during entity retrieval. Relationships are deeply integrated into Purview's graph-based metadata model and can impact:

  • Entity graph traversal and validation
  • Authorization and access checks
  • Internal joins and metadata aggregation
  • Response construction and filtering

As a result, entities with a large number of relationships can experience higher query latency compared to entities without relationships, even when relationships are excluded from the final response payload.

If you're encountering significant performance degradation at scale, it would be helpful to share:

  • Approximate number of relationships per entity
  • Total number of entities in the catalog
  • API being used (Atlas APIs, Data Map APIs, etc.)
  • Whether the issue occurs consistently or only during peak ingestion activity

For deeper insights into service behavior or undocumented limits, we recommend opening a support request, as those details are not publicly documented and may require investigation by the product engineering team.

Was this answer helpful?

1 person found this answer helpful.
0 comments

1 additional answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. Mofei Zhuang 160 Reputation points Microsoft Employee
    2026-06-18T23:31:21.3766667+00:00
    • For Submit Scenario

    Here is the sample data structure, we are using custom entity， please see response_1781823816744.txt

    and for QualifiedName field we use in this formart : pds://bench-rel-20260618050106/sub/0008ba92-f566-482b-8ea6-95b06465a802/appinsights/res-appinsights-0008ba92-0014

    From the test result I found that :

    we put 200 entities for a batch to submit , because when we add more than this it will occur an error like the URL is too long or some like that error info.

    If the request can success, the avg cost time will be around 4000 ms. (without any relationship， just the independent entities)

    So, my question is there any other way I can submit more for one time. Or is there any optimization I can do to accelerate the submit process.

    We're using DataMapClient.GetEntityClient().BatchCreateOrUpdateAsync to batch Submit

    • For Catalog search

    Here is the way we use catalog to search our target entities, I found that even I remove all the relationship the query perf is not so satisficed.

    Can you help me to check if is the right way to use the catalog search ? 

                `var prodSubResults = await _catalogService.SearchEntityByServiceIdAndAttributesAsync(`

    new Dictionary<string, string>

    {

    { "ServiceId", serviceId },

    { "Environment", "Prod" }

    },

    CustomEntityConstants.SUBSCRIPTION_CUSTOM_ENTITY_NAME,

    FilterOperator.Eq);

    public async Task<IEnumerable<SearchResultValue>> SearchEntityByServiceIdAndAttributesAsync(IReadOnlyDictionary<string, string> attrDic, string typeName, FilterOperator Oper)

    {

    var filters = new List<object>

    {

    new { typeName = typeName }

    };

    foreach (var kvp in attrDic)

    {

    filters.Add(new

    {

    attributeName = kvp.Key,

    @operator = Oper.GetEnumMemberValue(),

    attributeValue = kvp.Value

    });

    }

    var resList = new List<SearchResultValue>();

    string? continuationToken = null;

    var config = new QueryConfig

    {

    Limit = 1000,

    Filter = BinaryData.FromObjectAsJson(new

    {

    and = filters

    }),

    };

    do

    {

    var response = await purviewClient.DataMapClient.GetDiscoveryClient().QueryAsync(config);

    resList.AddRange(response.Value.Value ?? Enumerable.Empty<SearchResultValue>());

    continuationToken = response.Value.ContinuationToken;

    config.ContinuationToken = continuationToken;

    } while (!string.IsNullOrEmpty(continuationToken));

    return resList;

    }

    Was this answer helpful?

Your answer

Answers can be marked as 'Accepted' by the question author and 'Recommended' by moderators, which helps users know the answer solved the author's problem.