408 and 503 returns from Mistral Document AI

Hi Ian Van Dort,

Thank you for the detailed information.

I understand that you are experiencing intermittent HTTP 503 ("no healthy upstream") and HTTP 408 ("upstream request timeout") errors with the Mistral Document AI (mistral-document-ai-2512) model deployed through Azure AI Foundry. Given that this is affecting your production document-processing pipeline, I appreciate the urgency.

What These Errors Typically Indicate

Based on the behavior you described:

HTTP 503 – "no healthy upstream" usually indicates that the service was temporarily unable to route your request to a healthy backend instance. This often occurs when the serverless inference cluster is experiencing transient capacity constraints or backend health issues.

HTTP 408 – "upstream request timeout" indicates that the backend service did not complete processing within the allowed timeout window, which can happen if requests are queued, delayed during routing, or if backend processing takes longer than expected.

Why This Is Likely Not a Quota Issue

Your observations strongly suggest that this is not related to throttling or quota limits because:

• Your traffic is well below documented Azure quota limits
• You are applying additional client-side rate limiting
• You are not receiving HTTP 429 (Too Many Requests) responses
• The issue is occurring across multiple EU regions

This points more toward intermittent backend capacity, cluster availability, or service-side routing constraints rather than client-side overuse.

Recommended Mitigations

To improve resiliency while the issue is being investigated, we recommend the following:

1. Implement Retry Logic

Use exponential backoff with jitter for transient failures:

• Retry on HTTP 503 and 408
• Start with a short delay
• Retry up to 2–3 times

This helps absorb temporary capacity fluctuations without overwhelming the service.

2. Continue Cross-Region Failover

Your current multi-region design is a strong best practice.

• Distribute traffic evenly across your EU deployments
• Automatically reroute requests if a region begins returning elevated 503 or 408 errors

3. Monitor Request Characteristics

Please capture:

• Requests per second (QPS)
• Concurrent requests
• Average and peak document size
• Page count per document
• Failure rates by region

Large or highly concurrent workloads can occasionally exhaust shared serverless capacity, even when overall throughput remains below published quotas.

4. Validate in Foundry Portal

Testing identical requests in the Azure AI Foundry playground across each deployed region can help determine whether the issue is isolated to specific regions or more broadly distributed.

Regarding Scaling and Service Health

A recurring pattern of 503 and 408 responses under normal operating conditions may indicate temporary backend saturation, capacity contention, or regional service health degradation. In these cases, such errors can occur before explicit throttling (429) is triggered.

If an active service-side issue is identified, we will ensure that it is properly tracked and that you are kept informed of any updates or mitigation efforts.

Please refer this

Foundry Models sold directly by Azure (mistral-document-ai-2512 spec & region support): https://learn.microsoft.com/azure/foundry/foundry-models/concepts/models-sold-directly-by-azure?wt.mc_id=knowledgesearch_inproduct_azure-cxp-community-insider#mistral-models-sold-directly-by-azure

Troubleshoot model deployments in Azure AI Foundry: https://learn.microsoft.com/azure/ai-studio/how-to/troubleshoot-deploy-and-monitor

I Hope this helps. Do let me know if you have any further queries.

Thank you!

@SRILAKSHMI C

Thank you for the response.

1. We already have retries implemented, however these only make the problem worse currently as it just keeps receiving the same error.
2. Yes, we will continue doing this.
3. Yes we already log these things, thank you.
4. I will try this out.

I find it hard to believe this would be temporary backend saturation or any of the other things you list, as we have had this issue multiple times over the past few weeks. So either your service health monitor is inaccurate (it indicates no issues) or this is not the problem. Do you know which of the the two it is?

We would really like to have this resolved, can you indicate to me if the endpoints we have access to are experiencing issues? All our EU endpoints for this Mistral Document AI service we get these problems.

Hi @Ian Van Dort

Thank you for the update, I’ll check with our backend team and share an update with you as soon as possible.

Hello @SRILAKSHMI C,

Same issue also on our side. Is there any update?

Same in both poland central and sweden central - basically unusable