Azure Voice Live STT: How to enforce a hard per-session language lock (es-ES only)?

Question

Azure Voice Live STT: How to enforce a hard per-session language lock (es-ES only)?

Jurado, Jose Luis 0

Issue context:

We are using Azure Voice Live realtime STT in production voice sessions and need strict monolingual behavior per session.

Observed behavior:

We send session.update with input_audio_transcription.language set to es-ES.
- We also set turn_detection.type to azure_semantic_vad_multilingual.
- - Despite language=es-ES, STT still detects/transcribes other languages when speakers switch language.
- Expected behavior:
- - A hard language lock so STT only recognizes/transcribes Spanish (es-ES) for that session.
- Current payload example:
- {
- "type": "session.update",
- "session": {
- ```
"input_audio_transcription": {
```
- ```
  "model": "azure-speech",
```
- ```
  "language": "es-ES"
```
- ```
},
```
- ```
"turn_detection": {
```
- ```
  "type": "azure_semantic_vad_multilingual",
```
- ```
  "threshold": 0.5,
```
- ```
  "prefix_padding_ms": 500,
```
- ```
  "silence_duration_ms": 1400,
```
- ```
  "barge_in": true
```
- ```
}
```
- }
- }
Question:
1. Is strict hard language lock supported today for Voice Live STT per session?
2. 1. If yes, what exact parameter(s) and API version enforce it?
  2. 1. If not, what is the recommended workaround and roadmap?
  3. Any official guidance is appreciated.

AdityaJadhav-2653 0 Reputation points

2026-07-01T09:11:12.89+00:00
The issue isn't actually your transcription setting; it's your VAD (Voice Activity Detection).

When you set turn_detection.type to azure_semantic_vad_multilingual, you are essentially throwing a massive wrench into your own language lock. That specific VAD type tells Azure, "Hey, look out for speech boundaries across multiple different languages." Because it’s actively scanning for other languages to figure out when someone stops talking, it forces the downstream STT engine to switch into a multilingual mode. It completely overrides that "language": "es-ES" tag you passed it.

How to actually lock it down

To force a hard lock, you have to strip out the multilingual VAD. You need to use the standard, single-language semantic VAD instead.

If you drop the _multilingual suffix, the pipeline falls back strictly to a single language context. Here is exactly what your payload needs to look like to make that happen:

JSON

{

What to expect next

Once you swap azure_semantic_vad_multilingual to azure_semantic_vad, Azure will stop trying to be smart about other languages.

Because it's forced into a Spanish-only acoustic model, if a speaker switches to English or Hindi mid-session, the system won't transcribe it in that language anymore. Instead, it will either try to force those sounds into Spanish words (which will just look like gibberish phonetic Spanish) or it will drop the confidence score so low that it outputs nothing at all. Which, based on what you're building, sounds exactly like the guardrail you want.

Make sure you're hitting the latest API version (like 2026-04-10) when you push this config update, and that should solve the bleeding issue for you.
Jurado, Jose Luis 0 Reputation points

2026-07-01T09:44:30.9+00:00

I updated to turn_detection.type=azure_semantic_vad and api_version=2026-04-10, and my logs confirm both values are sent in session.update. However, English and Portuguese speech is still transcribed when input_audio_transcription.language=es-ES, so why is the change applied in payload but not enforcing a strict Spanish-only STT lock, and what exact setting should reject non-Spanish input?
Jurado, Jose Luis 0 Reputation points

2026-07-08T11:05:24.76+00:00

Thanks, this helps, but my latest runtime log seems to contradict the gpt-realtime explanation. I'm working in a multilingual project that needs to support 20+ languages, but each voice session must stay monolingual. In this test session, the service logs show the session is created with model=gpt-5.4, voice=es-ES-ElviraNeural, language=es-ES, then session.update is sent with input_audio_transcription.model="azure-speech" and language="es-ES", and Azure returns session.updated. After that, when I speak English, the transcript still comes back in English instead of being rejected or degraded. So this does not appear to be a gpt-realtime case. Can you clarify whether Voice Live with gpt-5.4 still treats azure-speech single-language mode only as a hint in this path, or whether the documented single-language restriction is not enforced for this model/service combination?
Jurado, Jose Luis 0 Reputation points

2026-07-08T11:10:46.2533333+00:00

One more clarification: I am also trying to use transcription confidence as a safeguard, but in my current Voice Live session the transcription completed event always gives me the transcript without a confidence value, so my backend always falls back to -1. Is a confidence score supposed to be available for conversation.item.input_audio_transcription.completed when using Azure Voice Live with Azure Speech transcription? If yes, could you point me to the exact event shape, model path, or configuration required to receive it? If no, what is the recommended confidence or language-identification signal for rejecting wrong-language or low-quality turns in a multilingual project where each session must stay monolingual?
Manish Deshpande 7,815 Reputation points Microsoft External Staff Moderator

2026-08-05T00:57:19.3966667+00:00
Hi @Jurado, Jose Luis ,

Thanks for the detail — the logs change the picture, and here's the short version on both points.

1) Language lock not holding on gpt-4.1

My earlier "language is just a hint" explanation was for gpt-realtime (native audio multimodal model). It doesn't apply here — gpt-4.1 is a text-based chat model, so per Microsoft's docs, Azure speech-to-text handles all transcription for non-multimodal models, and single-language config there is documented as a real restriction, not a hint.

So with your logged config (model: gpt-4.1, input_audio_transcription.model: azure-speech, language: es-ES, confirmed via session.updated) still transcribing English audio as English, that's not expected behavior — it looks like a bug where the lock isn't actually enforced for this model/transcription combo, or a runtime/config mismatch behind the scenes. This needs a support ticket with the session ID, timestamps, and the session.update/session.updated pair plus the offending transcript — that's the only way to get engineering eyes on it.

2) No confidence score available

Checked the current API reference — conversation.item.input_audio_transcription.completed only ever returns type, item_id, content_index, and transcript. No confidence field exists for this event, for any transcription model, including azure-speech. Your 1.0 fallback isn't a client bug; the field just isn't there.

For your rejection safeguard, you'll need to build it yourself:

Run a quick language-ID check against the returned transcript and reject/re-prompt on mismatch, or

Use plain Azure Speech-to-text (outside Voice Live) if you need native confidence scores — that surface returns them, Voice Live's wrapper currently doesn't.

Worth filing both as product feedback — the enforcement gap and the missing confidence/language-match signal are both blockers for a real per-session monolingual lock.

Thanks,
Manish

2 answers

Your answer

AdityaJadhav-2653 0 Reputation points

2026-07-01T09:11:12.89+00:00

The issue isn't actually your transcription setting; it's your VAD (Voice Activity Detection).

When you set turn_detection.type to azure_semantic_vad_multilingual, you are essentially throwing a massive wrench into your own language lock. That specific VAD type tells Azure, "Hey, look out for speech boundaries across multiple different languages." Because it’s actively scanning for other languages to figure out when someone stops talking, it forces the downstream STT engine to switch into a multilingual mode. It completely overrides that "language": "es-ES" tag you passed it.

How to actually lock it down

To force a hard lock, you have to strip out the multilingual VAD. You need to use the standard, single-language semantic VAD instead.

If you drop the _multilingual suffix, the pipeline falls back strictly to a single language context. Here is exactly what your payload needs to look like to make that happen:

JSON

{

What to expect next

Once you swap azure_semantic_vad_multilingual to azure_semantic_vad, Azure will stop trying to be smart about other languages.

Because it's forced into a Spanish-only acoustic model, if a speaker switches to English or Hindi mid-session, the system won't transcribe it in that language anymore. Instead, it will either try to force those sounds into Spanish words (which will just look like gibberish phonetic Spanish) or it will drop the confidence score so low that it outputs nothing at all. Which, based on what you're building, sounds exactly like the guardrail you want.

Make sure you're hitting the latest API version (like 2026-04-10) when you push this config update, and that should solve the bleeding issue for you.
Jurado, Jose Luis 0 Reputation points

2026-07-01T09:44:30.9+00:00

I updated to turn_detection.type=azure_semantic_vad and api_version=2026-04-10, and my logs confirm both values are sent in session.update. However, English and Portuguese speech is still transcribed when input_audio_transcription.language=es-ES, so why is the change applied in payload but not enforcing a strict Spanish-only STT lock, and what exact setting should reject non-Spanish input?
Jurado, Jose Luis 0 Reputation points

2026-07-08T11:05:24.76+00:00

Thanks, this helps, but my latest runtime log seems to contradict the gpt-realtime explanation. I'm working in a multilingual project that needs to support 20+ languages, but each voice session must stay monolingual. In this test session, the service logs show the session is created with model=gpt-5.4, voice=es-ES-ElviraNeural, language=es-ES, then session.update is sent with input_audio_transcription.model="azure-speech" and language="es-ES", and Azure returns session.updated. After that, when I speak English, the transcript still comes back in English instead of being rejected or degraded. So this does not appear to be a gpt-realtime case. Can you clarify whether Voice Live with gpt-5.4 still treats azure-speech single-language mode only as a hint in this path, or whether the documented single-language restriction is not enforced for this model/service combination?
Jurado, Jose Luis 0 Reputation points

2026-07-08T11:10:46.2533333+00:00

One more clarification: I am also trying to use transcription confidence as a safeguard, but in my current Voice Live session the transcription completed event always gives me the transcript without a confidence value, so my backend always falls back to -1. Is a confidence score supposed to be available for conversation.item.input_audio_transcription.completed when using Azure Voice Live with Azure Speech transcription? If yes, could you point me to the exact event shape, model path, or configuration required to receive it? If no, what is the recommended confidence or language-identification signal for rejecting wrong-language or low-quality turns in a multilingual project where each session must stay monolingual?

Answer 1

Jurado, Jose Luis 0

wrong placement

0 comments

Answer 2

Hello @Jurado, Jose Luis

Thanks for testing that so precisely the fact that it still leaks after moving to gpt-realtime is actually the key clue, and it points straight at the root cause. Two things to untangle here.

1. Turn detection is not your language control turn_detection (server_vad, azure_semantic_vad, azure_semantic_vad_multilingual) only governs when the service decides a turn started/ended (and, with a languages array, filler-word removal). It has no effect on which language your transcript comes back in. So swapping the _multilingual suffix didn't lock anything — you can set VAD to whatever endpoints best; it's orthogonal to language.

2. The real reason it leaks: gpt-realtime is a native-audio (multimodal) model. Multimodal models (gpt-realtime, gpt-realtime-mini, phi4-mm-realtime) listen to the audio directly. For these, input_audio_transcription.language is only a hint to improve transcription quality — not a hard lock, and the model returns results for languages not listed (just at lower quality). The transcription runs as a separate pass, so it can't constrain what the underlying model hears or answers. That's exactly why a Spanish-only setting still lets non-Spanish through.

Also note: with gpt-realtime/-mini, the available transcription models are the OpenAI ones (whisper-1, gpt-4o-transcribe, gpt-4o-mini-transcribe, gpt-4o-transcribe-diarize) — not azure-speech. So you can't get the azure-speech "single language" restriction while staying on a native-audio model.

How to actually enforce a per-session Spanish lock at the recognition layer: Use a non-multimodal (text) chat model — e.g. gpt-4.1 — paired with azure-speech transcription in single-language mode:

User's image

This is the documented "single language configuration," which restricts the transcription languages detected. Two things to watch:

language must be a single value with no comma. If it ever becomes "es-ES,en-US", Voice Live treats it as multi-language detection and will happily transcribe those other languages.
"Restrict" ≠ "reject." There's no flag that makes the recognizer refuse non-Spanish audio outright — a single-language config biases recognition to Spanish rather than returning a correct English/Portuguese transcript. If you need to hard-drop non-Spanish turns, do it at the application layer (run language identification / confidence-score filtering on the transcript and discard turns below your Spanish-confidence threshold).

A few validation/robustness steps:

Send session.update before streaming audio, and confirm the session.updated echo shows model: "azure-speech" and language: "es-ES" — a later session.update elsewhere in your code can silently reset it.
If you keep any native-audio model, pin the model's own behavior with an instruction and pin the output voice locale so replies don't drift:
```
  "instructions": "The caller always speaks Spanish (es-ES). Transcribe and respond only in Spanish, even if you detect another language in the input audio."
```
(This biases behavior; it's not a hard STT lock — only the text-model + azure-speech path gives you the recognition-layer restriction.)
Make sure you're on the GA API version (2025-10-01 or newer), since transcription/session behavior has changed across versions.

Links:
Voice Live supported languages (three language modes / single-language config)
https://learn.microsoft.com/en-us/azure/ai-services/speech-service/voice-live-language-support?tabs=speechinput
How to use the Voice Live API (audio input transcription, turn detection):
https://learn.microsoft.com/en-us/azure/ai-services/speech-service/voice-live-how-to

Speech-to-text language & locale support:

https://learn.microsoft.com/en-us/azure/ai-services/speech-service/language-support?tabs=stt

VAD isn't the lever. On a native-audio model like gpt-realtime, language is only a hint — no hard lock exists. For a true per-session es-ES lock, switch to a text chat model + azure-speech transcription with a single es-ES locale (no comma), and handle any hard rejection of non-Spanish turns at the app layer.

Thanks,
Manish.

Azure Voice Live STT: How to enforce a hard per-session language lock (es-ES only)?

How to actually lock it down

What to expect next

2 answers

Your answer