Get media transcription, translation, and language identification insights

Transcription, translation, and language identification features detect, transcribe, and translate speech in media files in more than 50 languages.

Azure AI Video Indexer processes speech in an audio file, extracts a transcription, and then translates that transcription into many languages. When you select a translation language, both the transcription and insights such as keywords, topics, labels, and OCR are translated into that language. You can use the transcription as-is or combine it with speaker insights that map and assign transcripts to speakers. The audio file can contain multiple speakers. Each speaker gets an ID that's displayed with their transcribed speech.

Language identification (LID) recognizes the supported dominant spoken language in the video file. For more information, see Applying LID.

Multi-language identification (MLID) automatically recognizes the spoken languages in different segments in the audio file and sends each segment to be transcribed in the identified languages. At the end of this process, all transcriptions are combined into the same file. For more information, see Applying MLID. Azure AI Video Indexer returns these insights in a categorized JSON list that includes ID, language, transcribed text, duration, and confidence score.

When Azure AI Video Indexer indexes media files with multiple speakers, it performs speaker diarization. It identifies each speaker in a video and attributes each transcribed line to a speaker. Speakers receive unique identities such as Speaker #1 and Speaker #2. This feature can help identify speakers during conversations and can be useful in scenarios such as doctor-patient conversations, agent-customer interactions, and court proceedings.

Media transcription, translation, and language identification use cases

• Promote accessibility by making content available to people with hearing loss. Use Azure AI Video Indexer to generate speech-to-text transcription and translation into multiple languages.
• Improve content distribution to a diverse audience in different regions and languages. Deliver content in multiple languages by using Azure AI Video Indexer's transcription and translation capabilities.
• Improve manual closed-caption and subtitle generation. Use Azure AI Video Indexer's transcription and translation capabilities and closed captions generated by Azure AI Video Indexer in one of the supported formats.
• Transcribe videos in unknown languages by using language identification (LID) or multi-language identification (MLID). These features allow Azure AI Video Indexer to automatically identify the languages appearing in the video and generate the transcription accordingly.

View insights JSON in the web portal

After you upload and index a video, download insights in JSON format from the web portal.

1. Select the Library tab.
2. Select the media file you want to analyze.
3. Select Download, then select Insights (JSON). The JSON file opens in a new browser tab.
4. Find the key-value pairs described in the example response.

Use the API

1. Use a Get Video Index request. Pass &includeSummarizedInsights=false.
2. Find the key-value pairs described in the example response.

Example response

The API returns detected languages in the video under sourceLanguage and sourceLanguages. Each instance in the transcript section includes the transcribed language.

    "insights": {
      "version": "1.0.0.0",
      "duration": "0:01:50.486",
      "sourceLanguage": "en-US",
      "sourceLanguages": [
        "es-ES",
        "en-US"
      ],
      "language": "en-US",
      "languages": [
        "en-US"
      ],
      "transcript": [
        {
          "id": 1,
          "text": "Hi, I'm Doug from office. We're talking about new features that office insiders will see first and I have a program manager,",
          "confidence": 0.8879,
          "speakerId": 1,
          "language": "en-US",
          "instances": [
            {
              "adjustedStart": "0:00:00",
              "adjustedEnd": "0:00:05.75",
              "start": "0:00:00",
              "end": "0:00:05.75"
            }
          ]
        },
        {
          "id": 2,
          "text": "Taylor Lee, with office graphics.",
          "confidence": 0.8879,
          "speakerId": 1,
          "language": "en-US",
          "instances": [
            {
              "adjustedStart": "0:00:05.75",
              "adjustedEnd": "0:00:07.01",
              "start": "0:00:05.75",
              "end": "0:00:07.01"
            }
          ]
        },

Sample code

See all samples for Azure AI Video Indexer.