Agent is not answering question which is based on Knowledge base,But i am getting response in Knowledge base

Hello @Suraj Panker

Thank you for reaching out to Microsoft Q&A.

From the screenshots, I can see that:

The Knowledge Base itself is working correctly. When you query the Knowledge Base directly in the Knowledge Base testing experience, it successfully retrieves the relevant content for "RBI/2021-22/78".

However, when you ask the same question through the Agent, it responds that "The indexed knowledge base does not contain any relevant content..." instead of using the retrieved information.

This indicates that the issue is likely not with the Knowledge Base indexing, but rather with how the Agent is configured to retrieve and ground its responses using the Knowledge Base.

Here are a few things to verify:

1. Verify the Knowledge Base is attached to the Agent

Open your Agent in Azure AI Foundry and confirm that the correct Knowledge Base has been added as a knowledge source and that the latest configuration has been saved and published.

2. Review the Agent's Retrieval Configuration

Check the Retrieval settings for the agent and ensure:

• Retrieval is enabled.
• The correct Knowledge Base is selected as the data source.
• There are no filters, relevance thresholds, or retrieval limits preventing the agent from retrieving the appropriate document.

If available, compare the retrieval results from the Knowledge Base Debug view with the agent's retrieval trace to determine whether the agent is retrieving the document but choosing not to use it, or whether retrieval is failing altogether.

3. Check the Agent's Scope Configuration

If your agent is configured with the in_scope parameter set to true, the agent will answer only using content that it determines to be within the configured knowledge scope. Depending on the configuration, this may result in fallback responses indicating that no relevant content was found.

As a troubleshooting step, you can try:

• Reviewing the in_scope setting.
• If appropriate for your scenario, temporarily changing it from true to false, saving the configuration, and testing the same query again.

4. Review the Agent Instructions

Check the system instructions or prompt configured for the agent.

If the instructions explicitly tell the model to avoid answering unless it has high confidence, or discourage using retrieved content, the agent may return a fallback response even when relevant information exists in the Knowledge Base.

5. Adjust Response Generation Parameters

If the agent is retrieving content but not consistently grounding its responses, you can also try tuning the generation parameters.

For example:

• Reduce Temperature (for example, around 0.2) to make responses more deterministic.
• Adjust Top P (for example, around 0.9) to reduce response variability.

These settings can help the model rely more consistently on retrieved knowledge.

6. If Using Azure AI Language Question Answering

If your implementation uses Azure AI Language Question Answering (FAQ-style matching), retrieval quality can also be affected by:

• Ambiguous or overlapping question phrasing.
• Insufficient alternate phrasings.
• Missing synonyms.
• Question wording that differs significantly from the indexed content.

In this case, consider:

• Adding alternate question phrasings.
• Including additional synonyms.
• Reducing overlap between similar Q&A pairs.
• Ensuring the user queries closely match the indexed content.

7. Republish the Agent

If the Knowledge Base was recently created or updated, save and republish the agent to ensure it is using the latest version of the Knowledge Base.

Please refer this

Azure OpenAI FAQ (training data/response behavior context): https://learn.microsoft.com/azure/ai-services/openai/faq

Resolve unanswered questions via in_scope (set in_scope true/false): https://learn.microsoft.com/azure/ai-services/openai/reference#azuresearchchatextensionparameters

Azure Bot sample showing how to call the knowledge base client: https://learn.microsoft.com/azure/bot-service/bot-builder-howto-answer-questions

I Hope this helps. Do let me know if you have any further queries.

Thank you!

Hi @Suraj Panker

Did you get any chance to review the above response. Do let me know if you have any further queries.

Thank you!