How to Debug Chatbot Answers

Review Chatbot History with Verbose Logging

FlowHunt provides a history of chatbot interactions, allowing you to review detailed log information about each chat session. If specific components, such as the Document Retriever, were used during a chat, the logs in verbose mode will indicate this, including which documents were discovered and leveraged to compose the chatbot’s answer.

Where to access verbose logs?

1. In your FlowHunt workspace, open History from the left side menu or use this link: https://app.flowhunt.io/aistudio/chatbot-history
2. Use the available filters. It’s recommended to use at least the session filter to specify the timeframe you want to check, to narrow down the results.
3. Open a specific conversation.
4. Change from Plain to Verbose on the top left side of the screen.
5. Look for Tool Call entries and see if (and which) Tool was used, along with detailed results.

Resolve identified issues in chatbot answers

If you find that the chatbot provided incorrect or suboptimal information even though the correct information is available, follow these steps to troubleshoot and improve the outcomes.

• Check if the Document Retriever was used. If the verbose history indicates that the Document Retriever was not used, update the system prompt of the tool-calling agent component to ensure it always leverages the Document Retriever when constructing answers. You can strengthen the prompt by adding instructions such as:

  <core_instructions>
  ALWAYS use Tool Call and utilize "Document Retriever" to provide concise responses with URLs from Document Retriever for more details.
  </core_instructions>
  

  This should ensure that the chatbot searches for and incorporates your documents and articles from Schedules as source material to compose responses, minimizing the risk of providing incorrect or suboptimal information based solely on its pre-trained data.

• Review the retrieved documents. If the verbose log indicates that the Document Retriever was used but it returned different or irrelevant articles, consider the following:

  • Improve your knowledge base: Add new articles or enhance existing ones to include keywords and phrases that are likely to appear in customer questions. This increases the likelihood that the right information will be retrieved.
  • Refine article metadata: Ensure articles have relevant titles, summaries, and tags to increase their discoverability by the Document Retriever component.
  • Create FAQs: For common or recurring questions, create dedicated FAQ entries with the correct answers. This ensures the chatbot can easily retrieve and present the right information in the future.
  • Review system prompt instructions: For situations when the chatbot should behave specifically and trigger email, or transfer to an agent, or answer in a certain specific way, try to update the system prompt of the tool-calling agent component to ensure it has specific case instructions, by adding instructions such as:

  <case_specific_instructions>
  - Issues with YOUR_PRODUCT_NAME:
    - Politely ask the user to provide their account name in the format: account.domain.com.
    - Only after the user provides their account name, immediately use the "LiveAgent Human Assist" tool to transfer the chat to a Human Support Agent.
  </case_specific_instructions>
  

Regular Review of Chatbot Answers

Especially in the first weeks after initial implementation, we advise customers to establish a routine for reviewing chatbot answers to identify any incorrect or suboptimal responses.

Tip: Use Tagging

Tagging directly in the chatbot history helps organize and track cases that you have already checked, as well as those that require further analysis or improvement.

• Tag conversations you’ve checked to avoid reviewing the same conversation twice.
• Tag problematic conversations for quick reference.
• Use tags such as ok, needs review, incorrect, or custom tags relevant to your needs.