AI eval failed

[akanshaaa19]

Summary

GitHub Issue Summary

Problem Statement: User's AI evaluation fails immediately without clear explanation.

Expected Behavior: User expected to successfully evaluate their AI assistant.

Actual Result: The evaluation process failed immediately with no further details provided.

Error Messages: None specified.

Reproduction Steps:

1. Attempt to evaluate an AI assistant.
2. Use a Golden QA CSV file without specified formats or prerequisites.
3. Evaluation instantly fails.

Additional Context:

• The user was prompted to ensure their Golden QA CSV had the correct format (two columns: question and answer), no blank rows, and a duplication factor between 1-5.
• It's also advised to check if at least one AI Assistant is configured and to select the correct assistant/version during the evaluation setup.
• A suggestion was made to try a small test run with a unique evaluation name and a smaller dataset (2–3 rows) to isolate the issue.

Further details regarding the exact AI assistant and the dataset used were not provided by the user.

Source

• Discord thread: https://discord.com/channels/717975833226248303/1511329315319451798
• Reported by: sai_ganesh_0004
• Title: AI eval failed

Original message

<@1132952719586492478> <@937979788587597824> I tried to evalute my ai assistant and instantly got failed

Full thread transcript

sai_ganesh_0004: <@1132952719586492478> <@937979788587597824> I tried to evalute my ai assistant and instantly got failed

Glific AI doc bot (bot): Hey! Sorry that happened—let’s get this fixed. Could you share the evaluation name and the AI assistant (and version) you selected?

Meanwhile, a few quick checks that commonly cause an immediate “Failed”:

• Golden QA CSV format: ensure the file has just two columns named question and answer, with no blank rows, and that the duplication factor you chose is between 1–5 .
• Prerequisites: confirm you have at least one AI Assistant configured and that you selected the correct assistant/version on the Create AI Evaluation screen .
• Try a small test run: give the evaluation a unique name and run with a tiny Golden QA (2–3 rows) to isolate if the issue is with the dataset or assistant selection .

Helpful references:

• AI Evaluations in Glific
• About Golden QnA

Glific AI doc bot (bot): Was this helpful in resolving your query? Or do you need further support?

tejasmahajan: <@1500359155893014669> can you pls look into this?

iamloak: Checking, will update here by EoD tomorrow

priyanshusingh3026: <@1290703630386593822> could you please try it now and let me know if the issue has been fixed?

[priyanshu6238]

We were able to identify the root cause of both issues:

1- Langfuse timeout issue – This is happening because we are directly fetching the response from Langfuse, and the request is timing out.
2- get_trace_info=true evaluation timeout issue – The root cause is the trace fetch rate limit. The current limit is 100 traces per request. SageX has around 40 questions with a duplication factor of 3, resulting in nearly 120 traces being fetched. This exceeds the free-tier rate limit and causes the request to time out.