Repository navigation
Flakey retrieval #91
Description
The following user queries lead to v1.2.0 not finding any matching documents during retrieval:
- Tell me about TP53
- Tell me about glycolysis
However some still do appear to work:
- Tell me about BRCA1
This is likely a bug introduced during the big refactor (#77) associated with adding support for profiles, along with the uniprot prototype (#78).
Certainly something in this diff: 1.1.2...1.2.0
There's probably a misnamed parameter that's not getting passed during retrieval.
Update
This issue is more mysterious than it initially seems.
The description above corresponded to running 1.2.0 vs 1.1.2 on my local machine, as well as observations on prod.
However I wasn't able to reproduce the issue on release.reactome.org...
Going to look into potential embedding bundle inconsistencies.
Activity
[-]Retrieval not finding certain topics in 1.2.0[/-][+]Flakey retrieval[/+]GFJHogue commentedon Sep 19, 2025
This is bizarre - it seems to be working in all cases now.
What I've observed in the past hour along with the accounts of team members, corroborates that this issue is not connected to our 1.2.0 release nor any embeddings-bundle corruptions/inconsistencies.
GFJHogue commentedon Sep 19, 2025
Okay, one more comment on this.
There is a likelihood that this issue is related to poor query rephrasing during the preprocessing step.
Specifically, the "Tell me about glycolysis" question is sensitive to context and seems to usually fail when it's the 1st message of a conversation, but will succeed if phrased slightly differently like "Tell me about the glycolysis pathway".
FYI @heliamoh
GFJHogue commentedon Sep 22, 2025
GFJHogue commentedon Sep 22, 2025
Using actual the tagged Docker image for 1.1.2 (
public.ecr.aws/reactome/reactome-chatbot:2044791) gives the following:(refusal, but at least it's giving some information)
I think this may be a key hint.
GFJHogue commentedon Sep 30, 2025
Copying from #94 (fixes):
CodeSharingPartially commentedon Nov 14, 2025
Hello! First of all thanks for the great work to improve and keep this tool alive. Really useful.
Second of all, I am testing the chatbot and , starting with very basic questions , I got semi-right answers?
My question was:
1- how could i retrieve with a cypher query, as a json, all the pathways, subpathways in molecules for the pathway "Immune system"?
Before raising this question, I already had in mind a specific event in the Event Hierarchy - Immune system https://reactome.org/PathwayBrowser/#/R-HSA-168256
The result was straightforawrd, and it went to the closest term with pathway instead of asking me if I refere to the "event". I think this is a common behavior with no agentic workflows. Ideally you would like to clarify the question I think, so the bot first will ask if i want "the pathway or the event"? otherwise it is assuming I want the pathway immune system within the disease event instead of the immune system event. anyway i left the snapshots here, (NOTE: It feels also that it though i was talking about pathways all the time and it did not gave me any description for the Event R-HSA-168256 )
NOTE2: In the last Reactome Hackathon, we used Google ADK Toolkit to develop a chatbot with agentic workflow: opentargets-archive/hackathon-tasks#107 and I think it may improve the chatbot workflow.
GFJHogue commentedon Nov 21, 2025
Hi @CodeSharingPartially,
Thank you for the valuable feedback and use-case!
The agentic design of the current system certainly has room for improvement, especially where Human-In-The-Loop interactions would be desirable.
In the semi-near-term we have plans to deploy an improved retrieval system that should reduce the chat's stubbornness of focusing on pathways.
CodeSharingPartially commentedon Nov 28, 2025
One last question, is Reactome running in AuraDB? or is it in house hosted (community version?)