[Feature]: Adding configurable llmservice endpoint which assumes backend stores the system prompt - #392
[Feature]: Adding configurable llmservice endpoint which assumes backend stores the system prompt#392JMN09 wants to merge 4 commits into
Conversation
… of AiEngine that supports a configurable backend, added the corresponding configuration variables, and integrated it in the factory method
…ture/llmservice
|
thanks for the contribution, i first need to merge refactoring_v1 PR and alter this PR to match the refactoring (if needed, hope not) what are the model files? 🤔 |
|
@JMN09 please refer to this comment #391 (comment) |
|
Hey. A model file is used to create an instance of LLMs with the system prompt wired into them. They are used to gain more context size in prompts without losing accuracy due to large prompts. Especially useful in smaller models such as llama3.1 8b |
|
please resolve conflicts |
|
@JMN09 all tests pass, you only need to resolve the conflicts, please lmk when i can merge it |
There was a problem hiding this comment.
A specialized backend that stores the system prompt may be useful, but this implementation conflicts with master and introduces separate endpoint/path settings alongside the existing OCO_API_URL without protocol tests. Please submit a new minimal design on top of master with a documented request/response contract and mock-server tests.
Relying on model files decreases the context size in prompts allowing LLMs to perform better. By adding this endpoint we allow communication with APIs that use model files.
I added a class that implements AiEngine to do so.
In addition to that added the possibility to confire OCO_AI_PROVIDER to llmservice
And added to configuration variable OCO_BACKEND_ENDPOINT and OCO_BACKEND_PATH
closes #386