Add a pool that is the "smart" routing pool. Effectively a pool that classifies prompts and sends them to the optimal endpoint. Prompts could be classified via any number of categories.
For example, say that a prompt comes in that is best answered by a specialized, legal endpoint. The smart pool classifies and routes it to the legal endpoint.
Likely involves traditional ML.
Add a pool that is the "smart" routing pool. Effectively a pool that classifies prompts and sends them to the optimal endpoint. Prompts could be classified via any number of categories.
For example, say that a prompt comes in that is best answered by a specialized, legal endpoint. The smart pool classifies and routes it to the legal endpoint.
Likely involves traditional ML.