Add MultimodalChatWithToolsProvider#400
Conversation
Signed-off-by: Lukas Schaefer <lukas@lschaefer.xyz>
852adf4 to
9254c32
Compare
…on for older providers Signed-off-by: Lukas Schaefer <lukas@lschaefer.xyz>
9254c32 to
60ee11d
Compare
Signed-off-by: Lukas Schaefer <lukas@lschaefer.xyz>
Signed-off-by: Lukas Schaefer <lukas@lschaefer.xyz>
Signed-off-by: Lukas Schaefer <lukas@lschaefer.xyz>
…context agent Signed-off-by: Lukas Schaefer <lukas@lschaefer.xyz>
bb8863b to
1808448
Compare
| $content = []; | ||
| foreach ($message['content'] as $item) { | ||
| if ($item['type'] === 'file') { | ||
| $content = array_merge($content, $this->openAiFileService->buildFileContentFromId($item['file_id'], $userId, $item['ocp_task_id'] ?? null)); |
There was a problem hiding this comment.
Not sure how to handle it for the fileId only path, but if it is from a task the getUserTask accepts a null value for userId.
| throw new ProcessingException('File is not readable'); | ||
| } | ||
| // Maximum file size for openai is 50MB. | ||
| if ($this->isUsingOpenAi() && $file->getSize() > self::MAX_FILE_SIZE_BYTES) { |
There was a problem hiding this comment.
Should we add a file size limit fro the general case as well?
There was a problem hiding this comment.
I can do the exact same limit as for openai unless you think a different one is better.
| $unset = '__unset__'; | ||
| $newValue = $this->appConfig->getValueString(Application::APP_ID, 'multimodal_image_enabled', $unset); | ||
| if ($newValue !== $unset) { | ||
| return; |
There was a problem hiding this comment.
This shifts the default state from 0 to 1, I think. Ideally, if it has been unset previously, we should set it to what used to be the default.
There was a problem hiding this comment.
Back when analyze image was created I don't think IONOS had a model with support for image input, but now that they do I think it might be worth it to change the default to on instead of off.
d6a95cb to
87ec6d9
Compare
Signed-off-by: Lukas Schaefer <lukas@lschaefer.xyz>
87ec6d9 to
1902b7f
Compare
There was a problem hiding this comment.
My tests worked great with images using google/gemini-3.1-flash-lite-image (Nano Banana 2 Lite) on OpenRouter.
I tried using gpt-audio on OpenRouter as well and it could take audio as input but never generated any audio output. The model claimed it had generated some audio but in the task output: "output_attachments":[].
Could this be an issue in this implementation or you think the model is just not outputting any audio content?
Are you sure |
|
@lukasdotcom I meant gpt-audio and audio input/output, sorry, typo. I edited my comment. |
I think the issue is that you need to pass an extra parameter, not like with the image models that don't require anything else, but not sure if we should even duplicate the code form AudioChat for multimodal chat. $extraParams = [
'modalities' => ['text', 'audio'],
'audio' => ['voice' => $outputVoice, 'format' => 'mp3'],
]; |
…tent of unsupported files Signed-off-by: Lukas Schaefer <lukas@lschaefer.xyz>
Signed-off-by: Lukas Schaefer <lukas@lschaefer.xyz>
Signed-off-by: Lukas Schaefer <lukas@lschaefer.xyz>
Signed-off-by: Lukas Schaefer <lukas@lschaefer.xyz>
9ac259e to
3bc687b
Compare
3 participants
Adds multimodal chat and also switches to enable also settings to enable different kinds of attachments. The new multimodal settings are also used for audio2audio chat and analyze image.
Look at nextcloud/assistant#602 to see all relevant pr's for this
🤖 AI (if applicable)