docs: Add Generative LLM support for INFERENCE data event#80
Draft
javoweb-fc wants to merge 2 commits into
Draft
docs: Add Generative LLM support for INFERENCE data event#80javoweb-fc wants to merge 2 commits into
javoweb-fc wants to merge 2 commits into
Conversation
9634b06 to
83a6321
Compare
Ref: FLCRM-20635
83a6321 to
403f985
Compare
Ref: FLCRM-20635
Contributor
There was a problem hiding this comment.
Pull request overview
Note
Copilot couldn't run its full agentic review because no GitHub Actions runner was available. Make sure your repository has a runner available to run Copilot's review, or add a copilot-setup-steps.yml file specifying one with the runs-on attribute. See the docs for more details.
Extends the INFERENCE data event documentation to cover on-device Generative LLM inference (in addition to existing Vision ML) including configuration parameters, output shape, and a new example.
Changes:
- Adds “Generative LLM” as a documented execution mode with an LLM-specific
configschema. - Updates model auto-detection docs and examples to include LLM model file extensions and outputs.
- Adds an on-device LLM summarization example and additional device resource guidance.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
50
to
+53
| | File Extension | Detected Model Type | Typical Use Cases | | ||
| | :--- | :--- | :--- | | ||
| | **`.tflite`** | **Vision ML** | Object detection | | ||
| | **`.litertlm`**, **`.task`** | **Generative LLM** | Text generation, text summarization, assistant chats, text classification | |
Comment on lines
+93
to
+96
| * `temperature` number (optional) - Controls randomness in generation. Must be non-negative. | ||
| * `topK` number (optional) - Restricts sampling to the top K most likely tokens. Must be a positive integer. | ||
| * `topP` number (optional) - Restricts sampling to cumulative probability P. Must be non-negative. | ||
| * `maxTokens` number (optional) - Maximum number of tokens to generate. Must be a positive integer. |
| * `box` array - The bounding box coordinates `[x, y, width, height]`. | ||
| * `score` number - The confidence score for the detection. | ||
| * `class` number - The detected class index. | ||
| * **For Generative LLM**: A `result.outputs` object containing `result.outputs.text` (the generated text response) and a `result.modelType` of `'LLM'`. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This PR extends the
INFERENCEdata event documentation (stacked on #79) to add support for on-device Generative LLM inference, alongside the existing Vision ML modes.Key Changes
.gguf,.litertlm, and.taskfile extensions as Generative LLM model types.configschema for Generative LLM (prompt,systemPrompt,temperature,topK,topP,maxTokens,contextSize,stopTokens).result.outputs.text,result.modelType).gemma-4-e2b.litertlm).JIRA / Issue Reference
Ref: FLCRM-20635
Notes
Part 2 of 2 splitting the original #76. Stacked on top of #79 — base branch should be updated to
v2once #79 merges.