Skip to content

docs: Add Generative LLM support for INFERENCE data event#80

Draft
javoweb-fc wants to merge 2 commits into
task/FLCRM-20635-onnx-litert-deprecationfrom
task/FLCRM-20635-llm-inference-support
Draft

docs: Add Generative LLM support for INFERENCE data event#80
javoweb-fc wants to merge 2 commits into
task/FLCRM-20635-onnx-litert-deprecationfrom
task/FLCRM-20635-llm-inference-support

Conversation

@javoweb-fc

@javoweb-fc javoweb-fc commented Jul 9, 2026

Copy link
Copy Markdown

Description

This PR extends the INFERENCE data event documentation (stacked on #79) to add support for on-device Generative LLM inference, alongside the existing Vision ML modes.

Key Changes

  1. Execution Modes: Adds Generative LLM as a supported execution mode, alongside Vision ML and Legacy Vision ML (ONNX).
  2. Model Detection: Adds .gguf, .litertlm, and .task file extensions as Generative LLM model types.
  3. Parameters: Documents the new config schema for Generative LLM (prompt, systemPrompt, temperature, topK, topP, maxTokens, contextSize, stopTokens).
  4. Callback Signature: Documents the LLM output shape (result.outputs.text, result.modelType).
  5. Device Warnings: Adds LLM-specific battery/memory guidance.
  6. New Example: Adds an on-device LLM text summarization example (gemma-4-e2b.litertlm).

JIRA / Issue Reference

Ref: FLCRM-20635

Notes

Part 2 of 2 splitting the original #76. Stacked on top of #79 — base branch should be updated to v2 once #79 merges.

@javoweb-fc javoweb-fc force-pushed the task/FLCRM-20635-llm-inference-support branch from 9634b06 to 83a6321 Compare July 9, 2026 21:23
@javoweb-fc javoweb-fc force-pushed the task/FLCRM-20635-llm-inference-support branch from 83a6321 to 403f985 Compare July 9, 2026 22:15
@javoweb-fc javoweb-fc requested a review from Copilot July 9, 2026 22:47

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Note

Copilot couldn't run its full agentic review because no GitHub Actions runner was available. Make sure your repository has a runner available to run Copilot's review, or add a copilot-setup-steps.yml file specifying one with the runs-on attribute. See the docs for more details.

Extends the INFERENCE data event documentation to cover on-device Generative LLM inference (in addition to existing Vision ML) including configuration parameters, output shape, and a new example.

Changes:

  • Adds “Generative LLM” as a documented execution mode with an LLM-specific config schema.
  • Updates model auto-detection docs and examples to include LLM model file extensions and outputs.
  • Adds an on-device LLM summarization example and additional device resource guidance.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines 50 to +53
| File Extension | Detected Model Type | Typical Use Cases |
| :--- | :--- | :--- |
| **`.tflite`** | **Vision ML** | Object detection |
| **`.litertlm`**, **`.task`** | **Generative LLM** | Text generation, text summarization, assistant chats, text classification |
Comment on lines +93 to +96
* `temperature` number (optional) - Controls randomness in generation. Must be non-negative.
* `topK` number (optional) - Restricts sampling to the top K most likely tokens. Must be a positive integer.
* `topP` number (optional) - Restricts sampling to cumulative probability P. Must be non-negative.
* `maxTokens` number (optional) - Maximum number of tokens to generate. Must be a positive integer.
* `box` array - The bounding box coordinates `[x, y, width, height]`.
* `score` number - The confidence score for the detection.
* `class` number - The detected class index.
* **For Generative LLM**: A `result.outputs` object containing `result.outputs.text` (the generated text response) and a `result.modelType` of `'LLM'`.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants