Skip to content

feat: revise Models documentation and add Model Artifact management s… #53

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
iasthc merged 1 commit into from
Dec 28, 2025
Merged

feat: revise Models documentation and add Model Artifact management s… #53

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
91 changes: 63 additions & 28 deletions src/content/docs/service/models.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,49 +1,84 @@
---
title: Models
description: Manage and monitor Large Language Models.
description: Manage and monitor AI/ML models and artifacts.
---

import { Steps, Aside } from '@astrojs/starlight/components';

The Models section allows you to manage and monitor Large Language Models (LLMs) deployed in the OtterScale cluster. It provides real-time insights into model performance and resource usage.

The Models page provides a unified view of all AI/ML models deployed in your OtterScale cluster. It aggregates information about model deployments and their artifacts, allowing you to monitor status, manage resources, and perform lifecycle operations.


## Introduction

The Models page displays a list of deployed LLMs. The table includes the following information:
The Models page displays a list of all models. The table includes the following columns (as shown in the UI):

| Column | Description |
| :---------------------- | :---------------------------------------------------------------------------------------- |
| **Model** | The name of the application hosting the model. Links to the application details. |
| **Name** | The specific name or identifier of the LLM (e.g., `llama3-8b`). |
| **Replicas** | The number of desired replicas for the model service. |
| **Healthies** | The number of currently healthy and ready replicas. |
| **GPU Cache** | The percentage of GPU cache currently in use. |
| **KV Cache** | The percentage of Key-Value (KV) cache usage, critical for transformer model performance. |
| **Requests** | The total request latency or load metric for the model. |
| **Time to First Token** | The average time taken to generate the first token of a response (latency metric). |
| Column | Description |
| :----------------- | :------------------------------------------------------------------------------------------------------------------- |
| **Name** | The name of the model. |
| **Model Name** | The unique identifier of the model (modelName/id). |
Comment on lines +18 to +19

@gemini-code-assist gemini-code-assist Bot Dec 28, 2025

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The distinction between Name and Model Name is unclear. Name is described as "The name of the model," while Model Name is "The unique identifier of the model (modelName/id)." This could be confusing for users. Please clarify the difference. For example, is Name a user-friendly display name, while Model Name is the technical identifier used by the system? Providing a more distinct description and an example would be helpful.

| **Name**           | A user-friendly display name for the model.                                                                  |
| **Model Name**     | The unique technical identifier of the model (e.g., `llama3-8b-instruct`).                                   |

| **Namespace** | The Kubernetes namespace where the model is deployed. |
| **Status** | The current status of the model (e.g., Running, Pending). |
| **Description** | The description of the model. |
| **Prefill** | Prefill configuration: vGPU memory %, replica, tensor (if available). |
| **Decode** | Decode configuration: vGPU memory %, replica, tensor (if available). |
| **First Deployed** | Timestamp of first deployment. |
| **Last Deployed** | Timestamp of last deployment. |
| **GPU Relation** | GPU resource relation (shown only if status is 'deployed'). |
| **Test** | Test button for model API (only available when the model is in the "ready" state; opens a dialog to test the model). |
| **Actions** | Management actions (update, delete, etc.). |

## Monitor Models

The dashboard integrates with Prometheus to provide real-time metrics for each model:

- **GPU & KV Cache**: Monitor these percentages to ensure your models are not running out of memory context, which could degrade performance or cause errors.
- **Latency Metrics**: Track "Requests" and "Time to First Token" to ensure the models are responsive and meeting service level objectives (SLOs).

Comment on lines 31 to 33

@gemini-code-assist gemini-code-assist Bot Dec 28, 2025

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

There are several formatting inconsistencies in the file:

  • Multiple consecutive blank lines are used here and on lines 52-53. A single blank line is sufficient for separating sections and improves source readability.
  • Line 37 contains trailing whitespace (...if available. ). This should be removed if a hard line break is not intended.

Please clean up these minor formatting issues to improve the document's consistency.

## Manage Models
### Pods Table

You can perform basic lifecycle actions on the models:

### Create a Model
The **Pods Table** (in the details view) lists all pods managed by this model, if available.
Click the expand icon at the beginning of a row to view detailed pod information.

To deploy a new model:
| Column | Description |
| :---------------------- | :--------------------------------------------------------------------------------------- |
| **Pod** | The unique name of the pod. |
| **Phase** | The current lifecycle phase of the pod (e.g., Running, Pending). |
| **Ready** | Number of ready containers vs total containers. |
| **Restarts** | Number of times containers in the pod have restarted. |
| **Conditions** | The most recent condition or error status for the pod. |
| **Time to First Token** | The sum of time (in seconds) taken to generate the first token for requests to this pod. |

@gemini-code-assist gemini-code-assist Bot Dec 28, 2025

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The description for Time to First Token states it is the "sum of time". This is an unusual metric for latency, as a sum would grow indefinitely and not provide a clear snapshot of current performance. Typically, latency is measured as an average, median, or percentile (e.g., p95). Please verify if "sum" is correct. If so, consider adding an explanation. If it's an average or another statistical measure, please update the description accordingly.

| **Time to First Token** | The average time (in seconds) taken to generate the first token for requests to this pod. |

| **Request Latency** | The 95th percentile end-to-end request latency (in seconds) for this pod. |
| **Log** | **Clickable**: Opens the log view for the pod. |
| **Create Time** | The timestamp when the pod was created. |

1. Click the **Create** button (plus icon) at the top of the page.
2. Follow the prompts to configure and deploy your LLM service.

### Delete a Model

To remove a model:
## Manage Models

1. Locate the model in the list.
2. Click the **Delete** button (trash icon) in the actions menu.
3. Confirm the action to remove the model deployment.
You can manage the lifecycle of your models using the **Actions** menu.

### Model Actions

The **Actions** menu (three dots icon) for each model provides:

#### Create
Create a new model.
<Steps>
1. Select **Create** from the actions menu or click the **Create** button.
2. You can search for models using the cloud icon next to the input box, or select a model from your model artifacts by clicking the archive icon.
3. Fill in the model configuration (name, namespace, prefill/decode, description, etc.).
4. Confirm to deploy the model.
</Steps>

#### Update
Modify the configuration of an existing model (such as prefill/decode, description, etc.).
<Steps>
1. Select **Update** from the actions menu.
2. Edit the desired fields.
3. Confirm to apply the changes.
</Steps>

#### Delete
Delete a model.
<Steps>
1. Select **Delete** from the actions menu.
2. Confirm deletion.
</Steps>
60 changes: 60 additions & 0 deletions src/content/docs/service/settings/05-model-artifact.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
---
title: Model Artifact
description: Manage model files and weights for AI/ML models.
---

import { Steps, Aside } from '@astrojs/starlight/components';



@gemini-code-assist gemini-code-assist Bot Dec 28, 2025

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

There are extra blank lines here and on line 29. To maintain source code consistency and readability, it's best to use only a single blank line between paragraphs or components.

The Model Artifact page allows you to manage model artifacts for your AI/ML models. You can upload, view, and delete model artifacts associated with different models and namespaces.


## Introduction

The Model Artifact page displays a list of all model artifacts. The table includes the following columns:

| Column | Description |
| :-------------- | :----------------------------------------------------------------- |
| **Name** | The name of the model artifact. |
| **Namespace** | The Kubernetes namespace where the artifact is stored. |
| **Model Name** | The name of the associated model. |
| **Status** | The current job status for downloading or processing the artifact. |
| **Phase** | The current phase/status of the artifact. |
Comment on lines +22 to +23

@gemini-code-assist gemini-code-assist Bot Dec 28, 2025

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The descriptions for Status and Phase are very similar and could be confusing. Status is "The current job status for downloading or processing the artifact," and Phase is "The current phase/status of the artifact." Please clarify the distinction between these two columns. For example, does Status refer to a background job (like Pending, Running, Succeeded) while Phase refers to the artifact's lifecycle state (like Downloading, Ready, Error)? Providing examples for each would be very helpful.

| **Status**      | The status of the background job processing the artifact (e.g., `Pending`, `Running`, `Succeeded`). |
| **Phase**       | The lifecycle phase of the artifact itself (e.g., `Downloading`, `Ready`, `Error`).                |

| **Size** | The size of the artifact (GB/TB). |
| **Volume** | The volume name where the artifact is stored. |
| **Create Time** | The timestamp when the artifact was created. |
| **Actions** | Management actions (delete). |


## Manage Model Artifacts

You can create new model artifacts or manage existing ones using the **Actions** menu.

### Create a New Model Artifact

To upload a new model artifact:

<Steps>
1. Click the **Create** button (plus icon) at the top of the page.
2. A modal window titled "Create Model Artifact" will appear.
3. **Configuration**:
- **Name**: Enter a unique name for the artifact.
- **Namespace**: Select the namespace (usually `llm-d`).
- **Model Name**: Select the model name (supports Hugging Face models).
- **Size**: Set the artifact size (GB/TB).
4. Click **Confirm** to create the model artifact.
</Steps>


### Model Artifact Actions

The **Actions** menu (trash icon) for each model artifact provides the following option:

#### Delete

Permanently removes the model artifact.

<Aside type="danger" title="Warning">
Deleting a model artifact will remove the file from storage. This action cannot be undone.
</Aside>
Loading