diff --git a/src/content/docs/demos/07-aidaptiv-operator.mdx b/src/content/docs/demos/07-aidaptiv-operator.mdx new file mode 100644 index 0000000..ad4ec3d --- /dev/null +++ b/src/content/docs/demos/07-aidaptiv-operator.mdx @@ -0,0 +1,96 @@ +--- +title: aiDAPTIV Operator +description: Learn how to install the aiDAPTIV Cache Operator for Phison aiDAPTIVCache device support in Kubernetes +--- + +import { Steps, Aside } from '@astrojs/starlight/components'; + +This guide demonstrates how to install the **aiDAPTIV Cache Operator**, which is required for Kubernetes to recognize and utilize Phison aiDAPTIVCache devices (`phison.com/ai100`). + +## Prerequisites + +Before installing the operator, ensure you have: + +- Access to OtterScale platform with Application management permissions +- A Kubernetes cluster with Phison aiDAPTIVCache hardware installed +- Sufficient cluster resources for the operator deployment + + + +## Installation Steps + + + +1. **Navigate to Application Store** + + In the OtterScale web interface: + - Go to **Application** → **Store** + - This opens the Helm chart repository + +2. **Add the aiDAPTIV Operator Chart** + + Click **Import** button and provide the following information: + + - **Chart URL**: + ``` + https://github.com/otterscale/charts/releases/download/aidaptivcache-operator-0.4.3/aidaptivcache-operator-0.4.3.tgz + ``` + + Click **Confirm** to import the chart into your store. + +3. **Install the Operator** + + - Find `aidaptivcache-inference` in the Store list + - Click the **Install** button + - In the Install Release dialog: + - Enter a **Name** for your deployment (e.g., `aidaptivcache-operator`) + - Enter a **Namespace** (e.g., `aidaptivcache-operator`) + - Click **View/Edit** button to open the configuration editor + + + + + +## What the Operator Does + +The aiDAPTIV Cache Operator performs the following functions: + +- **Resource Manager**: Manage aiDAPTIVCache devices +- **Device Discovery**: Automatically detects Phison aiDAPTIVCache devices on cluster nodes +- **Device Plugin**: Registers aiDAPTIVCache devices as Kubernetes resources, enabling targeted workload scheduling +- **Exporter**: Exposes metrics and cache information via an NVMe exporter + + + +## Troubleshooting + +### Operator Pod Not Starting + +If the operator pod fails to start: + +1. **Check pod logs via OtterScale UI**: + - Navigate to **Application** → **Workloads** + - Find the `aidaptivcache-operator` workload + - Click on the pod name to view details + - Click the **Logs** tab to view container logs + +2. Verify node labels and taints allow the operator to schedule + +3. Check that the cluster has sufficient resources + +## Next Steps + +After successfully installing the aiDAPTIV Cache Operator, you can: + +- Deploy fine-tuning jobs that utilize aiDAPTIVCache devices (see [Deploy aiDAPTIV Finetune](./08-aidaptiv-finetune)) +- Deploy inference services with aiDAPTIVCache acceleration (see [Deploy aiDAPTIV Inference](./09-aidaptiv-inference)) + + diff --git a/src/content/docs/demos/08-aidaptiv-finetune.mdx b/src/content/docs/demos/08-aidaptiv-finetune.mdx new file mode 100644 index 0000000..005340f --- /dev/null +++ b/src/content/docs/demos/08-aidaptiv-finetune.mdx @@ -0,0 +1,543 @@ +--- +title: aiDAPTIV Finetune +description: Fine-tune LLM models using aiDAPTIV Cache in OtterScale cluster. +--- + +import { Steps, Aside, LinkCard } from '@astrojs/starlight/components'; + +This guide demonstrates how to fine-tune Large Language Models (LLMs) using aiDAPTIV Cache within the OtterScale cluster. + +## Overview + +aiDAPTIV Finetune provides efficient model fine-tuning capabilities with support for LoRA, full parameter training, and other training modes. Key features: + +- Distributed training support (Multi-GPU) +- NFS storage integration for model input/output +- LoRA fine-tuning support +- Customizable training datasets +- Flexible resource allocation (vGPU, memory) + +## Prerequisites + + + +## Prepare Model and Data + +You need to make your model files accessible to the fine-tuning job. There are two methods to achieve this, both configured via the `prescript` field in the Helm chart values. + +### Method 1: Using NFS Storage (Recommended) + +This method uses OtterScale's NFS File System to store and access model files. + + + +1. **Create NFS File System in OtterScale**: + + Navigate to the Storage section: + - Go to `https:///scope//service/file-system` + - Create a new NFS File System or use an existing one + - Record the NFS server address (format: `10.102.197.0:/volumes/_nogroup/xxx`) + +2. **Upload your model files to NFS**: + + Mount the NFS on a machine that has access and copy your model files: + + ```bash + # Mount NFS on a node with access + mkdir -p /mnt/nfs + mount -t nfs4 10.102.197.0:/volumes/_nogroup/xxx /mnt/nfs + + # Copy your model files to NFS + cp -r /path/to/your-model /mnt/nfs/models/ + + # Verify files + ls /mnt/nfs/models/ + ``` + +3. **Configure prescript to mount NFS**: + + In the Helm chart values, you'll configure the `prescript` to mount this NFS (see NFS Mount Configuration section below). + + + + + +### Method 2: Using SCP to Copy Models + +This method copies model files directly into the pod using SCP during initialization. + + + +1. **Prepare a remote server with model files**: + + Ensure you have a remote server (accessible from the cluster) that contains your model files. + +2. **Configure prescript with SCP**: + + Use the following prescript template in your Helm chart values: + + ```yaml + prescript: | + # Install required tools + apt-get update && apt-get install -y sshpass + + # Create model directory + mkdir -p /mnt/data/models + + # Copy model from remote server using SCP + echo "Copying model from remote server..." + sshpass -p 'your-password' scp -o StrictHostKeyChecking=no -r \ + user@remote-host:/path/to/your-model /mnt/data/models/ + + if [ $? -eq 0 ]; then + echo "Model copied successfully!" + ls -lh /mnt/data/models/ + else + echo "Failed to copy model. Exiting..." + exit 1 + fi + ``` + + + You still need an NFS mount for storing training outputs. Add NFS mount commands after the SCP section: + + ```yaml + postscript: | + # Install tools + apt-get update && apt-get install -y sshpass nfs-common + + # Copy model via SCP + mkdir -p /mnt/data/models + echo "Copying model from remote server..." + sshpass -p 'your-password' scp -o StrictHostKeyChecking=no -r \ + user@remote-host:/path/to/model /mnt/data/models/ + + # Mount NFS for output storage + mkdir -p /mnt/data/output + echo "Mounting NFS for output storage..." + mount -t nfs4 -o nfsvers=4.1 10.102.197.0:/volumes/_nogroup/output-path /mnt/data/output + + echo "Setup complete!" + ``` + + + +## Install Helm Chart + + + +1. **Navigate to Application Store**: + + In the OtterScale web interface: + - Go to **Application** → **Store** + - This opens the Helm chart repository + +2. **Import Helm Chart**: + + - Click the **Import** button at the top of the page + - Enter the Helm Chart URL in the dialog: + ``` + https://github.com/otterscale/charts/releases/download/aidaptivcache-finetune-0.1.3/aidaptivcache-finetune-0.1.3.tgz + ``` + - Click **Confirm** to import + +3. **Install Chart**: + + - Find `aidaptivcache-finetune` in the Store list + - Click the **Install** button + - In the Install Release dialog: + - Enter a **Name** for your deployment (e.g., `ft1`) + - Enter a **Namespace** (e.g., `ft1`) + - Click **View/Edit** button to open the configuration editor + - Edit the `values.yaml` to configure your fine-tuning job (see Configuration Guide below) + - Click **Confirm** to start the installation + + + + + +## Configuration Guide + +Now that you've opened the configuration editor via **View/Edit**, you need to configure the following fields in the `values.yaml`. + +### Basic Configuration + +#### Image Settings + +```yaml +image: + repository: docker.io/library/aidaptiv + tag: vNXUN_2_05BA0 + pullPolicy: IfNotPresent +``` + +- `repository`: Container image address +- `tag`: Image version tag +- `pullPolicy`: Image pull policy (IfNotPresent/Always/Never) + +#### Job Configuration + +```yaml +job: + name: finetune-job + backoffLimit: 1 + restartPolicy: Never + ttlSecondsAfterFinished: 3600 +``` + +- `name`: Kubernetes Job name +- `backoffLimit`: Number of retry attempts on failure +- `restartPolicy`: Restart policy (Never/OnFailure) +- `ttlSecondsAfterFinished`: Time to retain Job after completion (seconds) + +### Training Configuration + +The training configuration is divided into three main sections: **expConfig**, **envConfig**, and **trainDataConfig**. + +#### 1. Experiment Configuration (expConfig) + +The `expConfig` section controls GPU resources, distributed training settings, and training hyperparameters. + +**Process Settings**: + +```yaml +expConfig: + processSettings: + numGpus: 1 # Number of GPUs to use + specifyGpus: null # Specific GPU IDs (e.g., "0,1,2,3") + masterPort: 8299 # Master port for distributed training + multiNodeSettings: + enable: false # Enable multi-node training + masterAddr: "127.0.0.1" # Master node address +``` + +**Run Settings**: + +```yaml +expConfig: + runSettings: + taskType: "text-generation" # Task type + taskMode: "train" # Mode: train/eval/inference + perDeviceTrainBatchSize: 4 # Batch size per device + perUpdateTotalBatchSize: 16 # Total batch size (gradient accumulation) + numTrainEpochs: 1 # Number of training epochs + maxIter: 12 # Maximum iterations + maxSeqLen: 2048 # Maximum sequence length + triton: true # Enable Triton optimization + precisionMode: 1 # Precision mode (0: FP32, 1: Mixed) +``` + +**LoRA Settings**: + +```yaml +expConfig: + runSettings: + lora: + enableLora: false # Enable LoRA fine-tuning + loraRank: 8 # LoRA rank + loraAlpha: 16 # LoRA alpha parameter + loraTaskType: "CAUSAL_LM" # Task type for LoRA + loraTargetModules: null # Target modules (null for auto) +``` + +**Learning Rate and Optimizer**: + +```yaml +expConfig: + runSettings: + lrScheduler: + mode: 1 # LR scheduler mode + learningRate: 0.000007 # Learning rate + + optimizer: + beta1: 0.9 # Adam beta1 + beta2: 0.95 # Adam beta2 + eps: 0.00000001 # Epsilon + weightDecay: 0.01 # Weight decay +``` + +#### 2. Environment Configuration (envConfig) + +The `envConfig` section defines all file paths used during training. + +```yaml +envConfig: + pathSettings: + modelNameOrPath: "/mnt/data/models/TinyLlama-1.1B-Chat-v1.0" # Model input path + nvmePath: "/mnt/nvme0" # NVMe cache path + outputDir: "/mnt/data/output" # Training output path + trainDataPath: # Training data config + - /config/train_data/QA_dataset_config.yaml + logName: "output.log" # Log file name +``` + +**Key Path Explanations**: + +- **`modelNameOrPath`** (Required): + - Path to the pre-trained model + - Must point to where NFS mounts the model via prescript + - Example: `/mnt/data/models/TinyLlama-1.1B-Chat-v1.0` + +- **`outputDir`** (Required): + - Where fine-tuned model weights will be saved + - Must be on NFS mount for persistence: `/mnt/data/output` + +- **`nvmePath`** (Required): + - NVMe device path for temporary storage and cache + - Typically uses node's NVMe: `/mnt/nvme0` + +- **`trainDataPath`**: + - Path to training data configuration file(s) + - Supports multiple datasets (array format) + + + +#### 3. Training Data Configuration (trainDataConfig) + +The `trainDataConfig` section defines the dataset format and prompts. + +```yaml +trainDataConfig: | + instruction-dataset: + data_path: "HuggingFaceH4/instruction-dataset" # HuggingFace dataset or local path + strategy: "QA" # Data strategy (QA/Chat) + system_prompt: "A chat between a curious user and an artificial intelligence assistant." + user_prompt: "{question}" # User prompt template + question_key: "prompt" # Column name for questions + answer_key: "completion" # Column name for answers + exp_type: train # Experiment type: train/eval/inference + label_key: "completion" # Label column (same as answer_key) +``` + +**Configuration Fields**: + +- **`data_path`**: HuggingFace dataset name or local file path +- **`strategy`**: Data processing strategy (QA/Chat/Custom) +- **`system_prompt`**: System instruction for the model +- **`user_prompt`**: Template for user questions (use `{question}` placeholder) +- **`question_key`**: Dataset column containing questions +- **`answer_key`**: Dataset column containing answers +- **`exp_type`**: Must be `train` for training +- **`label_key`**: Column used as training labels (typically same as `answer_key`) + + + +### Resource Configuration + +```yaml +resources: + limits: + otterscale.com/vgpu: 1 + otterscale.com/vgpumem-percentage: 60 + phison.com/ai100: 1 + requests: + otterscale.com/vgpu: 1 + otterscale.com/vgpumem-percentage: 60 + phison.com/ai100: 1 +``` + +- `otterscale.com/vgpu`: vGPU count +- `otterscale.com/vgpumem-percentage`: vGPU memory percentage (0-100) +- `phison.com/ai100`: Phison aiDAPTIVCache accelerator count + +### NFS Mount Configuration (Required) + +To access your model files and save training outputs, you need to configure the `prescript` to mount your NFS storage. + +```yaml +prescript: | + apt install -y nfs-common + echo "Starting NFS mount process..." + mkdir -p /mnt/data + TIMEOUT=300 + ELAPSED=0 + while [ $ELAPSED -lt $TIMEOUT ]; do + echo "Attempting to mount NFS to /mnt/data" + mount -t nfs4 -o nfsvers=4.1 -v 10.102.197.0:/volumes/_nogroup/your-nfs-path /mnt/data + if mountpoint -q /mnt/data; then + echo "NFS mount successful!" + break + else + echo "Mount failed, retrying in 5 seconds... (${ELAPSED}s/${TIMEOUT}s)" + sleep 5 + ELAPSED=$((ELAPSED + 5)) + fi + done + if [ $ELAPSED -ge $TIMEOUT ]; then + echo "NFS mount timeout after ${TIMEOUT} seconds. Exiting..." + exit 1 + fi +``` + +**Configuration Details**: +- Replace `10.102.197.0:/volumes/_nogroup/your-nfs-path` with your actual NFS server address from the File System page +- The script installs `nfs-common` package (required for NFS mounting) +- Implements retry logic with a 5-minute timeout +- Mounts NFS to `/mnt/data` which is used in the path configuration + +**Optional Post-execution Script**: + +```yaml +postscript: | + echo "Training job completed" + echo "Model saved to: $outputDir" +``` + + + +## Complete Configuration Example + +```yaml +image: + repository: docker.io/library/aidaptiv + tag: vNXUN_2_05BA0 + pullPolicy: IfNotPresent + +job: + name: finetune-llama-job + backoffLimit: 1 + restartPolicy: Never + ttlSecondsAfterFinished: 3600 + +securityContext: + privileged: true + +# NFS Mount Script (Required) +prescript: | + apt install -y nfs-common + echo "Starting NFS mount process..." + mkdir -p /mnt/data + TIMEOUT=300 + ELAPSED=0 + while [ $ELAPSED -lt $TIMEOUT ]; do + echo "Attempting to mount NFS to /mnt/data" + mount -t nfs4 -o nfsvers=4.1 -v 10.102.197.0:/volumes/_nogroup/my-nfs-path /mnt/data + if mountpoint -q /mnt/data; then + echo "NFS mount successful!" + break + else + echo "Mount failed, retrying in 5 seconds... (${ELAPSED}s/${TIMEOUT}s)" + sleep 5 + ELAPSED=$((ELAPSED + 5)) + fi + done + if [ $ELAPSED -ge $TIMEOUT ]; then + echo "NFS mount timeout after ${TIMEOUT} seconds. Exiting..." + exit 1 + fi + +envConfig: + pathSettings: + modelNameOrPath: "/mnt/data/models/TinyLlama-1.1B-Chat-v1.0" + nvmePath: "/mnt/nvme0" + outputDir: "/mnt/data/output" + trainDataPath: + - /config/train_data/QA_dataset_config.yaml + logName: "finetune_output.log" + +expConfig: + processSettings: + numGpus: 1 + masterPort: 8299 + multiNodeSettings: + enable: false + + runSettings: + taskType: "text-generation" + taskMode: "train" + perDeviceTrainBatchSize: 4 + perUpdateTotalBatchSize: 16 + numTrainEpochs: 1 + maxIter: 100 + maxSeqLen: 2048 + triton: true + + lrScheduler: + mode: 1 + learningRate: 0.000007 + + lora: + enableLora: true + loraRank: 8 + loraAlpha: 16 + loraTaskType: "CAUSAL_LM" + +trainDataConfig: | + instruction-dataset: + data_path: "HuggingFaceH4/instruction-dataset" + strategy: "QA" + system_prompt: "A chat between a curious user and an artificial intelligence assistant." + user_prompt: "{question}" + question_key: "prompt" + answer_key: "completion" + exp_type: train + label_key: "completion" + +resources: + limits: + otterscale.com/vgpu: 1 + otterscale.com/vgpumem-percentage: 60 + phison.com/ai100: 1 + requests: + otterscale.com/vgpu: 1 + otterscale.com/vgpumem-percentage: 60 + phison.com/ai100: 1 +``` + +## Monitor Training Progress + + + +1. **Check Job Status**: + + Navigate to the Jobs page: + ``` + https:///scope//service/job + ``` + + Find your fine-tune job under **Application → Job** and check its status. + +2. **Retrieve Output Model**: + + After training completes, the fine-tuned model will be saved in the path specified by `outputDir`: + + ```bash + # Access via the same NFS mount used during training + # Mount the NFS on your local machine or a node + mount -t nfs4 10.102.197.0:/volumes/_nogroup/your-nfs-path /mnt/nfs + ls /mnt/nfs/output/ + ``` + + + +## Related Resources + + + + diff --git a/src/content/docs/demos/09-aidaptiv-inference.mdx b/src/content/docs/demos/09-aidaptiv-inference.mdx new file mode 100644 index 0000000..b3aa571 --- /dev/null +++ b/src/content/docs/demos/09-aidaptiv-inference.mdx @@ -0,0 +1,619 @@ +--- +title: aiDAPTIV Inference +description: Deploy high-performance LLM inference service using aiDAPTIV Cache. +--- + +import { Steps, Aside, LinkCard } from '@astrojs/starlight/components'; + +This guide demonstrates how to deploy a high-performance LLM inference service using aiDAPTIV Cache within the OtterScale cluster. + +## Overview + +aiDAPTIV Inference provides high-performance LLM inference service based on vLLM, supporting: + +- High-throughput serving +- Tensor parallelism (Multi-GPU) +- PagedAttention memory optimization +- KV Cache offloading (DRAM/SSD) +- Dynamic LoRA loading +- Prefix Caching +- OpenAI-compatible API + +## Prerequisites + + + +## Prepare Model + +You need to make your model files accessible to the inference service. There are two methods to achieve this, both configured via the `prescript` field in the Helm chart values. + +### Method 1: Using NFS Storage (Recommended) + +This method uses OtterScale's NFS File System to store and access model files. + + + +1. **Create NFS File System in OtterScale**: + + Navigate to the Storage section: + - Go to `https:///scope//service/file-system` + - Create a new NFS File System or use an existing one + - Record the NFS server address (format: `10.102.197.0:/volumes/_nogroup/xxx`) + +2. **Upload your model files to NFS**: + + Mount the NFS on a machine that has access and copy your model files: + + ```bash + # Mount NFS on a node with access + mkdir -p /mnt/nfs + mount -t nfs4 10.102.197.0:/volumes/_nogroup/xxx /mnt/nfs + + # Copy your model files to NFS + cp -r /path/to/your-model /mnt/nfs/models/ + + # Verify files + ls /mnt/nfs/models/ + ``` + +3. **Configure prescript to mount NFS**: + + In the Helm chart values, you'll configure the `prescript` to mount this NFS (see NFS Mount Configuration section below). + + + + + +### Method 2: Using SCP to Copy Models + +This method copies model files directly into the pod using SCP during initialization. + + + +1. **Prepare a remote server with model files**: + + Ensure you have a remote server (accessible from the cluster) that contains your model files. + +2. **Configure prescript with SCP**: + + Use the following prescript template in your Helm chart values: + + ```yaml + prescript: | + # Install required tools + apt-get update && apt-get install -y sshpass + + # Create model directory + mkdir -p /mnt/data/models + + # Copy model from remote server using SCP + echo "Copying model from remote server..." + sshpass -p 'your-password' scp -o StrictHostKeyChecking=no -r \ + user@remote-host:/path/to/your-model /mnt/data/models/ + + if [ $? -eq 0 ]; then + echo "Model copied successfully!" + ls -lh /mnt/data/models/ + else + echo "Failed to copy model. Exiting..." + exit 1 + fi + ``` + + + +## Install Helm Chart + + + +1. **Navigate to Application Store**: + + In the OtterScale web interface: + - Go to **Application** → **Store** + - This opens the Helm chart repository + +2. **Import Helm Chart**: + + - Click the **Import** button at the top of the page + - Enter the Helm Chart URL in the dialog: + ``` + https://github.com/otterscale/charts/releases/download/aidaptivcache-inference-0.1.3/aidaptivcache-inference-0.1.3.tgz + ``` + - Click **Confirm** to import + +3. **Install Chart**: + + - Find `aidaptivcache-inference` in the Store list + - Click the **Install** button + - In the Install Release dialog: + - Enter a **Name** for your deployment (e.g., `llama-inference`) + - Enter a **Namespace** (e.g., `inference`) + - Click **View/Edit** button to open the configuration editor + - Edit the `values.yaml` to configure your inference service (see Configuration Guide below) + - Click **Confirm** to start the installation + + + + + +## Configuration Guide + +Now that you've opened the configuration editor via **View/Edit**, you need to configure the following fields in the `values.yaml`. + +### Basic Configuration + +#### Image Settings + +```yaml +image: + repository: docker.io/library/aidaptiv + tag: vNXUN_3_03AA + pullPolicy: IfNotPresent +``` + +- `repository`: Container image address +- `tag`: Image version tag +- `pullPolicy`: Image pull policy (IfNotPresent/Always/Never) + +#### Deployment Configuration + +```yaml +deployment: + name: vllm-api + replicas: 1 +``` + +- `name`: Kubernetes Deployment name +- `replicas`: Number of Pod replicas (typically 1 due to limited GPU resources) + +### vLLM Configuration + +#### Environment Variables + +```yaml +vllm: + env: + vllmUseV1: "1" + vllmWorkerMultiprocMethod: "spawn" + tiktokenEncodingsBase: "" +``` + +- `vllmUseV1`: Use vLLM v1 API +- `vllmWorkerMultiprocMethod`: Multi-process startup method (spawn/fork) + +#### Command Line Arguments (Important) + +```yaml +vllm: + args: + model: /mnt/data/model/Meta-Llama-3.1-8B-Instruct/ + nvmePath: /mnt/nvme0 + port: 8000 + gpuMemoryUtilization: 0.9 + maxModelLen: 32768 + tensorParallelSize: 4 + dramKvOffloadGb: 0 + ssdKvOffloadGb: 500 + noResumeKvCache: true + disableGpuReuse: false + enableChunkedPrefill: true +``` + +**Key Parameter Explanations**: + +- **`model`** (required): + - Model input path + - Must correspond to the container mount path + - Example: `/mnt/data/model/Meta-Llama-3.1-8B-Instruct/` + - **Note**: This is the container path, not the NFS server path! + +- **`nvmePath`** (required): + - NVMe cache path + - Used for KV Cache offloading + - Typically uses node NVMe: `/mnt/nvme0` + +- **`port`**: + - vLLM API service port + - Default 8000 + +- **`gpuMemoryUtilization`**: + - GPU memory utilization ratio (0.0-1.0) + - Recommended 0.8-0.9 to reserve some memory buffer + +- **`maxModelLen`**: + - Maximum sequence length + - Adjust based on model and GPU memory + +- **`tensorParallelSize`**: + - Number of GPUs for tensor parallelism + - Must match the `otterscale.com/vgpu` value in resource configuration + - Example: If `tensorParallelSize: 4`, you must set `otterscale.com/vgpu: 4` + +- **`dramKvOffloadGb`**: + - KV Cache offload to DRAM capacity (GB) + - Set to 0 to disable DRAM offload + +- **`ssdKvOffloadGb`**: + - KV Cache offload to SSD capacity (GB) + - Used with `nvmePath` + +- **`enableChunkedPrefill`**: + - Enable chunked prefill for better long-text performance + +**Optional Parameters** (commented by default): + +```yaml +# disableLongToken: true # Disable long token support +# resumeKvCache: true # Resume KV Cache +# cleanObsoleteKvCache: true # Clean obsolete KV Cache +# enablePrefixCaching: true # Enable prefix caching +# enforceEager: true # Force eager mode +``` + +#### LoRA Configuration + +```yaml +vllm: + lora: + enable: false + modules: "" + maxRank: 32 +``` + +**Enable LoRA Example**: + +```yaml +vllm: + lora: + enable: true + modules: "lora=/mnt/data/lora-adapters/llama3.1-8B-lora/" + maxRank: 32 +``` + +- `enable`: Set to `true` to enable LoRA +- `modules`: LoRA adapter path, format: `lora=/path/to/adapter/` +- `maxRank`: Maximum LoRA rank + +**Note**: All three parameters must be configured together to enable LoRA. + +### NFS Mount Configuration (For Method 1) + +If you're using NFS storage to access your model files, configure the `prescript` to mount your NFS storage. + +```yaml +prescript: | + apt install -y nfs-common + echo "Starting NFS mount process..." + mkdir -p /mnt/data + TIMEOUT=300 + ELAPSED=0 + while [ $ELAPSED -lt $TIMEOUT ]; do + echo "Attempting to mount NFS to /mnt/data" + mount -t nfs4 -o nfsvers=4.1 -v 10.102.197.0:/volumes/_nogroup/your-nfs-path /mnt/data + if mountpoint -q /mnt/data; then + echo "NFS mount successful!" + break + else + echo "Mount failed, retrying in 5 seconds... (${ELAPSED}s/${TIMEOUT}s)" + sleep 5 + ELAPSED=$((ELAPSED + 5)) + fi + done + if [ $ELAPSED -ge $TIMEOUT ]; then + echo "NFS mount timeout after ${TIMEOUT} seconds. Exiting..." + exit 1 + fi +``` + +**Configuration Details**: +- Replace `10.102.197.0:/volumes/_nogroup/your-nfs-path` with your actual NFS server address from the File System page +- The script installs `nfs-common` package (required for NFS mounting) +- Implements retry logic with a 5-minute timeout +- Mounts NFS to `/mnt/data` which is used in the model path configuration + + + +### Service Configuration + +```yaml +service: + type: NodePort + port: 8000 + targetPort: 8000 + # nodePort: 30299 +``` + +- `type`: Service type (NodePort/ClusterIP/LoadBalancer) +- `port`: Service external port +- `targetPort`: Container internal port +- `nodePort`: (Optional) Specify a fixed NodePort + +### Volume Configuration + +```yaml +volumes: + dshm: + enabled: true + sizeLimit: 30Gi +``` + +- **`dshm`**: + - `enabled`: Enable /dev/shm (shared memory) + - `sizeLimit`: Shared memory size (required for vLLM) + + + +### Resource Configuration + +```yaml +resources: + requests: + otterscale.com/vgpu: 1 + otterscale.com/vgpumem-percentage: 60 + phison.com/ai100: 1 + limits: + otterscale.com/vgpu: 1 + otterscale.com/vgpumem-percentage: 60 + phison.com/ai100: 1 +``` + +- **`otterscale.com/vgpu`**: vGPU count + - Must match `vllm.args.tensorParallelSize` for multi-GPU deployment + - Example: For 4-GPU tensor parallelism, set both `tensorParallelSize: 4` and `otterscale.com/vgpu: 4` +- **`otterscale.com/vgpumem-percentage`**: vGPU memory percentage (0-100) +- **`otterscale.com/vgpumem`**: Alternative to percentage, directly specify memory amount (MB) +- **`phison.com/ai100`**: Phison aiDAPTIVCache accelerator count + + + +## Complete Configuration Examples + +### Example: Basic Inference Service + +```yaml +image: + repository: docker.io/library/aidaptiv + tag: vNXUN_3_03AA + pullPolicy: IfNotPresent + +deployment: + name: llama3-inference + replicas: 1 + +vllm: + env: + vllmUseV1: "1" + vllmWorkerMultiprocMethod: "spawn" + + args: + model: /mnt/data/model/Meta-Llama-3.1-8B-Instruct/ + nvmePath: /mnt/nvme0 + port: 8000 + gpuMemoryUtilization: 0.9 + maxModelLen: 32768 + tensorParallelSize: 1 + ssdKvOffloadGb: 500 + enableChunkedPrefill: true + + lora: + enable: false + +service: + type: NodePort + port: 8000 + targetPort: 8000 + +securityContext: + privileged: true + +# NFS Mount Script +prescript: | + apt install -y nfs-common + echo "Starting NFS mount process..." + mkdir -p /mnt/data + TIMEOUT=300 + ELAPSED=0 + while [ $ELAPSED -lt $TIMEOUT ]; do + echo "Attempting to mount NFS to /mnt/data" + mount -t nfs4 -o nfsvers=4.1 -v 10.102.197.0:/volumes/_nogroup/models /mnt/data + if mountpoint -q /mnt/data; then + echo "NFS mount successful!" + break + else + echo "Mount failed, retrying in 5 seconds... (${ELAPSED}s/${TIMEOUT}s)" + sleep 5 + ELAPSED=$((ELAPSED + 5)) + fi + done + if [ $ELAPSED -ge $TIMEOUT ]; then + echo "NFS mount timeout after ${TIMEOUT} seconds. Exiting..." + exit 1 + fi + +volumes: + dshm: + enabled: true + sizeLimit: 30Gi + +resources: + requests: + otterscale.com/vgpu: 1 + otterscale.com/vgpumem-percentage: 80 + phison.com/ai100: 1 + limits: + otterscale.com/vgpu: 1 + otterscale.com/vgpumem-percentage: 80 + phison.com/ai100: 1 +``` +## Using the Inference Service + + + +1. **Get Service Address**: + + Navigate to the Services page: + ``` + https:///scope//service/services + ``` + + Find your inference service and note the NodePort. + +2. **Test API Connection**: + + ```bash + # Get Node IP + NODE_IP="your-node-ip" + NODE_PORT="your-node-port" + + # Test health check + curl http://${NODE_IP}:${NODE_PORT}/health + ``` + +3. **Use OpenAI-Compatible API**: + + ```bash + curl http://${NODE_IP}:${NODE_PORT}/v1/completions \ + -H "Content-Type: application/json" \ + -d '{ + "model": "Meta-Llama-3.1-8B-Instruct", + "prompt": "What is the capital of France?", + "max_tokens": 100, + "temperature": 0.7 + }' + ``` + +4. **Use Chat Completions API**: + + ```bash + curl http://${NODE_IP}:${NODE_PORT}/v1/chat/completions \ + -H "Content-Type: application/json" \ + -d '{ + "model": "Meta-Llama-3.1-8B-Instruct", + "messages": [ + {"role": "system", "content": "You are a helpful assistant."}, + {"role": "user", "content": "What is machine learning?"} + ], + "max_tokens": 200 + }' + ``` + +5. **Python Client Example**: + + ```python + from openai import OpenAI + + # Connect to vLLM service + client = OpenAI( + base_url=f"http://{NODE_IP}:{NODE_PORT}/v1", + api_key="dummy" # vLLM doesn't require a real API key + ) + + # Perform inference + response = client.chat.completions.create( + model="Meta-Llama-3.1-8B-Instruct", + messages=[ + {"role": "user", "content": "Explain quantum computing in simple terms"} + ], + max_tokens=150, + temperature=0.8 + ) + + print(response.choices[0].message.content) + ``` + + + +## Monitoring and Debugging + + + +1. **Check Deployment Status**: + + Navigate to the Workloads page: + ``` + https:///scope//service/workload + ``` + +2. **View Pod Logs**: + + - Click on the Pod corresponding to the Deployment + - View the **Logs** tab + - Monitor model loading, inference requests, etc. + +3. **Check Resource Usage**: + + - View GPU and memory usage in the Pod details page + - Check for OOM (Out of Memory) errors + + + +## Performance Tuning Recommendations + +### Memory Optimization + +- **Small models** (< 7B): + - `gpuMemoryUtilization: 0.9` + - `ssdKvOffloadGb: 0` (no offload needed) + +- **Medium models** (7B-13B): + - `gpuMemoryUtilization: 0.85` + - `ssdKvOffloadGb: 200-500` + +- **Large models** (> 13B): + - `gpuMemoryUtilization: 0.8` + - `ssdKvOffloadGb: 500-1000` + - Consider multi-GPU: `tensorParallelSize: 2-4` with `otterscale.com/vgpu: 2-4` + +### Latency vs Throughput Trade-off + +- **Low latency priority**: + ```yaml + maxModelLen: 8192 # Shorter sequences + enableChunkedPrefill: false + dramKvOffloadGb: 0 + ssdKvOffloadGb: 0 + ``` + +- **High throughput priority**: + ```yaml + maxModelLen: 32768 # Longer sequences + enableChunkedPrefill: true + enablePrefixCaching: true + ssdKvOffloadGb: 500 + ``` + +## Related Resources + + + + + +