diff --git a/docs/flagrelease_en/model_list.txt b/docs/flagrelease_en/model_list.txt index 0b2db3263..57c61059a 100644 --- a/docs/flagrelease_en/model_list.txt +++ b/docs/flagrelease_en/model_list.txt @@ -1,4 +1,5 @@ FlagRelease/BAAI-Cardiac-Agent-hygon-FlagOS +FlagRelease/Baguettotron-metax-FlagOS FlagRelease/C2S-Scale-Gemma-2-27B-hygon-FlagOS FlagRelease/C2S-Scale-Gemma-2-27B-nvidia-FlagOS FlagRelease/DeepSeek-R1-Distill-Qwen-32B-FlagOS-Cambricon @@ -25,6 +26,7 @@ FlagRelease/DeepSeek-V4-Pro-hygon-FlagOS FlagRelease/DeepSeek-V4-Pro-metax-FlagOS FlagRelease/DeepSeek-V4-Pro-mthreads-FlagOS FlagRelease/DeepSeek-V4-Pro-nvidia-FlagOS +FlagRelease/ERNIE-4.5-0.3B-PT FlagRelease/ERNIE-4.5-300B-A47B-PT-FlagOS FlagRelease/Emu3.5-FlagOS FlagRelease/GLM-4.5-FlagOS @@ -53,9 +55,16 @@ FlagRelease/HY-MT2-7B-mthreads-FlagOS FlagRelease/HY-MT2-7B-nvidia-FlagOS FlagRelease/HY-MT2-7B-zhenwu-FlagOS FlagRelease/Hunyuan-A13B-Instruct-FlagOS +FlagRelease/Hy3-hygon-FlagOS +FlagRelease/Hy3-iluvatar-FlagOS +FlagRelease/Hy3-metax-FlagOS +FlagRelease/Hy3-mthreads-FlagOS +FlagRelease/Hy3-nvidia-FlagOS-Express +FlagRelease/Hy3-tsingmicro-FlagOS +FlagRelease/Hy3-zhenwu-FlagOS FlagRelease/Kimi-K2-Instruct-FlagOS FlagRelease/Kimi-K2-Thinking-FlagOS -FlagRelease/Kimi-Linear-48B-A3B-Instruct-nvidia-FlagOS +FlagRelease/LFM2-2.6B-Exp-metax-FlagOS FlagRelease/MiniCPM-V-4-FlagOS FlagRelease/MiniCPM-V-4-metax-FlagOS FlagRelease/MiniCPM-o-4.5-ascend-FlagOS @@ -90,6 +99,8 @@ FlagRelease/MiniMax-M3-metax-FlagOS FlagRelease/MiniMax-M3-mthreads-FlagOS FlagRelease/MiniMax-M3-nvidia-FlagOS FlagRelease/MiniMax-M3-zhenwu-FlagOS +FlagRelease/Phi-3.5-mini-instruct-FlagOS +FlagRelease/Phi-3.5-mini-instruct-metax-FlagOS FlagRelease/QwQ-32B-FlagOS-Cambricon FlagRelease/QwQ-32B-FlagOS-Iluvatar FlagRelease/QwQ-32B-FlagOS-Nvidia @@ -147,8 +158,9 @@ FlagRelease/RoboBrain2.0-7B-metax-FlagOS FlagRelease/RoboBrain2.5-8B-FlagOS FlagRelease/RoboBrain2.5-8B-ascend-FlagOS FlagRelease/Seed-OSS-36B-Instruct-FlagOS -FlagRelease/Seed-OSS-36B-Instruct-iluvatar-FlagOS FlagRelease/TeleChat3-36B-Thinking-mthreads-FlagOS +FlagRelease/ZCK-Qwen3-8B-metax-FlagOS +FlagRelease/ZCK-Qwen3-8B-nvidia-FlagOS FlagRelease/deepseek-r1-1.5b-nvidia-FlagOS FlagRelease/farm_molecular_representation-hygon-FlagOS FlagRelease/farm_molecular_representation-nvidia-FlagOS diff --git a/docs/flagrelease_en/model_readmes/FlagRelease_Baguettotron-metax-FlagOS.md b/docs/flagrelease_en/model_readmes/FlagRelease_Baguettotron-metax-FlagOS.md new file mode 100644 index 000000000..9f94e73ff --- /dev/null +++ b/docs/flagrelease_en/model_readmes/FlagRelease_Baguettotron-metax-FlagOS.md @@ -0,0 +1,108 @@ +# Introduction +新模型介绍，待定.... + +### Integrated Deployment +- Out-of-the-box inference scripts with pre-configured hardware and software parameters +- Released **FlagOS-Metax** container image supporting deployment within minutes +### Consistency Validation +- Rigorously evaluated through benchmark testing: Performance and results from the FlagOS software stack are compared against native stacks on multiple public. + + +# Evaluation Results +## Benchmark Result +| Metrics | Baguettotron-metax-FlagOS-Origin | Baguettotron-metax-FlagOS-FlagOS | +|--------------|----------------------------------|----------------------------------| +| GPQA_Diamond | 0 | 0.0 | +| ERQA | - | - | +| Aime24 | - | - | + +# User Guide +Environment Setup + +| Item | Version | +|------------------|----------------------| +| Docker Version | 29.4.0 | +| Operating System | Ubuntu 22.04.3 LTS (Jammy Jellyfish) | + +## Operation Steps + +### Download FlagOS Image +```bash +docker pull harbor.baai.ac.cn/flagrelease-public/baguettotron-metax001-gems5.0.2-tree0.5.1-cxnone-plugin0.2.0-vllm0.20.2-cp312-pt28-maca37-x64-3.3.12:202607121209-v2 +``` + +### Download Open-source Model Weights +```bash +pip install modelscope +modelscope download --model FlagRelease/Baguettotron-metax-FlagOS --local_dir /data/Baguettotron-FlagOS +``` + +### Start the Container +```bash +docker run -d --name flagos --net=host --ipc=host --privileged --shm-size=64g --group-add video --ulimit memlock=-1 --security-opt seccomp=unconfined --security-opt apparmor=unconfined --device=/dev/dri --device=/dev/mxcd -v /data:/data harbor.baai.ac.cn/flagrelease-public/baguettotron-metax001-gems5.0.2-tree0.5.1-cxnone-plugin0.2.0-vllm0.20.2-cp312-pt28-maca37-x64-3.3.12:202607121209-v2 sleep infinity +``` +### Start the Server +```bash +vllm serve /data/Baguettotron-FlagOS --host 0.0.0.0 --port 8000 --served-model-name Baguettotron --tensor-parallel-size 1 --max-model-len 4096 --trust-remote-code +``` + +## Service Invocation +### Invocation Script +```bash +curl http://localhost:8000/v1/chat/completions \ + -H "Content-Type: application/json" \ + -d '{ + "model": "Baguettotron", + "messages": [{"role": "user", "content": "你好"}] + }' +``` + + +### AnythingLLM Integration Guide + +#### 1. Download & Install + +- Visit the official site: https://anythingllm.com/ +- Choose the appropriate version for your OS (Windows/macOS/Linux) +- Follow the installation wizard to complete the setup + +#### 2. Configuration + +- Launch AnythingLLM +- Open settings (bottom left, fourth tab) +- Configure core LLM parameters +- Click "Save Settings" to apply changes + +#### 3. Model Interaction + +- After model loading is complete: +- Click **"New Conversation"** +- Enter your question (e.g., "Explain the basics of quantum computing") +- Click the send button to get a response +# Technical Overview +**FlagOS** is a fully open-source system software stack designed to unify the "model–system–chip" layers and foster an open, collaborative ecosystem. It enables a "develop once, run anywhere" workflow across diverse AI accelerators, unlocking hardware performance, eliminating fragmentation among vendor-specific software stacks, and substantially lowering the cost of porting and maintaining AI workloads. With core technologies such as the **FlagScale**, together with vllm-plugin-fl, distributed training/inference framework, **FlagGems** universal operator library, **FlagCX** communication library, and **FlagTree** unified compiler, the **FlagRelease** platform leverages the **FlagOS** stack to automatically produce and release various combinations of \. This enables efficient and automated model migration across diverse chips, opening a new chapter for large model deployment and application. +## FlagGems +FlagGems is a high-performance, generic operator library implemented in [Triton](https://github.com/openai/triton) language. It is built on a collection of backend-neutral kernels that aims to accelerate LLM (Large-Language Models) training and inference across diverse hardware platforms. +## FlagTree +FlagTree is an open source, unified compiler for multiple AI chips project dedicated to developing a diverse ecosystem of AI chip compilers and related tooling platforms, thereby fostering and strengthening the upstream and downstream Triton ecosystem. Currently in its initial phase, the project aims to maintain compatibility with existing adaptation solutions while unifying the codebase to rapidly implement single-repository multi-backend support. For upstream model users, it provides unified compilation capabilities across multiple backends; for downstream chip manufacturers, it offers examples of Triton ecosystem integration. +## FlagScale and vllm-plugin-fl +Flagscale is a comprehensive toolkit designed to support the entire lifecycle of large models. It builds on the strengths of several prominent open-source projects, including [Megatron-LM](https://github.com/NVIDIA/Megatron-LM) and [vLLM](https://github.com/vllm-project/vllm), to provide a robust, end-to-end solution for managing and scaling large models. +vllm-plugin-fl is a vLLM plugin built on the FlagOS unified multi-chip backend, to help flagscale support multi-chip on vllm framework. +## **FlagCX** +FlagCX is a scalable and adaptive cross-chip communication library. It serves as a platform where developers, researchers, and AI engineers can collaborate on various projects, contribute to the development of cutting-edge AI solutions, and share their work with the global community. + +## **FlagEval Evaluation Framework** + FlagEval is a comprehensive evaluation system and open platform for large models launched in 2023. It aims to establish scientific, fair, and open benchmarks, methodologies, and tools to help researchers assess model and training algorithm performance. It features: + - **Multi-dimensional Evaluation**: Supports 800+ model evaluations across NLP, CV, Audio, and Multimodal fields, covering 20+ downstream tasks including language understanding and image-text generation. + - **Industry-Grade Use Cases**: Has completed horizontal evaluations of mainstream large models, providing authoritative benchmarks for chip-model performance validation. + +# Contributing + +We warmly welcome global developers to join us: + +1. Submit Issues to report problems +2. Create Pull Requests to contribute code +3. Improve technical documentation +4. Expand hardware adaptation support +# License +The model weights are derived from PleIAs/Baguettotron and are open‑sourced under the Apache License 2.0: https://www.apache.org/licenses/LICENSE-2.0.txt diff --git a/docs/flagrelease_en/model_readmes/FlagRelease_ERNIE-4.5-0.3B-PT.md b/docs/flagrelease_en/model_readmes/FlagRelease_ERNIE-4.5-0.3B-PT.md new file mode 100644 index 000000000..fd428031c --- /dev/null +++ b/docs/flagrelease_en/model_readmes/FlagRelease_ERNIE-4.5-0.3B-PT.md @@ -0,0 +1,134 @@ +--- +license: apache-2.0 +language: +- en +- zh +pipeline_tag: text-generation +tags: +- ERNIE4.5 +library_name: transformers +--- + +
+ + Chat + + + Hugging Face + + + Github + + + Blog + +
+ +
+ + License + +
+ +# ERNIE-4.5-0.3B + +> [!NOTE] +> Note: "**-Paddle**" models use [PaddlePaddle](https://github.com/PaddlePaddle/Paddle) weights, while "**-PT**" models use Transformer-style PyTorch weights. + +## ERNIE 4.5 Highlights + +The advanced capabilities of the ERNIE 4.5 models, particularly the MoE-based A47B and A3B series, are underpinned by several key technical innovations: + +1. **Multimodal Heterogeneous MoE Pre-Training:** Our models are jointly trained on both textual and visual modalities to better capture the nuances of multimodal information and improve performance on tasks involving text understanding and generation, image understanding, and cross-modal reasoning. To achieve this without one modality hindering the learning of another, we designed a *heterogeneous MoE structure*, incorporated *modality-isolated routing*, and employed *router orthogonal loss* and *multimodal token-balanced loss*. These architectural choices ensure that both modalities are effectively represented, allowing for mutual reinforcement during training. + +2. **Scaling-Efficient Infrastructure:** We propose a novel heterogeneous hybrid parallelism and hierarchical load balancing strategy for efficient training of ERNIE 4.5 models. By using intra-node expert parallelism, memory-efficient pipeline scheduling, FP8 mixed-precision training and finegrained recomputation methods, we achieve remarkable pre-training throughput. For inference, we propose *multi-expert parallel collaboration* method and *convolutional code quantization* algorithm to achieve 4-bit/2-bit lossless quantization. Furthermore, we introduce PD disaggregation with dynamic role switching for effective resource utilization to enhance inference performance for ERNIE 4.5 MoE models. Built on [PaddlePaddle](https://github.com/PaddlePaddle/Paddle), ERNIE 4.5 delivers high-performance inference across a wide range of hardware platforms. + +3. **Modality-Specific Post-Training:** To meet the diverse requirements of real-world applications, we fine-tuned variants of the pre-trained model for specific modalities. Our LLMs are optimized for general-purpose language understanding and generation. The VLMs focuses on visuallanguage understanding and supports both thinking and non-thinking modes. Each model employed a combination of *Supervised Fine-tuning (SFT)*, *Direct Preference Optimization (DPO)* or a modified reinforcement learning method named *Unified Preference Optimization (UPO)* for post-training. + +## Model Overview + +ERNIE-4.5-0.3B is a text dense Post-trained model. The following are the model configuration details: + +| Key | Value | +| -------------- | ------------ | +| Modality | Text | +| Training Stage | Posttraining | +| Params | 0.36B | +| Layers | 18 | +| Heads(Q/KV) | 16 / 2 | +| Context Length | 131072 | + +## Quickstart + +### Using `transformers` library + +**Note**: Before using the model, please ensure you have the `transformers` library installed +(upcoming version 4.54.0 or [the latest version](https://github.com/huggingface/transformers?tab=readme-ov-file#installation)) + +The following contains a code snippet illustrating how to use the model generate content based on given inputs. + +```python +import torch +from transformers import AutoModelForCausalLM, AutoTokenizer + +model_name = "baidu/ERNIE-4.5-0.3B-PT" + +# load the tokenizer and the model +tokenizer = AutoTokenizer.from_pretrained(model_name) +model = AutoModelForCausalLM.from_pretrained( + model_name, + device_map="auto", + torch_dtype=torch.bfloat16, +) + +# prepare the model input +prompt = "Give me a short introduction to large language model." +messages = [ + {"role": "user", "content": prompt} +] +text = tokenizer.apply_chat_template( + messages, + tokenize=False, + add_generation_prompt=True +) +model_inputs = tokenizer([text], add_special_tokens=False, return_tensors="pt").to(model.device) + +# conduct text completion +generated_ids = model.generate( + **model_inputs, + max_new_tokens=1024 +) +output_ids = generated_ids[0][len(model_inputs.input_ids[0]):].tolist() + +# decode the generated ids +generate_text = tokenizer.decode(output_ids, skip_special_tokens=True) +print("generate_text:", generate_text) +``` + +### vLLM inference + +[vllm](https://github.com/vllm-project/vllm/tree/main) github library. Python-only [build](https://docs.vllm.ai/en/latest/getting_started/installation/gpu.html#set-up-using-python-only-build-without-compilation). + +```bash +vllm serve baidu/ERNIE-4.5-0.3B-PT +``` + +## License + +The ERNIE 4.5 models are provided under the Apache License 2.0. This license permits commercial use, subject to its terms and conditions. Copyright (c) 2025 Baidu, Inc. All Rights Reserved. + +## Citation + +If you find ERNIE 4.5 useful or wish to use it in your projects, please kindly cite our technical report: + +```bibtex +@misc{ernie2025technicalreport, + title={ERNIE 4.5 Technical Report}, + author={Baidu ERNIE Team}, + year={2025}, + eprint={}, + archivePrefix={arXiv}, + primaryClass={cs.CL}, + url={} +} +``` diff --git a/docs/flagrelease_en/model_readmes/FlagRelease_Hy3-hygon-FlagOS.md b/docs/flagrelease_en/model_readmes/FlagRelease_Hy3-hygon-FlagOS.md new file mode 100644 index 000000000..2d8d454e9 --- /dev/null +++ b/docs/flagrelease_en/model_readmes/FlagRelease_Hy3-hygon-FlagOS.md @@ -0,0 +1,156 @@ +--- +base_model: +- "" +language: +- zh +- en +license: apache-2.0 +--- + +# Introduction +Hy3 is Tencent Hunyuan team's next-generation MoE large model with integrated fast and slow thinking: 295B total parameters, 21B active parameters (plus 3.8B MTP layer parameters), 192 experts with top-8 activation, supporting 256K context. Compared to the Preview version released in late April, Hy3 has achieved a comprehensive leap in intelligence through incorporating real-world business feedback, scaling up RL compute, and improving post-training data quality, significantly outperforming open-source models of similar size. + + +### Integrated Deployment +- Out-of-the-box inference scripts with pre-configured hardware and software parameters +- Released **FlagOS-Hygon** container image supporting deployment within minutes +### Consistency Validation +- Rigorously evaluated through benchmark testing: Performance and results from the FlagOS software stack are compared against native stacks on multiple public. + + +# Evaluation Results +## Benchmark Result +| Metrics | Hy3-Nvidia-Origin | Hy3-Hygon-FlagOS | +|--------------|-------------------|------------------| +| GPQA_Diamond | 66.16 | 63.78 | +| arc_challenge_chat | 96.33 | 96.5 | +| math_500 | 94.6 | 94.2 | + +# User Guide +Environment Setup + +| Item | Version | +|------------------|----------------------| +| Docker Version | Docker version 20.10.5, build 55c4c88 | +| Operating System | Ubuntu 22.04.4 LTS (Jammy Jellyfish) | + +## Operation Steps + +### Download FlagOS Image +```bash +docker pull harbor.baai.ac.cn/flagrelease-public/hy3-hygon001-gems5.4.0-treenone-cxnone-plugin0.2.0-vllm0.20.2-cp310-pt210-dtknone-x64-6.3.30-v1.4.1a +``` + +### Download Open-source Model Weights +```bash +pip install modelscope +modelscope download --model FlagRelease/Hy3-hygon-FlagOS --local_dir /data/Hy3 +``` + +### Start the Container +```bash +docker run --name flagos --network=host --ipc=host --device=/dev/kfd --device=/dev/mkfd --device=/dev/dri -v /opt/hyhal:/opt/hyhal -v /data:/data --group-add video --cap-add=SYS_PTRACE --security-opt seccomp=unconfined -itd harbor.baai.ac.cn/flagrelease-public/hy3-hygon001-gems5.4.0-treenone-cxnone-plugin0.2.0-vllm0.20.2-cp310-pt210-dtknone-x64-6.3.30-v1.4.1a bin/bash +docker exec -it flagos /bin/bash +``` +### Start the Server +master, fix NCCL/GLOO to your specific configs. +```bash +export VLLM_PLUGINS='fl' +export NCCL_IB_DISABLE=1 +export NCCL_SOCKET_IFNAME=bond1 +export GLOO_SOCKET_IFNAME=bond1 + +vllm serve /data/Hy3 \ + -tp 16 \ + --max-model-len 131072 \ + --served-model-name hygonhyv3 \ + --port 9010 \ + --nnodes 2 \ + --node-rank 0 \ + --master-addr `` \ + --reasoning-parser hy_v3 \ + --enforce-eager +``` + +worker + +```bash +export VLLM_PLUGINS='fl' +export NCCL_IB_DISABLE=1 +export NCCL_SOCKET_IFNAME=bond1 +export GLOO_SOCKET_IFNAME=bond1 + +vllm serve /data/Hy3 \ + -tp 16 \ + --max-model-len 131072 \ + --served-model-name hygonhyv3 \ + --port 9010 \ + --nnodes 2 \ + --node-rank 1 \ + --headless \ + --master-addr `` \ + --reasoning-parser hy_v3 \ + --enforce-eager +``` + +## Service Invocation +### Invocation Script +```bash +curl http://localhost:9010/v1/chat/completions \ + -H "Content-Type: application/json" \ + -d '{ + "model": "hygonhyv3", + "messages": [{"role": "user", "content": "你好"}] + }' +``` + + +### AnythingLLM Integration Guide + +#### 1. Download & Install + +- Visit the official site: https://anythingllm.com/ +- Choose the appropriate version for your OS (Windows/macOS/Linux) +- Follow the installation wizard to complete the setup + +#### 2. Configuration + +- Launch AnythingLLM +- Open settings (bottom left, fourth tab) +- Configure core LLM parameters +- Click "Save Settings" to apply changes + +#### 3. Model Interaction + +- After model loading is complete: +- Click **"New Conversation"** +- Enter your question (e.g., “Explain the basics of quantum computing”) +- Click the send button to get a response +# Technical Overview +**FlagOS** is a fully open-source system software stack designed to unify the "model–system–chip" layers and foster an open, collaborative ecosystem. It enables a “develop once, run anywhere” workflow across diverse AI accelerators, unlocking hardware performance, eliminating fragmentation among vendor-specific software stacks, and substantially lowering the cost of porting and maintaining AI workloads. With core technologies such as the **FlagScale**, together with vllm-plugin-fl, distributed training/inference framework, **FlagGems** universal operator library, **FlagCX** communication library, and **FlagTree** unified compiler, the **FlagRelease** platform leverages the **FlagOS** stack to automatically produce and release various combinations of \. This enables efficient and automated model migration across diverse chips, opening a new chapter for large model deployment and application. +## FlagGems +FlagGems is a high-performance, generic operator libraryimplemented in [Triton](https://github.com/openai/triton) language. It is built on a collection of backend-neutralkernels that aims to accelerate LLM (Large-Language Models) training and inference across diverse hardware platforms. +## FlagTree +FlagTree is an open source, unified compiler for multipleAI chips project dedicated to developing a diverse ecosystem of AI chip compilers and related tooling platforms, thereby fostering and strengthening the upstream and downstream Triton ecosystem. Currently in its initial phase, the project aims to maintain compatibility with existing adaptation solutions while unifying the codebase to rapidly implement single-repository multi-backend support. Forupstream model users, it provides unified compilation capabilities across multiple backends; for downstream chip manufacturers, it offers examples of Triton ecosystem integration. +## FlagScale and vllm-plugin-fl +Flagscale is a comprehensive toolkit designed to supportthe entire lifecycle of large models. It builds on the strengths of several prominent open-source projects, including [Megatron-LM](https://github.com/NVIDIA/Megatron-LM) and [vLLM](https://github.com/vllm-project/vllm), to provide a robust, end-to-end solution for managing and scaling large models. +vllm-plugin-fl is a vLLM plugin built on the FlagOS unified multi-chip backend, to help flagscale support multi-chip on vllm framework. +## **FlagCX** +FlagCX is a scalable and adaptive cross-chip communication library. It serves as a platform where developers, researchers, and AI engineers can collaborate on various projects, contribute to the development of cutting-edge AI solutions, and share their work with the global community. + +## **FlagEval Evaluation Framework** + FlagEval is a comprehensive evaluation system and open platform for large models launched in 2023. It aims to establish scientific, fair, and open benchmarks, methodologies, and tools to help researchers assess model and training algorithm performance. It features: + - **Multi-dimensional Evaluation**: Supports 800+ modelevaluations across NLP, CV, Audio, and Multimodal fields,covering 20+ downstream tasks including language understanding and image-text generation. + - **Industry-Grade Use Cases**: Has completed horizonta1 evaluations of mainstream large models, providing authoritative benchmarks for chip-model performance validation. + +# Contributing + +We warmly welcome global developers to join us: + +1. Submit Issues to report problems +2. Create Pull Requests to contribute code +3. Improve technical documentation +4. Expand hardware adaptation support +# License +The model weights are derived from Tencent-Hunyuan/Hy3 and are open‑sourced under the Apache License 2.0: https://www.apache.org/licenses/LICENSE-2.0.txt + diff --git a/docs/flagrelease_en/model_readmes/FlagRelease_Hy3-iluvatar-FlagOS.md b/docs/flagrelease_en/model_readmes/FlagRelease_Hy3-iluvatar-FlagOS.md new file mode 100644 index 000000000..73df9f03d --- /dev/null +++ b/docs/flagrelease_en/model_readmes/FlagRelease_Hy3-iluvatar-FlagOS.md @@ -0,0 +1,121 @@ +--- +base_model: +- "" +language: +- zh +- en +license: apache-2.0 +--- + +# Introduction +Hy3 is Tencent Hunyuan team's next-generation MoE large model with integrated fast and slow thinking: 295B total parameters, 21B active parameters (plus 3.8B MTP layer parameters), 192 experts with top-8 activation, supporting 256K context. Compared to the Preview version released in late April, Hy3 has achieved a comprehensive leap in intelligence through incorporating real-world business feedback, scaling up RL compute, and improving post-training data quality, significantly outperforming open-source models of similar size. + + +### Integrated Deployment +- Out-of-the-box inference scripts with pre-configured hardware and software parameters +- Released **FlagOS-Iluvatar** container image supporting deployment within minutes +### Consistency Validation +- Rigorously evaluated through benchmark testing: Performance and results from the FlagOS software stack are compared against native stacks on multiple public. + + +# Evaluation Results +## Benchmark Result +| Metrics | Hy3-Nvidia-Origin | Hy3-Iluvatar-FlagOS | +|--------------|--------------------------------|--------------------------------------| +| GPQA_Diamond | 83.33 | 81.82 | +| arc_challenge_chat | 96.33 | 95.82 | +| math_500 | 94.6 | 90.6 | + + +# User Guide +Environment Setup + +| Item | Version | +|------------------|----------------------| +| Docker Version | Docker version 20.10.25, build 20.10.25-0ubuntu1~20.04.1 | +| Operating System | Ubuntu 20.04.6 LTS | + +## Operation Steps + +### Download FlagOS Image +```bash +docker pull harbor.baai.ac.cn/flagrelease-public/hy3-iluvatar001-gems5.0.2-treenone-cxnone-pluginnone-vllm0.17.0-cp312-pt27-ixml44-x64-4.4.0 +``` + +### Download Open-source Model Weights +```bash +pip install modelscope +modelscope download --model FlagRelease/Hy3-iluvatar-FlagOS --local_dir /data/Hy3 +``` + +### Start the Container +```bash +docker run -dit -v /mnt:/mnt -v /usr/src:/usr/src -v /lib/modules:/lib/modules -v /dev:/dev -v /data:/data -v /home:/home --network=host --name=flagos --ipc=host --privileged --cap-add=ALL --pid=host harbor.baai.ac.cn/flagrelease-public/hy3-iluvatar001-gems5.0.2-treenone-cxnone-pluginnone-vllm0.17.0-cp312-pt27-ixml44-x64-4.4.0 /bin/bash +docker exec -it flagos /bin/bash +``` +### Start the Server +```bash +VLLM_W8A8_MOE_USE_W4A8=1 vllm serve /data/Hy3 -tp 16 -dp 1 --port 9010 --enforce-eager --reasoning-parser hy_v3 --served-model-name hyv3iluvatarw4a8 +``` + +## Service Invocation +### Invocation Script +```bash +curl http://localhost:9010/v1/chat/completions \ + -H "Content-Type: application/json" \ + -d '{ + "model": "hyv3iluvatarw4a8", + "messages": [{"role": "user", "content": "你好"}] + }' +``` + + +### AnythingLLM Integration Guide + +#### 1. Download & Install + +- Visit the official site: https://anythingllm.com/ +- Choose the appropriate version for your OS (Windows/macOS/Linux) +- Follow the installation wizard to complete the setup + +#### 2. Configuration + +- Launch AnythingLLM +- Open settings (bottom left, fourth tab) +- Configure core LLM parameters +- Click "Save Settings" to apply changes + +#### 3. Model Interaction + +- After model loading is complete: +- Click **"New Conversation"** +- Enter your question (e.g., “Explain the basics of quantum computing”) +- Click the send button to get a response +# Technical Overview +**FlagOS** is a fully open-source system software stack designed to unify the "model–system–chip" layers and foster an open, collaborative ecosystem. It enables a “develop once, run anywhere” workflow across diverse AI accelerators, unlocking hardware performance, eliminating fragmentation among vendor-specific software stacks, and substantially lowering the cost of porting and maintaining AI workloads. With core technologies such as the **FlagScale**, together with vllm-plugin-fl, distributed training/inference framework, **FlagGems** universal operator library, **FlagCX** communication library, and **FlagTree** unified compiler, the **FlagRelease** platform leverages the **FlagOS** stack to automatically produce and release various combinations of \. This enables efficient and automated model migration across diverse chips, opening a new chapter for large model deployment and application. +## FlagGems +FlagGems is a high-performance, generic operator libraryimplemented in [Triton](https://github.com/openai/triton) language. It is built on a collection of backend-neutralkernels that aims to accelerate LLM (Large-Language Models) training and inference across diverse hardware platforms. +## FlagTree +FlagTree is an open source, unified compiler for multipleAI chips project dedicated to developing a diverse ecosystem of AI chip compilers and related tooling platforms, thereby fostering and strengthening the upstream and downstream Triton ecosystem. Currently in its initial phase, the project aims to maintain compatibility with existing adaptation solutions while unifying the codebase to rapidly implement single-repository multi-backend support. Forupstream model users, it provides unified compilation capabilities across multiple backends; for downstream chip manufacturers, it offers examples of Triton ecosystem integration. +## FlagScale and vllm-plugin-fl +Flagscale is a comprehensive toolkit designed to supportthe entire lifecycle of large models. It builds on the strengths of several prominent open-source projects, including [Megatron-LM](https://github.com/NVIDIA/Megatron-LM) and [vLLM](https://github.com/vllm-project/vllm), to provide a robust, end-to-end solution for managing and scaling large models. +vllm-plugin-fl is a vLLM plugin built on the FlagOS unified multi-chip backend, to help flagscale support multi-chip on vllm framework. +## **FlagCX** +FlagCX is a scalable and adaptive cross-chip communication library. It serves as a platform where developers, researchers, and AI engineers can collaborate on various projects, contribute to the development of cutting-edge AI solutions, and share their work with the global community. + +## **FlagEval Evaluation Framework** + FlagEval is a comprehensive evaluation system and open platform for large models launched in 2023. It aims to establish scientific, fair, and open benchmarks, methodologies, and tools to help researchers assess model and training algorithm performance. It features: + - **Multi-dimensional Evaluation**: Supports 800+ modelevaluations across NLP, CV, Audio, and Multimodal fields,covering 20+ downstream tasks including language understanding and image-text generation. + - **Industry-Grade Use Cases**: Has completed horizonta1 evaluations of mainstream large models, providing authoritative benchmarks for chip-model performance validation. + +# Contributing + +We warmly welcome global developers to join us: + +1. Submit Issues to report problems +2. Create Pull Requests to contribute code +3. Improve technical documentation +4. Expand hardware adaptation support +# License +The model weights are derived from Tencent-Hunyuan/Hy3 and are open‑sourced under the Apache License 2.0: https://www.apache.org/licenses/LICENSE-2.0.txt + diff --git a/docs/flagrelease_en/model_readmes/FlagRelease_Hy3-metax-FlagOS.md b/docs/flagrelease_en/model_readmes/FlagRelease_Hy3-metax-FlagOS.md new file mode 100644 index 000000000..d76bc9a5e --- /dev/null +++ b/docs/flagrelease_en/model_readmes/FlagRelease_Hy3-metax-FlagOS.md @@ -0,0 +1,171 @@ +--- +base_model: +- "" +language: +- zh +- en +license: apache-2.0 +--- + +# Introduction +Hy3 is Tencent Hunyuan team's next-generation MoE large model with integrated fast and slow thinking: 295B total parameters, 21B active parameters (plus 3.8B MTP layer parameters), 192 experts with top-8 activation, supporting 256K context. Compared to the Preview version released in late April, Hy3 has achieved a comprehensive leap in intelligence through incorporating real-world business feedback, scaling up RL compute, and improving post-training data quality, significantly outperforming open-source models of similar size. + + +### Integrated Deployment +- Out-of-the-box inference scripts with pre-configured hardware and software parameters +- Released **FlagOS-Metax** container image supporting deployment within minutes +### Consistency Validation +- Rigorously evaluated through benchmark testing: Performance and results from the FlagOS software stack are compared against native stacks on multiple public. + + +# Evaluation Results +## Benchmark Result +| Metrics | Hy3-Nvidia-Origin | Hy3-Metax-FlagOS | +|--------------|--------------------------------|--------------------------------------| +| GPQA_Diamond | 83.33 | 84.85 | +| arc_challenge_chat | 96.33 | 96.59 | +| math_500 | 94.6 | 95.2 | + + +# User Guide +Environment Setup + +| Item | Version | +|------------------|----------------------| +| Docker Version | Docker version 28.0.4, build b8034c0 | +| Operating System | Ubuntu 22.04.4 LTS | + +## Operation Steps + +### Download FlagOS Image +```bash +docker pull harbor.baai.ac.cn/flagrelease-public/hy3-metax001-gems5.0.2-tree0.5.1-cxnone-plugin0.2.0-vllm0.20.2-cp312-pt28-maca37-x64-3.8.1:202607061058 +``` + +### Download Open-source Model Weights +```bash +pip install modelscope +modelscope download --model FlagRelease/Hy3-metax-FlagOS --local_dir /data/Hy3 +``` + +### Start the Container +```bash +docker run -d --name flagos --network host --shm-size 64g --device /dev/dri:/dev/dri:rwm --device /dev/mxcd:/dev/mxcd:rwm -v /data:/data harbor.baai.ac.cn/flagrelease-public/hy3-metax001-gems5.0.2-tree0.5.1-cxnone-plugin0.2.0-vllm0.20.2-cp312-pt28-maca37-x64-3.8.1:202607061058 sleep infinity +docker exec -it flagos /bin/bash +``` +### Start the Server +master, fix MCCL/NCCL/GLOO to your specific configs. +```bash +export MCCL_HC_PLUGIN=/opt/maca-3.3.0/lib/libmxccl_21605plugin.so +export MCCL_EXT_CCL_ENABLE=1 +export NCCL_FORCE_USE_MN_MIX=1 +export MCCL_SOCKET_IFNAME=inbond1 +export NCCL_SOCKET_IFNAME=inbond1 +export GLOO_SOCKET_IFNAME=inbond1 +export MCCL_DEBUG=WARN +export MCCL_PROTO=LL +export MCCL_P2P_LL_THRESHOLD=1073741824 +export MCCL_IB_DISABLE=1 +export NCCL_IB_DISABLE=1 +export NCCL_DEBUG=WARN +export MASTER_ADDR=192.168.2.127 +export MASTER_PORT=29501 +export VLLM_FL_FLAGOS_BLACKLIST="flash_attention_forward,mm,mm_out,bmm,addmm,addmm_out,baddbmm" +export VLLM_ENGINE_ITERATION_TIMEOUT_S=72000 +export NCCL_TIMEOUT=7200 +export VLLM_RPC_TIMEOUT=72000000 +export VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=7200 + +export GEMS_VENDOR=metax +export VLLM_PLUGINS=fl +vllm serve /data/Hy3 --host 0.0.0.0 --port 9010 --served-model-name metaxhy3 --reasoning-parser hy_v3 --tensor-parallel-size 16 --nnodes 2 --node-rank 0 --master-addr `` +``` + +worker +```bash +export MCCL_HC_PLUGIN=/opt/maca-3.3.0/lib/libmxccl_21605plugin.so +export MCCL_EXT_CCL_ENABLE=1 +export NCCL_FORCE_USE_MN_MIX=1 +export MCCL_SOCKET_IFNAME=inbond1 +export NCCL_SOCKET_IFNAME=inbond1 +export GLOO_SOCKET_IFNAME=inbond1 +export MCCL_DEBUG=WARN +export MCCL_PROTO=LL +export MCCL_P2P_LL_THRESHOLD=1073741824 +export MCCL_IB_DISABLE=1 +export NCCL_IB_DISABLE=1 +export NCCL_DEBUG=WARN +export MASTER_ADDR=192.168.2.127 +export MASTER_PORT=29501 +export VLLM_FL_FLAGOS_BLACKLIST="flash_attention_forward,mm,mm_out,bmm,addmm,addmm_out,baddbmm" +export VLLM_ENGINE_ITERATION_TIMEOUT_S=72000 +export NCCL_TIMEOUT=7200 +export VLLM_RPC_TIMEOUT=72000000 +export VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=7200 + +export GEMS_VENDOR=metax +export VLLM_PLUGINS=fl +vllm serve /data/Hy3 --host 0.0.0.0 --port 9010 --served-model-name metaxhy3 --reasoning-parser hy_v3 --tensor-parallel-size 16 --nnodes 2 --node-rank 1 --master-addr `` --headless +``` + +## Service Invocation +### Invocation Script +```bash +curl http://localhost:9010/v1/chat/completions \ + -H "Content-Type: application/json" \ + -d '{ + "model": "metaxhy3", + "messages": [{"role": "user", "content": "你好"}] + }' +``` + + +### AnythingLLM Integration Guide + +#### 1. Download & Install + +- Visit the official site: https://anythingllm.com/ +- Choose the appropriate version for your OS (Windows/macOS/Linux) +- Follow the installation wizard to complete the setup + +#### 2. Configuration + +- Launch AnythingLLM +- Open settings (bottom left, fourth tab) +- Configure core LLM parameters +- Click "Save Settings" to apply changes + +#### 3. Model Interaction + +- After model loading is complete: +- Click **"New Conversation"** +- Enter your question (e.g., “Explain the basics of quantum computing”) +- Click the send button to get a response +# Technical Overview +**FlagOS** is a fully open-source system software stack designed to unify the "model–system–chip" layers and foster an open, collaborative ecosystem. It enables a “develop once, run anywhere” workflow across diverse AI accelerators, unlocking hardware performance, eliminating fragmentation among vendor-specific software stacks, and substantially lowering the cost of porting and maintaining AI workloads. With core technologies such as the **FlagScale**, together with vllm-plugin-fl, distributed training/inference framework, **FlagGems** universal operator library, **FlagCX** communication library, and **FlagTree** unified compiler, the **FlagRelease** platform leverages the **FlagOS** stack to automatically produce and release various combinations of \. This enables efficient and automated model migration across diverse chips, opening a new chapter for large model deployment and application. +## FlagGems +FlagGems is a high-performance, generic operator libraryimplemented in [Triton](https://github.com/openai/triton) language. It is built on a collection of backend-neutralkernels that aims to accelerate LLM (Large-Language Models) training and inference across diverse hardware platforms. +## FlagTree +FlagTree is an open source, unified compiler for multipleAI chips project dedicated to developing a diverse ecosystem of AI chip compilers and related tooling platforms, thereby fostering and strengthening the upstream and downstream Triton ecosystem. Currently in its initial phase, the project aims to maintain compatibility with existing adaptation solutions while unifying the codebase to rapidly implement single-repository multi-backend support. Forupstream model users, it provides unified compilation capabilities across multiple backends; for downstream chip manufacturers, it offers examples of Triton ecosystem integration. +## FlagScale and vllm-plugin-fl +Flagscale is a comprehensive toolkit designed to supportthe entire lifecycle of large models. It builds on the strengths of several prominent open-source projects, including [Megatron-LM](https://github.com/NVIDIA/Megatron-LM) and [vLLM](https://github.com/vllm-project/vllm), to provide a robust, end-to-end solution for managing and scaling large models. +vllm-plugin-fl is a vLLM plugin built on the FlagOS unified multi-chip backend, to help flagscale support multi-chip on vllm framework. +## **FlagCX** +FlagCX is a scalable and adaptive cross-chip communication library. It serves as a platform where developers, researchers, and AI engineers can collaborate on various projects, contribute to the development of cutting-edge AI solutions, and share their work with the global community. + +## **FlagEval Evaluation Framework** + FlagEval is a comprehensive evaluation system and open platform for large models launched in 2023. It aims to establish scientific, fair, and open benchmarks, methodologies, and tools to help researchers assess model and training algorithm performance. It features: + - **Multi-dimensional Evaluation**: Supports 800+ modelevaluations across NLP, CV, Audio, and Multimodal fields,covering 20+ downstream tasks including language understanding and image-text generation. + - **Industry-Grade Use Cases**: Has completed horizonta1 evaluations of mainstream large models, providing authoritative benchmarks for chip-model performance validation. + +# Contributing + +We warmly welcome global developers to join us: + +1. Submit Issues to report problems +2. Create Pull Requests to contribute code +3. Improve technical documentation +4. Expand hardware adaptation support +# License +The model weights are derived from Tencent-Hunyuan/Hy3 and are open‑sourced under the Apache License 2.0: https://www.apache.org/licenses/LICENSE-2.0.txt + diff --git a/docs/flagrelease_en/model_readmes/FlagRelease_Hy3-mthreads-FlagOS.md b/docs/flagrelease_en/model_readmes/FlagRelease_Hy3-mthreads-FlagOS.md new file mode 100644 index 000000000..d3919befe --- /dev/null +++ b/docs/flagrelease_en/model_readmes/FlagRelease_Hy3-mthreads-FlagOS.md @@ -0,0 +1,150 @@ +--- +base_model: +- "" +language: +- zh +- en +license: apache-2.0 +--- + +# Introduction +Hy3 is Tencent Hunyuan team's next-generation MoE large model with integrated fast and slow thinking: 295B total parameters, 21B active parameters (plus 3.8B MTP layer parameters), 192 experts with top-8 activation, supporting 256K context. Compared to the Preview version released in late April, Hy3 has achieved a comprehensive leap in intelligence through incorporating real-world business feedback, scaling up RL compute, and improving post-training data quality, significantly outperforming open-source models of similar size. + + +### Integrated Deployment +- Out-of-the-box inference scripts with pre-configured hardware and software parameters +- Released **FlagOS-Mthreads** container image supporting deployment within minutes +### Consistency Validation +- Rigorously evaluated through benchmark testing: Performance and results from the FlagOS software stack are compared against native stacks on multiple public. + + +# Evaluation Results +## Benchmark Result +| Metrics | Hy3-Nvidia-Origin | Hy3-Mthreads-FlagOS | +|--------------|--------------------------------|--------------------------------------| +| GPQA_Diamond | 83.33 | 83.33 | +| arc_challenge_chat | 96.33 | 96.5 | +| math_500 | 94.6 | 91 | + + +# User Guide +Environment Setup + +| Item | Version | +|------------------|----------------------| +| Docker Version | Docker version 27.5.1, build 9f9e405 | +| Operating System | 22.04.4 LTS (Jammy Jellyfish) | + +## Operation Steps + +### Download FlagOS Image +```bash +docker pull harbor.baai.ac.cn/flagrelease-public/hy3-mthreads001-gems5.4.0-treenone-cxnone-plugin0.2.0-vllm0.20.2-cp310-pt27-musa43-x64-3.3.6-server:202607021123 +``` + +### Download Open-source Model Weights +```bash +pip install modelscope +modelscope download --model FlagRelease/Hy3-mthreads-FlagOS --local_dir /data/Hy3 +``` + +### Start the Container +```bash +docker run -itd --privileged --net host \ + --name flagos \ + -w /workspace \ + -v /data/:/data/ \ + -v /mnt/:/mnt/ \ + -v /public-ks3/:/public/ \ + --env MTHREADS_VISIBLE_DEVICES=all \ + --shm-size=560g \ + harbor.baai.ac.cn/flagrelease-public/hy3-mthreads001-gems5.4.0-treenone-cxnone-plugin0.2.0-vllm0.20.2-cp310-pt27-musa43-x64-3.3.6-server:202607021123 +docker exec -it flagos /bin/bash +``` +### Start the Server +```bash +# in node1 +export VLLM_FL_FLAGOS_WHITELIST=moe_sum,grouped_topk,moe_align_block_size,invoke_fused_moe_triton_kernel,embeddind,rsqrt,index_select,silu_and_mul,rand_like,cumsum +export VLLM_CONFIGURE_LOGGING=1 +vllm serve /data/Hy3 \ + --tensor-parallel-size 8 --pipeline-parallel-size 2\ + --port 8000 \ + --gpu-memory-utilization 0.95 \ + --served-model-name hy3 \ + --nnodes 2 --node-rank 0 \ + --master-addr --master-port 29500 \ + --reasoning-parser hy_v3 +# in node2 +export VLLM_FL_FLAGOS_WHITELIST=moe_sum,grouped_topk,moe_align_block_size,invoke_fused_moe_triton_kernel,embeddind,rsqrt,index_select,silu_and_mul,rand_like,cumsum +export VLLM_CONFIGURE_LOGGING=1 +vllm serve /data/Hy3 \ + --tensor-parallel-size 8 --pipeline-parallel-size 2 \ + --port 8000 \ + --gpu-memory-utilization 0.95 \ + --served-model-name hy3 \ + --nnodes 2 --node-rank 1 \ + --master-addr --master-port 29500 --headless \ + --reasoning-parser hy_v3 +``` + +## Service Invocation +### Invocation Script +```bash +curl http://localhost:8000/v1/chat/completions \ + -H "Content-Type: application/json" \ + -d '{ + "model": "flagOS", + "messages": [{"role": "user", "content": "hi!"}] + }' +``` + + +### AnythingLLM Integration Guide + +#### 1. Download & Install + +- Visit the official site: https://anythingllm.com/ +- Choose the appropriate version for your OS (Windows/macOS/Linux) +- Follow the installation wizard to complete the setup + +#### 2. Configuration + +- Launch AnythingLLM +- Open settings (bottom left, fourth tab) +- Configure core LLM parameters +- Click "Save Settings" to apply changes + +#### 3. Model Interaction + +- After model loading is complete: +- Click **"New Conversation"** +- Enter your question (e.g., “Explain the basics of quantum computing”) +- Click the send button to get a response +# Technical Overview +**FlagOS** is a fully open-source system software stack designed to unify the "model–system–chip" layers and foster an open, collaborative ecosystem. It enables a “develop once, run anywhere” workflow across diverse AI accelerators, unlocking hardware performance, eliminating fragmentation among vendor-specific software stacks, and substantially lowering the cost of porting and maintaining AI workloads. With core technologies such as the **FlagScale**, together with vllm-plugin-fl, distributed training/inference framework, **FlagGems** universal operator library, **FlagCX** communication library, and **FlagTree** unified compiler, the **FlagRelease** platform leverages the **FlagOS** stack to automatically produce and release various combinations of \. This enables efficient and automated model migration across diverse chips, opening a new chapter for large model deployment and application. +## FlagGems +FlagGems is a high-performance, generic operator libraryimplemented in [Triton](https://github.com/openai/triton) language. It is built on a collection of backend-neutralkernels that aims to accelerate LLM (Large-Language Models) training and inference across diverse hardware platforms. +## FlagTree +FlagTree is an open source, unified compiler for multipleAI chips project dedicated to developing a diverse ecosystem of AI chip compilers and related tooling platforms, thereby fostering and strengthening the upstream and downstream Triton ecosystem. Currently in its initial phase, the project aims to maintain compatibility with existing adaptation solutions while unifying the codebase to rapidly implement single-repository multi-backend support. Forupstream model users, it provides unified compilation capabilities across multiple backends; for downstream chip manufacturers, it offers examples of Triton ecosystem integration. +## FlagScale and vllm-plugin-fl +Flagscale is a comprehensive toolkit designed to supportthe entire lifecycle of large models. It builds on the strengths of several prominent open-source projects, including [Megatron-LM](https://github.com/NVIDIA/Megatron-LM) and [vLLM](https://github.com/vllm-project/vllm), to provide a robust, end-to-end solution for managing and scaling large models. +vllm-plugin-fl is a vLLM plugin built on the FlagOS unified multi-chip backend, to help flagscale support multi-chip on vllm framework. +## **FlagCX** +FlagCX is a scalable and adaptive cross-chip communication library. It serves as a platform where developers, researchers, and AI engineers can collaborate on various projects, contribute to the development of cutting-edge AI solutions, and share their work with the global community. + +## **FlagEval Evaluation Framework** + FlagEval is a comprehensive evaluation system and open platform for large models launched in 2023. It aims to establish scientific, fair, and open benchmarks, methodologies, and tools to help researchers assess model and training algorithm performance. It features: + - **Multi-dimensional Evaluation**: Supports 800+ modelevaluations across NLP, CV, Audio, and Multimodal fields,covering 20+ downstream tasks including language understanding and image-text generation. + - **Industry-Grade Use Cases**: Has completed horizonta1 evaluations of mainstream large models, providing authoritative benchmarks for chip-model performance validation. + +# Contributing + +We warmly welcome global developers to join us: + +1. Submit Issues to report problems +2. Create Pull Requests to contribute code +3. Improve technical documentation +4. Expand hardware adaptation support +# License +The model weights are derived from Tencent-Hunyuan/Hy3 and are open‑sourced under the Apache License 2.0: https://www.apache.org/licenses/LICENSE-2.0.txt + diff --git a/docs/flagrelease_en/model_readmes/FlagRelease_Hy3-nvidia-FlagOS-Express.md b/docs/flagrelease_en/model_readmes/FlagRelease_Hy3-nvidia-FlagOS-Express.md new file mode 100644 index 000000000..bcb8dcc09 --- /dev/null +++ b/docs/flagrelease_en/model_readmes/FlagRelease_Hy3-nvidia-FlagOS-Express.md @@ -0,0 +1,136 @@ +--- +license: apache-2.0 +language: +- zh +- en +--- + +# Introduction +Hy3 is Tencent Hunyuan team's next-generation MoE large model with integrated fast and slow thinking: 295B total parameters, 21B active parameters (plus 3.8B MTP layer parameters), 192 experts with top-8 activation, supporting 256K context. Compared to the Preview version released in late April, Hy3 has achieved a comprehensive leap in intelligence through incorporating real-world business feedback, scaling up RL compute, and improving post-training data quality, significantly outperforming open-source models of similar size. + + +### Integrated Deployment +- Out-of-the-box inference scripts with pre-configured hardware and software parameters +- Released **FlagOS-Nvidia** container image supporting deployment within minutes +### Consistency Validation +- Rigorously evaluated through benchmark testing: Performance and results from the FlagOS software stack are compared against native stacks on multiple public. + + +# Evaluation Results +## Benchmark Result +| Metrics | Hy3-Nvidia-Origin | Hy3-Nvidia-FlagOS | +|--------------|--------------------------------|-------------------| +| GPQA_Diamond | 83.33 | 85.86 | +| arc_challenge_chat | 96.33 | 95.99 | +| math_500 | 94.6 | 93.6 | + +## Performance Benchmark +| Test Scenario | 4k & 1k 64 Concurrent | 16k & 1k 64 Concurrent | 32k & 1k 64 Concurrent | +|----------------------------------------|-----------------------|------------------------|------------------------| +| Speedup Ratio (NV-flagos / NV-native) | 107.19% | 104.59% | 103.23% | + +# User Guide +Environment Setup + +| Item | Version | +|------------------|----------------------| +| Docker Version | Docker version 24.0.0, build 98fdcd7 | +| Operating System | 22.04.4 LTS (Jammy Jellyfish) | + +## Operation Steps + +### Download FlagOS Image +```bash +docker pull harbor.baai.ac.cn/flagrelease-public/hy3-nvidia003-gems5.4.0-tree0.6.0-cxnone-plugin0.2.0-vllm0.20.2-cp312-pt211-cu130-x64-580.95.05:202607021605 +``` + +### Download Open-source Model Weights +```bash +pip install modelscope +modelscope download --model FlagRelease/Hy3-nvidia-FlagOS --local_dir /data/Hy3 +``` + +### Start the Container +```bash +docker run -itd \ + --name flagos \ + --entrypoint /bin/bash \ + --gpus all \ + --ipc=host \ + --net host \ + --shm-size 512g \ + -v /data/:/data \ + harbor.baai.ac.cn/flagrelease-public/hy3-nvidia003-gems5.4.0-tree0.6.0-cxnone-plugin0.2.0-vllm0.20.2-cp312-pt211-cu130-x64-580.95.05:202607021605 +docker exec -it flagos /bin/bash +``` +### Start the Server +```bash +export VLLM_FL_FLAGOS_WHITELIST=invoke_fused_moe_triton_kernel,exponential_ +vllm serve /data/Hy3/ \ + --tensor-parallel-size 8 \ + --port 8000 \ + --gpu-memory-utilization 0.95 \ + --served-model-name hy3 \ + --reasoning-parser hy_v3 +``` + +## Service Invocation +### Invocation Script +```bash +curl http://localhost:8000/v1/chat/completions \ + -H "Content-Type: application/json" \ + -d '{ + "model": "hy3", + "messages": [{"role": "user", "content": "hi!"}] +}' +``` + + +### AnythingLLM Integration Guide + +#### 1. Download & Install + +- Visit the official site: https://anythingllm.com/ +- Choose the appropriate version for your OS (Windows/macOS/Linux) +- Follow the installation wizard to complete the setup + +#### 2. Configuration + +- Launch AnythingLLM +- Open settings (bottom left, fourth tab) +- Configure core LLM parameters +- Click "Save Settings" to apply changes + +#### 3. Model Interaction + +- After model loading is complete: +- Click **"New Conversation"** +- Enter your question (e.g., “Explain the basics of quantum computing”) +- Click the send button to get a response +# Technical Overview +**FlagOS** is a fully open-source system software stack designed to unify the "model–system–chip" layers and foster an open, collaborative ecosystem. It enables a “develop once, run anywhere” workflow across diverse AI accelerators, unlocking hardware performance, eliminating fragmentation among vendor-specific software stacks, and substantially lowering the cost of porting and maintaining AI workloads. With core technologies such as the **FlagScale**, together with vllm-plugin-fl, distributed training/inference framework, **FlagGems** universal operator library, **FlagCX** communication library, and **FlagTree** unified compiler, the **FlagRelease** platform leverages the **FlagOS** stack to automatically produce and release various combinations of \. This enables efficient and automated model migration across diverse chips, opening a new chapter for large model deployment and application. +## FlagGems +FlagGems is a high-performance, generic operator libraryimplemented in [Triton](https://github.com/openai/triton) language. It is built on a collection of backend-neutralkernels that aims to accelerate LLM (Large-Language Models) training and inference across diverse hardware platforms. +## FlagTree +FlagTree is an open source, unified compiler for multipleAI chips project dedicated to developing a diverse ecosystem of AI chip compilers and related tooling platforms, thereby fostering and strengthening the upstream and downstream Triton ecosystem. Currently in its initial phase, the project aims to maintain compatibility with existing adaptation solutions while unifying the codebase to rapidly implement single-repository multi-backend support. Forupstream model users, it provides unified compilation capabilities across multiple backends; for downstream chip manufacturers, it offers examples of Triton ecosystem integration. +## FlagScale and vllm-plugin-fl +Flagscale is a comprehensive toolkit designed to supportthe entire lifecycle of large models. It builds on the strengths of several prominent open-source projects, including [Megatron-LM](https://github.com/NVIDIA/Megatron-LM) and [vLLM](https://github.com/vllm-project/vllm), to provide a robust, end-to-end solution for managing and scaling large models. +vllm-plugin-fl is a vLLM plugin built on the FlagOS unified multi-chip backend, to help flagscale support multi-chip on vllm framework. +## **FlagCX** +FlagCX is a scalable and adaptive cross-chip communication library. It serves as a platform where developers, researchers, and AI engineers can collaborate on various projects, contribute to the development of cutting-edge AI solutions, and share their work with the global community. + +## **FlagEval Evaluation Framework** + FlagEval is a comprehensive evaluation system and open platform for large models launched in 2023. It aims to establish scientific, fair, and open benchmarks, methodologies, and tools to help researchers assess model and training algorithm performance. It features: + - **Multi-dimensional Evaluation**: Supports 800+ modelevaluations across NLP, CV, Audio, and Multimodal fields,covering 20+ downstream tasks including language understanding and image-text generation. + - **Industry-Grade Use Cases**: Has completed horizonta1 evaluations of mainstream large models, providing authoritative benchmarks for chip-model performance validation. + +# Contributing + +We warmly welcome global developers to join us: + +1. Submit Issues to report problems +2. Create Pull Requests to contribute code +3. Improve technical documentation +4. Expand hardware adaptation support +# License +The model weights are derived from Tencent-Hunyuan/Hy3 and are open‑sourced under the Apache License 2.0: https://www.apache.org/licenses/LICENSE-2.0.txt diff --git a/docs/flagrelease_en/model_readmes/FlagRelease_Hy3-tsingmicro-FlagOS.md b/docs/flagrelease_en/model_readmes/FlagRelease_Hy3-tsingmicro-FlagOS.md new file mode 100644 index 000000000..67ee90372 --- /dev/null +++ b/docs/flagrelease_en/model_readmes/FlagRelease_Hy3-tsingmicro-FlagOS.md @@ -0,0 +1,122 @@ +--- +license: apache-2.0 +language: +- zh +- en +--- + +# Introduction +Hy3 is Tencent Hunyuan team's next-generation MoE large model with integrated fast and slow thinking: 295B total parameters, 21B active parameters (plus 3.8B MTP layer parameters), 192 experts with top-8 activation, supporting 256K context. Compared to the Preview version released in late April, Hy3 has achieved a comprehensive leap in intelligence through incorporating real-world business feedback, scaling up RL compute, and improving post-training data quality, significantly outperforming open-source models of similar size. + +### Integrated Deployment +- Out-of-the-box inference scripts with pre-configured hardware and software parameters +- Released **FlagOS-Tsingmicro** container image supporting deployment within minutes +### Consistency Validation +- Rigorously evaluated through benchmark testing: Performance and results from the FlagOS software stack are compared against native stacks on multiple public. + + +# Evaluation Results +## Benchmark Result +| Metrics | Hy3-Nvidia-Origin | Hy3-Tsingmicro-FlagOS | +|--------------|-------------------|-----------------------| +| GPQA_Diamond | 66.16 | Evaluating | +| arc_challenge_chat | 96.33 | Evaluating | +| math_500 | 94.6 | 92 | + + +# User Guide +Environment Setup + +| Item | Version | +|------------------|----------------------| +| Docker Version | Docker version 29.1.3, build 29.1.3-0ubuntu3~22.04.2 | +| Operating System | Ubuntu 22.04.4 LTS | + +## Operation Steps + +### Download FlagOS Image +```bash +docker pull harbor.baai.ac.cn/flagrelease-public/hy3-tsingmicro001-gems4.2.1-treenone-cx0.1.0-plugin0.1.1-vllm0.20.2-cp310-pt24-raisa260629-x64-v260629145101:202607071144 +``` + +### Download Open-source Model Weights +```bash +pip install modelscope +modelscope download --model FlagRelease/Hy3-tsingmicro-FlagOS --local_dir /data/Hy3 +``` + +### Start the Container +```bash +docker run -d -name flagos --network host --shm-size=128g --privileged --ipc=host -v /dev:/dev -v /tmp:/tmp -v /lib/modules:/lib/modules -v /sys:/sys -v /data:/data -it harbor.baai.ac.cn/flagrelease-public/hy3-tsingmicro001-gems4.2.1-treenone-cx0.1.0-plugin0.1.1-vllm0.20.2-cp310-pt24-raisa260629-x64-v260629145101:202607071144 bash +docker exec -it flagos /bin/bash +``` +### Start the Server +```bash +# Enter the inference-related directory +cd /workspace/hy3_env +# Set environment variables, need to execute every time entering the container +source test_env.sh +# Run the inference serve script on 8 machines +bash ./hy3_server.sh +``` + +## Service Invocation +### Invocation Script +```bash +curl http://localhost:8000/v1/chat/completions \ + -H "Content-Type: application/json" \ + -d '{ + "model": "flagOS", + "messages": [{"role": "user", "content": "你好"}] + }' +``` + + +### AnythingLLM Integration Guide + +#### 1. Download & Install + +- Visit the official site: https://anythingllm.com/ +- Choose the appropriate version for your OS (Windows/macOS/Linux) +- Follow the installation wizard to complete the setup + +#### 2. Configuration + +- Launch AnythingLLM +- Open settings (bottom left, fourth tab) +- Configure core LLM parameters +- Click "Save Settings" to apply changes + +#### 3. Model Interaction + +- After model loading is complete: +- Click **"New Conversation"** +- Enter your question (e.g., “Explain the basics of quantum computing”) +- Click the send button to get a response +# Technical Overview +**FlagOS** is a fully open-source system software stack designed to unify the "model–system–chip" layers and foster an open, collaborative ecosystem. It enables a “develop once, run anywhere” workflow across diverse AI accelerators, unlocking hardware performance, eliminating fragmentation among vendor-specific software stacks, and substantially lowering the cost of porting and maintaining AI workloads. With core technologies such as the **FlagScale**, together with vllm-plugin-fl, distributed training/inference framework, **FlagGems** universal operator library, **FlagCX** communication library, and **FlagTree** unified compiler, the **FlagRelease** platform leverages the **FlagOS** stack to automatically produce and release various combinations of \. This enables efficient and automated model migration across diverse chips, opening a new chapter for large model deployment and application. +## FlagGems +FlagGems is a high-performance, generic operator libraryimplemented in [Triton](https://github.com/openai/triton) language. It is built on a collection of backend-neutralkernels that aims to accelerate LLM (Large-Language Models) training and inference across diverse hardware platforms. +## FlagTree +FlagTree is an open source, unified compiler for multipleAI chips project dedicated to developing a diverse ecosystem of AI chip compilers and related tooling platforms, thereby fostering and strengthening the upstream and downstream Triton ecosystem. Currently in its initial phase, the project aims to maintain compatibility with existing adaptation solutions while unifying the codebase to rapidly implement single-repository multi-backend support. Forupstream model users, it provides unified compilation capabilities across multiple backends; for downstream chip manufacturers, it offers examples of Triton ecosystem integration. +## FlagScale and vllm-plugin-fl +Flagscale is a comprehensive toolkit designed to supportthe entire lifecycle of large models. It builds on the strengths of several prominent open-source projects, including [Megatron-LM](https://github.com/NVIDIA/Megatron-LM) and [vLLM](https://github.com/vllm-project/vllm), to provide a robust, end-to-end solution for managing and scaling large models. +vllm-plugin-fl is a vLLM plugin built on the FlagOS unified multi-chip backend, to help flagscale support multi-chip on vllm framework. +## **FlagCX** +FlagCX is a scalable and adaptive cross-chip communication library. It serves as a platform where developers, researchers, and AI engineers can collaborate on various projects, contribute to the development of cutting-edge AI solutions, and share their work with the global community. + +## **FlagEval Evaluation Framework** + FlagEval is a comprehensive evaluation system and open platform for large models launched in 2023. It aims to establish scientific, fair, and open benchmarks, methodologies, and tools to help researchers assess model and training algorithm performance. It features: + - **Multi-dimensional Evaluation**: Supports 800+ modelevaluations across NLP, CV, Audio, and Multimodal fields,covering 20+ downstream tasks including language understanding and image-text generation. + - **Industry-Grade Use Cases**: Has completed horizonta1 evaluations of mainstream large models, providing authoritative benchmarks for chip-model performance validation. + +# Contributing + +We warmly welcome global developers to join us: + +1. Submit Issues to report problems +2. Create Pull Requests to contribute code +3. Improve technical documentation +4. Expand hardware adaptation support +# License +The model weights are derived from Tencent-Hunyuan/Hy3 and are open‑sourced under the Apache License 2.0: https://www.apache.org/licenses/LICENSE-2.0.txt diff --git a/docs/flagrelease_en/model_readmes/FlagRelease_Hy3-zhenwu-FlagOS.md b/docs/flagrelease_en/model_readmes/FlagRelease_Hy3-zhenwu-FlagOS.md new file mode 100644 index 000000000..253b816e5 --- /dev/null +++ b/docs/flagrelease_en/model_readmes/FlagRelease_Hy3-zhenwu-FlagOS.md @@ -0,0 +1,121 @@ +--- +base_model: +- "" +language: +- zh +- en +license: apache-2.0 +--- + +# Introduction +Hy3 is Tencent Hunyuan team's next-generation MoE large model with integrated fast and slow thinking: 295B total parameters, 21B active parameters (plus 3.8B MTP layer parameters), 192 experts with top-8 activation, supporting 256K context. Compared to the Preview version released in late April, Hy3 has achieved a comprehensive leap in intelligence through incorporating real-world business feedback, scaling up RL compute, and improving post-training data quality, significantly outperforming open-source models of similar size. + +### Integrated Deployment +- Out-of-the-box inference scripts with pre-configured hardware and software parameters +- Released **FlagOS-Zhenwu** container image supporting deployment within minutes +### Consistency Validation +- Rigorously evaluated through benchmark testing: Performance and results from the FlagOS software stack are compared against native stacks on multiple public. + + +# Evaluation Results +## Benchmark Result +| Metrics | Hy3-Nvidia-Origin | Hy3-Zhenwu-FlagOS | +|--------------|--------------------------------|--------------------------------------| +| GPQA_Diamond | 83.33 | 85.35 | +| arc_challenge_chat | 96.33 | 96.33 | +| math_500 | 94.6 | 93.8 | + +# User Guide +Environment Setup + +| Item | Version | +|------------------|----------------------| +| Docker Version | Docker version 28.1.0, build 4d8c241 | +| Operating System | Ubuntu 24.04.2 LTS | + +## Operation Steps +The image for this task is exported from Alibaba Cloud PAI and can be used on Alibaba Cloud EAS and DSW, both of which are container‑based resource services. For detailed instructions on how to use this image, please contact the PAI platform support team. The task released by BAAI is developed based on the container environment launched via the PAI platform. + +### Download FlagOS Image +```bash +docker pull harbor.baai.ac.cn/flagrelease-public/hy3-pp001-gems5.4.0-treenone-cxnone-plugin0.2.0-vllm0.20.2-cp312-pt210-hggc130-x64-1.3.2-d7f5a2:202607021622 +``` + +### Download Open-source Model Weights +```bash +pip install modelscope +modelscope download --model FlagRelease/Hy3-zhenwu-FlagOS --local_dir /data/Hy3 +``` + +### Start the Server +```bash +export VLLM_FL_FLAGOS_WHITELIST=attention_backend,grouped_topk,moe_align_block_size,invoke_fused_moe_triton_kernel,silu_and_mul,moe_sum,argmax,rand_like,arange,cat,to,where +export VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=3000 +vllm serve /data/Hy3 \ + --tensor-parallel-size 16 \ + --port 8000 \ + --reasoning-parser hy_v3 \ + --served-model-name hy3 +``` + +## Service Invocation +### Invocation Script +```bash +curl http://localhost:8000/v1/chat/completions \ + -H "Content-Type: application/json" \ + -d '{ + "model": "hy3", + "messages": [{"role": "user", "content": "hi!"}] + }' +``` + + +### AnythingLLM Integration Guide + +#### 1. Download & Install + +- Visit the official site: https://anythingllm.com/ +- Choose the appropriate version for your OS (Windows/macOS/Linux) +- Follow the installation wizard to complete the setup + +#### 2. Configuration + +- Launch AnythingLLM +- Open settings (bottom left, fourth tab) +- Configure core LLM parameters +- Click "Save Settings" to apply changes + +#### 3. Model Interaction + +- After model loading is complete: +- Click **"New Conversation"** +- Enter your question (e.g., “Explain the basics of quantum computing”) +- Click the send button to get a response +# Technical Overview +**FlagOS** is a fully open-source system software stack designed to unify the "model–system–chip" layers and foster an open, collaborative ecosystem. It enables a “develop once, run anywhere” workflow across diverse AI accelerators, unlocking hardware performance, eliminating fragmentation among vendor-specific software stacks, and substantially lowering the cost of porting and maintaining AI workloads. With core technologies such as the **FlagScale**, together with vllm-plugin-fl, distributed training/inference framework, **FlagGems** universal operator library, **FlagCX** communication library, and **FlagTree** unified compiler, the **FlagRelease** platform leverages the **FlagOS** stack to automatically produce and release various combinations of \. This enables efficient and automated model migration across diverse chips, opening a new chapter for large model deployment and application. +## FlagGems +FlagGems is a high-performance, generic operator libraryimplemented in [Triton](https://github.com/openai/triton) language. It is built on a collection of backend-neutralkernels that aims to accelerate LLM (Large-Language Models) training and inference across diverse hardware platforms. +## FlagTree +FlagTree is an open source, unified compiler for multipleAI chips project dedicated to developing a diverse ecosystem of AI chip compilers and related tooling platforms, thereby fostering and strengthening the upstream and downstream Triton ecosystem. Currently in its initial phase, the project aims to maintain compatibility with existing adaptation solutions while unifying the codebase to rapidly implement single-repository multi-backend support. Forupstream model users, it provides unified compilation capabilities across multiple backends; for downstream chip manufacturers, it offers examples of Triton ecosystem integration. +## FlagScale and vllm-plugin-fl +Flagscale is a comprehensive toolkit designed to supportthe entire lifecycle of large models. It builds on the strengths of several prominent open-source projects, including [Megatron-LM](https://github.com/NVIDIA/Megatron-LM) and [vLLM](https://github.com/vllm-project/vllm), to provide a robust, end-to-end solution for managing and scaling large models. +vllm-plugin-fl is a vLLM plugin built on the FlagOS unified multi-chip backend, to help flagscale support multi-chip on vllm framework. +## **FlagCX** +FlagCX is a scalable and adaptive cross-chip communication library. It serves as a platform where developers, researchers, and AI engineers can collaborate on various projects, contribute to the development of cutting-edge AI solutions, and share their work with the global community. + +## **FlagEval Evaluation Framework** + FlagEval is a comprehensive evaluation system and open platform for large models launched in 2023. It aims to establish scientific, fair, and open benchmarks, methodologies, and tools to help researchers assess model and training algorithm performance. It features: + - **Multi-dimensional Evaluation**: Supports 800+ modelevaluations across NLP, CV, Audio, and Multimodal fields,covering 20+ downstream tasks including language understanding and image-text generation. + - **Industry-Grade Use Cases**: Has completed horizonta1 evaluations of mainstream large models, providing authoritative benchmarks for chip-model performance validation. + +# Contributing + +We warmly welcome global developers to join us: + +1. Submit Issues to report problems +2. Create Pull Requests to contribute code +3. Improve technical documentation +4. Expand hardware adaptation support +# License +The model weights are derived from Tencent-Hunyuan/Hy3 and are open‑sourced under the Apache License 2.0: https://www.apache.org/licenses/LICENSE-2.0.txt + diff --git a/docs/flagrelease_en/model_readmes/FlagRelease_LFM2-2.6B-Exp-metax-FlagOS.md b/docs/flagrelease_en/model_readmes/FlagRelease_LFM2-2.6B-Exp-metax-FlagOS.md new file mode 100644 index 000000000..a8cbc6c1c --- /dev/null +++ b/docs/flagrelease_en/model_readmes/FlagRelease_LFM2-2.6B-Exp-metax-FlagOS.md @@ -0,0 +1,108 @@ +# Introduction +新模型介绍，待定.... + +### Integrated Deployment +- Out-of-the-box inference scripts with pre-configured hardware and software parameters +- Released **FlagOS-Metax** container image supporting deployment within minutes +### Consistency Validation +- Rigorously evaluated through benchmark testing: Performance and results from the FlagOS software stack are compared against native stacks on multiple public. + + +# Evaluation Results +## Benchmark Result +| Metrics | LFM2-2.6B-Exp-metax-FlagOS-Origin | LFM2-2.6B-Exp-metax-FlagOS-FlagOS | +|--------------|-----------------------------------|-----------------------------------| +| GPQA_Diamond | 0 | 0 | +| ERQA | - | - | +| Aime24 | - | - | + +# User Guide +Environment Setup + +| Item | Version | +|------------------|----------------------| +| Docker Version | 29.4.0 | +| Operating System | Ubuntu 22.04.3 LTS (Jammy Jellyfish) | + +## Operation Steps + +### Download FlagOS Image +```bash +docker pull harbor.baai.ac.cn/flagrelease-public/lfm2-2.6b-exp-metax001-gems5.0.2-tree0.5.1-cxnone-plugin0.2.0-vllm0.20.2-cp312-pt28-maca37-x64-3.3.12:202607171312-v2 +``` + +### Download Open-source Model Weights +```bash +pip install modelscope +modelscope download --model FlagRelease/LFM2-2.6B-Exp-metax-FlagOS --local_dir /data/LFM2-2.6B-Exp-FlagOS +``` + +### Start the Container +```bash +docker run -itd --name flagos --network=host --device /dev/dri --device /dev/mxcd -v /data:/data harbor.baai.ac.cn/flagrelease-public/lfm2-2.6b-exp-metax001-gems5.0.2-tree0.5.1-cxnone-plugin0.2.0-vllm0.20.2-cp312-pt28-maca37-x64-3.3.12:202607171312-v2 +``` +### Start the Server +```bash +vllm serve /data/LFM2-2.6B-Exp-FlagOS --host 0.0.0.0 --port 8000 --served-model-name LFM2-2.6B-Exp --tensor-parallel-size 1 --max-num-batched-tokens 16384 --max-num-seqs 256 --max-model-len 32768 --trust-remote-code +``` + +## Service Invocation +### Invocation Script +```bash +curl http://localhost:8000/v1/chat/completions \ + -H "Content-Type: application/json" \ + -d '{ + "model": "LFM2-2.6B-Exp", + "messages": [{"role": "user", "content": "你好"}] + }' +``` + + +### AnythingLLM Integration Guide + +#### 1. Download & Install + +- Visit the official site: https://anythingllm.com/ +- Choose the appropriate version for your OS (Windows/macOS/Linux) +- Follow the installation wizard to complete the setup + +#### 2. Configuration + +- Launch AnythingLLM +- Open settings (bottom left, fourth tab) +- Configure core LLM parameters +- Click "Save Settings" to apply changes + +#### 3. Model Interaction + +- After model loading is complete: +- Click **"New Conversation"** +- Enter your question (e.g., "Explain the basics of quantum computing") +- Click the send button to get a response +# Technical Overview +**FlagOS** is a fully open-source system software stack designed to unify the "model–system–chip" layers and foster an open, collaborative ecosystem. It enables a "develop once, run anywhere" workflow across diverse AI accelerators, unlocking hardware performance, eliminating fragmentation among vendor-specific software stacks, and substantially lowering the cost of porting and maintaining AI workloads. With core technologies such as the **FlagScale**, together with vllm-plugin-fl, distributed training/inference framework, **FlagGems** universal operator library, **FlagCX** communication library, and **FlagTree** unified compiler, the **FlagRelease** platform leverages the **FlagOS** stack to automatically produce and release various combinations of \. This enables efficient and automated model migration across diverse chips, opening a new chapter for large model deployment and application. +## FlagGems +FlagGems is a high-performance, generic operator library implemented in [Triton](https://github.com/openai/triton) language. It is built on a collection of backend-neutral kernels that aims to accelerate LLM (Large-Language Models) training and inference across diverse hardware platforms. +## FlagTree +FlagTree is an open source, unified compiler for multiple AI chips project dedicated to developing a diverse ecosystem of AI chip compilers and related tooling platforms, thereby fostering and strengthening the upstream and downstream Triton ecosystem. Currently in its initial phase, the project aims to maintain compatibility with existing adaptation solutions while unifying the codebase to rapidly implement single-repository multi-backend support. For upstream model users, it provides unified compilation capabilities across multiple backends; for downstream chip manufacturers, it offers examples of Triton ecosystem integration. +## FlagScale and vllm-plugin-fl +Flagscale is a comprehensive toolkit designed to support the entire lifecycle of large models. It builds on the strengths of several prominent open-source projects, including [Megatron-LM](https://github.com/NVIDIA/Megatron-LM) and [vLLM](https://github.com/vllm-project/vllm), to provide a robust, end-to-end solution for managing and scaling large models. +vllm-plugin-fl is a vLLM plugin built on the FlagOS unified multi-chip backend, to help flagscale support multi-chip on vllm framework. +## **FlagCX** +FlagCX is a scalable and adaptive cross-chip communication library. It serves as a platform where developers, researchers, and AI engineers can collaborate on various projects, contribute to the development of cutting-edge AI solutions, and share their work with the global community. + +## **FlagEval Evaluation Framework** + FlagEval is a comprehensive evaluation system and open platform for large models launched in 2023. It aims to establish scientific, fair, and open benchmarks, methodologies, and tools to help researchers assess model and training algorithm performance. It features: + - **Multi-dimensional Evaluation**: Supports 800+ model evaluations across NLP, CV, Audio, and Multimodal fields, covering 20+ downstream tasks including language understanding and image-text generation. + - **Industry-Grade Use Cases**: Has completed horizontal evaluations of mainstream large models, providing authoritative benchmarks for chip-model performance validation. + +# Contributing + +We warmly welcome global developers to join us: + +1. Submit Issues to report problems +2. Create Pull Requests to contribute code +3. Improve technical documentation +4. Expand hardware adaptation support +# License +The model weights are derived from LiquidAI/LFM2-2.6B-Exp and are open‑sourced under the Apache License 2.0: https://www.apache.org/licenses/LICENSE-2.0.txt diff --git a/docs/flagrelease_en/model_readmes/FlagRelease_MiniCPM5-1B-kunlunxin-FlagOS.md b/docs/flagrelease_en/model_readmes/FlagRelease_MiniCPM5-1B-kunlunxin-FlagOS.md index 03efdf47b..8c36bdf4d 100644 --- a/docs/flagrelease_en/model_readmes/FlagRelease_MiniCPM5-1B-kunlunxin-FlagOS.md +++ b/docs/flagrelease_en/model_readmes/FlagRelease_MiniCPM5-1B-kunlunxin-FlagOS.md @@ -70,14 +70,17 @@ docker exec -it flagos /bin/bash ``` ### Start the Server ```bash +export VLLM_FL_PLATFORM=kunlunxin +export VLLM_FL_PREFER=flagos +export VLLM_FL_FLAGOS_WHITELIST='rms_norm,silu_and_mul,rotary_embedding' +export FLAGCX_PATH=/env/FlagCX +export CUDA_VISIBLE_DEVICES=0 vllm serve /data/MiniCPM5-1B \ ---trust-remote-code \ ---dtype bfloat16 \ ---enforce-eager \ ---port 8000 \ ---host 0.0.0.0 \ ---served-model-name MiniCPM5-1B \ ---gpu-memory-utilization 0.85 + --served-model-name MiniCPM5-1B \ + --port 8000 \ + --tensor-parallel-size 1 \ + --gpu-memory-utilization 0.95 \ + --enforce-eager ``` ## Service Invocation diff --git a/docs/flagrelease_en/model_readmes/FlagRelease_Phi-3.5-mini-instruct-FlagOS.md b/docs/flagrelease_en/model_readmes/FlagRelease_Phi-3.5-mini-instruct-FlagOS.md new file mode 100644 index 000000000..28fba57a7 --- /dev/null +++ b/docs/flagrelease_en/model_readmes/FlagRelease_Phi-3.5-mini-instruct-FlagOS.md @@ -0,0 +1,108 @@ +# Introduction +新模型介绍，待定.... + +### Integrated Deployment +- Out-of-the-box inference scripts with pre-configured hardware and software parameters +- Released **FlagOS-Nvidia** container image supporting deployment within minutes +### Consistency Validation +- Rigorously evaluated through benchmark testing: Performance and results from the FlagOS software stack are compared against native stacks on multiple public. + + +# Evaluation Results +## Benchmark Result +| Metrics | Phi-3.5-mini-instruct-FlagOS-Origin | Phi-3.5-mini-instruct-FlagOS-FlagOS | +|--------------|-------------------------------------|-------------------------------------| +| GPQA_Diamond | 26.0 | 24.0 | +| ERQA | - | - | +| Aime24 | - | - | + +# User Guide +Environment Setup + +| Item | Version | +|------------------|----------------------| +| Docker Version | 24.0.0 | +| Operating System | Ubuntu 24.04.3 LTS (Noble Numbat) | + +## Operation Steps + +### Download FlagOS Image +```bash +docker pull harbor.baai.ac.cn/flagrelease-public/phi-3.5-mini-instruct-nvidia003-gems5.0.2-tree0.6.0-cxnone-plugin0.2.0-vllm0.20.2-cp312-pt211-cu130-x64-570.158.01:202607160833-v2 +``` + +### Download Open-source Model Weights +```bash +pip install modelscope +modelscope download --model FlagRelease/Phi-3.5-mini-instruct-FlagOS --local_dir /data/Phi-3.5-mini-instruct-FlagOS +``` + +### Start the Container +```bash +docker run -itd --name flagos --gpus=all --network=host -v /data:/data harbor.baai.ac.cn/flagrelease-public/phi-3.5-mini-instruct-nvidia003-gems5.0.2-tree0.6.0-cxnone-plugin0.2.0-vllm0.20.2-cp312-pt211-cu130-x64-570.158.01:202607160833-v2 +``` +### Start the Server +```bash +vllm serve /data/Phi-3.5-mini-instruct-FlagOS --host 0.0.0.0 --port 8000 --served-model-name Phi-3.5-mini-instruct --tensor-parallel-size 1 --max-model-len 32768 --trust-remote-code +``` + +## Service Invocation +### Invocation Script +```bash +curl http://localhost:8000/v1/chat/completions \ + -H "Content-Type: application/json" \ + -d '{ + "model": "Phi-3.5-mini-instruct", + "messages": [{"role": "user", "content": "你好"}] + }' +``` + + +### AnythingLLM Integration Guide + +#### 1. Download & Install + +- Visit the official site: https://anythingllm.com/ +- Choose the appropriate version for your OS (Windows/macOS/Linux) +- Follow the installation wizard to complete the setup + +#### 2. Configuration + +- Launch AnythingLLM +- Open settings (bottom left, fourth tab) +- Configure core LLM parameters +- Click "Save Settings" to apply changes + +#### 3. Model Interaction + +- After model loading is complete: +- Click **"New Conversation"** +- Enter your question (e.g., "Explain the basics of quantum computing") +- Click the send button to get a response +# Technical Overview +**FlagOS** is a fully open-source system software stack designed to unify the "model–system–chip" layers and foster an open, collaborative ecosystem. It enables a "develop once, run anywhere" workflow across diverse AI accelerators, unlocking hardware performance, eliminating fragmentation among vendor-specific software stacks, and substantially lowering the cost of porting and maintaining AI workloads. With core technologies such as the **FlagScale**, together with vllm-plugin-fl, distributed training/inference framework, **FlagGems** universal operator library, **FlagCX** communication library, and **FlagTree** unified compiler, the **FlagRelease** platform leverages the **FlagOS** stack to automatically produce and release various combinations of \. This enables efficient and automated model migration across diverse chips, opening a new chapter for large model deployment and application. +## FlagGems +FlagGems is a high-performance, generic operator library implemented in [Triton](https://github.com/openai/triton) language. It is built on a collection of backend-neutral kernels that aims to accelerate LLM (Large-Language Models) training and inference across diverse hardware platforms. +## FlagTree +FlagTree is an open source, unified compiler for multiple AI chips project dedicated to developing a diverse ecosystem of AI chip compilers and related tooling platforms, thereby fostering and strengthening the upstream and downstream Triton ecosystem. Currently in its initial phase, the project aims to maintain compatibility with existing adaptation solutions while unifying the codebase to rapidly implement single-repository multi-backend support. For upstream model users, it provides unified compilation capabilities across multiple backends; for downstream chip manufacturers, it offers examples of Triton ecosystem integration. +## FlagScale and vllm-plugin-fl +Flagscale is a comprehensive toolkit designed to support the entire lifecycle of large models. It builds on the strengths of several prominent open-source projects, including [Megatron-LM](https://github.com/NVIDIA/Megatron-LM) and [vLLM](https://github.com/vllm-project/vllm), to provide a robust, end-to-end solution for managing and scaling large models. +vllm-plugin-fl is a vLLM plugin built on the FlagOS unified multi-chip backend, to help flagscale support multi-chip on vllm framework. +## **FlagCX** +FlagCX is a scalable and adaptive cross-chip communication library. It serves as a platform where developers, researchers, and AI engineers can collaborate on various projects, contribute to the development of cutting-edge AI solutions, and share their work with the global community. + +## **FlagEval Evaluation Framework** + FlagEval is a comprehensive evaluation system and open platform for large models launched in 2023. It aims to establish scientific, fair, and open benchmarks, methodologies, and tools to help researchers assess model and training algorithm performance. It features: + - **Multi-dimensional Evaluation**: Supports 800+ model evaluations across NLP, CV, Audio, and Multimodal fields, covering 20+ downstream tasks including language understanding and image-text generation. + - **Industry-Grade Use Cases**: Has completed horizontal evaluations of mainstream large models, providing authoritative benchmarks for chip-model performance validation. + +# Contributing + +We warmly welcome global developers to join us: + +1. Submit Issues to report problems +2. Create Pull Requests to contribute code +3. Improve technical documentation +4. Expand hardware adaptation support +# License +The model weights are derived from LLM-Research/Phi-3.5-mini-instruct and are open‑sourced under the Apache License 2.0: https://www.apache.org/licenses/LICENSE-2.0.txt diff --git a/docs/flagrelease_en/model_readmes/FlagRelease_Phi-3.5-mini-instruct-metax-FlagOS.md b/docs/flagrelease_en/model_readmes/FlagRelease_Phi-3.5-mini-instruct-metax-FlagOS.md new file mode 100644 index 000000000..28c3a13b4 --- /dev/null +++ b/docs/flagrelease_en/model_readmes/FlagRelease_Phi-3.5-mini-instruct-metax-FlagOS.md @@ -0,0 +1,108 @@ +# Introduction +新模型介绍，待定.... + +### Integrated Deployment +- Out-of-the-box inference scripts with pre-configured hardware and software parameters +- Released **FlagOS-Metax** container image supporting deployment within minutes +### Consistency Validation +- Rigorously evaluated through benchmark testing: Performance and results from the FlagOS software stack are compared against native stacks on multiple public. + + +# Evaluation Results +## Benchmark Result +| Metrics | Phi-3.5-mini-instruct-metax-FlagOS-Origin | Phi-3.5-mini-instruct-metax-FlagOS-FlagOS | +|--------------|-------------------------------------------|-------------------------------------------| +| GPQA_Diamond | 0 | 28.0 | +| ERQA | - | - | +| Aime24 | - | - | + +# User Guide +Environment Setup + +| Item | Version | +|------------------|----------------------| +| Docker Version | 29.4.0 | +| Operating System | Ubuntu 22.04.3 LTS (Jammy Jellyfish) | + +## Operation Steps + +### Download FlagOS Image +```bash +docker pull harbor.baai.ac.cn/flagrelease-public/phi-3.5-mini-instruct-metax001-gems5.0.2-tree0.5.1-cxnone-plugin0.2.0-vllm0.20.2-cp312-pt28-maca37-x64-3.3.12:202607162145-v2 +``` + +### Download Open-source Model Weights +```bash +pip install modelscope +modelscope download --model FlagRelease/Phi-3.5-mini-instruct-metax-FlagOS --local_dir /data/Phi-3.5-mini-instruct-FlagOS +``` + +### Start the Container +```bash +docker run -v /data:/data -d --name flagos --net=host --ipc=host --privileged --shm-size=64g --group-add video --ulimit memlock=-1 --security-opt seccomp=unconfined --security-opt apparmor=unconfined --device=/dev/dri --device=/dev/mxcd -v /data/models/Phi-3.5-mini-instruct:/data/models/Phi-3.5-mini-instruct harbor.baai.ac.cn/flagrelease-public/phi-3.5-mini-instruct-metax001-gems5.0.2-tree0.5.1-cxnone-plugin0.2.0-vllm0.20.2-cp312-pt28-maca37-x64-3.3.12:202607162145-v2 sleep infinity +``` +### Start the Server +```bash +vllm serve /data/Phi-3.5-mini-instruct-FlagOS --host 0.0.0.0 --port 8000 --served-model-name Phi-3.5-mini-instruct --tensor-parallel-size 1 --max-model-len 32768 --trust-remote-code +``` + +## Service Invocation +### Invocation Script +```bash +curl http://localhost:8000/v1/chat/completions \ + -H "Content-Type: application/json" \ + -d '{ + "model": "Phi-3.5-mini-instruct", + "messages": [{"role": "user", "content": "你好"}] + }' +``` + + +### AnythingLLM Integration Guide + +#### 1. Download & Install + +- Visit the official site: https://anythingllm.com/ +- Choose the appropriate version for your OS (Windows/macOS/Linux) +- Follow the installation wizard to complete the setup + +#### 2. Configuration + +- Launch AnythingLLM +- Open settings (bottom left, fourth tab) +- Configure core LLM parameters +- Click "Save Settings" to apply changes + +#### 3. Model Interaction + +- After model loading is complete: +- Click **"New Conversation"** +- Enter your question (e.g., "Explain the basics of quantum computing") +- Click the send button to get a response +# Technical Overview +**FlagOS** is a fully open-source system software stack designed to unify the "model–system–chip" layers and foster an open, collaborative ecosystem. It enables a "develop once, run anywhere" workflow across diverse AI accelerators, unlocking hardware performance, eliminating fragmentation among vendor-specific software stacks, and substantially lowering the cost of porting and maintaining AI workloads. With core technologies such as the **FlagScale**, together with vllm-plugin-fl, distributed training/inference framework, **FlagGems** universal operator library, **FlagCX** communication library, and **FlagTree** unified compiler, the **FlagRelease** platform leverages the **FlagOS** stack to automatically produce and release various combinations of \. This enables efficient and automated model migration across diverse chips, opening a new chapter for large model deployment and application. +## FlagGems +FlagGems is a high-performance, generic operator library implemented in [Triton](https://github.com/openai/triton) language. It is built on a collection of backend-neutral kernels that aims to accelerate LLM (Large-Language Models) training and inference across diverse hardware platforms. +## FlagTree +FlagTree is an open source, unified compiler for multiple AI chips project dedicated to developing a diverse ecosystem of AI chip compilers and related tooling platforms, thereby fostering and strengthening the upstream and downstream Triton ecosystem. Currently in its initial phase, the project aims to maintain compatibility with existing adaptation solutions while unifying the codebase to rapidly implement single-repository multi-backend support. For upstream model users, it provides unified compilation capabilities across multiple backends; for downstream chip manufacturers, it offers examples of Triton ecosystem integration. +## FlagScale and vllm-plugin-fl +Flagscale is a comprehensive toolkit designed to support the entire lifecycle of large models. It builds on the strengths of several prominent open-source projects, including [Megatron-LM](https://github.com/NVIDIA/Megatron-LM) and [vLLM](https://github.com/vllm-project/vllm), to provide a robust, end-to-end solution for managing and scaling large models. +vllm-plugin-fl is a vLLM plugin built on the FlagOS unified multi-chip backend, to help flagscale support multi-chip on vllm framework. +## **FlagCX** +FlagCX is a scalable and adaptive cross-chip communication library. It serves as a platform where developers, researchers, and AI engineers can collaborate on various projects, contribute to the development of cutting-edge AI solutions, and share their work with the global community. + +## **FlagEval Evaluation Framework** + FlagEval is a comprehensive evaluation system and open platform for large models launched in 2023. It aims to establish scientific, fair, and open benchmarks, methodologies, and tools to help researchers assess model and training algorithm performance. It features: + - **Multi-dimensional Evaluation**: Supports 800+ model evaluations across NLP, CV, Audio, and Multimodal fields, covering 20+ downstream tasks including language understanding and image-text generation. + - **Industry-Grade Use Cases**: Has completed horizontal evaluations of mainstream large models, providing authoritative benchmarks for chip-model performance validation. + +# Contributing + +We warmly welcome global developers to join us: + +1. Submit Issues to report problems +2. Create Pull Requests to contribute code +3. Improve technical documentation +4. Expand hardware adaptation support +# License +The model weights are derived from LLM-Research/Phi-3.5-mini-instruct and are open‑sourced under the Apache License 2.0: https://www.apache.org/licenses/LICENSE-2.0.txt diff --git a/docs/flagrelease_en/model_readmes/FlagRelease_Qwen3.6-27B-metax-FlagOS.md b/docs/flagrelease_en/model_readmes/FlagRelease_Qwen3.6-27B-metax-FlagOS.md index fcf014a7a..fc3d1f1db 100644 --- a/docs/flagrelease_en/model_readmes/FlagRelease_Qwen3.6-27B-metax-FlagOS.md +++ b/docs/flagrelease_en/model_readmes/FlagRelease_Qwen3.6-27B-metax-FlagOS.md @@ -25,6 +25,11 @@ The first open-weight release of Qwen3.6 is now available. Building on the Qwen3 | GPQA_Diamond | 85.86 | 84.26 | | ERQA | 59.25 | 60 | +## Performance Benchmark Result +|Metric| 1k&1k 64 Concurrency| 4k&1k 64 Concurrency| 16k&1k 64 Concurrency| +|--------------|---------------------------|--------------------------|---| +|Equal Computing Power Ratio (flagos/H100)| 96.89% |97.43%| 83.48%| + # User Guide Environment Setup @@ -37,7 +42,7 @@ Environment Setup ### Download FlagOS Image ```bash -docker pull harbor.baai.ac.cn/flagrelease-public/flagrelease-qwen3.6-27b-metax-tree_0.5.1_metax3.0-gems_5.0.2-vllm_0.13.0_empty-plugin_0.1.0_vllm0.13.0-cx_0.8.0-python_3.12.11-torch_2.8.0_metax3.3.0.2-pcp_maca3.3.0.15-gpu_metax001-arc_amd64-driver_3.8.1:202606090203 +docker pull harbor.baai.ac.cn/flagrelease-public/qwen3.6-27b-metax001-gems5.4.0-tree0.5.1-cxnone-plugin0.2.0-vllm0.20.2-cp312-pt28-maca37-x64-3.8.1:202607160136 ``` ### Download Open-source Model Weights @@ -55,39 +60,39 @@ docker run -itd \ --security-opt seccomp=unconfined \ --security-opt apparmor=unconfined \ --shm-size '100gb' \ - --ulimit memlock=-1 \ - --group-add video \ - --device=/dev/dri \ - --device=/dev/mxcd \ - --device=/dev/mem \ - --device=/dev/infiniband \ - -v /usr/local/:/usr/local/ \ - -v /data/:/data/ \ - harbor.baai.ac.cn/flagrelease-public/flagrelease-qwen3.6-27b-metax-tree_0.5.1_metax3.0-gems_5.0.2-vllm_0.13.0_empty-plugin_0.1.0_vllm0.13.0-cx_0.8.0-python_3.12.11-torch_2.8.0_metax3.3.0.2-pcp_maca3.3.0.15-gpu_metax001-arc_amd64-driver_3.8.1:202606090203 \ - /bin/bash + --ulimit memlock=-1 \ + --group-add video \ + --device=/dev/dri \ + --device=/dev/mxcd \ + --device=/dev/mem \ + --device=/dev/infiniband \ + -v /usr/local/:/usr/local/ \ + -v /data/:/data/ \ + harbor.baai.ac.cn/flagrelease-public/qwen3.6-27b-metax001-gems5.4.0-tree0.5.1-cxnone-plugin0.2.0-vllm0.20.2-cp312-pt28-maca37-x64-3.8.1:202607160136 \ + /bin/bash docker exec -it flagos /bin/bash ``` ### Start the Server ```bash FLAGGEMS_VENDOR=metax \ -CUDA_VISIBLE_DEVICES=0,1 \ +CUDA_VISIBLE_DEVICES=2,3 \ VLLM_FL_FLAGOS_WHITELIST=cat,cos,cumsum,fill,full,gather,gt,le,lt,max,mul,sin,softmax,to,where,zeros,zeros_like \ -vllm serve /data/Qwen3.6-27B \ - --tensor-parallel-size 2 --port 8000 --trust-remote-code --dtype bfloat16 \ - --served-model-name flagOS \ - --max-num-batched-tokens 65536 --max-num-seqs 256 --async-scheduling +vllm serve /data/Qwen3.6-27B/ \ + --tensor-parallel-size 2 --port 8000 --trust-remote-code --dtype bfloat16 \ + --served-model-name qwen36-27b \ + --max-num-batched-tokens 16384 ``` ## Service Invocation ### Invocation Script ```bash curl http://localhost:8000/v1/chat/completions \ - -H "Content-Type: application/json" \ - -d '{ - "model": "flagOS", - "messages": [{"role": "user", "content": "你好"}] - }' + -H "Content-Type: application/json" \ + -d '{ + "model": "qwen36-27b", + "messages": [{"role": "user", "content": "你好"}] + }' ``` diff --git a/docs/flagrelease_en/model_readmes/FlagRelease_ZCK-Qwen3-8B-metax-FlagOS.md b/docs/flagrelease_en/model_readmes/FlagRelease_ZCK-Qwen3-8B-metax-FlagOS.md new file mode 100644 index 000000000..d35e618cc --- /dev/null +++ b/docs/flagrelease_en/model_readmes/FlagRelease_ZCK-Qwen3-8B-metax-FlagOS.md @@ -0,0 +1,48 @@ +--- +license: Apache License 2.0 +tags: [] + +#model-type: +##如 gpt、phi、llama、chatglm、baichuan 等 +#- gpt + +#domain: +##如 nlp、cv、audio、multi-modal +#- nlp + +#language: +##语言代码列表 https://help.aliyun.com/document_detail/215387.html?spm=a2c4g.11186623.0.0.9f8d7467kni6Aa +#- cn + +#metrics: +##如 CIDEr、Blue、ROUGE 等 +#- CIDEr + +#tags: +##各种自定义，包括 pretrained、fine-tuned、instruction-tuned、RL-tuned 等训练方法和其他 +#- pretrained + +#tools: +##如 vllm、fastchat、llamacpp、AdaSeq 等 +#- vllm +--- +### 当前模型的贡献者未提供更加详细的模型介绍。模型文件和权重，可浏览“模型文件”页面获取。 +#### 您可以通过如下git clone命令，或者ModelScope SDK来下载模型 + +SDK下载 +```bash +#安装ModelScope +pip install modelscope +``` +```python +#SDK模型下载 +from modelscope import snapshot_download +model_dir = snapshot_download('FlagRelease/ZCK-Qwen3-8B-metax-FlagOS') +``` +Git下载 +``` +#Git模型下载 +git clone https://www.modelscope.cn/FlagRelease/ZCK-Qwen3-8B-metax-FlagOS.git +``` + +

如果您是本模型的贡献者，我们邀请您根据模型贡献文档，及时完善模型卡片内容。

\ No newline at end of file diff --git a/docs/flagrelease_en/model_readmes/FlagRelease_ZCK-Qwen3-8B-nvidia-FlagOS.md b/docs/flagrelease_en/model_readmes/FlagRelease_ZCK-Qwen3-8B-nvidia-FlagOS.md new file mode 100644 index 000000000..5684cc03a --- /dev/null +++ b/docs/flagrelease_en/model_readmes/FlagRelease_ZCK-Qwen3-8B-nvidia-FlagOS.md @@ -0,0 +1,108 @@ +# Introduction +AI21-Jamba-1.5-Mini is an open‑source large language model. This release includes full adaptation for the Ascend platform, along with provided images, cache files, and performance results to facilitate rapid deployment and validation. + +### Integrated Deployment +- Out-of-the-box inference scripts with pre-configured hardware and software parameters +- Released **FlagOS-Nvidia** container image supporting deployment within minutes +### Consistency Validation +- Rigorously evaluated through benchmark testing: Performance and results from the FlagOS software stack are compared against native stacks on multiple public. + + +# Evaluation Results +## Benchmark Result +| Metrics | AI21-Jamba-1.5-Mini-ascend-FlagOS-Nvidia-Origin | AI21-Jamba-1.5-Mini-ascend-FlagOS-Nvidia-FlagOS | +|--------------|--------------------------------|--------------------------------------| +| GPQA_Diamond | - | - | +| ERQA | - | - | +| Aime24 | - | - | + +# User Guide +Environment Setup + +| Item | Version | +|------------------|----------------------| +| Docker Version | Docker version 24.0.0, build 98fdcd7 | +| Operating System | 22.04.4 LTS (Jammy Jellyfish) | + +## Operation Steps + +### Download FlagOS Image +```bash + +``` + +### Download Open-source Model Weights +```bash +pip install modelscope +modelscope download --model FlagRelease/AI21-Jamba-1.5-Mini-ascend-FlagOS --local_dir /data/AI21-Jamba-1.5-Mini-ascend-FlagOS +``` + +### Start the Container +```bash + +``` +### Start the Server +```bash + +``` + +## Service Invocation +### Invocation Script +```bash +curl http://localhost:8000/v1/chat/completions \ + -H "Content-Type: application/json" \ + -d '{ + "model": "flagOS", + "messages": [{"role": "user", "content": "你好"}] + }' +``` + + +### AnythingLLM Integration Guide + +#### 1. Download & Install + +- Visit the official site: https://anythingllm.com/ +- Choose the appropriate version for your OS (Windows/macOS/Linux) +- Follow the installation wizard to complete the setup + +#### 2. Configuration + +- Launch AnythingLLM +- Open settings (bottom left, fourth tab) +- Configure core LLM parameters +- Click "Save Settings" to apply changes + +#### 3. Model Interaction + +- After model loading is complete: +- Click **"New Conversation"** +- Enter your question (e.g., “Explain the basics of quantum computing”) +- Click the send button to get a response +# Technical Overview +**FlagOS** is a fully open-source system software stack designed to unify the "model–system–chip" layers and foster an open, collaborative ecosystem. It enables a “develop once, run anywhere” workflow across diverse AI accelerators, unlocking hardware performance, eliminating fragmentation among vendor-specific software stacks, and substantially lowering the cost of porting and maintaining AI workloads. With core technologies such as the **FlagScale**, together with vllm-plugin-fl, distributed training/inference framework, **FlagGems** universal operator library, **FlagCX** communication library, and **FlagTree** unified compiler, the **FlagRelease** platform leverages the **FlagOS** stack to automatically produce and release various combinations of \. This enables efficient and automated model migration across diverse chips, opening a new chapter for large model deployment and application. +## FlagGems +FlagGems is a high-performance, generic operator libraryimplemented in [Triton](https://github.com/openai/triton) language. It is built on a collection of backend-neutralkernels that aims to accelerate LLM (Large-Language Models) training and inference across diverse hardware platforms. +## FlagTree +FlagTree is an open source, unified compiler for multipleAI chips project dedicated to developing a diverse ecosystem of AI chip compilers and related tooling platforms, thereby fostering and strengthening the upstream and downstream Triton ecosystem. Currently in its initial phase, the project aims to maintain compatibility with existing adaptation solutions while unifying the codebase to rapidly implement single-repository multi-backend support. Forupstream model users, it provides unified compilation capabilities across multiple backends; for downstream chip manufacturers, it offers examples of Triton ecosystem integration. +## FlagScale and vllm-plugin-fl +Flagscale is a comprehensive toolkit designed to supportthe entire lifecycle of large models. It builds on the strengths of several prominent open-source projects, including [Megatron-LM](https://github.com/NVIDIA/Megatron-LM) and [vLLM](https://github.com/vllm-project/vllm), to provide a robust, end-to-end solution for managing and scaling large models. +vllm-plugin-fl is a vLLM plugin built on the FlagOS unified multi-chip backend, to help flagscale support multi-chip on vllm framework. +## **FlagCX** +FlagCX is a scalable and adaptive cross-chip communication library. It serves as a platform where developers, researchers, and AI engineers can collaborate on various projects, contribute to the development of cutting-edge AI solutions, and share their work with the global community. + +## **FlagEval Evaluation Framework** + FlagEval is a comprehensive evaluation system and open platform for large models launched in 2023. It aims to establish scientific, fair, and open benchmarks, methodologies, and tools to help researchers assess model and training algorithm performance. It features: + - **Multi-dimensional Evaluation**: Supports 800+ modelevaluations across NLP, CV, Audio, and Multimodal fields,covering 20+ downstream tasks including language understanding and image-text generation. + - **Industry-Grade Use Cases**: Has completed horizonta1 evaluations of mainstream large models, providing authoritative benchmarks for chip-model performance validation. + +# Contributing + +We warmly welcome global developers to join us: + +1. Submit Issues to report problems +2. Create Pull Requests to contribute code +3. Improve technical documentation +4. Expand hardware adaptation support +# License +The model weights are derived from AI-ModelScope/AI21-Jamba-1.5-Mini and are open‑sourced under the Apache License 2.0: https://www.apache.org/licenses/LICENSE-2.0.txt