Skip to content

Commit f0fbbbe

Browse files
cyyehclaude
andcommitted
Say at the top of both READMEs that the model can also run on vLLM
The opening list still described only transformers on your own machine. - Models: the model also runs on vLLM, in Docker started from the sidebar's What runs the model, or on a server of your own at VLLM_URL - MTP speculative decoding: on vLLM with a GPU both models decode speculatively (Qwen3.5 with its own MTP layer), by default in Docker - The paragraph below: the token details come from your machine or from what the vLLM server sends back, which depends on its flags Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 parent fd746af commit f0fbbbe

2 files changed

Lines changed: 6 additions & 6 deletions

File tree

‎README.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -8,14 +8,14 @@ English | [繁體中文](README.zh-TW.md)
88

99
This project is for learning: it shows how an agent loop works underneath, in a way more intuitive and visual than reading about it. Behind an agent's chat box runs a loop: step after step, the model proposes an action and the harness carries it out, until it stops. A chat app shows only the question and the answer; this page draws every step as it happens, down to each token the model writes, so the whole mechanism is laid out in front of you. Each part of the page matches a section of the Unit 01 reading of [Agent Atlas](https://cyyeh.github.io/agent-atlas-learning/) (in Traditional Chinese), the course this project belongs to, and [Experiments to try](#experiments-to-try) turns those ideas into things you can try yourself.
1010

11-
- Models: pick [google/gemma-4-E2B-it](https://huggingface.co/google/gemma-4-E2B-it) or [Qwen/Qwen3.5-0.8B](https://huggingface.co/Qwen/Qwen3.5-0.8B) in the sidebar; they run on your machine with Hugging Face `transformers`
12-
- MTP speculative decoding: for Gemma 4 E2B you can switch it on in the sidebar (it is off by default); it uses [google/gemma-4-E2B-it-assistant](https://huggingface.co/google/gemma-4-E2B-it-assistant) as the drafter, so one forward pass produces several tokens
11+
- Models: pick [google/gemma-4-E2B-it](https://huggingface.co/google/gemma-4-E2B-it) or [Qwen/Qwen3.5-0.8B](https://huggingface.co/Qwen/Qwen3.5-0.8B) in the sidebar; they run on your machine with Hugging Face `transformers`, or on [vLLM](https://docs.vllm.ai/): pick vLLM (Docker) under What runs the model in the sidebar and the page starts a vLLM server in Docker for you, on the GPU if there is an NVIDIA one and on the CPU otherwise, or set `VLLM_URL` to use a server of your own, on this machine or on a Linux machine with a GPU ([Running the model on a vLLM server](#running-the-model-on-a-vllm-server))
12+
- MTP speculative decoding: one forward pass produces several tokens. With `transformers` you can switch it on in the sidebar for Gemma 4 E2B (it is off by default), with [google/gemma-4-E2B-it-assistant](https://huggingface.co/google/gemma-4-E2B-it-assistant) as the drafter; on vLLM with a GPU both models decode speculatively, Gemma 4 E2B with that drafter and Qwen3.5 with the MTP layer in its own checkpoint, and vLLM in Docker does so by default
1313
- Files: attach images and videos (both models) and sound (Gemma 4 E2B, which also takes a recording from the microphone) with the + in the input box, or pick one of the sample files above it (a receipt, a recording, a video, sales data) and ask your own question about it; each becomes a run of tokens in the context, and the page shows them
1414
- Writing code (optional): Let the model write code in the sidebar gives the model a `run_python` tool, which runs its Python in a Docker sandbox, reads the data files you attach (CSV, TSV, JSON, text, Markdown, Excel) and draws charts with matplotlib, which appear in the chat. Docker is needed for this alone
1515
- Languages: the page and the How it works guide are in English and Traditional Chinese. They start in your browser's language, the switch at the top right changes it, and the language you pick is remembered for your next visit
1616
- License: [Apache License 2.0](LICENSE)
1717

18-
On the left is a chat like ChatGPT's; on the right the agent loop is drawn step by step: the model reads the context and proposes an action, the harness carries it out, and the result goes back into the context, until it stops. The model runs on your own computer, so every token's probability, the candidate tokens, the special tokens, the prefill and the generation speed are all in view. The models run the same harness, so switching models changes only their text formats, which makes them easy to compare.
18+
On the left is a chat like ChatGPT's; on the right the agent loop is drawn step by step: the model reads the context and proposes an action, the harness carries it out, and the result goes back into the context, until it stops. The model runs on your own computer, or on a vLLM server that sends back each token's log probabilities, so every token's probability, the candidate tokens, the special tokens, the prefill and the generation speed are all in view (on vLLM, as much of it as the server's flags let it send). The models run the same harness, so switching models changes only their text formats, which makes them easy to compare.
1919

2020
[How it works](https://cyyeh.github.io/agent-loop-visualizer/en/) maps each step of one turn, from sending a question to the reply, to the code that does it, with file names, line numbers and excerpts. It lives on GitHub Pages; How it works at the top of the app opens it in a new tab, in the language the app is in.
2121

‎README.zh-TW.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -8,14 +8,14 @@
88

99
這個專案是學習用的:用更直覺、更視覺化的方式,讓你看懂 agent 迴圈在底層怎麼運作。Agent 的聊天框背後跑著一個迴圈:一步接著一步,模型提出動作、harness 去執行,直到停止。一般的聊天 app 只讓你看到問題和答案;這個網頁在每一步發生的當下把它畫出來,細到模型寫下的每一個 token,整個機制攤開在你眼前。畫面上的每個部分都對應 [Agent Atlas 技術與實戰交流](https://cyyeh.github.io/agent-atlas-learning/) unit 01 導讀的一節,本專案就是它的專案之一;[可以試試看的實驗](#可以試試看的實驗)則把這些概念變成可以自己動手試的實驗。
1010

11-
- 模型:側欄可以選 [google/gemma-4-E2B-it](https://huggingface.co/google/gemma-4-E2B-it) 或 [Qwen/Qwen3.5-0.8B](https://huggingface.co/Qwen/Qwen3.5-0.8B),在本機用 Hugging Face `transformers` 推論
12-
- MTP 推測解碼:Gemma 4 E2B 可以在側欄打開(預設關閉),用 [google/gemma-4-E2B-it-assistant](https://huggingface.co/google/gemma-4-E2B-it-assistant) 當草稿模型,一次 forward 產生好幾個 token
11+
- 模型:側欄可以選 [google/gemma-4-E2B-it](https://huggingface.co/google/gemma-4-E2B-it) 或 [Qwen/Qwen3.5-0.8B](https://huggingface.co/Qwen/Qwen3.5-0.8B),在本機用 Hugging Face `transformers` 推論,也可以用 [vLLM](https://docs.vllm.ai/) 跑:在側欄的「用什麼跑模型」選 vLLM (Docker),網頁會替你在 Docker 裡啟動一台 vLLM 伺服器,有 NVIDIA GPU 就用 GPU,否則用 CPU;或是設定 `VLLM_URL`,連到你自己的伺服器,在這台電腦上或有 GPU 的 Linux 機器上都行(見[在 vLLM 伺服器上跑模型](#在-vllm-伺服器上跑模型))
12+
- MTP 推測解碼:一次 forward 產生好幾個 token。用 `transformers` 時,Gemma 4 E2B 可以在側欄打開(預設關閉),用 [google/gemma-4-E2B-it-assistant](https://huggingface.co/google/gemma-4-E2B-it-assistant) 當草稿模型;在有 GPU 的 vLLM 上兩個模型都會推測解碼,Gemma 4 E2B 用同一個草稿模型,Qwen3.5 用它自己 checkpoint 裡的 MTP 層,由網頁在 Docker 裡啟動的 vLLM 預設就打開
1313
- 檔案:用輸入框左邊的 + 附上圖片和影片(兩個模型都行)、聲音(Gemma 4 E2B,也可以直接用麥克風錄音),或是在輸入框上方選一個範例檔案(收據、錄音、影片、銷售資料),自己問它問題;每個檔案在脈絡裡變成一串 token,畫面上看得到
1414
- 寫程式(選用):在側欄打開「讓模型寫程式」,模型多一個 `run_python` 工具,它寫的 Python 在 Docker 沙箱裡執行,讀得到你附上的資料檔(CSV、TSV、JSON、純文字、Markdown、Excel),也能用 matplotlib 畫圖,圖會出現在聊天區。只有這個功能需要 Docker
1515
- 語言:網頁和運作說明都有繁體中文和英文版,預設跟著瀏覽器的語言,右上角可以切換;選過的語言會記住,下次打開直接用它
1616
- 授權:[Apache License 2.0](LICENSE)
1717

18-
左邊是像 ChatGPT 的聊天介面,右邊一步一步畫出 agent 迴圈:模型讀進脈絡、提出動作、harness 去執行、結果接回脈絡,直到停止。模型跑在你自己的電腦上,所以每個 token 的機率、候選 token、特殊 token、prefill 和生成速度都看得到。不同模型跑的是同一個 harness,換模型時只有它們的文字格式不同,正好可以對照。
18+
左邊是像 ChatGPT 的聊天介面,右邊一步一步畫出 agent 迴圈:模型讀進脈絡、提出動作、harness 去執行、結果接回脈絡,直到停止。模型跑在你自己的電腦上,或是跑在一台會把每個 token 的 log 機率傳回來的 vLLM 伺服器上,所以每個 token 的機率、候選 token、特殊 token、prefill 和生成速度都看得到(用 vLLM 時,看得到多少要看伺服器開了哪些參數)。不同模型跑的是同一個 harness,換模型時只有它們的文字格式不同,正好可以對照。
1919

2020
[運作說明](https://cyyeh.github.io/agent-loop-visualizer/zh/)把一輪對話從送出到回覆的每一步對應到程式碼,附上檔名、行號和程式片段。它放在 GitHub Pages 上,app 上方的「運作說明」會在新分頁打開它,語言和網頁當下的語言一樣。
2121

0 commit comments

Comments
 (0)