Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 12 additions & 12 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ default-members = [
]

[workspace.package]
version = "0.1.5"
version = "0.1.6"
edition = "2021"
rust-version = "1.88"
authors = ["swananan <jt26wzz@gmail.com>"]
Expand Down
57 changes: 46 additions & 11 deletions README-zh.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
</p>

<p>
<img src="https://img.shields.io/badge/版本-0.1.5-blue.svg" alt="版本"/>
<img src="https://img.shields.io/badge/版本-0.1.6-blue.svg" alt="版本"/>
<img src="https://img.shields.io/badge/协议-GPL-green.svg" alt="协议"/>
<img src="https://img.shields.io/badge/Linux-4.4+-orange.svg" alt="Linux 4.4+"/>
<img src="https://img.shields.io/badge/Rust-1.88.0-red.svg" alt="Rust 1.88.0"/>
Expand All @@ -18,16 +18,17 @@

## 概述

GhostScope 是一个 **面向源码语义的用户态追踪器**。有 DWARF 调试信息时,它可以在不停住目标进程的前提下,按函数、源码行或指令粒度设点,并打印真正重要的信息
GhostScope 是一个 **面向源码语义的用户态追踪器**。有 DWARF 调试信息时,它可以在不停住目标进程的前提下,按函数、源码行或指令粒度设点,打印真正重要的信息,并输出源码感知的调用栈

> *"The most effective debugging tool is still careful thought, coupled with judiciously placed print statements."* — Brian Kernighan

### 什么时候适合用 GhostScope

- 你在排查线上正在运行的服务,不能接受 GDB 式 stop-the-world 带来的巨大性能扰动,同时又更希望使用基于 eBPF 的工作流,获得比传统内核模块式探测更好的安全边界和更低开销。
- 你关心的是源码行和真实变量值,而不只是函数入口参数。
- 你想把“这里要是能加一条 printf 就好了”快速变成一个可运行的追踪脚本。
- 你希望让 AI agent 基于 GhostScope 文档、源码路径和 DWARF 二进制直接产出追踪命令。
- 你需要在源码行级探针里读取真实的局部变量、参数、全局变量和复杂数据,而不只是函数入口参数。
- 你想在同一个 live probe 里既打印关键状态,又输出基于 DWARF unwind 的调用栈 (backtrace)。
- 你需要解释一次请求、函数、源码行或代码路径是如何被执行到的,同时又不能停止目标进程。
- 你想把“这里要是能加一条 printf 就好了”快速变成一个可重复运行的 CLI 追踪脚本。

### 什么时候不适合用 GhostScope

Expand All @@ -54,15 +55,15 @@ curl -fsSL https://raw.githubusercontent.com/swananan/ghostscope/main/scripts/sk

如果你想强制指定目标,可以追加 `--codex`、`--claude` 或 `--all`。安装后需要重启 Codex 或 Claude Code。

当 AI 和你共享同一个 workspace 时,它通常可以自己发现源码 checkout 路径以及 DWARF/调试符号状态。只有在这些信息无法可靠判断时,它才应该先追问,再生成带源码依据的结果。一个典型的会话可以拆成四步
当 AI 和你共享同一个 workspace 时,它通常可以自己发现源码 checkout 路径以及 DWARF/调试符号状态。只有在这些信息无法可靠判断时,它才应该先追问,再生成带源码依据的结果。两个常用 prompt 模式是:先让它展示你关心的变量值,再让它展示解释执行路径的调用栈

这个例子里,AI 可以在本地自行发现的上下文:

- 源码 checkout:`/mnt/500g/code/openresty/openresty-1.27.1.1/build/nginx-1.27.1`
- 目标二进制:`/usr/local/openresty/nginx/sbin/nginx`
- 调试信息状态:二进制内带有 DWARF 调试信息

提问:
#### 提问 1:查看运行时变量

```text
$ghostscope-runtime-analysis 跟踪正在运行的 nginx worker,并把请求 body 的原始字节打印出来
Expand Down Expand Up @@ -91,15 +92,49 @@ trace /mnt/500g/code/openresty/openresty-1.27.1.1/build/nginx-1.27.1/src/http/ng

演示效果:

![GhostScope CLI 演示](assets/demo-cli.gif)
![Prompt 1 CLI 演示](assets/demo-prompt1.gif)

#### 提问 2:补上调用上下文

```text
$ghostscope-runtime-analysis 跟踪正在运行的 nginx worker,展示哪个请求到达了 ngx_http_process_request,并输出源码感知的调用栈
```

生成的命令:

```bash
WORKER_PID=$(pgrep -n -f 'nginx: worker process')
sudo ghostscope -p "$WORKER_PID" --script-file /tmp/ghostscope-nginx-request-stack.gs --script-output pretty
```

生成的脚本:

```ghostscope
trace ngx_http_process_request {
let method_len = r.method_name.len;
let uri_len = r.uri.len;
let line_len = r.request_line.len;

print "nginx pid={} method={:s.method_len$} uri={:s.uri_len$} line={:s.line_len$}",
$pid, r.method_name.data, r.uri.data, r.request_line.data;

bt full;
}
```

这里的 `print` 回答“哪个请求状态到达了这个探针?”,`bt full` 回答“nginx 是沿着哪条调用路径到达这里的?”,并输出基于 DWARF unwind 的源码感知栈帧。

演示效果:

![Prompt 2 CLI 演示](assets/demo-prompt2.gif)

为了得到最好效果,请确保相关源码树可用、你关心的模块带有 DWARF 调试信息,并且 GhostScope 具备加载 eBPF 程序所需的权限。如果这些信息无法在本地可靠发现,skill 应该先追问,再生成带源码依据的追踪结果。拉取仓库更新后,重新执行同一个安装脚本即可;安装的 skill 自带版本号,版本变化时会自动刷新。

### 理想中的 Printf

GhostScope 把编译后的二进制重新变成“可观测系统”。在 TUI 里,这个过程是递进展开的:先定位到感兴趣的函数或源码行,看清楚当前可见的变量,再从那个位置进入 Script Mode 设点,最后一边让目标进程继续运行,一边在输出面板里看实时结果。它不是一个泛泛的监控面板,而是一个沿着源码路径逐步展开的运行时 printf 调试界面。
GhostScope 把编译后的二进制重新变成“可观测系统”。在 TUI 里,这个过程是递进展开的:先定位到感兴趣的函数或源码行,看清楚当前可见的变量,再从那个位置进入 Script Mode 设点,最后一边让目标进程继续运行,一边在输出面板里看变量值和调用栈。它不是一个泛泛的监控面板,而是一个带执行上下文的源码导向运行时 printf 调试界面。

下面这个演示就是按这条路径展开的:先在带 DWARF 的 nginx worker 里找到目标代码,再在对应行写一小段脚本,最后立刻看到条件判断、按源码语义访问变量,以及运行中进程的实时输出。
下面这个演示就是按这条路径展开的:先在带 DWARF 的 nginx worker 里找到目标代码,再在对应行写一小段脚本,最后立刻看到条件判断、按源码语义访问变量、调用栈上下文,以及运行中进程的实时输出。

<br />

Expand All @@ -112,7 +147,7 @@ GhostScope 把编译后的二进制重新变成“可观测系统”。在 TUI

想象一下,你面对的是一片广袤的二进制数据荒野 —— 内存地址、寄存器值、栈帧数据 —— 没有上下文,它们只是毫无意义的数字。**DWARF 调试信息就是我们的地图**:它告诉我们栈地址 `RSP-0x18` 存储着局部变量 `count`,堆地址 `0x5621a8c0` 处的结构体是 `user` 对象,其偏移 `+0x20` 处是字符串指针 `user.name`;它追踪每个变量在程序执行过程中的位置变化 —— 参数 `x` 现在在寄存器 `RDI` 中,之后会被移到栈上 `RSP-0x10` 的位置。

有了这张地图,GhostScope 利用 **eBPF 和 uprobe** 技术从运行中程序的指令点安全地提取二进制数据。这种组合威力强大:DWARF 揭示进程虚拟地址空间中数据的含义,eBPF 安全地获取我们需要的数据。结果呢?你可以在不停止、不修改程序的情况下打印变量值(局部变量或全局变量)、函数参数和复杂数据结构
有了这张地图,GhostScope 利用 **eBPF 和 uprobe** 技术从运行中程序的指令点安全地提取二进制数据。这种组合威力强大:DWARF 揭示进程虚拟地址空间中数据的含义,eBPF 安全地获取我们需要的数据。结果呢?你可以在不停止、不修改程序的情况下打印变量值(局部变量或全局变量)、函数参数、复杂数据结构,以及基于 DWARF unwind 的调用栈

## ✨ 核心特性

Expand Down
57 changes: 46 additions & 11 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
</p>

<p>
<img src="https://img.shields.io/badge/version-0.1.5-blue.svg" alt="Version"/>
<img src="https://img.shields.io/badge/version-0.1.6-blue.svg" alt="Version"/>
<img src="https://img.shields.io/badge/license-GPL-green.svg" alt="License"/>
<img src="https://img.shields.io/badge/Linux-4.4+-orange.svg" alt="Linux 4.4+"/>
<img src="https://img.shields.io/badge/Rust-1.88.0-red.svg" alt="Rust 1.88.0"/>
Expand All @@ -22,16 +22,17 @@

## Overview

GhostScope is a **source-aware userspace tracer** for live Linux processes. With DWARF debug info, it lets you attach at function, source-line, or instruction granularity and print the values that matter without stopping the target.
GhostScope is a **source-aware userspace tracer** for live Linux processes. With DWARF debug info, it lets you attach at function, source-line, or instruction granularity, print the values that matter, and emit source-aware call stacks without stopping the target.

> *"The most effective debugging tool is still careful thought, coupled with judiciously placed print statements."* — Brian Kernighan

### When To Use GhostScope

- You are diagnosing a production service that must stay online: GDB-style stop-the-world debugging would cause too much disruption, and you prefer an eBPF-based workflow with stronger safety boundaries and lower overhead than traditional kernel-module instrumentation.
- You care about source lines and real variable values, not just function entry arguments.
- You want a low-friction path from "I wish I had one more printf here" to a runnable trace script.
- You want an AI agent to turn GhostScope docs, source paths, and DWARF-backed binaries into concrete tracing commands.
- You need source-line probes that can read real locals, parameters, globals, and complex data, not just function entry arguments.
- You want DWARF-unwound stack backtraces from the same live probe that prints the relevant state.
- You need to explain how execution reached a specific source line, function, request, or code path without stopping the process.
- You want a low-friction path from "I wish I had one more printf here" to a repeatable CLI trace script.

### When Not To Use GhostScope

Expand Down Expand Up @@ -59,15 +60,15 @@ If you already have the repository checked out locally, `./scripts/skills/instal

Use `--codex`, `--claude`, or `--all` when you want to force a target. Restart Codex or Claude Code after installation.

When the agent shares your workspace, it can often discover the source checkout path and DWARF/debug-symbol status on its own. If it cannot determine them reliably, it should ask before generating a source-backed trace. A typical session looks like this.
When the agent shares your workspace, it can often discover the source checkout path and DWARF/debug-symbol status on its own. If it cannot determine them reliably, it should ask before generating a source-backed trace. Two useful prompt patterns are: ask for the values you want to see, then ask for the call stack that explains how execution reached that point.

Context the agent can discover locally in this example:

- Source checkout: `/mnt/500g/code/openresty/openresty-1.27.1.1/build/nginx-1.27.1`
- Target binary: `/usr/local/openresty/nginx/sbin/nginx`
- Debug info: embedded DWARF is available

Prompt:
#### Prompt 1: Inspect live values

```text
$ghostscope-runtime-analysis trace the running nginx worker and show the raw request body bytes
Expand Down Expand Up @@ -96,15 +97,49 @@ trace /mnt/500g/code/openresty/openresty-1.27.1.1/build/nginx-1.27.1/src/http/ng

Demo:

![GhostScope CLI demo](assets/demo-cli.gif)
![Prompt 1 CLI demo](assets/demo-prompt1.gif)

#### Prompt 2: Add call context

```text
$ghostscope-runtime-analysis trace the running nginx worker and show which request reached ngx_http_process_request, plus the source-aware call stack
```

Generated command:

```bash
WORKER_PID=$(pgrep -n -f 'nginx: worker process')
sudo ghostscope -p "$WORKER_PID" --script-file /tmp/ghostscope-nginx-request-stack.gs --script-output pretty
```

Generated script:

```ghostscope
trace ngx_http_process_request {
let method_len = r.method_name.len;
let uri_len = r.uri.len;
let line_len = r.request_line.len;

print "nginx pid={} method={:s.method_len$} uri={:s.uri_len$} line={:s.line_len$}",
$pid, r.method_name.data, r.uri.data, r.request_line.data;

bt full;
}
```

Here `print` answers "what request state reached this probe?", while `bt full` answers "how did nginx get here?" using DWARF-unwound, source-aware stack frames.

Demo:

![Prompt 2 CLI demo](assets/demo-prompt2.gif)

For best results, make sure the relevant source tree is available, the modules you care about carry DWARF debug information, and GhostScope has the privileges needed to load eBPF programs. When that information is not discoverable locally, the skill should ask for it before generating a source-backed trace. Re-run the same installer after pulling updates; the installed skill is versioned and refreshes automatically when the version changes.

### The Printf That Should Have Been

GhostScope turns compiled binaries into observable systems. In the TUI, that experience unfolds progressively: first find the function or source line you care about, then inspect the variables visible at that point, enter Script Mode from that location, and finally watch the live output panel update while the target keeps running. It feels less like a generic dashboard and more like source-guided runtime printf debugging.
GhostScope turns compiled binaries into observable systems. In the TUI, that experience unfolds progressively: first find the function or source line you care about, then inspect the variables visible at that point, enter Script Mode from that location, and finally watch values and call stacks update while the target keeps running. It feels less like a generic dashboard and more like source-guided runtime printf debugging with execution context.

The demo below follows exactly that path on a DWARF-enabled nginx worker: locate the code path, drop a trace at the right line, add a small script, and immediately see conditional logic, source-oriented variable access, and live output from the running process.
The demo below follows exactly that path on a DWARF-enabled nginx worker: locate the code path, drop a trace at the right line, add a small script, and immediately see conditional logic, source-oriented variable access, call-stack context, and live output from the running process.

<br />

Expand All @@ -117,7 +152,7 @@ The demo below follows exactly that path on a DWARF-enabled nginx worker: locate

Imagine navigating a vast, uncharted forest of binary data — memory addresses, register values, stack frames — all meaningless numbers without context. **DWARF debug information is our map**: it tells us that stack address `RSP-0x18` stores local variable `count`, heap address `0x5621a8c0` is a `user` object with string pointer `user.name` at offset `+0x20`; it tracks where each variable lives throughout program execution — parameter `x` is in register `RDI` now but will move to stack offset `RSP-0x10` later.

With this map in hand, GhostScope leverages **eBPF and uprobe** technology to safely extract binary data from instruction points in your running program. The combination is powerful: DWARF reveals the meaning of bytes in the process's virtual address space, while eBPF safely retrieves exactly what we need. The result? You can print variable values (local or global), function arguments, and complex data structures without stopping or modifying the program.
With this map in hand, GhostScope leverages **eBPF and uprobe** technology to safely extract binary data from instruction points in your running program. The combination is powerful: DWARF reveals the meaning of bytes in the process's virtual address space, while eBPF safely retrieves exactly what we need. The result? You can print variable values (local or global), function arguments, complex data structures, and DWARF-unwound call stacks without stopping or modifying the program.

## ✨ Highlights

Expand Down
File renamed without changes
Binary file added assets/demo-prompt2.gif
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
2 changes: 1 addition & 1 deletion bins/debuginfod-client/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,6 @@ path = "src/main.rs"
[dependencies]
anyhow.workspace = true
clap.workspace = true
ghostscope-debuginfod = { version = "0.1.5", path = "../../ghostscope-debuginfod" }
ghostscope-debuginfod = { version = "0.1.6", path = "../../ghostscope-debuginfod" }
object.workspace = true
tokio.workspace = true
4 changes: 2 additions & 2 deletions bins/dwarf-tool/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -19,8 +19,8 @@ name = "dwarf-tool"
path = "src/main.rs"

[dependencies]
ghostscope-dwarf = { version = "0.1.5", path = "../../ghostscope-dwarf" }
ghostscope-process = { version = "0.1.5", path = "../../ghostscope-process" }
ghostscope-dwarf = { version = "0.1.6", path = "../../ghostscope-dwarf" }
ghostscope-process = { version = "0.1.6", path = "../../ghostscope-process" }

# Command line parsing
clap.workspace = true
Expand Down
2 changes: 1 addition & 1 deletion docs/install.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ Restart the terminal and run `ghostscope --version` to verify.

2. Extract the binary:
```bash
tar -xzf ghostscope-v0.1.5-x86_64-linux.tar.gz
tar -xzf ghostscope-v0.1.6-x86_64-linux.tar.gz
```

3. Move to system path (optional):
Expand Down
6 changes: 6 additions & 0 deletions docs/tutorial.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,12 @@ Then ask for a concrete tracing workflow:
$ghostscope-runtime-analysis trace the running nginx worker and show the raw request body bytes
```

When you need call context instead of only values, ask for the stack explicitly:

```text
$ghostscope-runtime-analysis trace the running nginx worker and show which request reached ngx_http_process_request, plus the source-aware call stack
```

This works especially well when you want a repeatable CLI flow with `--script-file`. After the AI gives you a starting point, the TUI is still the best place to inspect source, tweak trace points, and iterate on the script interactively. If you want the hands-on details of using GhostScope itself, continue with the tutorial below.

### Launch GhostScope
Expand Down
Loading
Loading