Revitalize AppleTrace: faster hook, Perfetto-only, richer events

.gitattributes

@@ -1,4 +1,3 @@
-catapult/* linguist-vendored
 sampledata/* linguist-vendored
 sample/* linguist-vendored
 

.github/workflows/python-tests.yml

@@ -36,3 +36,12 @@ jobs:
         run: chmod +x scripts/test_objc_msgsend_hook.sh scripts/test_objc_msgsend_hook_experimental.sh
       - name: Run objc_msgSend experimental test
         run: ./scripts/test_objc_msgsend_hook_experimental.sh
+
+  batching-stress:
+    runs-on: macos-latest
+    steps:
+      - uses: actions/checkout@v4
+      - name: Make stress script executable
+        run: chmod +x scripts/test_batching_stress.sh
+      - name: Run batched-writer stress test (text + binary)
+        run: ./scripts/test_batching_stress.sh

.gitignore

@@ -11,3 +11,6 @@ appletrace/appletrace.xcodeproj/xcuserdata/
 *.zip
 *.dylib
 build/
+__pycache__/
+*.pyc
+.pytest_cache/

AGENT.md

@@ -3,46 +3,46 @@
 This reference is for AI agents and contributors working inside the AppleTrace repository. It summarizes how the project is organized, how to run and verify changes, and the expectations for contributions.
 
 ## Project Overview
-- AppleTrace instruments iOS apps so you can analyze performance hotspots with Chrome's tracing viewer.
-- Developers can either add manual `APTBeginSection` / `APTEndSection` markers or hook every `objc_msgSend` via HookZz (arm64 only).
-- `merge.py`, `scripts/appletrace_cli.py`, Catapult's `trace2html`, and the helper `go.sh` script transform sandbox data into `trace.json` and `trace.html`.
+- AppleTrace instruments iOS apps so you can analyze performance hotspots in [Perfetto](https://ui.perfetto.dev).
+- Developers can either add manual `APTBeginSection` / `APTEndSection` markers (plus `APTInstant` / `APTCounter` / `APTAsyncBegin` / `APTAsyncEnd` events) or hook every `objc_msgSend` via a fishhook-style direct symbol rebind (arm64 only; see `appletrace/appletrace/src/objc/hook_objc_msgSend.m`).
+- `merge.py` and `scripts/appletrace_cli.py` (and the helper `go.sh`) merge sandbox fragments into a `trace.json` you open directly in Perfetto. Visualization is Perfetto-only; there is no Catapult/Chrome HTML pipeline.
 - Releases bundle a loader tweaked for arm64, but the source can be rebuilt via the included Xcode projects.
 
 ## Repository Map
 - `appletrace/` — Core framework sources (`appletrace.xcodeproj`, Objective-C runtime hooks, exported headers).
 - `loader/` — Loader/packaging project plus `resign.sh` for re-signing the embedded `appletrace.framework`.
-- `sample/ManualSectionDemo` and `sample/TraceAllMsgDemo` — Xcode samples that show manual instrumentation and HookZz-based tracing.
+- `sample/ManualSectionDemo` and `sample/TraceAllMsgDemo` — Xcode samples that show manual instrumentation and automatic `objc_msgSend` tracing.
 - `springboard/AppleTraceSpringBoard` — Additional loader project for SpringBoard-focused experiments.
-- `hookzz/` — Embedded HookZz dependency used to hook `objc_msgSend`.
-- `go.sh`, `merge.py`, `scripts/appletrace_cli.py`, `get_catapult.sh` — Scripts for merging trace files, converting them with Catapult, and downloading Catapult.
-- `sampledata/` — Ready-made traces (`trace.html`) for verifying the visualization pipeline.
-- `release/` — Notes and artifacts for the prebuilt loader (arm64 only).
+- `hookzz/` — Legacy embedded HookZz dependency (the current `objc_msgSend` hook uses a direct symbol rebind instead).
+- `go.sh`, `merge.py`, `scripts/appletrace_cli.py` — Scripts for merging trace fragments into `trace.json` and opening Perfetto.
+- `sampledata/` — Ready-made trace (`trace.json`) for verifying the visualization pipeline in Perfetto.
+- `release/` — Notes and artifacts for the prebuilt loader (arm64).
 - `image/`, `wechat.png` — Documentation assets.
 
 ## Running the Project Locally
 1. **Clone & prerequisites**
    - `git clone https://github.com/everettjf/AppleTrace.git`
-   - Install Xcode, Python 3, Chrome, LLDB, and `ldid` (for re-signing loader builds).
+   - Install Xcode, Python 3, LLDB, and `ldid` (for re-signing loader builds).
    - Optional: `python3 -m pip install -r requirements.txt` for local test tooling.
-   - Run `sh get_catapult.sh` once to fetch Catapult (`catapult/tracing/bin/trace2html` must exist before generating HTML reports).
+   - Visualization is browser-based at [ui.perfetto.dev](https://ui.perfetto.dev); nothing to download.
 2. **Build instrumentation**
    - For manual tracing, open `appletrace/appletrace.xcodeproj`, build the framework, and embed it into your target (see `sample/ManualSectionDemo`).
-   - For automatic tracing, build the HookZz-based dynamic library (see `sample/TraceAllMsgDemo`). This mode must run on arm64 under LLDB.
+   - For automatic tracing, build the dynamic library (see `sample/TraceAllMsgDemo`). This mode runs on arm64 under LLDB.
 3. **Collect data**
    - Run the instrumented app; trace segments are written to `<app sandbox>/Library/appletracedata`.
    - Pull the folder from the Simulator or device.
 4. **Process traces**
-   - Quick path: `sh go.sh <path-to-appletracedata>` to run merge + `trace2html` and open Chrome.
-   - Manual path: `python3 merge.py -d <path>` followed by `python3 catapult/tracing/bin/trace2html <path>/trace.json --output=<path>/trace.html`.
-   - Unified path: `python3 scripts/appletrace_cli.py all <path-to-appletracedata> --open`.
+   - Quick path: `sh go.sh <path-to-appletracedata>` to merge and open Perfetto.
+   - Manual path: `python3 merge.py -d <path>` then drag `<path>/trace.json` into [ui.perfetto.dev](https://ui.perfetto.dev).
+   - Unified path: `python3 scripts/appletrace_cli.py open <path-to-appletracedata>`.
 
 ## Testing
 - Automated coverage exists for the Python trace merge pipeline:
   - `python3 -m pytest tests`
 - Runtime and loader validation is still manual:
-  - Run the sample projects and confirm the generated `trace.json`/`trace.html`.
-  - Inspect traces in Chrome to ensure new instrumentation appears as expected.
-  - When touching the loader or HookZz code, test on a real arm64 device under LLDB.
+  - Run the sample projects and confirm the generated `trace.json`.
+  - Inspect traces in Perfetto to ensure new instrumentation appears as expected.
+  - When touching the loader or hook code, test on a real arm64 device under LLDB.
 
 ## Linting & Formatting
 - No dedicated Objective-C lint/format pipeline exists. Follow existing Objective-C/C/C++/Python conventions in the repo (clang/Xcode defaults, 4-space indentation in Python).
@@ -51,23 +51,24 @@ This reference is for AI agents and contributors working inside the AppleTrace r
 ## Build & Release
 - **Frameworks**: Use `appletrace.xcodeproj` targets. Make sure exported headers remain in `appletrace.framework`.
 - **Loader**: After swapping in a rebuilt framework (`loader/AppleTraceLoader/Package/Library/Frameworks/appletrace.framework`), run `loader/resign.sh` to re-sign with `ldid`.
-- **Catapult**: Keep the downloaded Catapult copy in sync if `trace2html` changes; document updates in README/AGENT when bumping instructions.
-- **Deliverables**: The `release/` folder is arm64-only; highlight this in release notes and README when publishing new binaries.
+- **Visualization**: Traces open in Perfetto (`ui.perfetto.dev`) directly; there is no bundled HTML exporter to keep in sync.
+- **Deliverables**: The `release/` folder targets arm64; highlight this in release notes and README when publishing new binaries.
 
 ## Coding Style & Conventions
 - Prefer concise Objective-C with explicit `APTBeginSection` markers; avoid introducing new macros unless necessary.
 - Keep Objective-C source under `appletrace/src`; Python utilities now live both at repo root (`merge.py`) and under `scripts/` for higher-level workflows.
-- Use descriptive section names inside traces to keep Chrome timelines meaningful.
+- Use descriptive section names inside traces to keep Perfetto timelines meaningful.
 
 ## Debugging
-- Use LLDB breakpoints around the HookZz injection points (`appletrace/src/objc/hook_objc_msgSend.m`) when troubleshooting automatic tracing.
+- Use LLDB breakpoints around the rebinding/wrapper code (`appletrace/appletrace/src/objc/hook_objc_msgSend.m`) when troubleshooting automatic tracing.
 - Inspect intermediate `trace.appletrace` files before merging to ensure data is written.
-- Open `chrome://tracing` with `trace.json` to verify event ordering before generating HTML.
-- Compare against `sampledata/trace.html` if output looks incorrect.
+- Open `trace.json` in [ui.perfetto.dev](https://ui.perfetto.dev) to verify event ordering and timing.
+- Compare against `sampledata/trace.json` if output looks incorrect.
 
 ## Rules for Making Changes
 - Keep changes scoped: avoid mixing instrumentation updates with tooling refactors or documentation tweaks.
-- Maintain compatibility with existing arm64-only assumption unless explicitly widening platform support.
+- Target arm64 only; other architectures (arm64e, x86_64) are out of scope. The `objc_msgSend` hook hard-errors if built for arm64e.
+- Visualization is Perfetto-only — do not reintroduce a Catapult/Chrome HTML pipeline.
 - Update README/AGENT/wiki when changing workflows, scripts, or dependencies.
 - Never remove diagnostic scripts (`merge.py`, `go.sh`) without providing replacements.
 - Preserve existing assets (images, sample traces) so documentation stays accurate.

CONTRIBUTING.md

@@ -0,0 +1,65 @@
+# Contributing to AppleTrace
+
+Thanks for your interest in improving AppleTrace! This guide covers how to get
+set up, what we expect from changes, and how to validate them.
+
+## Getting Started
+
+```bash
+git clone https://github.com/everettjf/AppleTrace.git
+cd AppleTrace
+python3 -m pip install -r requirements.txt
+```
+
+You will need Xcode 12+ (macOS 10.15+) to build the framework and samples, and a
+recent Python 3 for the tooling and tests.
+
+## Project Layout
+
+- `appletrace/` — core tracing framework (Objective-C/C++ runtime, public headers).
+- `appletrace/appletrace/src/objc/hook_objc_msgSend.m` — arm64 `objc_msgSend` hook.
+- `loader/`, `springboard/` — loader/packaging projects.
+- `merge.py`, `scripts/appletrace_cli.py`, `go.sh` — tooling (merge + open in Perfetto).
+- `tests/` — Python regression tests.
+
+See [AGENT.md](AGENT.md) for a deeper map and build/release details, and
+[ROADMAP.md](ROADMAP.md) for where the project is headed.
+
+## Making Changes
+
+- Keep changes scoped: don't mix instrumentation, tooling, and docs in one PR.
+- The `objc_msgSend` hook targets arm64 only — preserve that assumption
+  (arm64e and other architectures are out of scope; the hook hard-errors on arm64e).
+- Avoid adding work to the tracing hot path; prefer caching/interning and
+  per-thread state over per-event allocation.
+- Update `README.md` / `README_CN.md` / `AGENT.md` when workflows or APIs change.
+
+## Testing
+
+```bash
+# Python tooling
+python3 -m pytest tests
+
+# objc_msgSend hook smoke tests (run on a Mac)
+./scripts/test_objc_msgsend_hook.sh
+./scripts/test_objc_msgsend_hook_experimental.sh
+
+# Verify the merge + export pipeline on real data
+python3 merge.py -d <path-to-appletracedata>
+```
+
+When touching the framework, build the relevant Xcode targets and confirm a
+trace renders correctly in [Perfetto](https://ui.perfetto.dev).
+
+## Code Style
+
+- **Objective-C:** [Google Objective-C Style Guide](https://google.github.io/styleguide/objcguide.html)
+- **Python:** [PEP 8](https://www.python.org/dev/peps/pep-0008/)
+- **Shell:** [Google Shell Style Guide](https://google.github.io/styleguide/shellguide.html)
+
+## Pull Requests
+
+1. Fork and create a feature branch.
+2. Make your change with tests where applicable.
+3. Run the test commands above and note any manual testing (device, iOS version).
+4. Open a PR describing the change and its motivation.