Sr. Staff AI/ML Engineer, 10+ years. I build multi-agent systems that ship, and I measure them before I believe them.
Built work, with receipts. Every project below runs, and every claim traces to an eval, a probe, or a logged run. The receipts that proved me wrong are here too.
A wrong number in a dashboard fails loudly. A generative system fails plausibly: a mouth trailing the audio by four frames, a cited statistic with nothing behind it. Those are invisible to a type check, obvious to a person, and different again tomorrow. So I capture human judgment as labels, compile it into thresholds, and wire it into gates that can refuse to spend.
| if you want | go to |
|---|---|
| one project, read deeply | the featured pipeline |
| range across problems | the table of five |
| how I work | principles I work by |
3d-filmmaking-ads-multimodal-evals
13 probes · 16 of 16 gating thresholds derived from labelled exemplars · 4 guards, 3 of which fail open and say so · 1 credit per scheduled render · 77 views per frame
An AI filmmaking pipeline for advertising-grade avatar video: multimodal evals govern a cloned voice, a consistent generated character, and light-field 3D output. Taste is captured as labels, compiled into thresholds, and enforced by gates that block bad output before it spends money.
It runs on a schedule, against metered vendor APIs, with nobody awake. That is the design constraint: an unattended pipeline cannot be corrected mid flight, so every gate fires before the credit is spent, and anything short of a clean success pages loudly instead of passing quietly.
The repo publishes the measurements that falsified my own claims, including two failures the shipped render still does not pass, and a speedup whose number held while the explanation I wrote for it turned out to be wrong.
Ranked by priority, one per repository, nothing listed twice.
| Focus | Project | What it solves | Who benefits |
|---|---|---|---|
Voice AIConversational AI Realtime ASR + TTS Twilio PSTN LLM‑as‑Judge Evals |
realtime-voice-agent-turn-taking-stack | Answering a real phone line with no human on it, and knowing when to stop talking | Support, scheduling, and anyone shipping voice agents |
Reinforcement LearningMulti‑Agent RL PPO PyTorch FastAPI |
multi-agent-rl-mapf-drone-navigation | Many drones planning conflict free paths at once, in real time | Robotics, drone, and simulation teams |
LLM EvaluationPrompt Engineering Self‑Hosted Llama Ablation Benchmarks Statistical Analysis |
self-hosted-llm-evals-lab | Knowing which prompt strategy actually wins, by statistics instead of vibes | ML platform and LLM eval teams |
RAG PipelinesLLM Orchestration Claude + GPT APIs NLP Extraction Document Processing |
research_doc_extraction_rag_agent | Thousands of open ended survey answers nobody has time to read | Research, insights, and product teams |
MCP ServersModel Context Protocol Tool Design OAuth 2.1 Self‑Hosted Atlassian |
onprem-bitbucket-jiraconfluence-mcp | AI agents that cannot reach self-hosted Bitbucket, Jira, or Confluence | Enterprise platform and developer experience teams |
A suite only covers what someone thought to measure. Eleven probes scored the face, the motion and the timing in a rendered frame. None asked whether the subject was visible against the background. A dark top on a black matte separated by 22 levels of luma where the face separated by 134, so the torso dissolved and the render shipped a floating head. Every probe passed. The gap was not a bug in any of them; it was a question nobody had written down.
Three green checkmarks over two empty files. In a multi-agent system every step reported success and both artifacts persisted empty. The agent cards advertised localhost, correct on one machine and wrong across a container network, so discovery kept succeeding while delivery failed silently. The client now rebases the address and logs that it did, and every artifact writes its byte size.
- Chat output is not evidence. The database write, the rendered file, and the logged artifact size are the evidence.
- Degrade loudly, never silently. Every fallback is logged; a green checkmark over an empty artifact is the worst bug in agentic systems.
- Spend where the artifact is judged. Flagship models on deliverables, fast models on hot paths; cost per accepted deliverable beats cost per token.
- Languages: Python, TypeScript, Bash
- LLMs and agents: Multi-agent systems, agentic AI, LangGraph, MCP (Model Context Protocol), A2A (agent-to-agent), RAG, tool design, prompt engineering
- Evaluation: LLM evaluation, LLM-as-a-judge, guardrails, ablation studies, statistical analysis, regression testing
- Voice and speech: Conversational AI, voice AI, real-time ASR and TTS, speech recognition, text-to-speech, Twilio, PSTN, WebSocket, voice cloning
- Machine learning: Reinforcement learning, PPO, PyTorch, NLP, generative AI, computer vision, monocular depth estimation
- Retrieval: Vector search, embeddings, semantic search, reranking
- Platform and MLOps: MLOps, FastAPI, Docker, CI/CD, Cloudflare Workers, observability
- Models: Claude (Anthropic), GPT (OpenAI), Llama
Sr. Staff / Principal AI engineering: agentic systems, LLM evals, and the guardrails that make them safe to run unattended. San Francisco, or remote.
Role titles I map to: Staff AI Engineer, Principal AI Engineer, Applied AI Engineer, Machine Learning Engineer, AI/ML Engineer, Forward Deployed Engineer (FDE).
The fastest way to evaluate me is to not take my word for it. No accounts, no API keys,
no GPU. Needs python3 and ffmpeg:
git clone https://github.com/jameswniu/3d-filmmaking-ads-multimodal-evals
cd 3d-filmmaking-ads-multimodal-evals && pip install -r requirements.txt
python3 evals/derive.py
It prints every gating threshold in that pipeline beside the labelled pass and the labelled reject that bracket it, recomputes the shipped exemplars rather than reciting them, and exits nonzero if any constant has drifted outside its own evidence.
Hiring, or want the walkthrough: LinkedIn | +1 (917) 355-7504