Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 11 additions & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,17 @@ jobs:
- name: Python syntax
run: python -m py_compile kit/*.py
- name: JavaScript syntax
run: node --check kit/render.js
run: |
node --check kit/render.js
# scene files are ES modules; parse them as .mjs so --check does not
# treat `import` as a CommonJS syntax error
for f in $(ls kit/lib/*.js films/*/scene.js templates/scene.js 2>/dev/null); do
cp "$f" /tmp/check.mjs
node --check /tmp/check.mjs
echo "ok $f"
done
- name: build.py ships scene-kit.js
run: grep -q "scene-kit.js" kit/build.py
- name: film.json files parse
run: |
python - <<'PY'
Expand Down
38 changes: 35 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -119,6 +119,38 @@ Captions are on by default and follow the voice word by word: the word being spo
{ "captions": true, "captionAccent": "#ffd45e" }
```

## Scenes, cast and lights

A film can cut between locations. `kit/lib/scene-kit.js` provides the pieces a
`scene.js` used to re-invent: a stage manager, a poseable cast, a lighting rig,
set geometry helpers and the HUD.

```js
import * as K from './scene-kit.js';

const app = K.createApp({ w: 1080, h: 1920, shadows: true });
const rig = await K.loadRig('./assets/readyplayer.me.glb');

const stages = K.createStages(app.scene, [
{ key: 'office', t0: -1, t1: T.s3, build: office }, // cut on a measured beat
{ key: 'plain', t0: T.s3, t1: T.s7, build: plain }
]);

const clerk = K.makeHuman(rig, { height: 1.70, top: 0x35507a, skin: 0x9a7150 });
K.poseHuman(clerk, K.POSE.write);
K.speak(clerk, K.mouthAt(envelope, t)); // mouth follows the narration envelope
```

Each stage owns a group, a camera path and optionally its own lights, so five
locations in one film is a list of five entries rather than five engines. Inside a
stage, `shotPicker` cuts between camera setups (wide, close-up, low angle), and
`walkCycle`, `headLook`, `motes`, `snowfall` and `handheld` keep the frame moving. Full
reference in [`docs/scene-kit.md`](docs/scene-kit.md).

What it deliberately does not do: no walk cycles, no IK, no facial animation
beyond the mouth, hard cuts only (no dissolves), and one rig recoloured per
character.

## Quality checks

`qa.py` decodes the finished MP4 at low resolution and reports black frames, blown frames, still runs, hard pops and loudness. It exits non-zero on failure, so CI can gate on it.
Expand All @@ -134,10 +166,10 @@ integrated loudness -14.2 LUFS · true peak -1.6 dBTP

```
kit/ voice, music, mix, render, encode, qa, doctor, build, make
kit/lib/ three.js, loaders, OutlineEffect, Devanagari font, rigged avatar
films/ one directory per film: film.json and scene.js
kit/lib/ three.js, loaders, scene-kit, OutlineEffect, Devanagari font, rigged avatar
films/ one directory per film: film.json and scene.js (built on scene-kit)
templates/ starter film.json, scene.js and index.html
docs/ format reference, styles, safe areas, speed, portability, qa
docs/ format reference, scene-kit, styles, safe areas, speed, portability, qa
assets/ logo and banner
```

Expand Down
8 changes: 8 additions & 0 deletions START-HERE.md
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,14 @@ One sentence per line works best: each line becomes one spoken clip, one caption
- **Slow.** Expected without a GPU. Try `--draft` first.
- **Anything else.** Run `make doctor` for an environment report, or `make test` to check the pipeline on a three-line film. Both print what is broken.


## More than one set

`make.py` writes a single-set film. If your story moves between places, write the
stages yourself with `kit/lib/scene-kit.js` — each stage is a set, a camera path
and its own lights, cut on a measured beat. See
[`docs/scene-kit.md`](docs/scene-kit.md).

## Going further

The file `make.py` writes is ordinary JSON: lines, colours, music cues, captions, safe areas. Edit it and rebuild:
Expand Down
21 changes: 14 additions & 7 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ kit/build.py ──► build/<slug>/site/
timeline.js generated: FPS, DUR, T{}, LINES{}
film.js generated: the whole film.json as FILM
envelope.json generated
three.module.js, jsm/, fonts/, assets/ from kit/lib/
three.module.js, scene-kit.js, jsm/, fonts/, assets/ from kit/lib/

kit/render.js ──► build/<slug>/frames/f00000.jpg …
kit/encode.py ──► build/<slug>/<slug>.mp4
Expand All @@ -29,19 +29,26 @@ timeline.js export const T = { s5: 29.15, ... }
scene.js if (t < T.s5) return camLerp(...)
```

## Layers in the scene
## Layers in a film

1. **Set**, static geometry, built once at load: shop, counter, shelves, street, houses
2. **Cast**, cloned skinned meshes, posed per frame through bones
3. **Camera**, keyed positions blended with easing
4. **HUD**, Canvas 2D drawn *after* the 3D pass, in screen space
1. **Stages**, one per location: a group of static geometry with its own lights and
its own camera path. Only one is visible at a time, so moving between locations
is a visibility change rather than a reload. Built with `kit/lib/scene-kit.js`.
2. **Shots**, inside a stage: `shotPicker` selects the active camera setup — wide,
close-up, low angle — and runs its move, so a location is not one locked-off
camera for its whole duration.
3. **Cast**, cloned skinned meshes from one GLB, posed per frame through bones,
walking with `walkCycle` and speaking with `mouthAt` off `envelope.json`.
4. **Life**, dust motes, snow, drifting far clouds and a few tenths of a degree of
handheld camera drift, so no frame reads as a still.
5. **HUD**, Canvas 2D drawn *after* the 3D pass, in screen space.

The HUD lives in 2D deliberately. 3D text under a 9:16 perspective gets cropped by tight framing and shears at the edges; a 2D pass is always crisp, always correctly placed, and free to render.

## Per-frame pipeline

```
update(t) pose bones, place props, move camera
stages.update(t) pick the stage and the shot, pose bones, move camera
renderer.render() 3D pass (WebGL)
drawHUD(t, ctx2d) 2D pass, cards, captions, progress bar
vignette + fade Canvas 2D gradient, cheap
Expand Down
133 changes: 133 additions & 0 deletions docs/scene-kit.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,133 @@
# scene-kit — stages, cast, lights, HUD

`kit/lib/scene-kit.js` is the reusable half of a film. Before 2.1 every film
re-invented the renderer, the lights, the cast, the camera path and the HUD
inside its own `scene.js`; `templates/scene.js` was 1,100 lines of exactly that,
wired to one shop set. This module is that work, extracted and parameterised.

A film's `scene.js` becomes a description of its sets instead of an engine.

```js
import * as K from './scene-kit.js';

const app = K.createApp({ w: 1080, h: 1920, exposure: 1.22, shadows: true });
const rig = await K.loadRig('./assets/readyplayer.me.glb');
const envelope = await fetch('./envelope.json').then(r => r.json());

const stages = K.createStages(app.scene, [
{ key: 'office', t0: -1, t1: T.s3, build: office },
{ key: 'plain', t0: T.s3, t1: T.s7, build: plain }
]);

const hud = K.makeHud({ film: FILM, lines: LINES, safe: K.SAFE });
K.run(app, { fps: FPS, dur: DUR, hud, update: t => stages.update(t) });
```

## Stages

`createStages(scene, defs)` takes a list of `{ key, t0, t1, build }`. `build(group)`
adds meshes to that stage's group and returns `{ camera(u), update(t, u) }`.

- Only one stage is visible at a time, so a film can cut between locations.
- `t0`/`t1` are seconds, normally beat times from `timeline.js` (`T.s3`), so the
cut follows the measured speech instead of a hand-typed number.
- `camera(u)` gets the stage's local progress 0..1, eased; `update(t, u)` gets
absolute time.
- Cuts are hard. There is no built-in cross-fade; a fade belongs in the HUD.
- Lights inside a stage's group are only active while that stage is visible.

## Cast

```js
const rig = await K.loadRig('./assets/readyplayer.me.glb'); // 67 bones, RPM sample
const clerk = K.makeHuman(rig, {
height: 1.70, top: 0x35507a, bottom: 0x1e2434, skin: 0x9a7150,
hide: ['Wolf3D_Headwear', 'Wolf3D_Beard']
});
clerk.root.position.set(0.75, 0, -3.9);
group.add(clerk.root);

K.poseHuman(clerk, K.POSE.write); // bone quaternions, blended from rest
K.speak(clerk, K.mouthAt(envelope, t)); // mouth from the narration envelope
```

- `makeHuman` clones the rig, converts PBR materials to Lambert, recolours
outfit/skin/hair, scales to a height in metres, drops the feet to y=0 and adds
a contact shadow.
- `POSE` ships `idle, talk, present, pointUp, listen, write, sad, happy`. They are
static blends from the rest pose — no walk cycle, no IK.
- `mouthAt(envelope, t)` reads the RMS envelope `voice.py` already writes, so lip
movement matches the narration with no extra pass.
- `speaking(lines, id, t)` says whether a line is being spoken, which is how a
stage decides who moves their mouth.

## Lights

`K.threePoint(group, opts)` builds key/fill/rim plus ambient and an optional
practical point light. Only the key casts shadows.

```js
K.threePoint(group, {
keyColor: 0xffc98a, keyIntensity: 3.4, keyPos: [3.6, 3.6, 2.2], shadowMap: 512,
fillColor: 0x7d9ac6, fillIntensity: 2.1,
ambientColor: 0x8fa2bd, ambient: 2.3,
practical: { color: 0xffcf94, intensity: 16, distance: 10, pos: [0.6, 2.9, -2.2] }
});
```

Shadow maps are the expensive part on a CPU rasteriser: 512 is enough at 720p, and
`shadows: false` plus the contact shadow is the cheap option that still reads.

## Geometry helpers

`mat`, `flat`, `plane`, `box`, `cyl`, `canvasTex`, `gradientTex`, `softDot`,
`strip` (a height profile over x, triangulated by hand because `ShapeGeometry`
drops vertices on these profiles), `ridgePts`, `snowLine`, `peakGroup`,
`contactShadow`.

## HUD

`makeHud({ film, lines, safe })` draws the title card, steps panel, proof card,
end card, burned-in captions and the vignette inside the platform safe area.
Timing comes from `times: { stepsIn, stepsOut, stepBeats, proofIn, proofOut }`.
Cards sit in the left column (`column: {x, w}`) so they do not cover the subject.

## What this does not do

- No walk cycles, no IK, no facial animation beyond `mouthOpen`/`mouthSmile`.
- One rig, recoloured per character. Distinct faces need distinct GLBs.
- Hard cuts only; no transitions, wipes or dissolves.
- No automatic framing: you place the camera and the props yourself, per stage.

That is the honest boundary. A walking crowd is not something this kit will give
you.

## Motion, shots and life

A locked-off camera on a still set reads as a slideshow. Three additions fix that.

**Shots.** `K.shotPicker([{ t0, t1, cam(u, t) }])` returns `pick(t)`: it selects the
active camera setup and runs its move. A stage can cut several times — wide to
close-up — without becoming a new stage.

```js
const pick = K.shotPicker([
{ t0: -1, t1: T.s2, cam(u, t) { /* wide, walking in */ } },
{ t0: T.s2, t1: 999, cam(u, t) { /* close-up */ } }
]);
return { camera(u, t) { pick(t); }, update(t) { /* ... */ } };
```

**A moving cast.** The rig has no animation clips, so `K.walkCycle(h, t, opts)`
drives the legs and arms procedurally, and `K.headLook(h, yaw, pitch)` turns the
head. Keep character position a pure function of `t`
(`root.position.x = K.lerp(a, b, K.smooth(K.seg(t, t0, t1)))`) so a resumed render
matches an uninterrupted one.

**Life in the frame.** `K.motes()` for dust or far clouds, `K.snowfall()` for snow,
`K.handheld(t, amp)` for a few tenths of a degree of camera drift, and
`K.windowOpening()` to cut a real hole in a wall so the same mountain can be seen
outside every interior — which is what makes five locations feel like one world.

All of it is deterministic: no `Math.random`, no `Date`, no state carried between
frames.
Loading
Loading