df4e06f047
- stage: detectVisemes() maps CC/ARKit/plain morph names -> canonical
A/E/I/O/U/MBP/FV/L at character load; visemeTargets(id) + setMorph(id,name,w)
for the timeline to drive lip sync; visemeSelfCheck() asserts the matcher
- dock: inspector 🗣 badge (greyed when a rig has no mouth morphs)
- stage: camera/light default source:{type:'none'} (PLAN §4.1); applyState
skips only upload mesh/backdrop bytes -> rig survives save/load (SYNC4)
Verified live at :8020: selfcheck ok=[A,FV,L,MBP,U]; save/load keeps cam+light.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
188 lines
13 KiB
Markdown
188 lines
13 KiB
Markdown
# Lane A log — append-only, newest at bottom
|
||
|
||
Format per session:
|
||
|
||
## <date> session N
|
||
DONE: …
|
||
DECISIONS: …
|
||
BLOCKED/REQUESTS: …
|
||
NEXT: …
|
||
|
||
(no sessions yet)
|
||
|
||
## 2026-07-18 session 1
|
||
DONE: M1 shipped and verified end-to-end in a browser.
|
||
- `web/room3d.js` — ported the rigroom retarget as a pure ES module (no DOM/globals).
|
||
Exports parseAny/canon/boneMap/captureRest/bakeRetarget/disposeRoot/stats + `_selftest`.
|
||
bakeRetarget is now `(clip, srcRoot, srcRest, tgtRoot, tgtRest, opts?)` — target is a
|
||
parameter instead of the closed-over `char`. Algorithm kept byte-for-byte (30fps world-delta
|
||
bake, hips-height ratio, quat sign continuity, dup-hierarchy dedupe). Added `.bvh` to parseAny.
|
||
- `web/stage.js` — `class Stage` per PLAN §4.2. IBL (RoomEnvironment+PMREM), grid+circle floor,
|
||
OrbitControls director cam, TransformControls gizmo (W/E/R), raycast select, shadow-casting.
|
||
addEntity handles character/prop/backdrop(plane|corner|dome)/camera/light; wrapper-Group holds
|
||
all transforms, loaded root normalized+grounded underneath. prepareClip caches parse-per-path and
|
||
bake-per-(entity,path,index). captureState/applyState round-trip entities.
|
||
- `web/dock.js` — asset dock (Characters/Props/Backdrops/Animations tabs from `/assets/tree`,
|
||
offline→file-drop) + inspector (transform/params editors, ⏺key dispatches `scenegod:capturekey`,
|
||
delete). Local drop: image→backdrop, mesh→character, clip-onto-selected-character→bake+play.
|
||
- `web/index.html` + `web/style.css` — CSS-grid layout (dock/view/inspector/timeline), rigroom
|
||
importmap (three 0.160.0), `?selftest=1` badge. Timeline loaded via guarded dynamic import so the
|
||
page never breaks when Lane B's file is absent.
|
||
|
||
VERIFICATION (browser, in-app):
|
||
- `?selftest=1`: PASS — two 3-bone skeletons, different rests+namespaces, baked 1s clip,
|
||
world-delta match maxErr=6.4e-12, srcMoved=0.224rad (guard against a silently-frozen source).
|
||
- Real end-to-end (Female_mixamo.fbx char + "Running.fbx" clip + a jpg backdrop, served from a
|
||
throwaway scratchpad harness): char loaded (65 bones, 18.6k tris, normalized), gizmo-moved,
|
||
clip retargeted (66 tracks) and animating live. Screenshot: `logs/shots/m1-lady-running.png`.
|
||
- Page boots with zero console errors with BOTH timeline.js and the server absent.
|
||
|
||
DECISIONS:
|
||
- selftest uses underscore namespaces (`mixamorig1_`/`mixamorig4_`), not colon — a colon in a bone
|
||
name breaks three's PropertyBinding so the source clip wouldn't bind (silent pass). canon() still
|
||
collapses both to the same key, which is the thing under test.
|
||
- Added `Stage.prepareClipUpload(id, buf, name, idx)` (bake from in-memory bytes) so the offline
|
||
file-drop path can retarget without a server. `Stage.setMixerAuto(false)` lets Lane B take over
|
||
mixer stepping at SYNC 1 (default true so dropped clips preview standalone now).
|
||
- backdrop param edits (mode/image/width) don't rebuild the mesh yet — marked M2. fov/light
|
||
color/intensity are live.
|
||
|
||
BLOCKED/REQUESTS:
|
||
- @Lane C: please confirm the item shape inside `/assets/tree` arrays. dock.js tolerates both a
|
||
bare path string and `{path,name,files:[{path,ext}]}` (rigroom-style) for now — tell me the real
|
||
one and I'll drop the fallback. Also confirm the page is served such that absolute `/web/*.js`
|
||
imports resolve (I used `/web/...` to be URL-independent).
|
||
|
||
NEXT (M2, do not start until orchestrator flips the milestone line):
|
||
- multiple cameras + PiP overlay via renderActiveCamera, light entity param plumbing polish,
|
||
backdrop rebuild-on-param-change, drag-from-dock onto viewport (currently click-to-add).
|
||
|
||
## 2026-07-18 session 2 (M2)
|
||
DONE: M2 Lane-A scope shipped and verified in the *integrated* app (Lane C server :8020 +
|
||
Lane B timeline/tlui mounted). Worked the orchestrator's priority order:
|
||
- **P0 /assets/tree shape** — dropped the bare-string fallback; dock.js now reads the confirmed
|
||
`{name, path, formats, thumb}` and renders `thumb` (→ `/assets/file?path=`) when present.
|
||
- **P1 eventize clip drops** — `dock._applyClip(id,path,idx)`: when `window.timeline` exists it
|
||
dispatches `scenegod:clipdrop {detail:{id,path,clipIndex}}` (Lane B places the block); only
|
||
bakes+previews standalone when no timeline. Uploaded/local clip files can't hand off a server
|
||
path, so with a timeline up they toast a "use the dock" note instead of a dead preview.
|
||
- **P2 onChange on every entity** — `Stage.addEntity`/`removeEntity` now fire `_fireChange(en)`,
|
||
so tlui's `onChange→_syncRows` builds a row per entity, incl. programmatic `applyState` loads.
|
||
Fixed the empty-names bug on both add and remove. Also fixed a latent bug: the gizmo `mouseUp`
|
||
fired `_fireChange` with the *id* not the entity, so the inspector never refreshed on drag.
|
||
- **P4 cameras/lights/PiP/backdrop/drag**:
|
||
- `Stage` PiP: overlay `<canvas class="pip">` in `#view`, rendered each frame from the active
|
||
cam (render active→main canvas, blit to pip, then render director last so the main view is
|
||
correct). `setActiveCamera` shows/hides it. Verified 16,495 lit pixels (real content).
|
||
- `renderActiveCamera(canvas|null)` hardened as the M3 final-render target; added
|
||
`pause()/resume()` so Lane C can freeze the director loop and drive frames deterministically.
|
||
- dock "add rig" toolbar: **+ camera / + key light / + ambient** (these have no asset file);
|
||
inspector gains **◉ set active camera** for cameras; light `castShadow` param plumbed.
|
||
- `setParam` rebuilds the backdrop mesh on mode/image/width change (`_rebuildBackdrop`).
|
||
- drag-from-dock onto the viewport: cards are `draggable`; `#view` drop places chars/props at
|
||
the cursor (`screenToFloor`) and bakes clips onto the character under the cursor (`entityAt`).
|
||
|
||
VERIFICATION (in-app, :8020, zero console errors throughout):
|
||
- Added man + key light + ambient + camera via toolbar → all 4 appear as tlui rows immediately
|
||
(P2 proven). Man lit by the scene lights. Screenshot: `logs/shots/m2-cameras-lights-dome.png`.
|
||
- Clicked ANIMATIONS→hiphop on the selected man → `scenegod:clipdrop` fired with
|
||
`{id:'e1', path:'animations/test/hiphop.fbx', clipIndex:0}` (P1 proven). Lane B DOES consume it
|
||
(timeline.js:98) but the block doesn't land — see the integration gap below.
|
||
- capturekey button → `scenegod:capturekey {id:'e1'}`. backdrop plane→dome rebuild swapped the
|
||
root mesh with no error.
|
||
|
||
DECISIONS:
|
||
- PiP renders the active cam into the *main* WebGL canvas then blits to the 2D pip canvas (one
|
||
shared context); director view is rendered last each frame so it always wins the main canvas.
|
||
Same primitive is the M3 render path: `pause()` the loop, `renderActiveCamera(null)`, read pixels.
|
||
- Local/upload clip files are a no-op-with-note when a timeline is mounted (no server path to place
|
||
a block). ponytail: revisit only if John wants drag-in clip files to become timeline blocks.
|
||
|
||
BLOCKED/REQUESTS — INTEGRATION GAP (orchestrator, please route):
|
||
- Live dock/stage-added entities never reach the timeline's model. `Timeline.scene.entities`
|
||
starts `[]` (timeline.js:80) and is only filled by `timeline.load(sceneJson)`. Both
|
||
`scenegod:clipdrop`→`addClipDrop` and `scenegod:capturekey`→`addKey` call `_tracks(id)`, which
|
||
throws `no entity <id>` for anything added through the dock. Verified live at :8020:
|
||
`manOnStage=true, inTimelineModel=false, addClipDropErr="no entity e1"`. So on today's build,
|
||
keyframing/clip-dropping a dock-added entity is dead — it only works after a scene `load()`.
|
||
- This is a cross-lane seam, not a Lane-A-file fix (timeline.js is Lane B's). The natural hook is
|
||
already in place on my side: **`stage.onChange(en)` now fires on every add AND remove**
|
||
(my P2 fix). Suggested resolution (Lane B, ~4 lines): in the Timeline ctor,
|
||
`stage.onChange(en => this._mirror(en))` where `_mirror` upserts/removes a `scene.entities`
|
||
entry from `{id,kind,label,source,params, transform: stage.entityTransform(id), tracks:{}}`.
|
||
I can instead dispatch a dedicated `scenegod:entityadd/entityremove` event from the dock if the
|
||
orchestrator prefers an explicit contract over reusing onChange — say the word and I'll add it.
|
||
|
||
NEXT: once the entity-mirror seam is decided, the "lady + man + street + 2 cameras + sunset" SYNC-2
|
||
demo should play. Holding at the M2 boundary until the orchestrator flips the milestone line.
|
||
|
||
## 2026-07-18 session 3 (M3-polish)
|
||
DONE: both M3-polish items the orchestrator flagged, verified live at :8020.
|
||
- **PiP cap + swap.** Was fixed 256×144 and (with the CSS `aspect-ratio`/`%` combo) blew up to
|
||
340×413 portrait on a `<canvas>` — CSS aspect-ratio is unreliable on replaced elements. Now sized
|
||
in `_resize()`: `clamp(180, view.width*0.24, 340)` wide × 16:9, bottom-right. Click the inset to
|
||
swap main↔active-cam (`_pipSwap`). Refactored `renderActiveCamera`→ private `_render(cam,canvas)`
|
||
that restores the cam's aspect after a pip blit, so swapping never distorts either camera
|
||
(verified: dirCam.aspect stays 1.79 while it's shown in the inset). `renderActiveCamera(canvas)`
|
||
is now a one-line wrapper over `_render(this._activeCam(), canvas)` — M3 contract unchanged;
|
||
`pause()/resume()` untouched. Verified 180×101 @1.78 ratio, swap flips the views cleanly.
|
||
- **lady.glb flat-white diagnosed + fixed.** It's the ASSET: a Reallusion/Character-Creator export
|
||
with `extensionsRequired:["KHR_materials_pbrSpecularGlossiness"]` — diffuse lives in that
|
||
extension, which three r160 dropped, so all 26 materials fell back to white MeshStandard. (rigroom
|
||
would show the same — no port regression.) Since SCENEGOD must eat any dropped character, added a
|
||
minimal `SpecGlossPlugin` registered on GLTFLoader in room3d.js: spec-gloss → MeshStandard
|
||
(diffuse→map/color, roughness≈1−glossiness, metalness 0). Only touches materials carrying the
|
||
extension; metallic-rough GLBs and FBX untouched. Verified: lady loads with withMap=26/26 and
|
||
renders fully textured (skin/clothes/face). Shot: `logs/shots/m3-pip-swap.png`.
|
||
|
||
DECISIONS:
|
||
- Runtime spec-gloss shim over an offline gltf-transform convert: keeps CC characters "just work"
|
||
in the drop-on-stage flow and in the render pipeline, no per-asset conversion step. Approximation
|
||
(ignores the specular map) is fine for machinima lighting; upgrade path = full spec→metal convert
|
||
if a hero asset needs it.
|
||
- PiP size lives in JS, not CSS — `<canvas>` + `aspect-ratio` is the trap that produced the giant
|
||
inset. CSS keeps only position/border.
|
||
|
||
BLOCKED/REQUESTS: none.
|
||
|
||
NEXT: hold at M3-polish boundary. Available for SYNC-3 findings (finalRender drives
|
||
`pause`/`resume`/`renderActiveCamera` — kept stable per the hard contract).
|
||
|
||
## 2026-07-18 session 4 (SYNC4 fix + M4-A visemes)
|
||
DONE:
|
||
- **applyState/source clean fix.** Absorbed the orchestrator's SYNC4 hotfix and did the schema-correct
|
||
version: `addEntity` defaults camera/light `source` to `{type:'none'}` (PLAN §4.1); `captureState`
|
||
emits it; `applyState` skips only entities that need bytes it lacks (`type:'upload'` AND
|
||
kind∈{character,prop,backdrop}), so `'none'` always rebuilds and legacy `'upload'` cams still
|
||
rebuild. Verified live: save→load of {character, camera, light} → srcTypes
|
||
`[character:assets, camera:none, light:none]`, all 3 come back (rig no longer lost).
|
||
- **M4-A viseme lane** (my deliverable = detection + setMorph hook + badge; Rhubarb→keys is Lane B):
|
||
- `detectVisemes(root)` (stage.js): scans every skinned mesh's `morphTargetDictionary`, loosely
|
||
maps names → canonical visemes A/E/I/O/U/MBP/FV/L (covers CC `V_Open/V_Explosive/V_Dental_Lip/
|
||
V_Tongue_*`, ARKit `jawOpen/mouthPucker/mouthFunnel/mouthClose`, plain vowels). Built once at
|
||
character load; stored on the entity as `{morphs, visemes}` (head/teeth/tongue meshes sharing a
|
||
morph name are grouped so one call drives them all).
|
||
- `stage.visemeTargets(id)` → canonical names present; `stage.setMorph(id, name, weight)` drives a
|
||
canonical viseme OR a raw morph name across all mapped meshes.
|
||
- inspector 🗣 badge (dock.js): lists detected visemes, greyed `🗣 no visemes` when the rig has none.
|
||
- `visemeSelfCheck()` exported: synthetic dict of CC+ARKit+blink names → asserts A/U/MBP/FV/L
|
||
detected, E/I/O NOT (no blink/brow false positives), and setMorph('A') drives the mapped indices
|
||
only. Verified in-browser: `ok=true, found=[A,FV,L,MBP,U]`.
|
||
|
||
VERIFICATION (:8020, zero console errors): viseme self-check ok; man/lady show greyed 🗣 badge
|
||
(neither rig carries mouth morphs — lady.glb is a CC body export with 0 morph targets, confirmed by
|
||
reading the GLB); save/load round-trip keeps cameras+lights.
|
||
|
||
DECISIONS:
|
||
- Viseme matcher is intentionally loose (orchestrator asked "match loosely") and detection-only; the
|
||
authoritative Rhubarb-letter→canonical mapping + keyframing is Lane B's. `setMorph` also accepts
|
||
raw morph names so B can drive anything the matcher missed.
|
||
|
||
BLOCKED/REQUESTS:
|
||
- @orchestrator: no viseme-bearing test character in `assets/` (man.fbx and lady.glb both have zero
|
||
mouth morphs), so the LIT badge + a visible mouth move are proven only by the synthetic self-check,
|
||
not on a real asset. If you can drop a CC/ARKit head with visemes into `assets/characters/test/`,
|
||
I'll confirm live and Lane B can validate Rhubarb end-to-end.
|
||
|
||
NEXT: hold at M4-A boundary. Available for a live viseme asset check and any SYNC findings.
|