Compare commits
No commits in common. "866551f266d68c5adf5824f107699c176c049d32" and "df4e06f047a276853f2f52af3987b8ebb072886b" have entirely different histories.
866551f266
...
df4e06f047
@ -25,10 +25,7 @@ three.js, same conventions as MESHGOD (`~/Documents/MESHGOD`).
|
|||||||
4. **Commit early, commit often**, on `main`, prefixed `[laneA]`/`[laneB]`/
|
4. **Commit early, commit often**, on `main`, prefixed `[laneA]`/`[laneB]`/
|
||||||
`[laneC]`. `git pull --rebase` before every commit — other lanes commit to
|
`[laneC]`. `git pull --rebase` before every commit — other lanes commit to
|
||||||
the same branch. File ownership makes conflicts near-zero; if you hit one
|
the same branch. File ownership makes conflicts near-zero; if you hit one
|
||||||
anyway, resolve only your own files and log it. **Stage your own files
|
anyway, resolve only your own files and log it.
|
||||||
explicitly (`git add <paths>`) — never `git add -A`/`.`**: another lane's
|
|
||||||
uncommitted work may be sitting in the shared working dir, and -A sweeps
|
|
||||||
it into your commit (happened once; attribution mess).
|
|
||||||
5. **Don't exceed your lane's current milestone.** Milestones and sync points
|
5. **Don't exceed your lane's current milestone.** Milestones and sync points
|
||||||
are in PLAN.md §5. Finishing early = polish + tests + log, not starting
|
are in PLAN.md §5. Finishing early = polish + tests + log, not starting
|
||||||
the next milestone uninvited.
|
the next milestone uninvited.
|
||||||
|
|||||||
41
README.md
41
README.md
@ -1,41 +0,0 @@
|
|||||||
# SCENEGOD — machinima director's sandbox
|
|
||||||
|
|
||||||
Drop characters, Mixamo clips, and photo backdrops onto a 3D stage; keyframe
|
|
||||||
transforms, cameras (with cuts), and lights on a timeline; render an mp4 with
|
|
||||||
muxed audio. The Movies / Source Filmmaker energy on the GODVERSE stack.
|
|
||||||
|
|
||||||
## Run
|
|
||||||
|
|
||||||
```
|
|
||||||
.venv/bin/uvicorn scenegod.server:app --port 8020
|
|
||||||
# → http://127.0.0.1:8020
|
|
||||||
```
|
|
||||||
|
|
||||||
Assets live in `assets/{characters,animations,props,backdrops,audio}/`
|
|
||||||
(plain folders, scanned live — files sharing dir+stem are one entry; a
|
|
||||||
same-stem image becomes the thumbnail). Scenes save to `scenes/*.json`,
|
|
||||||
renders to `renders/` (reaped after 48h).
|
|
||||||
|
|
||||||
## What works (v1)
|
|
||||||
|
|
||||||
- GLB/FBX/OBJ characters & props; spec-gloss (Character Creator) GLBs are
|
|
||||||
converted on load. Image backdrops: flat plane, corner, or 360 dome.
|
|
||||||
- Mixamo-style clip retarget onto any mixamorig-named skeleton, client-side
|
|
||||||
(world-space rotation-delta bake, ported from MESHGOD rigroom).
|
|
||||||
- Timeline: transform + param keyframes with easing, clip blocks with
|
|
||||||
trim/loop/crossfade, multiple cameras + cut list, keyframable sun/ambient,
|
|
||||||
audio tracks (Web Audio preview; server-side ffmpeg mux in the final mp4),
|
|
||||||
viseme/morph track with Rhubarb JSON import (`GET /rhubarb?path=` if the
|
|
||||||
rhubarb binary is installed).
|
|
||||||
- Render: draft webm (MediaRecorder) or deterministic PNG-sequence →
|
|
||||||
ffmpeg h264 (+aac) mp4 at any resolution.
|
|
||||||
- `scripts/graft_limb.py`: headless-Blender armature graft (add a rigged
|
|
||||||
hand/limb to a rigged body; refuses when the bones already exist).
|
|
||||||
- `scripts/test_server.py` (9 groups) + `node scenegod/web/timeline_test.mjs`.
|
|
||||||
|
|
||||||
## Repo docs
|
|
||||||
|
|
||||||
`PLAN.md` — architecture, scene-JSON schema, module contracts, milestones.
|
|
||||||
`CLAUDE.md` — lane-agent protocol (3 parallel Opus lanes + orchestrator),
|
|
||||||
stack rules, inherited gotchas. `lanes/` + `logs/` — per-lane instructions
|
|
||||||
and session logs; the build history of this repo is readable there.
|
|
||||||
@ -158,18 +158,3 @@ M3-polish list (order):
|
|||||||
present, greyed otherwise. Add `stage.setMorph(id, name, weight)` for the
|
present, greyed otherwise. Add `stage.setMorph(id, name, weight)` for the
|
||||||
timeline to drive. Rhubarb-JSON → keyframes lands on Lane B's side; your
|
timeline to drive. Rhubarb-JSON → keyframes lands on Lane B's side; your
|
||||||
deliverable is detection + the setMorph hook + the badge.
|
deliverable is detection + the setMorph hook + the badge.
|
||||||
|
|
||||||
### 2026-07-18 — session 4 ACCEPTED with one matcher hotfix. **v1 COMPLETE — no open A work.**
|
|
||||||
applyState source:'none' fix verified live (rig survives save/load). Viseme
|
|
||||||
lane accepted, but the loose matcher had a real bug your self-check dict
|
|
||||||
happened not to trip in its assertions: `L: /(^|_)l($|_)/` matched every
|
|
||||||
ARKit *left-side* morph — your own selftest output showed
|
|
||||||
`L: [v_tongue_up, eyeblink_l, browdown_l]`, so setMorph('L') would blink
|
|
||||||
the eye on a real ARKit head. Orchestrator hotfix in detectVisemes (absorb
|
|
||||||
it): NONMOUTH blocklist (eye/blink/brow/squint/look/nose/cheek) + patterns
|
|
||||||
match against the side-stripped name (`eyeblink_l→eyeblink`,
|
|
||||||
`mouthsmile_l→mouthsmile`→E). Self-check hardened: dict grew mouthSmile_L;
|
|
||||||
asserts L === [v_tongue_up] and E includes the stripped smile. Lesson worth
|
|
||||||
keeping: assert group CONTENTS, not just which groups exist.
|
|
||||||
Still owed by John, not you: a viseme-bearing test character for a live
|
|
||||||
badge check. Standing by.
|
|
||||||
|
|||||||
@ -137,14 +137,3 @@ refresh + snap select seen live; tests green). M4-B scope (PLAN §5):
|
|||||||
`stage.setMorph(id, name, weight)` with lerp between keys; plus
|
`stage.setMorph(id, name, weight)` with lerp between keys; plus
|
||||||
`importRhubarb(id, rhubarbJson, t0)` mapping mouthShape cues → morph keys.
|
`importRhubarb(id, rhubarbJson, t0)` mapping mouthShape cues → morph keys.
|
||||||
Guard: only offer when `stage.visemeTargets(id).length > 0`.
|
Guard: only offer when `stage.visemeTargets(id).length > 0`.
|
||||||
|
|
||||||
### 2026-07-18 — session 4 ACCEPTED. **v1 COMPLETE — no required B work.**
|
|
||||||
Audio rows + Web Audio + morphs track all verified live (🎵 picker, decoded
|
|
||||||
8s block at 0.5s offset, gain input, param/cut/sun lanes, zero console
|
|
||||||
errors). Your ownership note is acted on: CLAUDE.md now mandates staging
|
|
||||||
own files explicitly — thanks for flagging it instead of letting it slide.
|
|
||||||
Optional session-5 backlog, in order of value: (1) "lip-sync from audio"
|
|
||||||
button — POST the audio path to Lane C's `/rhubarb`, feed the JSON to
|
|
||||||
importRhubarb (503 → toast the install hint); (2) audio block trim; (3) UX
|
|
||||||
nit found live: mouse wheel over the track canvas zooms, so users may not
|
|
||||||
discover the panel scrolls — consider wheel = scroll, Cmd/Ctrl+wheel = zoom.
|
|
||||||
|
|||||||
@ -134,48 +134,3 @@ BLOCKED/REQUESTS: none. Verified with a throwaway harness (`_check.html`,
|
|||||||
NEXT: M2 polish complete. Undo already wired (Ctrl+Z). Open for M4 timeline
|
NEXT: M2 polish complete. Undo already wired (Ctrl+Z). Open for M4 timeline
|
||||||
work when unlocked (audio tracks on the timeline UI + Web Audio sync, per
|
work when unlocked (audio tracks on the timeline UI + Web Audio sync, per
|
||||||
PLAN §5 M4) or any SYNC 3 fixes. Holding until orchestrator flips M4.
|
PLAN §5 M4) or any SYNC 3 fixes. Holding until orchestrator flips M4.
|
||||||
|
|
||||||
## 2026-07-18 session 4 (M4-B: audio + visemes)
|
|
||||||
DONE: M4-B all three items.
|
|
||||||
1. Audio rows in tlui from scene.audio[]: one lane per clip below Cameras,
|
|
||||||
green block at the DECODED buffer width (fallback width until decoded),
|
|
||||||
gain <input> in the name column, drag = offset (moveAudio), right-click =
|
|
||||||
delete. 🎵 button reads /assets/tree audio category into a small floating
|
|
||||||
menu → addAudio at the playhead.
|
|
||||||
2. Web Audio draft playback synced to the master clock: lazy AudioContext,
|
|
||||||
decode+cache buffers (/assets/file), _audioStart schedules
|
|
||||||
AudioBufferSourceNodes at (start - time) with per-clip gain, _audioStop on
|
|
||||||
pause/natural-end, user seeks reschedule (_seek helper). step()/render
|
|
||||||
never touch audio — server muxes from scene.audio.
|
|
||||||
3. morphs track kind: _evalMorphs lerps per viseme name → stage.setMorph;
|
|
||||||
importRhubarb(id, json, t0) maps mouthCues → morph keys; tlui shows a
|
|
||||||
"↳ visemes" lane guarded by stage.visemeTargets(id).length > 0.
|
|
||||||
Model (timeline.js, stagestub.js, timeline_test.mjs) committed ba29d0c;
|
|
||||||
tests cover morph lerp, rhubarb keys, audio mutators+undo. All green.
|
|
||||||
VERIFIED (real server :8020, real assets): 🎵 picker listed + added
|
|
||||||
audio/test/tone.wav, buffer decoded (8s), green block drawn, Web Audio
|
|
||||||
scheduled/rescheduled/stopped (ctx running), viseme lane rendered
|
|
||||||
importRhubarb keys, setMorph fired while scrubbing. Contract match confirmed:
|
|
||||||
real stage.js ships setMorph(id,name,weight) + visemeTargets(id) (returns
|
|
||||||
Object.keys(visemes)) — my `.length>0` guard works against both.
|
|
||||||
DECISIONS:
|
|
||||||
- Audio duration is a client draw concern (decode+cache in tlui), NOT stored
|
|
||||||
in the {path,start,gain} schema — server already knows durations at mux.
|
|
||||||
- importRhubarb = simple per-cue ramp (shape 1 at cue start → 0 at end);
|
|
||||||
hold/crossfade tuning deferred (ponytail marker in timeline.js) until it
|
|
||||||
looks wrong on a real blendshape character.
|
|
||||||
- Audio blocks can extend past scene.duration (tone.wav is 8s in a 6s scene)
|
|
||||||
— left as-is; the ruler just clips visually. Trim is a later nicety.
|
|
||||||
BLOCKED/REQUESTS:
|
|
||||||
- **Ownership note for orchestrator**: my tlui.js M4-B changes (+146 lines)
|
|
||||||
were committed inside Lane A's `df4e06f [laneA] M4-A ...`, not a [laneB]
|
|
||||||
commit — Lane A's commit swept my then-uncommitted tlui.js working changes
|
|
||||||
along with their dock.js/stage.js. Code is intact + verified; only the
|
|
||||||
attribution is off. Heads-up that our git hygiene let a cross-lane file
|
|
||||||
ride in on another lane's commit; a `git add <own files>` (not `-A`) avoids
|
|
||||||
it. Nothing for me to fix.
|
|
||||||
NEXT: M4-B complete. Open items if a session 5 lands: audio block trim/clip to
|
|
||||||
duration, viseme import UI button (currently importRhubarb is API-only; Lane
|
|
||||||
C has a /rhubarb endpoint — could wire a "lip-sync from audio" button that
|
|
||||||
POSTs the clip and feeds the result to importRhubarb). Holding for SYNC/next
|
|
||||||
orchestrator update.
|
|
||||||
|
|||||||
@ -26,12 +26,8 @@ function detectVisemes(root){
|
|||||||
for(const [name, idx] of Object.entries(o.morphTargetDictionary)){
|
for(const [name, idx] of Object.entries(o.morphTargetDictionary)){
|
||||||
const k = name.toLowerCase(); (morphs.get(k) || morphs.set(k, []).get(k)).push({mesh:o, idx}); } });
|
const k = name.toLowerCase(); (morphs.get(k) || morphs.set(k, []).get(k)).push({mesh:o, idx}); } });
|
||||||
const visemes = {};
|
const visemes = {};
|
||||||
// SYNC5: ARKit side suffixes (_L/_R) made /(^|_)l($|_)/ swallow eyeblink_l & co —
|
|
||||||
// never match non-mouth regions, and match against the side-stripped name.
|
|
||||||
const NONMOUTH = /eye|blink|brow|squint|look|nose|cheek/;
|
|
||||||
for(const [canon, pats] of Object.entries(VISEMES)){
|
for(const [canon, pats] of Object.entries(VISEMES)){
|
||||||
const hits = [...morphs.keys()].filter(k => !NONMOUTH.test(k)
|
const hits = [...morphs.keys()].filter(k => pats.some(p => p.test(k)));
|
||||||
&& pats.some(p => p.test(k.replace(/(.)[_-][lr]$/, '$1'))));
|
|
||||||
if(hits.length) visemes[canon] = hits;
|
if(hits.length) visemes[canon] = hits;
|
||||||
}
|
}
|
||||||
return { morphs, visemes };
|
return { morphs, visemes };
|
||||||
@ -39,16 +35,12 @@ function detectVisemes(root){
|
|||||||
// runnable check for the matcher (assert-based). Call visemeSelfCheck() from the console.
|
// runnable check for the matcher (assert-based). Call visemeSelfCheck() from the console.
|
||||||
export function visemeSelfCheck(){
|
export function visemeSelfCheck(){
|
||||||
const dict = {'V_Open':0,'V_Explosive':1,'jawOpen':2,'mouthPucker':3,'mouthFunnel':4,
|
const dict = {'V_Open':0,'V_Explosive':1,'jawOpen':2,'mouthPucker':3,'mouthFunnel':4,
|
||||||
'mouthClose':5,'V_Dental_Lip':6,'V_Tongue_up':7,'EyeBlink_L':8,'browDown_L':9,'mouthSmile_L':10};
|
'mouthClose':5,'V_Dental_Lip':6,'V_Tongue_up':7,'EyeBlink_L':8,'browDown_L':9};
|
||||||
const mesh = { isMesh:true, morphTargetDictionary:dict, morphTargetInfluences:new Array(11).fill(0) };
|
const mesh = { isMesh:true, morphTargetDictionary:dict, morphTargetInfluences:new Array(10).fill(0) };
|
||||||
const { morphs, visemes } = detectVisemes({ traverse(cb){ cb(mesh); } });
|
const { morphs, visemes } = detectVisemes({ traverse(cb){ cb(mesh); } });
|
||||||
const has = v => v in visemes;
|
const has = v => v in visemes;
|
||||||
const detectOk = ['A','U','MBP','FV','L'].every(has) // open/jaw, pucker/funnel, explos/close, dental, tongue
|
const detectOk = ['A','U','MBP','FV','L'].every(has) // open/jaw, pucker/funnel, explos/close, dental, tongue
|
||||||
&& !has('I') && !has('O')
|
&& !has('E') && !has('I') && !has('O'); // blink/brow must NOT register as visemes
|
||||||
// SYNC5 regression guard: side-suffixed ARKit names must land on the RIGHT viseme —
|
|
||||||
// L is tongue only (no blink/brow/smile bleed), E gets the side-stripped smile.
|
|
||||||
&& visemes.L.length === 1 && visemes.L[0] === 'v_tongue_up'
|
|
||||||
&& has('E') && visemes.E.includes('mouthsmile_l');
|
|
||||||
// canonical 'A' drives every mesh morph mapped to it (jawOpen idx2 + V_Open idx0)
|
// canonical 'A' drives every mesh morph mapped to it (jawOpen idx2 + V_Open idx0)
|
||||||
for(const r of visemes.A) for(const {idx} of morphs.get(r)) mesh.morphTargetInfluences[idx] = 0.7;
|
for(const r of visemes.A) for(const {idx} of morphs.get(r)) mesh.morphTargetInfluences[idx] = 0.7;
|
||||||
const driveOk = mesh.morphTargetInfluences[0] === 0.7 && mesh.morphTargetInfluences[2] === 0.7
|
const driveOk = mesh.morphTargetInfluences[0] === 0.7 && mesh.morphTargetInfluences[2] === 0.7
|
||||||
|
|||||||
Loading…
Reference in New Issue
Block a user