f12b6e0847
- GET /api/splat serves data/work/splat.ply; manifest reports has_splat - scene3d: lazy-load @mkkellogg/gaussian-splats-3d DropInViewer when a splat exists (rigs/anchors/paths overlay as before); point-cloud fallback intact - scripts/splat_via_modelbeast.sh: train the splat on the MODELBEAST fleet (colmap_poses -> brush_train via the mb queue) and drop it in place - docs/modelbeast-crossover.md: full crossover map (fleet compute, volumetric capture w/ CorridorKey + audio sync, per-moment splats) Verified in-browser: 32MB room splat rendered via /api/splat with overlays live.
25 lines
3.0 KiB
Markdown
25 lines
3.0 KiB
Markdown
# Festival 4D × MODELBEAST — crossover map
|
||
|
||
*2026-07-16. Both projects live on the same fleet (M3 Ultra 256GB primary, M1 Ultra 128GB, M4 Pro 24GB — see MODELBEAST's `CLUSTER.md`). They stay separate products — asset factory vs. experience viewer — but share infrastructure and pipelines.*
|
||
|
||
## Shipped: the splat upgrade (ideas.md #1) ✅
|
||
|
||
`docs/ideas.md`'s top item ("replace the sparse point cloud with 3D Gaussian Splatting") is now wired:
|
||
|
||
- **Backend:** `GET /api/splat` serves `data/work/splat.ply` when present; `/api/manifest` reports `has_splat`.
|
||
- **Viewer:** `scene3d.js` lazy-loads `@mkkellogg/gaussian-splats-3d` and renders the splat as a `DropInViewer` in the existing scene (camera rigs, anchors, and fly-paths overlay as before). Falls back to the point cloud when absent or on load error.
|
||
- **Training:** `scripts/splat_via_modelbeast.sh` sends this project's frames to the MODELBEAST queue (`colmap_poses` → `brush_train` on whichever GPU node is free) and drops the trained `.ply` into place. No local GPU blocking; the farm handles it.
|
||
|
||
## The bigger crossover map
|
||
|
||
1. **Fleet compute for every heavy step.** MODELBEAST's queue pools gpu (1 job/Mac) and cpu lanes across all three nodes. Festival 4D's COLMAP/splat/classification steps can submit there (`mb` CLI + token) instead of hogging the local machine.
|
||
2. **Poor-man's volumetric capture** (with CorridorKey): several phones film one subject on a green screen → *our GCC-PHAT audio sync* aligns them with no clapboard → CorridorKey (neural keyer, tuned on this fleet — see MODELBEAST's `CORRIDORKEY.md`) extracts true-edge mattes per synced frame → per-instant multi-view cutouts → 3D-gen or per-moment splats. DIY volumetric video from phones; every part exists in-house today.
|
||
3. **Per-moment splats = pragmatic "4D".** True temporal 4DGS has no credible Mac port yet (honest limit). But we already detect *moments*: pool all cameras' frames in a ±1s window around each moment → COLMAP → splat → one splat per moment → scrub between them on the timeline. Static splats, time-indexed — feels 4D where it counts. Caveat: needs decent multi-camera coverage; 3 fan phones is thin.
|
||
4. **Back-flow to MODELBEAST:** our audio-sync module is the missing piece for any multi-camera capture rig there; our moment detection could auto-pick hero frames for video→3D.
|
||
5. **GVM wildcard:** CorridorKey's ~80GB auto-matting module (untested on Metal, fits only the M3 256GB) would key *real concert footage* — performer/crowd separation without green screens — feeding use case 2 with actual festival material.
|
||
|
||
## Practical notes
|
||
- The `mb` CLI needs `MB_HOST` + `MB_TOKEN` (MODELBEAST Settings → Users). The M3 primary queue is at `100.89.131.57:8777` on the tailnet.
|
||
- Splat training cost: a ~40s 4K room clip → ~190 frames → COLMAP (~1 min, GLOMAP) → Brush 30k steps (~15 min on the M3). Concert scenes with crowds will be harder than rooms — expect floaters; SuperSplat can crop/clean.
|
||
- License note for use case 2: CorridorKey is CC BY-NC-SA (non-commercial).
|