monster/festifun
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
56e569fd7a
festifun/docs/modelbeast-crossover.md
m3ultra f12b6e0847 Splat upgrade (ideas.md #1) + MODELBEAST crossover 
- GET /api/splat serves data/work/splat.ply; manifest reports has_splat
- scene3d: lazy-load @mkkellogg/gaussian-splats-3d DropInViewer when a splat
  exists (rigs/anchors/paths overlay as before); point-cloud fallback intact
- scripts/splat_via_modelbeast.sh: train the splat on the MODELBEAST fleet
  (colmap_poses -> brush_train via the mb queue) and drop it in place
- docs/modelbeast-crossover.md: full crossover map (fleet compute, volumetric
  capture w/ CorridorKey + audio sync, per-moment splats)
Verified in-browser: 32MB room splat rendered via /api/splat with overlays live.
2026-07-16 22:32:01 +10:00

3.0 KiB
Raw Blame History

Festival 4D × MODELBEAST — crossover map

2026-07-16. Both projects live on the same fleet (M3 Ultra 256GB primary, M1 Ultra 128GB, M4 Pro 24GB — see MODELBEAST's CLUSTER.md). They stay separate products — asset factory vs. experience viewer — but share infrastructure and pipelines.

Shipped: the splat upgrade (ideas.md #1) ✅

docs/ideas.md's top item ("replace the sparse point cloud with 3D Gaussian Splatting") is now wired:

  • Backend: GET /api/splat serves data/work/splat.ply when present; /api/manifest reports has_splat.
  • Viewer: scene3d.js lazy-loads @mkkellogg/gaussian-splats-3d and renders the splat as a DropInViewer in the existing scene (camera rigs, anchors, and fly-paths overlay as before). Falls back to the point cloud when absent or on load error.
  • Training: scripts/splat_via_modelbeast.sh sends this project's frames to the MODELBEAST queue (colmap_poses → brush_train on whichever GPU node is free) and drops the trained .ply into place. No local GPU blocking; the farm handles it.

The bigger crossover map

  1. Fleet compute for every heavy step. MODELBEAST's queue pools gpu (1 job/Mac) and cpu lanes across all three nodes. Festival 4D's COLMAP/splat/classification steps can submit there (mb CLI + token) instead of hogging the local machine.
  2. Poor-man's volumetric capture (with CorridorKey): several phones film one subject on a green screen → our GCC-PHAT audio sync aligns them with no clapboard → CorridorKey (neural keyer, tuned on this fleet — see MODELBEAST's CORRIDORKEY.md) extracts true-edge mattes per synced frame → per-instant multi-view cutouts → 3D-gen or per-moment splats. DIY volumetric video from phones; every part exists in-house today.
  3. Per-moment splats = pragmatic "4D". True temporal 4DGS has no credible Mac port yet (honest limit). But we already detect moments: pool all cameras' frames in a ±1s window around each moment → COLMAP → splat → one splat per moment → scrub between them on the timeline. Static splats, time-indexed — feels 4D where it counts. Caveat: needs decent multi-camera coverage; 3 fan phones is thin.
  4. Back-flow to MODELBEAST: our audio-sync module is the missing piece for any multi-camera capture rig there; our moment detection could auto-pick hero frames for video→3D.
  5. GVM wildcard: CorridorKey's ~80GB auto-matting module (untested on Metal, fits only the M3 256GB) would key real concert footage — performer/crowd separation without green screens — feeding use case 2 with actual festival material.

Practical notes

  • The mb CLI needs MB_HOST + MB_TOKEN (MODELBEAST Settings → Users). The M3 primary queue is at 100.89.131.57:8777 on the tailnet.
  • Splat training cost: a ~40s 4K room clip → ~190 frames → COLMAP (~1 min, GLOMAP) → Brush 30k steps (~15 min on the M3). Concert scenes with crowds will be harder than rooms — expect floaters; SuperSplat can crop/clean.
  • License note for use case 2: CorridorKey is CC BY-NC-SA (non-commercial).