7d0a3b7f7c
pose_engine.py: queue video -> HSMR/SKEL -> BVH@fps + sidecar JSON per the treaty. - detection: torchvision mobilenet (engine_patches), every-Nth frame + single best-box reuse (single-subject) -> ~1 pose/frame aligned timeline. mobilenet swap + detect-every address the CPU-detector wall-clock finding. - recovery: HSMR ViT-H on MPS -> SKEL params (poses q46, betas, cam_t). - SKEL forward -> per-joint global orientations; rigid-bone BVH via rest-aware FK (offset[j]=rest_J[j]-rest_J[parent]; W[j]=Rg[j]@rest_R[j]^T; local Q=W[parent]^T@W[j]). Verified: rest-aware FK reconstructs SKEL joints to ~4cm (residual = SKEL biomechanical coupled translations, which rigid BVH can't carry), in-BVH FK self-consistency 0.0mm. - camera->world axis fix: HSMR emits Y-down (OpenCV, verified head.y<pelvis.y); diag(1,-1,-1) -> Y-up, faces -Z (treaty). Blender import: 24 bones, upright, 1.59m, motion present. - B3 jitter: savgol on hemisphere-aligned quaternions (rotations) + root xyz separately. - sidecar: root_space=camera (per-frame HSMR), joint_order, root_trajectory, model, fps. - build models once + per-video try/except: queue of 6 drained unattended, 2 bad clips (no-person, garbage) failed gracefully, 4 produced valid BVH; heartbeat per clip. - runtime steady-state ~1.6x realtime on ultra (detect+recover), under the <2x bar. - --selfcheck (engine-free): BVH FK exact + Euler round-trip exact. spec/skel_to_mixamorig.json: SKEL-24 -> mixamorig name map for Lane C's retarget (20 mapped, 4 null twist/heel; validated against both skeletons). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
78 lines
3.7 KiB
Python
78 lines
3.7 KiB
Python
"""
|
|
Person detector — torchvision drop-in replacement for HSMR's detectron2/ViTDet.
|
|
|
|
Why: detectron2 + chumpy won't build from source on Apple Silicon (the Lane B blocker),
|
|
and this module's `import detectron2` was a load-time landmine for everything that touches
|
|
`lib.kits.hsmr_demo` (smoke_test.py, the future pose_engine.py). torchvision ships a
|
|
COCO-pretrained Faster R-CNN with ungated weights and no build step.
|
|
|
|
Same public contract as the original, so run_demo.py / smoke_test.py / pose_engine.py
|
|
import it unchanged:
|
|
|
|
build_detector(...) -> callable(raw_imgs) -> (dets, downsample_ratios)
|
|
|
|
dets[i] : dict of CPU tensors (empty tensors if no person) —
|
|
pred_classes : all 0 (== hsmr_demo CLASS_HUMAN_ID)
|
|
scores : float
|
|
pred_boxes : (N, 4) left-upper-right-bottom pixels
|
|
downsample_ratios : one float per image; boxes are in native-resolution pixels -> 1.0
|
|
|
|
Contract is checked by test_detector.py (runs today, no SKEL needed).
|
|
|
|
# ponytail: ~35 lines replaces the whole detectron2 stack. Detection on CPU is the pose-engine
|
|
# wall-clock bottleneck, so this uses the fast mobilenet backbone and callers detect every-Nth frame.
|
|
"""
|
|
import numpy as np
|
|
import torch
|
|
from tqdm import tqdm
|
|
from torchvision.models.detection import (
|
|
fasterrcnn_mobilenet_v3_large_fpn,
|
|
FasterRCNN_MobileNet_V3_Large_FPN_Weights,
|
|
)
|
|
|
|
_COCO_PERSON = 1 # torchvision COCO label id for "person" (0 == background)
|
|
|
|
|
|
class _PersonDetector:
|
|
"""Callable over a list of RGB HxWx3 images (what HSMR's load_inputs yields).
|
|
uint8 or float, [0,255] or [0,1] are all handled."""
|
|
|
|
def __init__(self, device='cpu', score_thresh=0.25):
|
|
# score_thresh 0.25 mirrors the old detectron2 test_score_thresh; hsmr_demo's
|
|
# _img_det2patches re-filters at 0.5, so this only widens recall a little.
|
|
# mobilenet_v3 (not resnet50) — ~5-10x faster on CPU, which is the pose-engine wall-clock
|
|
# bottleneck (Lane B finding). Plenty accurate for one prominent, in-frame subject.
|
|
weights = FasterRCNN_MobileNet_V3_Large_FPN_Weights.DEFAULT
|
|
self.model = fasterrcnn_mobilenet_v3_large_fpn(
|
|
weights=weights, box_score_thresh=score_thresh,
|
|
).eval().to(device)
|
|
self.device = device
|
|
|
|
@torch.no_grad()
|
|
def __call__(self, raw_imgs):
|
|
dets, ratios = [], []
|
|
for img in tqdm(raw_imgs, desc='Detecting'):
|
|
t = torch.as_tensor(np.ascontiguousarray(img), device=self.device).float()
|
|
if float(t.max()) > 1.5: # [0,255] -> [0,1]
|
|
t = t / 255.0
|
|
t = t.permute(2, 0, 1) # HWC -> CHW (the model's own transform caps size)
|
|
out = self.model([t])[0]
|
|
keep = out['labels'] == _COCO_PERSON
|
|
dets.append({
|
|
'pred_classes': torch.zeros(int(keep.sum()), dtype=torch.long), # human -> 0
|
|
'scores' : out['scores'][keep].cpu(),
|
|
'pred_boxes' : out['boxes'][keep].cpu(), # xyxy == left-upper-right-bottom
|
|
})
|
|
ratios.append(1.0) # detected at native resolution
|
|
return dets, ratios
|
|
|
|
|
|
def build_detector(batch_size=1, max_img_size=512, device='cpu'):
|
|
"""Signature-compatible with the old detectron2 builder. batch_size / max_img_size are
|
|
accepted for parity but unused — detection runs per-frame at native resolution.
|
|
|
|
# ponytail: detector pinned to CPU regardless of `device`. It's cheap next to the
|
|
# ViT-H HMR pass, and CPU sidesteps the MPS coverage gaps in RoIAlign/NMS. Flip to
|
|
# `device` only if per-frame detection ever dominates on long videos."""
|
|
return _PersonDetector(device='cpu')
|