diff --git a/README.md b/README.md index 7d34dcf..e6fb2c1 100644 --- a/README.md +++ b/README.md @@ -4,14 +4,24 @@ GGML/C++ implementation of NVIDIA's Kimodo text-to-motion model. ## Status -`Kimodo-SMPLX-RP-v1` accepts either a UTF-8 prompt or a precomputed LLM2Vec -embedding and generates unconstrained SMPL-X22 local rotations and root -translations on CPU or Vulkan. The text encoder uses eight-layer Vulkan chunks -by default; set `KIMODO_TEXT_LAYER_CHUNK=1..32` to tune VRAM use. +The five released Kimodo motion checkpoints accept either a UTF-8 prompt or a +precomputed LLM2Vec embedding and generate local rotations plus root +translations on CPU or Vulkan: + +- SMPL-X RP v1: 22 joints +- SOMA RP/SEED v1.1: the predicted compact 30-joint control skeleton +- G1 RP/SEED v1: 34 Unitree G1 joints + +NVIDIA's Python API expands SOMA's predicted 30 joints to a relaxed-hand +77-joint presentation skeleton. The native API currently returns the 30 joints +the model actually predicts. The text encoder uses eight-layer Vulkan chunks by +default; set `KIMODO_TEXT_LAYER_CHUNK=1..32` to tune VRAM use. Included: checked GGUF loading, safetensors conversion, DDIM sampling, C/C++ -APIs, CPU/Vulkan parity tests, and a local text-to-motion demo. Constraints, -SOMA, G1, GLB export, and quantised models are not implemented yet. +APIs, conditioned multi-prompt transitions, CPU/Vulkan parity tests, +skeleton-only GLB export, and a local text-to-motion demo. General constraint +input, 77-joint SOMA expansion, skinned-mesh GLB export, and quantised models +are not implemented yet. ## Build and test on Linux @@ -21,7 +31,7 @@ support. GGML is a pinned Git submodule: ```sh git submodule update --init --recursive -scripts/download_gguf_weights.sh --output "$PWD" +scripts/download_gguf_weights.sh --output "$PWD" --model soma-rp-v1.1 cmake --preset debug cmake --build --preset debug ctest --preset debug @@ -56,8 +66,9 @@ to the process. The GGUF parser fuzzer requires Clang. `include/kimodo/kimodo_capi.h` is the C API. Model loading checks the motion GGUF and text bundle before inference. Use `kimodo_generate_embedding` for -4096 F32 values or `kimodo_generate` for text. Both return SMPL-X22 root -translations and local XYZW rotations. +4096 F32 values or `kimodo_generate` for text. Both return the selected model's +root translations and local XYZW rotations; query the joint count from the +result rather than assuming a fixed skeleton. ## Demo @@ -69,27 +80,43 @@ go run ./demo -addr 0.0.0.0:8094 Open `http://localhost:8094`. The left sidebar contains the prompt and a persistent history; choosing a previous animation restores its prompt for a -new generation. +new generation. Every successful animation also writes a standalone +`animation.glb` beside its raw streams, for example +`demo-output//animation.glb`. It contains the selected animated +node hierarchy (no mesh), ready to copy into a Three.js project. It is also +available from `/api/animations//animation.glb` while the demo +is running. ## Weights Ready-to-run native GGML weights are published under the Hugging Face `LocalAI-io` organisation (not GitHub's `localai-org`). The reusable [Llama-3-Kimodo-GGML](https://huggingface.co/LocalAI-io/Llama-3-Kimodo-GGML) -text encoder and the upstream-linked -[Kimodo-SMPLX-RP-v1-GGML](https://huggingface.co/LocalAI-io/Kimodo-SMPLX-RP-v1-GGML) -diffusion model are separate, so users download rather than recreate them: +text encoder is separate from the four redistributable motion repositories, +each of which preserves a one-to-one relationship to its NVIDIA upstream: + +- [Kimodo-SOMA-RP-v1.1-GGML](https://huggingface.co/LocalAI-io/Kimodo-SOMA-RP-v1.1-GGML) +- [Kimodo-SOMA-SEED-v1.1-GGML](https://huggingface.co/LocalAI-io/Kimodo-SOMA-SEED-v1.1-GGML) +- [Kimodo-G1-RP-v1-GGML](https://huggingface.co/LocalAI-io/Kimodo-G1-RP-v1-GGML) +- [Kimodo-G1-SEED-v1-GGML](https://huggingface.co/LocalAI-io/Kimodo-G1-SEED-v1-GGML) + +Download one or repeat `--model` to install several: ```sh -scripts/download_gguf_weights.sh --output "$PWD" +scripts/download_gguf_weights.sh --output "$PWD" \ + --model soma-rp-v1.1 --model g1-rp-v1 ``` The installer verifies each published manifest and SHA-256 hashes. Use `--motion-only` when supplying a precomputed 4096-float LLM2Vec embedding. +SMPL-X RP is deliberately absent from the published-weight installer: its +internal-R&D licence prohibits distributing derivative models, so it must be +converted locally after the user obtains the upstream checkpoint under its +gated terms. -The GGUF bundle includes converted Meta Llama 3 material and Kimodo is -non-commercial research-only. Review the published model card and upstream -licences before downloading or redistributing. +The text bundle includes converted Meta Llama 3 material and retains its +separate terms. Review every selected model card before downloading or +redistributing. ## License @@ -97,6 +124,22 @@ The C++ port and its original tooling are licensed under Apache-2.0; see [LICENSE](LICENSE). GGML and the model weights retain their respective licences. +| Motion checkpoint | Upstream terms | Commercial use | +| --- | --- | --- | +| Kimodo-SMPLX-RP-v1 | [NVIDIA Internal Scientific Research and Development Model License](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-internal-scientific-research-and-development-model-license/) | No; internal, non-production R&D only; derivative model redistribution is prohibited | +| SOMA RP/SEED v1.1 | [NVIDIA Open Model License](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/) | Permitted by the model licence | +| G1 RP/SEED v1 | [NVIDIA Open Model License](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/) | Permitted by the model licence | + +The SMPL-X warning is about NVIDIA's trained Kimodo checkpoint, not the mere +fact that its output uses an SMPL-X-shaped 22-joint hierarchy. Converting that +checkpoint to GGUF is a new runtime representation of the same weights and does +not replace its licence. Skeleton names, parent links, and the Apache-2.0 port +source do not by themselves make the SOMA or G1 checkpoints non-commercial. +The SMPL-X Hugging Face metadata, model card, and access terms identify the +internal-R&D licence; treat those restrictive terms as controlling even though +an apparently inconsistent `LICENSE` file has also appeared in that upstream +repository. + ### Regenerating the bundle This is only needed to reproduce a conversion. The SMPL-X checkpoint and Llama @@ -105,8 +148,9 @@ authenticating, download the exact revisions and hash manifests with: ```sh nix develop path:. --command hf auth login -nix develop path:. --command scripts/download_weights.sh \ - --output "$PWD/models" --with-text +scripts/download_weights.sh --output "$PWD/models" --with-text \ + --model smplx-rp-v1 --model soma-rp-v1.1 --model soma-seed-v1.1 \ + --model g1-rp-v1 --model g1-seed-v1 ``` Convert the local LLM2Vec model to the native component bundle with: @@ -120,9 +164,10 @@ Validate a prospective release without network access, then explicitly upload it from an account allowed to publish to `LocalAI-io`: ```sh -nix develop path:. --command python scripts/publish_gguf.py --component motion nix develop path:. --command python scripts/publish_gguf.py --component motion \ - --upload --confirm-upstream-licences + --motion-model soma-rp-v1.1 +nix develop path:. --command python scripts/publish_gguf.py --component motion \ + --motion-model soma-rp-v1.1 --upload --confirm-upstream-licences nix develop path:. --command python scripts/publish_gguf.py --component text \ --upload --confirm-upstream-licences ``` diff --git a/demo/index.html b/demo/index.html index 1f14746..380bbb2 100644 --- a/demo/index.html +++ b/demo/index.html @@ -1,20 +1,21 @@ Kimodo text to motion
Camera tracks motion · Drag: rotate · wheel: zoom · Shift/right-drag: pan · double-click: reset
- +
Camera tracks motion · Drag: rotate · wheel: zoom · Shift/right-drag: pan · double-click: reset
+ diff --git a/demo/main.go b/demo/main.go index 703a459..8de0243 100644 --- a/demo/main.go +++ b/demo/main.go @@ -51,13 +51,19 @@ type promptSegment struct { Frames int `json:"frames"` } type motionModel struct { - ID string `json:"id"` - Label string `json:"label"` - Skeleton string `json:"skeleton"` - Upstream string `json:"upstream"` - Available bool `json:"available"` - Reason string `json:"reason,omitempty"` - Motion string `json:"-"` + ID string `json:"id"` + Label string `json:"label"` + Skeleton string `json:"skeleton"` + SkeletonKey string `json:"skeleton_key"` + Upstream string `json:"upstream"` + License string `json:"license"` + LicenseURL string `json:"license_url"` + Commercial bool `json:"commercial"` + Available bool `json:"available"` + Reason string `json:"reason,omitempty"` + Parents []int `json:"parents"` + Offsets [][3]float32 `json:"offsets"` + Motion string `json:"-"` } type gallery struct { mu sync.RWMutex @@ -94,14 +100,6 @@ func (g *gallery) list() []*animation { return result } -func copyFile(dst, src string) error { - b, err := os.ReadFile(src) - if err != nil { - return err - } - return os.WriteFile(dst, b, 0600) -} - func readF32(path string) ([]float32, error) { b, err := os.ReadFile(path) if err != nil { @@ -125,79 +123,159 @@ func writeF32(path string, values []float32) error { return os.WriteFile(path, b, 0600) } -func blendQuaternion(a, b []float32, alpha float32) { - dot := a[0]*b[0] + a[1]*b[1] + a[2]*b[2] + a[3]*b[3] - if dot < 0 { - for i := range b { - b[i] = -b[i] - } - } - length := float32(0) - for i := range a { - a[i] = alpha*a[i] + (1-alpha)*b[i] - length += a[i] * a[i] - } - if length > 0 { - length = 1 / float32(math.Sqrt(float64(length))) - for i := range a { - a[i] *= length - } - } +// Each motion is exported as a node-only GLB: it deliberately has no mesh or +// skin, so consumers can attach their own Three.js geometry to the named +// joints. Kimodo stores root translations and local XYZW rotations. +var smplx22Parents = [...]int{-1, 0, 0, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 9, 9, 12, 13, 14, 16, 17, 18, 19} +var smplx22Names = [...]string{"pelvis", "left_hip", "right_hip", "spine1", "left_knee", "right_knee", "spine2", "left_ankle", "right_ankle", "spine3", "left_foot", "right_foot", "neck", "left_collar", "right_collar", "head", "left_shoulder", "right_shoulder", "left_elbow", "right_elbow", "left_wrist", "right_wrist"} +var smplx22Offsets = [...][3]float32{{}, {.052299, -.093936, -.027607}, {-.057193, -.106548, -.022218}, {-.001496, .11293, -.024981}, {.058867, -.416442, -.006557}, {-.048074, -.39756, -.014061}, {.0069, .145636, -.006859}, {-.041738, -.437584, -.029512}, {.014489, -.446853, -.01803}, {-.010334, .056082, .021116}, {.049294, -.065279, .126259}, {-.040575, -.065287, .127076}, {-.011026, .171365, -.028827}, {.047725, .087643, -.008375}, {-.046636, .086612, -.014864}, {.024654, .175391, .024463}, {.126285, .05768, -.013885}, {-.109342, .053674, -.009118}, {.272907, -.069853, -.039094}, {-.292029, -.03544, -.024565}, {.276174, .021254, -.002478}, {-.271878, -.004835, -.016445}} + +type skeletonDefinition struct { + key string + names []string + parents []int + offsets [][3]float32 } -// stitchSegments joins independently sampled demo segments. The overlap is -// blended in root space and by normalized linear interpolation for quaternions. -// Native observed-motion conditioning is deliberately a later parity step. -func stitchSegments(output string, dirs []string, overlap int) error { - var roots, rotations []float32 - for index, dir := range dirs { - root, err := readF32(filepath.Join(dir, "root_positions.f32")) - if err != nil { - return err - } - rot, err := readF32(filepath.Join(dir, "local_rotations_xyzw.f32")) - if err != nil { - return err - } - frames := len(root) / 3 - if frames == 0 || len(rot) != frames*22*4 { - return fmt.Errorf("invalid motion segment %d", index+1) - } - if index == 0 { - roots, rotations = root, rot - continue - } - n := overlap - if n > frames { - n = frames - } - if n > len(roots)/3 { - n = len(roots) / 3 - } - for frame := 0; frame < n; frame++ { - alpha := float32(0.5) - if n > 1 { - alpha = 1 - float32(frame)/float32(n-1) - } - old := (len(roots)/3 - n + frame) * 3 - newest := frame * 3 - for axis := 0; axis < 3; axis++ { - roots[old+axis] = alpha*roots[old+axis] + (1-alpha)*root[newest+axis] - } - for joint := 0; joint < 22; joint++ { - oldQ := (len(rotations)/4 - n*22 + frame*22 + joint) * 4 - newQ := (frame*22 + joint) * 4 - blendQuaternion(rotations[oldQ:oldQ+4], append([]float32(nil), rot[newQ:newQ+4]...), alpha) - } - } - roots = append(roots, root[n*3:]...) - rotations = append(rotations, rot[n*22*4:]...) +var skeletonDefinitions = map[string]skeletonDefinition{ + "smplx22": {key: "smplx22", names: smplx22Names[:], parents: smplx22Parents[:], offsets: smplx22Offsets[:]}, +} + +type gltfBufferView struct { + Buffer int `json:"buffer"` + ByteOffset int `json:"byteOffset,omitempty"` + ByteLength int `json:"byteLength"` +} +type gltfAccessor struct { + BufferView int `json:"bufferView"` + ComponentType int `json:"componentType"` + Count int `json:"count"` + Type string `json:"type"` +} + +func appendF32(dst []byte, values []float32) []byte { + for _, value := range values { + var b [4]byte + binary.LittleEndian.PutUint32(b[:], math.Float32bits(value)) + dst = append(dst, b[:]...) } - if err := writeF32(filepath.Join(output, "root_positions.f32"), roots); err != nil { + return dst +} + +func writeSkeletonGLB(path string, roots, rotations []float32, skeleton skeletonDefinition) error { + frames := len(roots) / 3 + joints := len(skeleton.parents) + if frames < 1 || joints < 1 || len(skeleton.names) != joints || len(skeleton.offsets) != joints || len(roots) != frames*3 || len(rotations) != frames*joints*4 { + return fmt.Errorf("invalid %s motion for GLB export", skeleton.key) + } + times := make([]float32, frames) + for i := range times { + times[i] = float32(i) / 30 + } + bin := make([]byte, 0, (frames+frames*3+frames*22*4)*4) + views := make([]gltfBufferView, 0, 24) + addView := func(values []float32) int { + offset := len(bin) + bin = appendF32(bin, values) + views = append(views, gltfBufferView{Buffer: 0, ByteOffset: offset, ByteLength: len(bin) - offset}) + return len(views) - 1 + } + timeView, rootView := addView(times), addView(roots) + rotationViews := make([]int, joints) + for joint := range rotationViews { + track := make([]float32, frames*4) + for frame := 0; frame < frames; frame++ { + copy(track[frame*4:], rotations[(frame*joints+joint)*4:(frame*joints+joint+1)*4]) + } + rotationViews[joint] = addView(track) + } + accessors := []gltfAccessor{{BufferView: timeView, ComponentType: 5126, Count: frames, Type: "SCALAR"}, {BufferView: rootView, ComponentType: 5126, Count: frames, Type: "VEC3"}} + for _, view := range rotationViews { + accessors = append(accessors, gltfAccessor{BufferView: view, ComponentType: 5126, Count: frames, Type: "VEC4"}) + } + nodes := make([]map[string]any, joints) + for joint := range nodes { + node := map[string]any{"name": skeleton.names[joint]} + if joint != 0 { + node["translation"] = skeleton.offsets[joint] + } + children := make([]int, 0, 3) + for child, parent := range skeleton.parents { + if parent == joint { + children = append(children, child) + } + } + if len(children) != 0 { + node["children"] = children + } + nodes[joint] = node + } + samplers := make([]map[string]any, 0, 23) + channels := make([]map[string]any, 0, 23) + addChannel := func(node, output int, path string) { + samplers = append(samplers, map[string]any{"input": 0, "output": output, "interpolation": "LINEAR"}) + channels = append(channels, map[string]any{"sampler": len(samplers) - 1, "target": map[string]any{"node": node, "path": path}}) + } + addChannel(0, 1, "translation") + for joint := 0; joint < joints; joint++ { + addChannel(joint, joint+2, "rotation") + } + document := map[string]any{ + "asset": map[string]string{"version": "2.0", "generator": "kimodo.cpp skeleton exporter"}, + "scene": 0, + "scenes": []map[string]any{{"nodes": []int{0}}}, + "nodes": nodes, + "buffers": []map[string]int{{"byteLength": len(bin)}}, + "bufferViews": views, + "accessors": accessors, + "animations": []map[string]any{{"name": "KimodoMotion", "samplers": samplers, "channels": channels}}, + "extras": map[string]any{"skeleton": skeleton.key, "fps": 30, "rotation_order": "xyzw"}, + } + jsonChunk, err := json.Marshal(document) + if err != nil { return err } - return writeF32(filepath.Join(output, "local_rotations_xyzw.f32"), rotations) + for len(jsonChunk)%4 != 0 { + jsonChunk = append(jsonChunk, ' ') + } + for len(bin)%4 != 0 { + bin = append(bin, 0) + } + total := 12 + 8 + len(jsonChunk) + 8 + len(bin) + out := make([]byte, 0, total) + putU32 := func(value uint32) { + var b [4]byte + binary.LittleEndian.PutUint32(b[:], value) + out = append(out, b[:]...) + } + putU32(0x46546c67) + putU32(2) + putU32(uint32(total)) + putU32(uint32(len(jsonChunk))) + putU32(0x4e4f534a) + out = append(out, jsonChunk...) + putU32(uint32(len(bin))) + putU32(0x004e4942) + out = append(out, bin...) + return os.WriteFile(path, out, 0600) } + +func exportSkeletonGLB(dir, skeletonKey string) error { + skeleton, ok := skeletonDefinitions[skeletonKey] + if !ok { + return fmt.Errorf("unsupported skeleton %q", skeletonKey) + } + roots, err := readF32(filepath.Join(dir, "root_positions.f32")) + if err != nil { + return err + } + rotations, err := readF32(filepath.Join(dir, "local_rotations_xyzw.f32")) + if err != nil { + return err + } + return writeSkeletonGLB(filepath.Join(dir, "animation.glb"), roots, rotations, skeleton) +} + func (g *gallery) worker() { for id := range g.queue { g.mu.Lock() @@ -239,6 +317,9 @@ func (g *gallery) worker() { err = fmt.Errorf("sequence: %w: %s", runErr, strings.TrimSpace(string(output))) } } + if err == nil { + err = exportSkeletonGLB(dir, model.SkeletonKey) + } } } g.mu.Lock() @@ -259,6 +340,10 @@ func (g *gallery) worker() { func main() { addr := flag.String("addr", "127.0.0.1:8090", "listen address") motion := flag.String("motion-model", "models/kimodo-smplx-rp-v1-f32.gguf", "motion GGUF") + somaRP := flag.String("soma-rp-model", "models/kimodo-soma-rp-v1.1-f32.gguf", "SOMA RP v1.1 motion GGUF") + somaSEED := flag.String("soma-seed-model", "models/kimodo-soma-seed-v1.1-f32.gguf", "SOMA SEED v1.1 motion GGUF") + g1RP := flag.String("g1-rp-model", "models/kimodo-g1-rp-v1-f32.gguf", "G1 RP v1 motion GGUF") + g1SEED := flag.String("g1-seed-model", "models/kimodo-g1-seed-v1-f32.gguf", "G1 SEED v1 motion GGUF") text := flag.String("text-bundle", "generated/llm2vec-text-bundle", "native LLM2Vec component directory") generator := flag.String("generator", "build/debug/kmd-generate", "native text-to-motion command") output := flag.String("output", "demo-output", "persistent gallery directory") @@ -266,12 +351,24 @@ func main() { if err := os.MkdirAll(*output, 0755); err != nil { log.Fatal(err) } + makeModel := func(id, label, skeletonLabel, skeletonKey, upstream, license, licenseURL, path string, commercial bool) motionModel { + definition := skeletonDefinitions[skeletonKey] + model := motionModel{ID: id, Label: label, Skeleton: skeletonLabel, SkeletonKey: skeletonKey, Upstream: upstream, License: license, LicenseURL: licenseURL, Commercial: commercial, Parents: definition.parents, Offsets: definition.offsets, Motion: path} + if info, err := os.Stat(path); err == nil && info.Mode().IsRegular() { + model.Available = true + } else { + model.Reason = "GGUF not found at " + path + } + return model + } + const internalLicense = "https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-internal-scientific-research-and-development-model-license/" + const openLicense = "https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/" models := map[string]motionModel{ - "smplx-rp-v1": {ID: "smplx-rp-v1", Label: "SMPL-X RP v1", Skeleton: "SMPL-X 22 joints", Upstream: "nvidia/Kimodo-SMPLX-RP-v1", Available: true, Motion: *motion}, - "soma-rp-v1.1": {ID: "soma-rp-v1.1", Label: "SOMA RP v1.1", Skeleton: "SOMA 30 joints", Upstream: "nvidia/Kimodo-SOMA-RP-v1.1", Reason: "SOMA decoder and GGML conversion are being added"}, - "soma-seed-v1.1": {ID: "soma-seed-v1.1", Label: "SOMA SEED v1.1", Skeleton: "SOMA 30 joints", Upstream: "nvidia/Kimodo-SOMA-SEED-v1.1", Reason: "SOMA decoder and GGML conversion are being added"}, - "g1-rp-v1": {ID: "g1-rp-v1", Label: "G1 RP v1", Skeleton: "Unitree G1 34 joints", Upstream: "nvidia/Kimodo-G1-RP-v1", Reason: "G1 decoder and GGML conversion are being added"}, - "g1-seed-v1": {ID: "g1-seed-v1", Label: "G1 SEED v1", Skeleton: "Unitree G1 34 joints", Upstream: "nvidia/Kimodo-G1-SEED-v1", Reason: "G1 decoder and GGML conversion are being added"}, + "smplx-rp-v1": makeModel("smplx-rp-v1", "SMPL-X RP v1", "SMPL-X 22 joints", "smplx22", "nvidia/Kimodo-SMPLX-RP-v1", "NVIDIA Internal Scientific R&D (non-commercial)", internalLicense, *motion, false), + "soma-rp-v1.1": makeModel("soma-rp-v1.1", "SOMA RP v1.1", "SOMA compact 30-joint control skeleton", "soma30", "nvidia/Kimodo-SOMA-RP-v1.1", "NVIDIA Open Model License", openLicense, *somaRP, true), + "soma-seed-v1.1": makeModel("soma-seed-v1.1", "SOMA SEED v1.1", "SOMA compact 30-joint control skeleton", "soma30", "nvidia/Kimodo-SOMA-SEED-v1.1", "NVIDIA Open Model License", openLicense, *somaSEED, true), + "g1-rp-v1": makeModel("g1-rp-v1", "G1 RP v1", "Unitree G1 34 joints", "g1skel34", "nvidia/Kimodo-G1-RP-v1", "NVIDIA Open Model License", openLicense, *g1RP, true), + "g1-seed-v1": makeModel("g1-seed-v1", "G1 SEED v1", "Unitree G1 34 joints", "g1skel34", "nvidia/Kimodo-G1-SEED-v1", "NVIDIA Open Model License", openLicense, *g1SEED, true), } g := &gallery{items: map[string]*animation{}, output: *output, queue: make(chan string, 32), generator: *generator, motion: *motion, text: *text, models: models} entries, _ := filepath.Glob(filepath.Join(*output, "*.json")) @@ -283,6 +380,15 @@ func main() { var a animation if json.Unmarshal(b, &a) == nil { g.items[a.ID] = &a + if a.Status == "ready" { + model, ok := models[a.Model] + if !ok { + model = models["smplx-rp-v1"] + } + if err := exportSkeletonGLB(filepath.Join(*output, a.ID), model.SkeletonKey); err != nil && !os.IsNotExist(err) { + log.Printf("export existing animation %s: %v", a.ID, err) + } + } } } go g.worker() @@ -400,7 +506,7 @@ func main() { }) mux.HandleFunc("/api/animations/", func(w http.ResponseWriter, r *http.Request) { parts := strings.Split(strings.TrimPrefix(r.URL.Path, "/api/animations/"), "/") - if len(parts) != 2 || (parts[1] != "root.f32" && parts[1] != "rotations.f32") { + if len(parts) != 2 || (parts[1] != "root.f32" && parts[1] != "rotations.f32" && parts[1] != "animation.glb") { http.NotFound(w, r) return } @@ -415,7 +521,23 @@ func main() { if parts[1] == "rotations.f32" { name = "local_rotations_xyzw.f32" } - w.Header().Set("Content-Type", "application/octet-stream") + if parts[1] == "animation.glb" { + name = "animation.glb" + w.Header().Set("Content-Type", "model/gltf-binary") + w.Header().Set("Content-Disposition", "attachment; filename=kimodo-"+a.ID+".glb") + // A GLB is a compact asset; read it directly so browsers always receive + // it as a download rather than invoking any path-cleaning redirects. + data, err := os.ReadFile(filepath.Join(g.output, a.ID, name)) + if err != nil { + http.NotFound(w, r) + return + } + w.Header().Set("Content-Length", fmt.Sprint(len(data))) + _, _ = w.Write(data) + return + } else { + w.Header().Set("Content-Type", "application/octet-stream") + } w.Header().Set("Cache-Control", "no-store") http.ServeFile(w, r, filepath.Join(g.output, a.ID, name)) }) diff --git a/demo/main_test.go b/demo/main_test.go new file mode 100644 index 0000000..53e13c3 --- /dev/null +++ b/demo/main_test.go @@ -0,0 +1,55 @@ +package main + +import ( + "encoding/binary" + "encoding/json" + "os" + "path/filepath" + "testing" +) + +func TestWriteSkeletonGLB(t *testing.T) { + for _, skeletonKey := range []string{"smplx22", "soma30", "g1skel34"} { + t.Run(skeletonKey, func(t *testing.T) { + skeleton := skeletonDefinitions[skeletonKey] + path := filepath.Join(t.TempDir(), "animation.glb") + roots := []float32{0, 0, 0, 1, 0, 0} + rotations := make([]float32, 2*len(skeleton.parents)*4) + for frame := 0; frame < 2; frame++ { + for joint := range skeleton.parents { + rotations[(frame*len(skeleton.parents)+joint)*4+3] = 1 + } + } + if err := writeSkeletonGLB(path, roots, rotations, skeleton); err != nil { + t.Fatal(err) + } + assertSkeletonGLB(t, path, len(skeleton.parents)) + }) + } +} + +func assertSkeletonGLB(t *testing.T, path string, expectedJoints int) { + t.Helper() + b, err := os.ReadFile(path) + if err != nil { + t.Fatal(err) + } + if len(b) < 20 || binary.LittleEndian.Uint32(b) != 0x46546c67 || binary.LittleEndian.Uint32(b[4:]) != 2 || int(binary.LittleEndian.Uint32(b[8:])) != len(b) { + t.Fatalf("invalid GLB header") + } + jsonLength := int(binary.LittleEndian.Uint32(b[12:])) + if binary.LittleEndian.Uint32(b[16:]) != 0x4e4f534a || 20+jsonLength > len(b) { + t.Fatalf("invalid GLB JSON chunk") + } + var document struct { + Asset map[string]string `json:"asset"` + Nodes []json.RawMessage `json:"nodes"` + Animations []json.RawMessage `json:"animations"` + } + if err := json.Unmarshal(b[20:20+jsonLength], &document); err != nil { + t.Fatal(err) + } + if document.Asset["version"] != "2.0" || len(document.Nodes) != expectedJoints || len(document.Animations) != 1 { + t.Fatalf("unexpected GLB document: %s", b[20:20+jsonLength]) + } +} diff --git a/demo/models.js b/demo/models.js index 7272c71..05de78b 100644 --- a/demo/models.js +++ b/demo/models.js @@ -5,6 +5,7 @@ window.addEventListener('load', async () => { const generate = document.querySelector('#generate'); if (!prompt || !form || !generate) return; const models = await fetch('/api/models').then(r => r.json()); + window.kimodoModels = models; const modelLabel = document.createElement('label'); modelLabel.htmlFor = 'motionModel'; modelLabel.textContent = 'Motion model'; @@ -19,7 +20,14 @@ window.addEventListener('load', async () => { const modelHint = document.createElement('div'); modelHint.className = 'hint'; const updateModel = () => { const model = models.find(item => item.id === select.value); - modelHint.textContent = model.available ? `${model.skeleton} · ${model.upstream}` : `${model.skeleton} · ${model.reason}`; + if (!model) return; + const title = document.querySelector('.eyebrow'); + if (title) title.textContent = `${model.label} · Vulkan`; + modelHint.classList.toggle('license-warning', !model.commercial); + const terms = model.commercial ? 'commercial use permitted under NVIDIA Open Model License' : '⚠ non-commercial research use only'; + const detail = model.available ? `${model.skeleton} · ${model.upstream} · ` : `${model.skeleton} · ${model.reason} · `; + const link = document.createElement('a'); link.href = model.license_url; link.target = '_blank'; link.rel = 'noreferrer'; link.textContent = terms; + modelHint.replaceChildren(document.createTextNode(detail), link); }; select.onchange = updateModel; form.insertBefore(modelLabel, prompt); form.insertBefore(select, prompt); form.insertBefore(modelHint, prompt); updateModel(); diff --git a/demo/skeletons_extra.go b/demo/skeletons_extra.go new file mode 100644 index 0000000..51b3ada --- /dev/null +++ b/demo/skeletons_extra.go @@ -0,0 +1,19 @@ +package main + +// Copied from NVIDIA Kimodo's Apache-2.0 skeleton definitions. Offsets are +// the parent-local differences extracted from the official neutral-joint +// assets in the trusted reference container. +func init() { + skeletonDefinitions["soma30"] = skeletonDefinition{ + key: "soma30", + names: []string{"Hips", "Spine1", "Spine2", "Chest", "Neck1", "Neck2", "Head", "Jaw", "LeftEye", "RightEye", "LeftShoulder", "LeftArm", "LeftForeArm", "LeftHand", "LeftHandThumbEnd", "LeftHandMiddleEnd", "RightShoulder", "RightArm", "RightForeArm", "RightHand", "RightHandThumbEnd", "RightHandMiddleEnd", "LeftLeg", "LeftShin", "LeftFoot", "LeftToeBase", "RightLeg", "RightShin", "RightFoot", "RightToeBase"}, + parents: []int{-1, 0, 1, 2, 3, 4, 5, 6, 6, 6, 3, 10, 11, 12, 13, 13, 3, 16, 17, 18, 19, 19, 0, 22, 23, 24, 0, 26, 27, 28}, + offsets: [][3]float32{{0, 0, 0}, {-.00013727, .0500376256, -.00053726669}, {-1.86574103e-9, .0712530139, -.000298248546}, {-5.75188398e-9, .0755006305, -.00815970992}, {-.00181676517, .263112953, -.00553348292}, {-2.85102231e-8, .0770939664, .0230258546}, {-4.5975437e-8, .0612891595, .0195370861}, {2.63687901e-5, .0047559225, .0309494062}, {.0320638079, .0538020513, .0758688308}, {-.0322244017, .05361869, .0755823359}, {.0162165175, .232371641, .0511341324}, {.149198457, 2.19397873e-8, -.0550232576}, {.287393078, 2.50268389e-9, -2.58787737e-5}, {.270939812, -7.06625108e-9, 2.60897248e-5}, {.122686267, -.0322017573, .0483306876}, {.190119595, -.00312878387, -.000339570373}, {-.0138011824, .231803086, .0521415786}, {-.150371962, 1.17387901e-7, -.0554560437}, {-.287366393, 1.87628082e-8, -2.59709359e-5}, {-.271336198, -1.16767401e-9, 2.61269368e-5}, {-.122642483, -.0321145448, .0480403904}, {-.190005945, -.00306615542, -.0003157343}, {.10043214, -.0843452671, .0259565473}, {-1e-8, -.432217537, -.00802912805}, {1e-8, -.421550959, -.0348152298}, {0, -.0505947206, .132315294}, {-.10047278, -.0829525995, .0262031695}, {1e-8, -.433622059, -.00805555828}, {2e-8, -.421173943, -.0347839785}, {-3.42907669e-9, -.0507960932, .132841956}}, + } + skeletonDefinitions["g1skel34"] = skeletonDefinition{ + key: "g1skel34", + names: []string{"pelvis_skel", "left_hip_pitch_skel", "left_hip_roll_skel", "left_hip_yaw_skel", "left_knee_skel", "left_ankle_pitch_skel", "left_ankle_roll_skel", "left_toe_base", "right_hip_pitch_skel", "right_hip_roll_skel", "right_hip_yaw_skel", "right_knee_skel", "right_ankle_pitch_skel", "right_ankle_roll_skel", "right_toe_base", "waist_yaw_skel", "waist_roll_skel", "waist_pitch_skel", "left_shoulder_pitch_skel", "left_shoulder_roll_skel", "left_shoulder_yaw_skel", "left_elbow_skel", "left_wrist_roll_skel", "left_wrist_pitch_skel", "left_wrist_yaw_skel", "left_hand_roll_skel", "right_shoulder_pitch_skel", "right_shoulder_roll_skel", "right_shoulder_yaw_skel", "right_elbow_skel", "right_wrist_roll_skel", "right_wrist_pitch_skel", "right_wrist_yaw_skel", "right_hand_roll_skel"}, + parents: []int{-1, 0, 1, 2, 3, 4, 5, 6, 0, 8, 9, 10, 11, 12, 13, 0, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 17, 26, 27, 28, 29, 30, 31, 32}, + offsets: [][3]float32{{0, 0, 0}, {.064452, -.1027, 0}, {.052, -.030465, 0}, {0, -.12412, .025001}, {.0021489, -.17734, -.078273}, {-.000094445, -.30001, 0}, {0, -.017558, 0}, {0, -.035, .14}, {-.064452, -.1027, 0}, {-.052, -.030465, 0}, {0, -.12412, .025001}, {-.0021489, -.17734, -.078273}, {.000094445, -.30001, 0}, {0, -.017558, 0}, {0, -.035, .14}, {0, 0, 0}, {0, .044, -.0039635}, {0, 0, 0}, {.10022, .24778, .0039563}, {.038, -.013831, 0}, {.00624, -.1032, 0}, {0, -.080518, .015783}, {.00188791, -.01, .1}, {0, 0, .038}, {0, 0, .046}, {0, 0, .1}, {-.10021, .24778, .0039563}, {-.038, -.013831, 0}, {-.00624, -.1032, 0}, {0, -.080518, .015783}, {-.00188791, -.01, .1}, {0, 0, .038}, {0, 0, .046}, {0, 0, .1}}, + } +} diff --git a/reference/dump_kimodo_reference.py b/reference/dump_kimodo_reference.py index cd3292a..fce0221 100644 --- a/reference/dump_kimodo_reference.py +++ b/reference/dump_kimodo_reference.py @@ -32,7 +32,7 @@ def parse_args() -> argparse.Namespace: parser.add_argument("--seed", required=True, type=int) parser.add_argument("--device", default="cuda") parser.add_argument("--checkpoint-dir", type=Path, - help="local directory containing Kimodo-SMPLX-RP-v1; never download at capture time") + help="local directory containing the selected Kimodo checkpoint; never download at capture time") parser.add_argument("--zero-embedding", action="store_true", help="use a deterministic [1,1,4096] zero embedding; enables motion-only fixtures") parser.add_argument("--text-base", type=Path, @@ -113,8 +113,21 @@ def main() -> None: raise SystemExit("--zero-embedding cannot be combined with real text model paths") if args.checkpoint_dir: checkpoint = args.checkpoint_dir.resolve() - if not (checkpoint / "Kimodo-SMPLX-RP-v1" / "config.yaml").is_file(): - raise SystemExit("--checkpoint-dir must contain Kimodo-SMPLX-RP-v1/config.yaml") + model_folders = { + "kimodo-smplx-rp": "Kimodo-SMPLX-RP-v1", + "kimodo-smplx-rp-v1": "Kimodo-SMPLX-RP-v1", + "kimodo-soma-rp": "Kimodo-SOMA-RP-v1.1", + "kimodo-soma-rp-v1.1": "Kimodo-SOMA-RP-v1.1", + "kimodo-soma-seed": "Kimodo-SOMA-SEED-v1.1", + "kimodo-soma-seed-v1.1": "Kimodo-SOMA-SEED-v1.1", + "kimodo-g1-rp": "Kimodo-G1-RP-v1", + "kimodo-g1-rp-v1": "Kimodo-G1-RP-v1", + "kimodo-g1-seed": "Kimodo-G1-SEED-v1", + "kimodo-g1-seed-v1": "Kimodo-G1-SEED-v1", + } + folder = model_folders.get(args.model) + if folder is None or not (checkpoint / folder / "config.yaml").is_file(): + raise SystemExit("--checkpoint-dir does not contain the selected official Kimodo model") # This is deliberately set only for the reference subprocess. It # prevents a missing local model from silently falling back to HF. import os diff --git a/scripts/convert_motion_to_gguf.py b/scripts/convert_motion_to_gguf.py index 6ee6e07..cdc0f2f 100755 --- a/scripts/convert_motion_to_gguf.py +++ b/scripts/convert_motion_to_gguf.py @@ -1,5 +1,5 @@ #!/usr/bin/env python3 -"""Convert the Kimodo SMPL-X safetensors checkpoint to a self-describing GGUF. +"""Convert a supported Kimodo motion safetensors checkpoint to GGUF. This converter deliberately implements only the safe safetensors and NPY formats. It never imports torch, never deserializes pickle, and writes to a @@ -22,6 +22,15 @@ GGUF_MAGIC, GGUF_VERSION, GGML_TYPE_F32 = 0x46554747, 3, 0 TYPE_UINT64, TYPE_STRING, TYPE_FLOAT32 = 10, 8, 6 TYPE_UINT32 = 4 +MODEL_SPECS = { + "nvidia/Kimodo-SMPLX-RP-v1": ("smplx22", "SMPLXSkeleton22", 22, False, + "nvidia-internal-scientific-research-and-development-model-license"), + "nvidia/Kimodo-SOMA-RP-v1.1": ("soma30", "SOMASkeleton30", 30, True, "nvidia-open-model-license"), + "nvidia/Kimodo-SOMA-SEED-v1.1": ("soma30", "SOMASkeleton30", 30, True, "nvidia-open-model-license"), + "nvidia/Kimodo-G1-RP-v1": ("g1skel34", "G1Skeleton34", 34, True, "nvidia-open-model-license"), + "nvidia/Kimodo-G1-SEED-v1": ("g1skel34", "G1Skeleton34", 34, True, "nvidia-open-model-license"), +} + @dataclass(frozen=True) class Tensor: name: str @@ -183,7 +192,7 @@ def copy_range(dst, tensor: Tensor) -> None: def main() -> None: p = argparse.ArgumentParser(description=__doc__) - p.add_argument("--input", required=True, type=Path, help="downloaded Kimodo-SMPLX-RP-v1 directory") + p.add_argument("--input", required=True, type=Path, help="downloaded Kimodo model directory") p.add_argument("--output", required=True, type=Path) args = p.parse_args() root = args.input.resolve() @@ -195,22 +204,33 @@ def main() -> None: for part in ("global_root", "local_root", "body"): for stat in ("mean", "std"): tensors.append(read_npy(root / "stats" / "motion" / part / f"{stat}.npy", f"stats.{part}.{stat}")) - revision = (root / "REVISION").read_text(encoding="utf-8").split()[0] + revision_fields = (root / "REVISION").read_text(encoding="utf-8").split() + if len(revision_fields) != 2 or revision_fields[1] not in MODEL_SPECS: + raise SystemExit("REVISION does not identify a supported official Kimodo model") + revision, model_id = revision_fields + skeleton, skeleton_class, joints, commercial, license_name = MODEL_SPECS[model_id] + config = (root / "config.yaml").read_text(encoding="utf-8") + if f"_target_: kimodo.skeleton.{skeleton_class}" not in config: + raise SystemExit("config.yaml skeleton does not match REVISION model identity") + motion_dim = 9 + 12 * joints + body_dim = motion_dim - 5 meta = [ metadata_string("general.architecture", "kimodo-motion"), - metadata_string("general.name", "Kimodo-SMPLX-RP-v1"), + metadata_string("general.name", model_id.removeprefix("nvidia/")), # GGML's own loader requires general.alignment to be UINT32. metadata_uint32("general.alignment", ALIGNMENT), metadata_uint("kimodo.format_version", 1), - metadata_string("kimodo.skeleton", "smplx22"), - metadata_string("kimodo.model_identity", f"nvidia/Kimodo-SMPLX-RP-v1@{revision}"), + metadata_string("kimodo.skeleton", skeleton), + metadata_string("kimodo.model_identity", f"{model_id}@{revision}"), + metadata_string("kimodo.license", license_name), + metadata_uint("kimodo.commercial_use", int(commercial)), metadata_string("kimodo.source_revision", revision), metadata_string("kimodo.source_sha256", sha256(ckpt)), metadata_uint("kimodo.text_embedding_width", 4096), - metadata_uint("kimodo.motion_dim", 273), + metadata_uint("kimodo.motion_dim", motion_dim), metadata_uint("kimodo.global_root_dim", 5), metadata_uint("kimodo.local_root_dim", 4), - metadata_uint("kimodo.body_dim", 268), + metadata_uint("kimodo.body_dim", body_dim), metadata_uint("kimodo.hidden_size", 1024), metadata_uint("kimodo.layers", 16), metadata_uint("kimodo.heads", 8), diff --git a/scripts/download_gguf_weights.sh b/scripts/download_gguf_weights.sh index 04b953e..dbef17c 100755 --- a/scripts/download_gguf_weights.sh +++ b/scripts/download_gguf_weights.sh @@ -4,19 +4,20 @@ set -euo pipefail export HF_HUB_DISABLE_PROGRESS_BARS=1 ORG="${GGUF_ORG:-LocalAI-io}" -MOTION_REPO_DEFAULT="$ORG/Kimodo-SMPLX-RP-v1-GGML" TEXT_REPO_DEFAULT="$ORG/Llama-3-Kimodo-GGML" usage() { - printf '%s\n' "usage: $0 --output DIR [--motion-repo HF_REPO] [--text-repo HF_REPO] [--revision REVISION] [--motion-only]" >&2 + printf '%s\n' "usage: $0 --output DIR [--model MODEL]... [--motion-repo HF_REPO] [--text-repo HF_REPO] [--revision REVISION] [--motion-only]" >&2 + printf '%s\n' "models: soma-rp-v1.1, soma-seed-v1.1, g1-rp-v1, g1-seed-v1" >&2 exit 2 } -output='' motion_repo="$MOTION_REPO_DEFAULT" text_repo="$TEXT_REPO_DEFAULT" revision='main' motion_only=0 +output='' motion_repo_override='' text_repo="$TEXT_REPO_DEFAULT" revision='main' motion_only=0 models=() while [ "$#" -gt 0 ]; do case "$1" in --output) [ "$#" -ge 2 ] || usage; output=$2; shift 2 ;; - --motion-repo) [ "$#" -ge 2 ] || usage; motion_repo=$2; shift 2 ;; + --model) [ "$#" -ge 2 ] || usage; models+=("$2"); shift 2 ;; + --motion-repo) [ "$#" -ge 2 ] || usage; motion_repo_override=$2; shift 2 ;; --text-repo) [ "$#" -ge 2 ] || usage; text_repo=$2; shift 2 ;; --revision) [ "$#" -ge 2 ] || usage; revision=$2; shift 2 ;; --motion-only) motion_only=1; shift ;; @@ -25,6 +26,8 @@ while [ "$#" -gt 0 ]; do done [ -n "$output" ] || usage command -v hf >/dev/null || { echo "hf not found; enter the Nix shell first" >&2; exit 1; } +[ "${#models[@]}" -gt 0 ] || models=(soma-rp-v1.1) +[ -z "$motion_repo_override" ] || [ "${#models[@]}" -eq 1 ] || { echo "--motion-repo requires exactly one --model" >&2; exit 2; } mkdir -p "$output" @@ -70,7 +73,17 @@ print("verified native Kimodo GGUF bundle") PY } -download_and_verify "$motion_repo" "models/kimodo-smplx-rp-v1-f32.gguf" +for model in "${models[@]}"; do + case "$model" in + soma-rp-v1.1) motion_repo="$ORG/Kimodo-SOMA-RP-v1.1-GGML"; motion_file='models/kimodo-soma-rp-v1.1-f32.gguf' ;; + soma-seed-v1.1) motion_repo="$ORG/Kimodo-SOMA-SEED-v1.1-GGML"; motion_file='models/kimodo-soma-seed-v1.1-f32.gguf' ;; + g1-rp-v1) motion_repo="$ORG/Kimodo-G1-RP-v1-GGML"; motion_file='models/kimodo-g1-rp-v1-f32.gguf' ;; + g1-seed-v1) motion_repo="$ORG/Kimodo-G1-SEED-v1-GGML"; motion_file='models/kimodo-g1-seed-v1-f32.gguf' ;; + *) echo "unknown motion model: $model" >&2; usage ;; + esac + [ -z "$motion_repo_override" ] || motion_repo=$motion_repo_override + download_and_verify "$motion_repo" "$motion_file" +done if [ "$motion_only" -eq 0 ]; then download_and_verify "$text_repo" "generated/llm2vec-text-bundle/*" fi diff --git a/scripts/download_weights.sh b/scripts/download_weights.sh index ff25c5d..0fe2314 100755 --- a/scripts/download_weights.sh +++ b/scripts/download_weights.sh @@ -6,15 +6,17 @@ set -euo pipefail export HF_HUB_DISABLE_PROGRESS_BARS=1 usage() { - printf '%s\n' "usage: $0 --output DIR [--revision REVISION] [--with-text]" >&2 + printf '%s\n' "usage: $0 --output DIR [--revision REVISION] [--model NAME]... [--with-text]" >&2 exit 2 } output='' revision='main' with_text=0 +models=() while [ "$#" -gt 0 ]; do case "$1" in --output) [ "$#" -ge 2 ] || usage; output=$2; shift 2 ;; --revision) [ "$#" -ge 2 ] || usage; revision=$2; shift 2 ;; + --model) [ "$#" -ge 2 ] || usage; models+=("$2"); shift 2 ;; --with-text) with_text=1; shift ;; *) usage ;; esac @@ -52,7 +54,18 @@ download() { printf '%s %s\n' "$sha" "$repo" > "$target/REVISION" } -download nvidia/Kimodo-SMPLX-RP-v1 Kimodo-SMPLX-RP-v1 +if [ "${#models[@]}" -eq 0 ]; then models=(smplx-rp-v1); fi +for model in "${models[@]}"; do + case "$model" in + smplx-rp-v1) repo=nvidia/Kimodo-SMPLX-RP-v1; folder=Kimodo-SMPLX-RP-v1 ;; + soma-rp-v1.1) repo=nvidia/Kimodo-SOMA-RP-v1.1; folder=Kimodo-SOMA-RP-v1.1 ;; + soma-seed-v1.1) repo=nvidia/Kimodo-SOMA-SEED-v1.1; folder=Kimodo-SOMA-SEED-v1.1 ;; + g1-rp-v1) repo=nvidia/Kimodo-G1-RP-v1; folder=Kimodo-G1-RP-v1 ;; + g1-seed-v1) repo=nvidia/Kimodo-G1-SEED-v1; folder=Kimodo-G1-SEED-v1 ;; + *) echo "Unknown Kimodo model: $model" >&2; usage ;; + esac + download "$repo" "$folder" +done if [ "$with_text" -eq 1 ]; then # The MNTP repo is a LoRA adapter, not the Llama base checkpoint. Keep all # three identities separately so converter provenance cannot confuse them. diff --git a/scripts/hf/Kimodo-G1-RP-v1-GGML/NOTICE b/scripts/hf/Kimodo-G1-RP-v1-GGML/NOTICE new file mode 100644 index 0000000..492050b --- /dev/null +++ b/scripts/hf/Kimodo-G1-RP-v1-GGML/NOTICE @@ -0,0 +1,4 @@ +Licensed by NVIDIA Corporation under the NVIDIA Open Model License. + +Upstream model: nvidia/Kimodo-G1-RP-v1. This GGUF conversion grants no +additional rights. diff --git a/scripts/hf/Kimodo-G1-RP-v1-GGML/README.md b/scripts/hf/Kimodo-G1-RP-v1-GGML/README.md new file mode 100644 index 0000000..a0e49b3 --- /dev/null +++ b/scripts/hf/Kimodo-G1-RP-v1-GGML/README.md @@ -0,0 +1,25 @@ +--- +license: other +library_name: ggml +base_model: nvidia/Kimodo-G1-RP-v1 +base_model_relation: quantized +tags: [gguf, ggml, text-to-motion, unitree-g1, kimodo] +--- + +# Kimodo-G1-RP-v1-GGML + +Native F32 GGML/GGUF conversion of +[nvidia/Kimodo-G1-RP-v1](https://huggingface.co/nvidia/Kimodo-G1-RP-v1), +targeting the 34-joint Unitree G1 skeleton. Its reusable Llama-derived text +encoder is distributed separately as +[`Llama-3-Kimodo-GGML`](https://huggingface.co/LocalAI-io/Llama-3-Kimodo-GGML). + +The model is installed at `models/kimodo-g1-rp-v1-f32.gguf` by +`scripts/download_gguf_weights.sh --output "$PWD" --model g1-rp-v1`. + +## Provenance and licence + +Converted from upstream revision `3020ad8c419c244e0429d360163730c63c4ed011`. +`MANIFEST.json` records the source revision and SHA-256 of the GGUF. The model +remains subject to the [NVIDIA Open Model License](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/). +This conversion grants no additional rights. diff --git a/scripts/hf/Kimodo-G1-SEED-v1-GGML/NOTICE b/scripts/hf/Kimodo-G1-SEED-v1-GGML/NOTICE new file mode 100644 index 0000000..508d7cb --- /dev/null +++ b/scripts/hf/Kimodo-G1-SEED-v1-GGML/NOTICE @@ -0,0 +1,4 @@ +Licensed by NVIDIA Corporation under the NVIDIA Open Model License. + +Upstream model: nvidia/Kimodo-G1-SEED-v1. This GGUF conversion grants no +additional rights. diff --git a/scripts/hf/Kimodo-G1-SEED-v1-GGML/README.md b/scripts/hf/Kimodo-G1-SEED-v1-GGML/README.md new file mode 100644 index 0000000..73cb48b --- /dev/null +++ b/scripts/hf/Kimodo-G1-SEED-v1-GGML/README.md @@ -0,0 +1,25 @@ +--- +license: other +library_name: ggml +base_model: nvidia/Kimodo-G1-SEED-v1 +base_model_relation: quantized +tags: [gguf, ggml, text-to-motion, unitree-g1, kimodo] +--- + +# Kimodo-G1-SEED-v1-GGML + +Native F32 GGML/GGUF conversion of +[nvidia/Kimodo-G1-SEED-v1](https://huggingface.co/nvidia/Kimodo-G1-SEED-v1), +targeting the 34-joint Unitree G1 skeleton. Its reusable Llama-derived text +encoder is distributed separately as +[`Llama-3-Kimodo-GGML`](https://huggingface.co/LocalAI-io/Llama-3-Kimodo-GGML). + +The model is installed at `models/kimodo-g1-seed-v1-f32.gguf` by +`scripts/download_gguf_weights.sh --output "$PWD" --model g1-seed-v1`. + +## Provenance and licence + +Converted from upstream revision `5e6f2c7e18c2ab834c8d7983b9dcce701e5c6097`. +`MANIFEST.json` records the source revision and SHA-256 of the GGUF. The model +remains subject to the [NVIDIA Open Model License](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/). +This conversion grants no additional rights. diff --git a/scripts/hf/Kimodo-SMPLX-RP-v1-GGML/NOTICE b/scripts/hf/Kimodo-SMPLX-RP-v1-GGML/NOTICE index 8a6804e..a183f6c 100644 --- a/scripts/hf/Kimodo-SMPLX-RP-v1-GGML/NOTICE +++ b/scripts/hf/Kimodo-SMPLX-RP-v1-GGML/NOTICE @@ -1,3 +1,4 @@ -Kimodo-SMPLX-RP-v1 source model: NVIDIA. This converted distribution remains -subject to the NVIDIA Internal Scientific Research and Development Model License -and is for non-commercial research use only. +Kimodo-SMPLX-RP-v1 source model: NVIDIA. The NVIDIA Internal Scientific +Research and Development Model License limits derivative models to internal, +non-production R&D and prohibits their distribution. This notice is retained +for local conversion records; it does not authorize a converted distribution. diff --git a/scripts/hf/Kimodo-SMPLX-RP-v1-GGML/README.md b/scripts/hf/Kimodo-SMPLX-RP-v1-GGML/README.md index ed0f37a..bb5d79f 100644 --- a/scripts/hf/Kimodo-SMPLX-RP-v1-GGML/README.md +++ b/scripts/hf/Kimodo-SMPLX-RP-v1-GGML/README.md @@ -6,30 +6,31 @@ base_model_relation: quantized tags: [gguf, ggml, text-to-motion, smplx, kimodo] --- -# Kimodo-SMPLX-RP-v1-GGML +# Oops — Kimodo-SMPLX-RP-v1 is local-conversion only -Native F32 GGML/GGUF conversion of +The kimodo.cpp converter can produce a local F32 GGML/GGUF representation of [nvidia/Kimodo-SMPLX-RP-v1](https://huggingface.co/nvidia/Kimodo-SMPLX-RP-v1), the SMPL-X 22-joint text-and-constraint conditioned motion diffusion model. -This repository contains only the diffusion model; its reusable Llama-derived -text encoder is distributed separately as -[`Llama-3-Kimodo-GGML`](https://huggingface.co/LocalAI-io/Llama-3-Kimodo-GGML). +It is not a redistributable GGUF release. -From a kimodo.cpp checkout with the Hugging Face CLI installed, install both with: +We originally published the converted weights here, then noticed that the +upstream NVIDIA Internal Scientific Research and Development Model License +explicitly prohibits distributing derivative models. Oops. The GGUF, manifest, +and checksums have therefore been removed; this card remains so that existing +links explain what happened instead of becoming a mysterious 404. -```sh -scripts/download_gguf_weights.sh --output "$PWD" -``` - -The model is installed at `models/kimodo-smplx-rp-v1-f32.gguf`. Use -`--motion-only` when supplying a precomputed LLM2Vec embedding. +If someone at NVIDIA is willing to give LocalAI-io written permission to +redistribute this checkpoint as GGML/GGUF, that would be very welcome. We would +be happy to restore the conversion with its upstream provenance and verified +parity results. ## Provenance and licence -Converted by kimodo.cpp from upstream commit -`1419ba56b734c48bbafb41fefa84088ca94583b5`. `MANIFEST.json` records the -source revision and SHA-256 of the GGUF. +The local converter targets upstream revision +`1419ba56b734c48bbafb41fefa84088ca94583b5`. -Kimodo-SMPLX-RP-v1 is for non-commercial research use only and remains subject -to the [NVIDIA Internal Scientific Research and Development Model License](https://huggingface.co/nvidia/Kimodo-SMPLX-RP-v1). -This conversion grants no additional rights. +Kimodo-SMPLX-RP-v1 remains subject to the +[NVIDIA Internal Scientific Research and Development Model License](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-internal-scientific-research-and-development-model-license/). +That licence limits the checkpoint and derivative models to internal, +non-production R&D and prohibits their distribution. Converting the weights to +GGUF does not grant additional rights. diff --git a/scripts/hf/Kimodo-SOMA-RP-v1.1-GGML/NOTICE b/scripts/hf/Kimodo-SOMA-RP-v1.1-GGML/NOTICE new file mode 100644 index 0000000..6207972 --- /dev/null +++ b/scripts/hf/Kimodo-SOMA-RP-v1.1-GGML/NOTICE @@ -0,0 +1,4 @@ +Licensed by NVIDIA Corporation under the NVIDIA Open Model License. + +Upstream model: nvidia/Kimodo-SOMA-RP-v1.1. This GGUF conversion grants no +additional rights. diff --git a/scripts/hf/Kimodo-SOMA-RP-v1.1-GGML/README.md b/scripts/hf/Kimodo-SOMA-RP-v1.1-GGML/README.md new file mode 100644 index 0000000..e3558b5 --- /dev/null +++ b/scripts/hf/Kimodo-SOMA-RP-v1.1-GGML/README.md @@ -0,0 +1,25 @@ +--- +license: other +library_name: ggml +base_model: nvidia/Kimodo-SOMA-RP-v1.1 +base_model_relation: quantized +tags: [gguf, ggml, text-to-motion, soma, kimodo] +--- + +# Kimodo-SOMA-RP-v1.1-GGML + +Native F32 GGML/GGUF conversion of +[nvidia/Kimodo-SOMA-RP-v1.1](https://huggingface.co/nvidia/Kimodo-SOMA-RP-v1.1). +The model predicts the compact SOMA 30-joint control skeleton. Its reusable +Llama-derived text encoder is distributed separately as +[`Llama-3-Kimodo-GGML`](https://huggingface.co/LocalAI-io/Llama-3-Kimodo-GGML). + +The model is installed at `models/kimodo-soma-rp-v1.1-f32.gguf` by +`scripts/download_gguf_weights.sh --output "$PWD" --model soma-rp-v1.1`. + +## Provenance and licence + +Converted from upstream revision `6c9233af1180b8151e3c4703477104af5dce9dd5`. +`MANIFEST.json` records the source revision and SHA-256 of the GGUF. The model +remains subject to the [NVIDIA Open Model License](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/). +This conversion grants no additional rights. diff --git a/scripts/hf/Kimodo-SOMA-SEED-v1.1-GGML/NOTICE b/scripts/hf/Kimodo-SOMA-SEED-v1.1-GGML/NOTICE new file mode 100644 index 0000000..8c07b35 --- /dev/null +++ b/scripts/hf/Kimodo-SOMA-SEED-v1.1-GGML/NOTICE @@ -0,0 +1,4 @@ +Licensed by NVIDIA Corporation under the NVIDIA Open Model License. + +Upstream model: nvidia/Kimodo-SOMA-SEED-v1.1. This GGUF conversion grants no +additional rights. diff --git a/scripts/hf/Kimodo-SOMA-SEED-v1.1-GGML/README.md b/scripts/hf/Kimodo-SOMA-SEED-v1.1-GGML/README.md new file mode 100644 index 0000000..5e36a12 --- /dev/null +++ b/scripts/hf/Kimodo-SOMA-SEED-v1.1-GGML/README.md @@ -0,0 +1,25 @@ +--- +license: other +library_name: ggml +base_model: nvidia/Kimodo-SOMA-SEED-v1.1 +base_model_relation: quantized +tags: [gguf, ggml, text-to-motion, soma, kimodo] +--- + +# Kimodo-SOMA-SEED-v1.1-GGML + +Native F32 GGML/GGUF conversion of +[nvidia/Kimodo-SOMA-SEED-v1.1](https://huggingface.co/nvidia/Kimodo-SOMA-SEED-v1.1). +The model predicts the compact SOMA 30-joint control skeleton. Its reusable +Llama-derived text encoder is distributed separately as +[`Llama-3-Kimodo-GGML`](https://huggingface.co/LocalAI-io/Llama-3-Kimodo-GGML). + +The model is installed at `models/kimodo-soma-seed-v1.1-f32.gguf` by +`scripts/download_gguf_weights.sh --output "$PWD" --model soma-seed-v1.1`. + +## Provenance and licence + +Converted from upstream revision `aae3af194322c60d21bc44062b64c3fec912be50`. +`MANIFEST.json` records the source revision and SHA-256 of the GGUF. The model +remains subject to the [NVIDIA Open Model License](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/). +This conversion grants no additional rights. diff --git a/scripts/publish_gguf.py b/scripts/publish_gguf.py index 8cacc4c..44844e4 100755 --- a/scripts/publish_gguf.py +++ b/scripts/publish_gguf.py @@ -23,15 +23,54 @@ ROOT = Path(__file__).resolve().parent.parent HF_ORG = "LocalAI-io" # Hugging Face organisation; GitHub is localai-org. DEFAULT_REPOS = { "text": f"{HF_ORG}/Llama-3-Kimodo-GGML", - "motion": f"{HF_ORG}/Kimodo-SMPLX-RP-v1-GGML", } -MOTION_NAME = "kimodo-smplx-rp-v1-f32.gguf" +MOTION_MODELS = { + "smplx-rp-v1": { + "repo": f"{HF_ORG}/Kimodo-SMPLX-RP-v1-GGML", + "source": "nvidia/Kimodo-SMPLX-RP-v1", + "folder": "Kimodo-SMPLX-RP-v1", + "revision": "1419ba56b734c48bbafb41fefa84088ca94583b5", + "file": "kimodo-smplx-rp-v1-f32.gguf", + "redistributable": False, + }, + "soma-rp-v1.1": { + "repo": f"{HF_ORG}/Kimodo-SOMA-RP-v1.1-GGML", + "source": "nvidia/Kimodo-SOMA-RP-v1.1", + "folder": "Kimodo-SOMA-RP-v1.1", + "revision": "6c9233af1180b8151e3c4703477104af5dce9dd5", + "file": "kimodo-soma-rp-v1.1-f32.gguf", + "redistributable": True, + }, + "soma-seed-v1.1": { + "repo": f"{HF_ORG}/Kimodo-SOMA-SEED-v1.1-GGML", + "source": "nvidia/Kimodo-SOMA-SEED-v1.1", + "folder": "Kimodo-SOMA-SEED-v1.1", + "revision": "aae3af194322c60d21bc44062b64c3fec912be50", + "file": "kimodo-soma-seed-v1.1-f32.gguf", + "redistributable": True, + }, + "g1-rp-v1": { + "repo": f"{HF_ORG}/Kimodo-G1-RP-v1-GGML", + "source": "nvidia/Kimodo-G1-RP-v1", + "folder": "Kimodo-G1-RP-v1", + "revision": "3020ad8c419c244e0429d360163730c63c4ed011", + "file": "kimodo-g1-rp-v1-f32.gguf", + "redistributable": True, + }, + "g1-seed-v1": { + "repo": f"{HF_ORG}/Kimodo-G1-SEED-v1-GGML", + "source": "nvidia/Kimodo-G1-SEED-v1", + "folder": "Kimodo-G1-SEED-v1", + "revision": "5e6f2c7e18c2ab834c8d7983b9dcce701e5c6097", + "file": "kimodo-g1-seed-v1-f32.gguf", + "redistributable": True, + }, +} TEXT_NAMES = ( "tokenizer.gguf", "embedding.gguf", "final-norm.gguf", *(f"layer-{index:02d}.gguf" for index in range(32)), ) SOURCE_REVISIONS = { - "nvidia/Kimodo-SMPLX-RP-v1": "1419ba56b734c48bbafb41fefa84088ca94583b5", "meta-llama/Meta-Llama-3-8B-Instruct": "8afb486c1db24fe5011ec46dfbe5b5dccdb575c2", "McGill-NLP/LLM2Vec-Meta-Llama-3-8B-Instruct-mntp": "31474e395ada192e8ed1586db6be79fb3b70c9c0", "McGill-NLP/LLM2Vec-Meta-Llama-3-8B-Instruct-mntp-supervised": "baa8ebf04a1c2500e61288e7dad65e8ae42601a7", @@ -47,24 +86,23 @@ def digest(path: Path) -> str: return value.hexdigest() -def require_revision(repo: str) -> None: - revision = ROOT / "models" / { - "nvidia/Kimodo-SMPLX-RP-v1": "Kimodo-SMPLX-RP-v1", +def require_revision(repo: str, expected: str, folder: str | None = None) -> None: + revision = ROOT / "models" / (folder or { "meta-llama/Meta-Llama-3-8B-Instruct": "llama3-8b-instruct-base", "McGill-NLP/LLM2Vec-Meta-Llama-3-8B-Instruct-mntp": "llm2vec-mntp-adapter", "McGill-NLP/LLM2Vec-Meta-Llama-3-8B-Instruct-mntp-supervised": "llm2vec-adapter", - }[repo] / "REVISION" + }[repo]) / "REVISION" if not revision.is_file(): raise ValueError(f"missing provenance file: {revision}") actual = revision.read_text(encoding="utf-8").split()[0] - if actual != SOURCE_REVISIONS[repo]: - raise ValueError(f"unexpected {repo} revision: {actual} (expected {SOURCE_REVISIONS[repo]})") + if actual != expected: + raise ValueError(f"unexpected {repo} revision: {actual} (expected {expected})") -def artifacts(component: str, motion: Path, bundle: Path) -> list[tuple[Path, str]]: +def artifacts(component: str, motion: Path, motion_name: str, bundle: Path) -> list[tuple[Path, str]]: result: list[tuple[Path, str]] = [] if component == "motion": - result.append((motion, f"models/{MOTION_NAME}")) + result.append((motion, f"models/{motion_name}")) else: result.extend((bundle / name, f"generated/llm2vec-text-bundle/{name}") for name in TEXT_NAMES) for source, destination in result: @@ -82,7 +120,9 @@ def artifacts(component: str, motion: Path, bundle: Path) -> list[tuple[Path, st def main() -> int: parser = argparse.ArgumentParser(description=__doc__) - parser.add_argument("--motion", type=Path, default=ROOT / "models" / MOTION_NAME) + parser.add_argument("--motion", type=Path, default=None, + help="motion GGUF (defaults to the selected model's converted output)") + parser.add_argument("--motion-model", choices=tuple(MOTION_MODELS), default="soma-rp-v1.1") parser.add_argument("--text-bundle", type=Path, default=ROOT / "generated/llm2vec-text-bundle") parser.add_argument("--component", choices=("text", "motion"), required=True, @@ -93,21 +133,28 @@ def main() -> int: parser.add_argument("--confirm-upstream-licences", action="store_true", help="required with --upload; confirms authority to redistribute all inputs") args = parser.parse_args() - repo = args.repo or DEFAULT_REPOS[args.component] - card_dir = ROOT / "scripts/hf" / ("Llama-3-Kimodo-GGML" if args.component == "text" else "Kimodo-SMPLX-RP-v1-GGML") + motion_spec = MOTION_MODELS[args.motion_model] + if args.component == "motion" and args.upload and not motion_spec["redistributable"]: + print("error: the SMPL-X checkpoint licence prohibits distributing Derivative Models; local conversion only", + file=sys.stderr) + return 2 + motion = args.motion or ROOT / "models" / motion_spec["file"] + repo = args.repo or (DEFAULT_REPOS["text"] if args.component == "text" else motion_spec["repo"]) + card_dir = ROOT / "scripts/hf" / ("Llama-3-Kimodo-GGML" if args.component == "text" else repo.rsplit("/", 1)[1]) card = card_dir / "README.md" notice = card_dir / "NOTICE" relevant_sources = (SOURCE_REVISIONS if args.component == "text" - else {"nvidia/Kimodo-SMPLX-RP-v1": SOURCE_REVISIONS["nvidia/Kimodo-SMPLX-RP-v1"]}) + else {motion_spec["source"]: motion_spec["revision"]}) try: if not card.is_file() or not notice.is_file(): raise ValueError("version-controlled model card or NOTICE is missing") if args.component == "text" and not LLAMA_LICENSE.is_file(): raise ValueError("Meta Llama 3 licence is missing") - for source_repo in relevant_sources: - require_revision(source_repo) - files = artifacts(args.component, args.motion, args.text_bundle) + for source_repo, revision in relevant_sources.items(): + folder = motion_spec["folder"] if args.component == "motion" else None + require_revision(source_repo, revision, folder) + files = artifacts(args.component, motion, motion_spec["file"], args.text_bundle) except ValueError as error: print(f"error: {error}", file=sys.stderr) return 1 diff --git a/src/denoiser.cpp b/src/denoiser.cpp index 4060db4..91dbbc4 100644 --- a/src/denoiser.cpp +++ b/src/denoiser.cpp @@ -90,9 +90,10 @@ std::expected, std::string> run_two_stage_denoiser( std::span embedding, std::span timesteps, std::span headings, std::span mask, std::size_t batch, std::size_t frames) { - if (!batch || !frames || x.size()!=batch*frames*546 || mask.size()!=batch*frames) + const size_t dim=weights.motion_dim(), root_input_dim=2*dim, body_input_dim=2*dim-1; + if (!batch || !frames || !dim || x.size()!=batch*frames*root_input_dim || mask.size()!=batch*frames) return std::unexpected("invalid two-stage denoiser input dimensions"); - auto root=run_motion_transformer(weights,"root_model.",x,546,embedding,timesteps,headings,batch,frames); + auto root=run_motion_transformer(weights,"root_model.",x,root_input_dim,embedding,timesteps,headings,batch,frames); if(!root)return std::unexpected(root.error()); auto gm=weights.f32_values("stats.global_root.mean"), gs=weights.f32_values("stats.global_root.std"), lm=weights.f32_values("stats.local_root.mean"), ls=weights.f32_values("stats.local_root.std"); if(!gm)return std::unexpected(gm.error()); @@ -101,16 +102,16 @@ std::expected, std::string> run_two_stage_denoiser( if(!ls)return std::unexpected(ls.error()); auto local=global_root_to_local_root(*root,mask,batch,frames,*gm,*gs,*lm,*ls); if(!local)return std::unexpected(local.error()); - std::vector body_input(batch*frames*545); + std::vector body_input(batch*frames*body_input_dim); for(std::size_t b=0;bdata()+(b*frames+t)*4,4*sizeof(float)); - std::memcpy(body_input.data()+dst+4,x.data()+src+5,541*sizeof(float)); + std::memcpy(body_input.data()+dst+4,x.data()+src+5,(root_input_dim-5)*sizeof(float)); } - auto body=run_motion_transformer(weights,"body_model.",body_input,545,embedding,timesteps,headings,batch,frames); + auto body=run_motion_transformer(weights,"body_model.",body_input,body_input_dim,embedding,timesteps,headings,batch,frames); if(!body)return std::unexpected(body.error()); - std::vector output(batch*frames*273); - for(std::size_t b=0;bdata()+r,5*sizeof(float));std::memcpy(output.data()+o+5,body->data()+q,268*sizeof(float));} + std::vector output(batch*frames*dim); + for(std::size_t b=0;bdata()+r,5*sizeof(float));std::memcpy(output.data()+o+5,body->data()+q,(dim-5)*sizeof(float));} return output; } @@ -118,7 +119,7 @@ std::expected, std::string> run_separated_cfg_denoiser( const ggml_motion_weights &weights, std::span motion, std::span embedding, float timestep, float text_weight, float constraint_weight, std::size_t frames) { - const std::vector empty(frames*273, 0.f); + const std::vector empty(frames*weights.motion_dim(), 0.f); return run_separated_cfg_denoiser_conditioned(weights, motion, embedding, empty, empty, timestep, 0.f, text_weight, constraint_weight, frames); } @@ -128,25 +129,26 @@ std::expected, std::string> run_separated_cfg_denoiser_condit std::span embedding, std::span observed, std::span observed_mask, float timestep, float heading, float text_weight, float constraint_weight, std::size_t frames) { - if (motion.size()!=frames*273 || embedding.size()!=4096 || !std::isfinite(timestep) || !std::isfinite(text_weight) || !std::isfinite(constraint_weight)) + const size_t dim=weights.motion_dim(); + if (!dim || motion.size()!=frames*dim || embedding.size()!=4096 || !std::isfinite(timestep) || !std::isfinite(text_weight) || !std::isfinite(constraint_weight)) return std::unexpected("invalid separated CFG denoiser input"); - if (observed.size()!=frames*273 || observed_mask.size()!=frames*273 || !std::isfinite(heading)) + if (observed.size()!=frames*dim || observed_mask.size()!=frames*dim || !std::isfinite(heading)) return std::unexpected("invalid separated CFG condition dimensions"); - constexpr size_t cfg_batch=3; std::vector extended(cfg_batch*frames*546), text(cfg_batch*4096), times(cfg_batch,timestep), headings(cfg_batch,heading), mask(cfg_batch*frames,1.f); + constexpr size_t cfg_batch=3; std::vector extended(cfg_batch*frames*2*dim), text(cfg_batch*4096), times(cfg_batch,timestep), headings(cfg_batch,heading), mask(cfg_batch*frames,1.f); for(size_t b=0;b result(frames*273); + std::vector result(frames*dim); for(size_t i=0;i, std::string> sample_motion_from_noise( const ggml_motion_weights &weights, std::span initial, std::span embedding, std::size_t frames, unsigned steps, float text_weight, float constraint_weight) { - if(initial.size()!=frames*273) return std::unexpected("invalid initial motion noise dimensions"); + if(initial.size()!=frames*weights.motion_dim()) return std::unexpected("invalid initial motion noise dimensions"); auto schedule=make_cosine_schedule(1000,steps); if(!schedule)return std::unexpected(schedule.error()); std::vector state(initial.begin(),initial.end()), next(state.size()); for(unsigned i=steps;i-->0;) { @@ -173,7 +175,7 @@ std::expected, std::string> sample_motion_from_noise_conditio std::span embedding, std::span observed, std::span observed_mask, float heading, std::size_t frames, unsigned steps, float text_weight, float constraint_weight) { - if(initial.size()!=frames*273 || observed.size()!=initial.size() || observed_mask.size()!=initial.size()) + if(initial.size()!=frames*weights.motion_dim() || observed.size()!=initial.size() || observed_mask.size()!=initial.size()) return std::unexpected("invalid conditioned motion noise dimensions"); auto schedule=make_cosine_schedule(1000,steps); if(!schedule)return std::unexpected(schedule.error()); std::vector state(initial.begin(),initial.end()), next(state.size()); diff --git a/src/denoiser.hpp b/src/denoiser.hpp index 504b16f..8749607 100644 --- a/src/denoiser.hpp +++ b/src/denoiser.hpp @@ -34,15 +34,15 @@ std::expected, std::string> run_motion_transformer( std::span headings, std::size_t batch, std::size_t frames); // Exact two-stage Kimodo denoiser for concatenated motion/mask inputs -// [B,T,546]. Returned clean prediction is [B,T,273]. +// [B,T,2*motion_dim]. Returned clean prediction is [B,T,motion_dim]. std::expected, std::string> run_two_stage_denoiser( const ggml_motion_weights &weights, std::span motion_and_mask, std::span text_embedding, std::span timesteps, std::span headings, std::span motion_mask, std::size_t batch, std::size_t frames); -// Unconstrained separated CFG wrapper. `motion` is [T,273], embedding is -// [4096], and the result is one clean [T,273] prediction. +// Unconstrained separated CFG wrapper. `motion` is [T,motion_dim], embedding +// is [4096], and the result is one clean prediction. std::expected, std::string> run_separated_cfg_denoiser( const ggml_motion_weights &weights, std::span motion, std::span embedding, float timestep, float text_weight, @@ -54,7 +54,7 @@ std::expected, std::string> sample_motion_from_noise( std::span embedding, std::size_t frames, unsigned steps, float text_weight, float constraint_weight); -// Multi-prompt transition sampler. `observed` and `observed_mask` are [T,273] +// Multi-prompt transition sampler. `observed` and `observed_mask` are [T,motion_dim] // normalized motion-representation values/masks. This mirrors the upstream // concat-mask denoiser: text, constraint, and unconditional CFG branches. std::expected, std::string> sample_motion_from_noise_conditioned( diff --git a/src/generate.cpp b/src/generate.cpp index 4f73168..63ebd58 100644 --- a/src/generate.cpp +++ b/src/generate.cpp @@ -41,7 +41,7 @@ int main(int argc, char **argv) try { const std::filesystem::path output(argv[7]); std::filesystem::create_directories(output); write_f32(output / "root_positions.f32", motion->root_positions); write_f32(output / "local_rotations_xyzw.f32", motion->local_rotations_xyzw); - std::cout << "generated " << motion->frames << " SMPL-X22 sequence frames\n"; + std::cout << "generated " << motion->frames << " frames with " << motion->joints << " joints\n"; return 0; } if (argc != 8) { @@ -63,7 +63,7 @@ int main(int argc, char **argv) try { std::filesystem::create_directories(output); write_f32(output / "root_positions.f32", motion->root_positions); write_f32(output / "local_rotations_xyzw.f32", motion->local_rotations_xyzw); - std::cout << "generated " << motion->frames << " SMPL-X22 frames\n"; + std::cout << "generated " << motion->frames << " frames with " << motion->joints << " joints\n"; return 0; } catch (const std::exception &error) { std::cerr << error.what() << '\n'; diff --git a/src/ggml_weights.cpp b/src/ggml_weights.cpp index 499ba81..4fe751f 100644 --- a/src/ggml_weights.cpp +++ b/src/ggml_weights.cpp @@ -61,6 +61,9 @@ std::expected, std::string> ggml_motion_wei if (!checked) return std::unexpected(checked.error()); if (auto valid = validate_motion_gguf(*checked); !valid) return std::unexpected(valid.error()); auto result = std::unique_ptr(new ggml_motion_weights); + result->skeleton_ = checked->strings.at("kimodo.skeleton"); + result->motion_dim_ = static_cast(checked->uints.at("kimodo.motion_dim")); + result->body_dim_ = static_cast(checked->uints.at("kimodo.body_dim")); gguf_init_params params{true, &result->context_}; result->gguf_ = gguf_init_from_file(std::string(path).c_str(), params); if (!result->gguf_ || !result->context_) return std::unexpected("GGML could not load checked motion GGUF"); diff --git a/src/ggml_weights.hpp b/src/ggml_weights.hpp index 376a884..796fd72 100644 --- a/src/ggml_weights.hpp +++ b/src/ggml_weights.hpp @@ -1,5 +1,6 @@ #pragma once +#include #include #include #include @@ -27,6 +28,9 @@ public: ggml_tensor *tensor(std::string_view name) const; std::expected, std::string> f32_values(std::string_view name) const; ggml_backend *backend() const noexcept { return backend_; } + std::string_view skeleton_key() const noexcept { return skeleton_; } + std::size_t motion_dim() const noexcept { return motion_dim_; } + std::size_t body_dim() const noexcept { return body_dim_; } private: ggml_motion_weights() = default; @@ -34,6 +38,9 @@ private: gguf_context *gguf_ = nullptr; ggml_backend *backend_ = nullptr; ggml_backend_buffer *buffer_ = nullptr; + std::string skeleton_; + std::size_t motion_dim_ = 0; + std::size_t body_dim_ = 0; }; } // namespace kimodo::detail diff --git a/src/gguf.cpp b/src/gguf.cpp index 363cdf6..5c49220 100644 --- a/src/gguf.cpp +++ b/src/gguf.cpp @@ -1,4 +1,5 @@ #include "gguf.hpp" +#include "skeleton.hpp" #include #include @@ -106,8 +107,14 @@ std::expected validate_motion_gguf(const gguf_file &file) { if (format == file.uints.end() || format->second != 1) return std::unexpected("unsupported Kimodo motion GGUF format"); const auto skeleton = file.strings.find("kimodo.skeleton"); - if (skeleton == file.strings.end() || skeleton->second != "smplx22") - return std::unexpected("first runtime supports only smplx22 skeletons"); + if (skeleton == file.strings.end() || !find_skeleton(skeleton->second)) + return std::unexpected("motion GGUF has an unsupported skeleton"); + const auto &spec = *find_skeleton(skeleton->second); + const auto motion_dim = file.uints.find("kimodo.motion_dim"); + const auto body_dim = file.uints.find("kimodo.body_dim"); + if (motion_dim == file.uints.end() || motion_dim->second != spec.motion_dim() || + body_dim == file.uints.end() || body_dim->second != spec.body_dim()) + return std::unexpected("motion GGUF dimensions do not match its skeleton"); const auto width = file.uints.find("kimodo.text_embedding_width"); if (width == file.uints.end() || width->second != 4096) return std::unexpected("motion GGUF has incompatible text embedding width"); diff --git a/src/inspect.cpp b/src/inspect.cpp index a1ce402..8563fbd 100644 --- a/src/inspect.cpp +++ b/src/inspect.cpp @@ -22,5 +22,5 @@ int main(int argc, char **argv) { } gguf_free(file); ggml_free(tensor_context); #endif - std::puts("Kimodo SMPL-X motion GGUF: valid (414 F32 tensors)"); + std::puts("Kimodo motion GGUF: valid (414 F32 tensors)"); } diff --git a/src/model.cpp b/src/model.cpp index 70de857..a3e5779 100644 --- a/src/model.cpp +++ b/src/model.cpp @@ -1,5 +1,6 @@ #include #include "gguf.hpp" +#include "skeleton.hpp" #ifdef KIMODO_HAVE_GGML #include "ggml_weights.hpp" #include "denoiser.hpp" @@ -15,6 +16,7 @@ namespace kimodo { struct model::impl { detail::gguf_file motion; std::string motion_path; + const detail::skeleton_spec *skeleton = nullptr; #ifdef KIMODO_HAVE_GGML mutable std::unique_ptr weights; std::unique_ptr text; @@ -30,6 +32,7 @@ std::expected, std::string> model::load(std::string_view auto state = std::make_unique(); state->motion = std::move(*file); state->motion_path = std::string(motion_path); + state->skeleton = detail::find_skeleton(state->motion.strings.at("kimodo.skeleton")); #ifdef KIMODO_HAVE_GGML if (!text_path.empty()) { auto text = detail::llm_text_encoder::load(text_path); @@ -74,7 +77,8 @@ std::expected model::generate_embedding( } std::mt19937_64 rng(seed); std::normal_distribution normal(0.f, 1.f); - std::vector noise(static_cast(frames)*273); + const size_t motion_dim=impl_->skeleton->motion_dim(); + std::vector noise(static_cast(frames)*motion_dim); for (float &value : noise) value = normal(rng); auto sampled = detail::sample_motion_from_noise(*impl_->weights, noise, embedding, frames, steps, text_cfg, constraint_cfg); if (!sampled) return std::unexpected(sampled.error()); @@ -84,10 +88,10 @@ std::expected model::generate_embedding( if (!global_std) return std::unexpected(global_std.error()); if (!body_mean) return std::unexpected(body_mean.error()); if (!body_std) return std::unexpected(body_std.error()); - auto decoded=detail::decode_smplx22(*sampled,frames,*global_mean,*global_std,*body_mean,*body_std); + auto decoded=detail::decode_motion(*sampled,frames,*impl_->skeleton,*global_mean,*global_std,*body_mean,*body_std); if (!decoded) return std::unexpected(decoded.error()); motion_data result; - result.frames=frames; result.joints=22; + result.frames=frames; result.joints=static_cast(impl_->skeleton->joints()); result.local_rotations_xyzw=std::move(decoded->local_xyzw); result.root_positions=std::move(decoded->root_positions); return result; @@ -127,22 +131,23 @@ std::expected model::generate_text_sequence( const auto sampled_frames = static_cast(segment.frames) + (index == 0 ? 0 : transition_frames); embeddings.push_back(*embedding); - noise.emplace_back(sampled_frames*273); + noise.emplace_back(sampled_frames*impl_->skeleton->motion_dim()); for (float &value : noise.back()) value=normal(rng); sampled.push_back({embeddings.back(), noise.back(), segment.frames}); } auto joined=detail::sample_motion_sequence_from_noise(*impl_->weights,sampled,transition_frames,steps,text_cfg,constraint_cfg); if (!joined) return std::unexpected(joined.error()); - const auto frames=static_cast(joined->size()/273); + const size_t motion_dim=impl_->skeleton->motion_dim(), body_dim=impl_->skeleton->body_dim(); + const auto frames=static_cast(joined->size()/motion_dim); auto normalized=*joined; for (size_t row=0; rowskeleton,*gm,*gs,*bm,*bs); if (!decoded) return std::unexpected(decoded.error()); - motion_data result; result.frames=frames; result.joints=22; + motion_data result; result.frames=frames; result.joints=static_cast(impl_->skeleton->joints()); result.local_rotations_xyzw=std::move(decoded->local_xyzw); result.root_positions=std::move(decoded->root_positions); return result; #else diff --git a/src/motion_decode.cpp b/src/motion_decode.cpp index 31edd04..41d936f 100644 --- a/src/motion_decode.cpp +++ b/src/motion_decode.cpp @@ -1,13 +1,14 @@ #include "motion_decode.hpp" -#include +#include "skeleton.hpp" #include +#include namespace kimodo::detail { namespace { -constexpr int parent[22]={-1,0,0,0,1,2,3,4,5,6,7,8,9,9,9,12,13,14,16,17,18,19}; struct M{float v[9];}; M mul(const M&a,const M&b){M r{};for(int i=0;i<3;++i)for(int j=0;j<3;++j)for(int k=0;k<3;++k)r.v[i*3+j]+=a.v[i*3+k]*b.v[k*3+j];return r;} M tr(const M&a){M r{};for(int i=0;i<3;++i)for(int j=0;j<3;++j)r.v[i*3+j]=a.v[j*3+i];return r;} M six(const float*x){float n=std::sqrt(x[0]*x[0]+x[1]*x[1]+x[2]*x[2]);float a[3]={x[0]/n,x[1]/n,x[2]/n};float z[3]={a[1]*x[5]-a[2]*x[4],a[2]*x[3]-a[0]*x[5],a[0]*x[4]-a[1]*x[3]};n=std::sqrt(z[0]*z[0]+z[1]*z[1]+z[2]*z[2]);for(float&v:z)v/=n;float b[3]={z[1]*a[2]-z[2]*a[1],z[2]*a[0]-z[0]*a[2],z[0]*a[1]-z[1]*a[0]};return M{{a[0],b[0],z[0],a[1],b[1],z[1],a[2],b[2],z[2]}};} void quat(const M&m,float*q){float w,x,y,z,t=m.v[0]+m.v[4]+m.v[8];if(t>0){float s=2*std::sqrt(t+1);w=.25f*s;x=(m.v[7]-m.v[5])/s;y=(m.v[2]-m.v[6])/s;z=(m.v[3]-m.v[1])/s;}else if(m.v[0]>m.v[4]&&m.v[0]>m.v[8]){float s=2*std::sqrt(1+m.v[0]-m.v[4]-m.v[8]);w=(m.v[7]-m.v[5])/s;x=.25f*s;y=(m.v[1]+m.v[3])/s;z=(m.v[2]+m.v[6])/s;}else if(m.v[4]>m.v[8]){float s=2*std::sqrt(1+m.v[4]-m.v[0]-m.v[8]);w=(m.v[2]-m.v[6])/s;x=(m.v[1]+m.v[3])/s;y=.25f*s;z=(m.v[5]+m.v[7])/s;}else{float s=2*std::sqrt(1+m.v[8]-m.v[0]-m.v[4]);w=(m.v[3]-m.v[1])/s;x=(m.v[2]+m.v[6])/s;y=(m.v[5]+m.v[7])/s;z=.25f*s;}q[0]=x;q[1]=y;q[2]=z;q[3]=w;} } -std::expected decode_smplx22(std::spanx,size_t T,std::spangm,std::spangs,std::spanbm,std::spanbs){if(x.size()!=T*273||gm.size()!=5||gs.size()!=5||bm.size()!=268||bs.size()!=268)return std::unexpected("invalid SMPL-X decode inputs");decoded_motion o;o.local_xyzw.resize(T*22*4);o.root_positions.resize(T*3);auto scale=[](float s){return std::sqrt(s*s+1.e-5f);};for(size_t t=0;t f{};for(int i=0;i<5;++i)f[i]=in[i]*scale(gs[i])+gm[i];for(int i=0;i<268;++i)f[5+i]=in[5+i]*scale(bs[i])+bm[i];o.root_positions[t*3]=f[0]+f[5];o.root_positions[t*3+1]=f[6];o.root_positions[t*3+2]=f[2]+f[7];M g[22],l[22];for(int j=0;j<22;++j)g[j]=six(f.data()+71+j*6);for(int j=0;j<22;++j)l[j]=parent[j]<0?g[j]:mul(tr(g[parent[j]]),g[j]);for(int j=0;j<22;++j)quat(l[j],o.local_xyzw.data()+(t*22+j)*4);}return o;} +std::expected decode_motion(std::spanx,size_t T,const skeleton_spec&s,std::spangm,std::spangs,std::spanbm,std::spanbs){const size_t D=s.motion_dim(),J=s.joints(),body=D-5,rotation=5+3*J;if(x.size()!=T*D||gm.size()!=5||gs.size()!=5||bm.size()!=body||bs.size()!=body)return std::unexpected("invalid "+std::string(s.key)+" decode inputs");decoded_motion o;o.local_xyzw.resize(T*J*4);o.root_positions.resize(T*3);auto scale=[](float v){return std::sqrt(v*v+1.e-5f);};std::vectorf(D);std::vectorg(J),l(J);for(size_t t=0;t(s.parents[j])]),g[j]);for(size_t j=0;j decode_smplx22(std::spanx,size_t T,std::spangm,std::spangs,std::spanbm,std::spanbs){return decode_motion(x,T,smplx22_spec,gm,gs,bm,bs);} } diff --git a/src/motion_decode.hpp b/src/motion_decode.hpp index 2aed827..16c8831 100644 --- a/src/motion_decode.hpp +++ b/src/motion_decode.hpp @@ -4,6 +4,8 @@ #include #include namespace kimodo::detail { +struct skeleton_spec; struct decoded_motion { std::vector local_xyzw, root_positions; }; +std::expected decode_motion(std::span normalized, std::size_t frames, const skeleton_spec &skeleton, std::span global_mean, std::span global_std, std::span body_mean, std::span body_std); std::expected decode_smplx22(std::span normalized, std::size_t frames, std::span global_mean, std::span global_std, std::span body_mean, std::span body_std); } diff --git a/src/sample_fixture.cpp b/src/sample_fixture.cpp index 23f1264..021e372 100644 --- a/src/sample_fixture.cpp +++ b/src/sample_fixture.cpp @@ -3,6 +3,7 @@ #include "denoiser.hpp" #include "ggml_weights.hpp" #include "motion_decode.hpp" +#include "skeleton.hpp" #include #include @@ -54,10 +55,11 @@ int main(int argc, char **argv) try { const std::filesystem::path fixture(argv[2]), output(argv[5]); const auto embedding = read_f32(fixture / "text_features.f32"); const auto noise = read_f32(fixture / "sampling_initial_noise.f32"); - if (embedding.size() != 4096 || noise.size() != frames * 273) - throw std::runtime_error("fixture does not match requested [1,1,4096] embedding and [1,T,273] noise"); auto weights = kimodo::detail::ggml_motion_weights::load(argv[1]); if (!weights) throw std::runtime_error(weights.error()); + const auto *skeleton=kimodo::detail::find_skeleton((*weights)->skeleton_key()); + if (!skeleton || embedding.size() != 4096 || noise.size() != frames * skeleton->motion_dim()) + throw std::runtime_error("fixture does not match the requested embedding and model motion dimensions"); auto sampled = kimodo::detail::sample_motion_from_noise(**weights, noise, embedding, frames, steps, 2.f, 2.f); if (!sampled) throw std::runtime_error(sampled.error()); auto gm = (**weights).f32_values("stats.global_root.mean"); @@ -65,7 +67,7 @@ int main(int argc, char **argv) try { auto bm = (**weights).f32_values("stats.body.mean"); auto bs = (**weights).f32_values("stats.body.std"); if (!gm || !gs || !bm || !bs) throw std::runtime_error("missing motion normalisation tensors"); - auto decoded = kimodo::detail::decode_smplx22(*sampled, frames, *gm, *gs, *bm, *bs); + auto decoded = kimodo::detail::decode_motion(*sampled, frames, *skeleton, *gm, *gs, *bm, *bs); if (!decoded) throw std::runtime_error(decoded.error()); std::filesystem::create_directories(output); write_f32(output / "sampling_final_state.f32", *sampled); diff --git a/src/sequence.cpp b/src/sequence.cpp index 031c253..6dbf6da 100644 --- a/src/sequence.cpp +++ b/src/sequence.cpp @@ -1,132 +1,72 @@ #include "denoiser.hpp" #include "ggml_weights.hpp" +#include "skeleton.hpp" #include -#include #include +#include namespace kimodo::detail { namespace { -constexpr int parent[22]={-1,0,0,0,1,2,3,4,5,6,7,8,9,9,9,12,13,14,16,17,18,19}; -constexpr float offset[22][3]={{0,0,0},{.052299179F,-.093935639F,-.027606763F},{-.057192899F,-.106548190F,-.022217851F},{-.001495834F,.112929940F,-.024981268F},{.058866613F,-.416441321F,-.006556974F},{-.048074268F,-.397559673F,-.014061437F},{.006900469F,.145636231F,-.006858510F},{-.041737989F,-.437583506F,-.029511765F},{.014489345F,-.446852267F,-.018029511F},{-.010334037F,.056081813F,.021115851F},{.049293540F,-.065279245F,.126259089F},{-.040575184F,-.065286517F,.127075911F},{-.011025756F,.171365142F,-.028827066F},{.047724526F,.087643057F,-.008375450F},{-.046636276F,.086612143F,-.014864366F},{.024654359F,.175390735F,.024463326F},{.126284808F,.057680372F,-.013885141F},{-.109341696F,.053674292F,-.009117880F},{.272907287F,-.069853373F,-.039094493F},{-.292028785F,-.035440356F,-.024564851F},{.276173830F,.021254137F,-.002478220F},{-.271878421F,-.004834589F,-.016445294F}}; struct mat { double v[9]; }; mat mul(const mat&a,const mat&b){mat r{};for(int i=0;i<3;++i)for(int j=0;j<3;++j)for(int k=0;k<3;++k)r.v[i*3+j]+=a.v[i*3+k]*b.v[k*3+j];return r;} mat trans(const mat&a){mat r{};for(int i=0;i<3;++i)for(int j=0;j<3;++j)r.v[i*3+j]=a.v[j*3+i];return r;} -mat cont6(const float *x) { double a[3]={x[0],x[1],x[2]}, n=std::sqrt(a[0]*a[0]+a[1]*a[1]+a[2]*a[2]); for(double &q:a)q/=n; double z[3]={a[1]*x[5]-a[2]*x[4],a[2]*x[3]-a[0]*x[5],a[0]*x[4]-a[1]*x[3]}; n=std::sqrt(z[0]*z[0]+z[1]*z[1]+z[2]*z[2]);for(double&q:z)q/=n; double b[3]={z[1]*a[2]-z[2]*a[1],z[2]*a[0]-z[0]*a[2],z[0]*a[1]-z[1]*a[0]};return {{a[0],b[0],z[0],a[1],b[1],z[1],a[2],b[2],z[2]}}; } -void rotate(const mat&m,const float *x,float *o){for(int i=0;i<3;++i)o[i]=static_cast(m.v[i*3]*x[0]+m.v[i*3+1]*x[1]+m.v[i*3+2]*x[2]);} +mat cont6(const float*x){double a[3]={x[0],x[1],x[2]},n=std::sqrt(a[0]*a[0]+a[1]*a[1]+a[2]*a[2]);for(double&q:a)q/=n;double z[3]={a[1]*x[5]-a[2]*x[4],a[2]*x[3]-a[0]*x[5],a[0]*x[4]-a[1]*x[3]};n=std::sqrt(z[0]*z[0]+z[1]*z[1]+z[2]*z[2]);for(double&q:z)q/=n;double b[3]={z[1]*a[2]-z[2]*a[1],z[2]*a[0]-z[0]*a[2],z[0]*a[1]-z[1]*a[0]};return{{a[0],b[0],z[0],a[1],b[1],z[1],a[2],b[2],z[2]}};} +void rotate(const mat&m,const std::array&x,float*o){for(int i=0;i<3;++i)o[i]=static_cast(m.v[i*3]*x[0]+m.v[i*3+1]*x[1]+m.v[i*3+2]*x[2]);} + +// Reconstruct the full-body/end-effector condition used by upstream +// `_multiprompt`, generalized over the three released skeleton layouts. +float condition_row(const float *raw, const skeleton_spec &s, float *value) { + const size_t D=s.motion_dim(),J=s.joints(),rotation_begin=5+3*J; + std::copy_n(raw,D,value); + std::vector decoded(J),local(J),global(J); + for(size_t j=0;j(s.parents[j])]),decoded[j]); + const float root[3]={value[0]+value[5],value[6],value[2]+value[7]}; + std::vector> posed(J); + for(size_t j=0;j(parent)],local[j]);float offset[3];rotate(global[static_cast(parent)],s.offsets[j],offset);for(int k=0;k<3;++k)posed[j][k]=posed[static_cast(parent)][k]+offset[k];}} + const auto right=s.hips[0],left=s.hips[1]; + const float angle=std::atan2(posed[right][2]-posed[left][2],-(posed[right][0]-posed[left][0])); + value[1]=root[1];value[3]=std::cos(angle);value[4]=std::sin(angle); + for(size_t j=0;j(d)]=static_cast(global[joint].v[(d%3)*3+d/3]); + return angle; } +} + std::expected prepare_sequence_transition( - const ggml_motion_weights &, std::span previous, + const ggml_motion_weights &weights, std::span previous, std::size_t continuation_frames, unsigned transition_frames) { - constexpr size_t features = 273; - const size_t overlap=transition_frames; - if (!overlap || overlap>=continuation_frames || previous.size()<=(overlap*features)) + const auto *s=find_skeleton(weights.skeleton_key()); + if(!s)return std::unexpected("unsupported sequence skeleton"); + const size_t D=s->motion_dim(),J=s->joints(),rotation_begin=5+3*J,rotation_end=rotation_begin+6*J,overlap=transition_frames; + if(!overlap||overlap>=continuation_frames||previous.size()<=overlap*D||previous.size()%D) return std::unexpected("invalid sequence transition"); sequence_transition result; - result.observed.resize((continuation_frames+overlap)*features); + result.observed.resize((continuation_frames+overlap)*D); result.observed_mask.resize(result.observed.size()); - const size_t previous_start=previous.size()-overlap*features; - constexpr std::array, 3> constrained = {{{0, 71}, {113, 125}, {191, 203}}}; - for (size_t frame=0; frame value{}; - const float *raw=previous.data()+previous_start+base; - std::copy_n(raw,features,value.data()); - mat decoded[22],local[22],global[22]; for(int j=0;j<22;++j)decoded[j]=cont6(value.data()+71+j*6); - for(int j=0;j<22;++j) local[j]=parent[j]<0?decoded[j]:mul(trans(decoded[parent[j]]),decoded[j]); - float root[3]={value[0]+value[5],value[6],value[2]+value[7]}, posed[22][3]{}; - for(int j=0;j<22;++j){if(parent[j]<0){global[j]=local[j];posed[j][0]=root[0];posed[j][1]=root[1];posed[j][2]=root[2];}else{global[j]=mul(global[parent[j]],local[j]);float d[3];rotate(global[parent[j]],offset[j],d);for(int k=0;k<3;++k)posed[j][k]=posed[parent[j]][k]+d[k];}} - // FullBodyConstraintSet: smooth root, root Y, heading, all joint - // positions; EndEffectorConstraintSet adds its four rotation blocks. - value[0]=value[0]; value[1]=root[1]; value[2]=value[2]; - // `compute_heading_angle`: right hip minus left hip. - const float dx=posed[2][0]-posed[1][0], dz=posed[2][2]-posed[1][2], angle=std::atan2(dz,-dx); - value[3]=std::cos(angle); value[4]=std::sin(angle); - for(int j=0;j<22;++j){value[5+j*3]=posed[j][0]-value[0];value[6+j*3]=posed[j][1];value[7+j*3]=posed[j][2]-value[2];} - for(int j: {7,8,20,21}) for(int d=0;d<6;++d) - value[71+j*6+d]=static_cast(global[j].v[(d%3)*3+d/3]); - std::copy_n(value.data(),203,result.observed.data()+base); - for (const auto &[first,last] : constrained) - std::fill(result.observed_mask.begin()+static_cast(base+first), - result.observed_mask.begin()+static_cast(base+last),1.F); - } - result.origin_x=result.observed[0]; - result.origin_z=result.observed[2]; - for (size_t frame=0; frame value{}; std::copy_n(raw,features,value.data()); - mat decoded[22],local[22],global[22]; for(int j=0;j<22;++j)decoded[j]=cont6(value.data()+71+j*6); - for(int j=0;j<22;++j)local[j]=parent[j]<0?decoded[j]:mul(trans(decoded[parent[j]]),decoded[j]); - float root[3]={value[0]+value[5],value[6],value[2]+value[7]}, posed[22][3]{}; - for(int j=0;j<22;++j){if(parent[j]<0){global[j]=local[j];for(int k=0;k<3;++k)posed[j][k]=root[k];}else{global[j]=mul(global[parent[j]],local[j]);float d[3];rotate(global[parent[j]],offset[j],d);for(int k=0;k<3;++k)posed[j][k]=posed[parent[j]][k]+d[k];}} - result.first_heading=std::atan2(posed[2][2]-posed[1][2],-(posed[2][0]-posed[1][0])); + const size_t previous_start=previous.size()-overlap*D; + std::vector value(D); + for(size_t frame=0;frame(base),result.observed_mask.begin()+static_cast(base+rotation_begin),1.F);for(unsigned joint:s->end_effectors){const size_t first=base+rotation_begin+joint*6;std::fill(result.observed_mask.begin()+static_cast(first),result.observed_mask.begin()+static_cast(first+6),1.F);}} + result.origin_x=result.observed[0];result.origin_z=result.observed[2]; + for(size_t frame=0;frame, std::string> sample_motion_sequence_from_noise( const ggml_motion_weights &weights, std::span segments, unsigned transition_frames, unsigned steps, float text_weight, float constraint_weight) { - constexpr size_t features = 273; - if (segments.empty() || !transition_frames) - return std::unexpected("sequence requires segments and a transition"); - auto gm=weights.f32_values("stats.global_root.mean"), gs=weights.f32_values("stats.global_root.std"); - auto bm=weights.f32_values("stats.body.mean"), bs=weights.f32_values("stats.body.std"); - if (!gm || !gs || !bm || !bs) return std::unexpected("motion GGUF lacks motion statistics"); - // Upstream Stats normalizes with sqrt(std^2 + 1e-5), rather than raw std. - auto scale = [](float stddev) { return std::sqrt(stddev * stddev + 1.e-5F); }; - auto unnormalize = [&](std::vector &motion) { for(size_t row=0;row &motion) { for(size_t row=0;row joined, previous; - for (size_t index=0; index current; - if (!index) { - auto sampled=sample_motion_from_noise(weights,segment.initial_noise,segment.embedding, - sampled_frames,steps,text_weight,constraint_weight); - if (!sampled) return std::unexpected(sampled.error()); - current=std::move(*sampled); - unnormalize(current); - } else { - const size_t overlap=transition_frames; - if (overlap >= segment.frames || previous.size()origin_x; - const float origin_z=transition->origin_z; - normalize(transition->observed); - auto sampled=sample_motion_from_noise_conditioned(weights,segment.initial_noise,segment.embedding, - transition->observed,transition->observed_mask,transition->first_heading,sampled_frames,steps,text_weight,constraint_weight); - if (!sampled) return std::unexpected(sampled.error()); - current=std::move(*sampled); - unnormalize(current); - for (size_t frame=0; frame(overlap*features),current.end()); - } - if (!index) joined=current; - previous=std::move(current); - } + const size_t D=weights.motion_dim(),body=D-5; + if(segments.empty()||!transition_frames||!D)return std::unexpected("sequence requires segments and a transition"); + auto gm=weights.f32_values("stats.global_root.mean"),gs=weights.f32_values("stats.global_root.std"); + auto bm=weights.f32_values("stats.body.mean"),bs=weights.f32_values("stats.body.std"); + if(!gm||!gs||!bm||!bs||gm->size()!=5||gs->size()!=5||bm->size()!=body||bs->size()!=body)return std::unexpected("motion GGUF lacks compatible motion statistics"); + auto scale=[](float stddev){return std::sqrt(stddev*stddev+1.e-5F);}; + auto unnormalize=[&](std::vector&motion){for(size_t row=0;row&motion){for(size_t row=0;row joined,previous; + for(size_t index=0;indexcurrent;if(!index){auto sampled=sample_motion_from_noise(weights,segment.initial_noise,segment.embedding,sampled_frames,steps,text_weight,constraint_weight);if(!sampled)return std::unexpected(sampled.error());current=std::move(*sampled);unnormalize(current);}else{const size_t overlap=transition_frames;if(overlap>=segment.frames||previous.size()origin_x,origin_z=transition->origin_z;normalize(transition->observed);auto sampled=sample_motion_from_noise_conditioned(weights,segment.initial_noise,segment.embedding,transition->observed,transition->observed_mask,transition->first_heading,sampled_frames,steps,text_weight,constraint_weight);if(!sampled)return std::unexpected(sampled.error());current=std::move(*sampled);unnormalize(current);for(size_t frame=0;frame(overlap*D),current.end());}if(!index)joined=current;previous=std::move(current);} return joined; } } // namespace kimodo::detail diff --git a/src/skeleton.hpp b/src/skeleton.hpp new file mode 100644 index 0000000..47c3f22 --- /dev/null +++ b/src/skeleton.hpp @@ -0,0 +1,101 @@ +#pragma once + +#include +#include +#include +#include + +namespace kimodo::detail { + +// Joint names and parent graphs are copied from NVIDIA Kimodo's Apache-2.0 +// kimodo/skeleton/definitions.py. Parent-local offsets were extracted from +// the accompanying joints.p assets in the trusted reference container. +struct skeleton_spec { + std::string_view key; + std::span names; + std::span parents; + std::span> offsets; + std::array hips; // right, left + std::array end_effectors; // left foot, right foot, left hand, right hand + + [[nodiscard]] constexpr std::size_t joints() const noexcept { return names.size(); } + [[nodiscard]] constexpr std::size_t motion_dim() const noexcept { return 9 + 12 * joints(); } + [[nodiscard]] constexpr std::size_t body_dim() const noexcept { return motion_dim() - 5; } +}; + +inline constexpr std::array smplx22_names{ + std::string_view{"pelvis"}, "left_hip", "right_hip", "spine1", "left_knee", "right_knee", + "spine2", "left_ankle", "right_ankle", "spine3", "left_foot", "right_foot", "neck", + "left_collar", "right_collar", "head", "left_shoulder", "right_shoulder", "left_elbow", + "right_elbow", "left_wrist", "right_wrist"}; +inline constexpr std::array smplx22_parents{-1,0,0,0,1,2,3,4,5,6,7,8,9,9,9,12,13,14,16,17,18,19}; +inline constexpr std::array,22> smplx22_offsets{{ + {0,0,0},{.052299179F,-.093935639F,-.027606763F},{-.057192899F,-.106548190F,-.022217851F}, + {-.001495834F,.112929940F,-.024981268F},{.058866613F,-.416441321F,-.006556974F}, + {-.048074268F,-.397559673F,-.014061437F},{.006900469F,.145636231F,-.006858510F}, + {-.041737989F,-.437583506F,-.029511765F},{.014489345F,-.446852267F,-.018029511F}, + {-.010334037F,.056081813F,.021115851F},{.049293540F,-.065279245F,.126259089F}, + {-.040575184F,-.065286517F,.127075911F},{-.011025756F,.171365142F,-.028827066F}, + {.047724526F,.087643057F,-.008375450F},{-.046636276F,.086612143F,-.014864366F}, + {.024654359F,.175390735F,.024463326F},{.126284808F,.057680372F,-.013885141F}, + {-.109341696F,.053674292F,-.009117880F},{.272907287F,-.069853373F,-.039094493F}, + {-.292028785F,-.035440356F,-.024564851F},{.276173830F,.021254137F,-.002478220F}, + {-.271878421F,-.004834589F,-.016445294F}}}; + +inline constexpr std::array soma30_names{ + std::string_view{"Hips"}, "Spine1", "Spine2", "Chest", "Neck1", "Neck2", "Head", "Jaw", + "LeftEye", "RightEye", "LeftShoulder", "LeftArm", "LeftForeArm", "LeftHand", + "LeftHandThumbEnd", "LeftHandMiddleEnd", "RightShoulder", "RightArm", "RightForeArm", + "RightHand", "RightHandThumbEnd", "RightHandMiddleEnd", "LeftLeg", "LeftShin", "LeftFoot", + "LeftToeBase", "RightLeg", "RightShin", "RightFoot", "RightToeBase"}; +inline constexpr std::array soma30_parents{-1,0,1,2,3,4,5,6,6,6,3,10,11,12,13,13,3,16,17,18,19,19,0,22,23,24,0,26,27,28}; +inline constexpr std::array,30> soma30_offsets{{ + {0,0,0},{-.00013727F,.0500376256F,-.00053726669F},{-1.86574103e-9F,.0712530139F,-.000298248546F}, + {-5.75188398e-9F,.0755006305F,-.00815970992F},{-.00181676517F,.263112953F,-.00553348292F}, + {-2.85102231e-8F,.0770939664F,.0230258546F},{-4.5975437e-8F,.0612891595F,.0195370861F}, + {2.63687901e-5F,.0047559225F,.0309494062F},{.0320638079F,.0538020513F,.0758688308F}, + {-.0322244017F,.05361869F,.0755823359F},{.0162165175F,.232371641F,.0511341324F}, + {.149198457F,2.19397873e-8F,-.0550232576F},{.287393078F,2.50268389e-9F,-2.58787737e-5F}, + {.270939812F,-7.06625108e-9F,2.60897248e-5F},{.122686267F,-.0322017573F,.0483306876F}, + {.190119595F,-.00312878387F,-.000339570373F},{-.0138011824F,.231803086F,.0521415786F}, + {-.150371962F,1.17387901e-7F,-.0554560437F},{-.287366393F,1.87628082e-8F,-2.59709359e-5F}, + {-.271336198F,-1.16767401e-9F,2.61269368e-5F},{-.122642483F,-.0321145448F,.0480403904F}, + {-.190005945F,-.00306615542F,-.0003157343F},{.10043214F,-.0843452671F,.0259565473F}, + {-1e-8F,-.432217537F,-.00802912805F},{1e-8F,-.421550959F,-.0348152298F}, + {0,-.0505947206F,.132315294F},{-.10047278F,-.0829525995F,.0262031695F}, + {1e-8F,-.433622059F,-.00805555828F},{2e-8F,-.421173943F,-.0347839785F}, + {-3.42907669e-9F,-.0507960932F,.132841956F}}}; + +inline constexpr std::array g1skel34_names{ + std::string_view{"pelvis_skel"}, "left_hip_pitch_skel", "left_hip_roll_skel", "left_hip_yaw_skel", + "left_knee_skel", "left_ankle_pitch_skel", "left_ankle_roll_skel", "left_toe_base", + "right_hip_pitch_skel", "right_hip_roll_skel", "right_hip_yaw_skel", "right_knee_skel", + "right_ankle_pitch_skel", "right_ankle_roll_skel", "right_toe_base", "waist_yaw_skel", + "waist_roll_skel", "waist_pitch_skel", "left_shoulder_pitch_skel", "left_shoulder_roll_skel", + "left_shoulder_yaw_skel", "left_elbow_skel", "left_wrist_roll_skel", "left_wrist_pitch_skel", + "left_wrist_yaw_skel", "left_hand_roll_skel", "right_shoulder_pitch_skel", + "right_shoulder_roll_skel", "right_shoulder_yaw_skel", "right_elbow_skel", + "right_wrist_roll_skel", "right_wrist_pitch_skel", "right_wrist_yaw_skel", "right_hand_roll_skel"}; +inline constexpr std::array g1skel34_parents{-1,0,1,2,3,4,5,6,0,8,9,10,11,12,13,0,15,16,17,18,19,20,21,22,23,24,17,26,27,28,29,30,31,32}; +inline constexpr std::array,34> g1skel34_offsets{{ + {0,0,0},{.064452F,-.1027F,0},{.052F,-.030465F,0},{0,-.12412F,.025001F}, + {.0021489F,-.17734F,-.078273F},{-.000094445F,-.30001F,0},{0,-.017558F,0},{0,-.035F,.14F}, + {-.064452F,-.1027F,0},{-.052F,-.030465F,0},{0,-.12412F,.025001F},{-.0021489F,-.17734F,-.078273F}, + {.000094445F,-.30001F,0},{0,-.017558F,0},{0,-.035F,.14F},{0,0,0},{0,.044F,-.0039635F}, + {0,0,0},{.10022F,.24778F,.0039563F},{.038F,-.013831F,0},{.00624F,-.1032F,0}, + {0,-.080518F,.015783F},{.00188791F,-.01F,.1F},{0,0,.038F},{0,0,.046F},{0,0,.1F}, + {-.10021F,.24778F,.0039563F},{-.038F,-.013831F,0},{-.00624F,-.1032F,0}, + {0,-.080518F,.015783F},{-.00188791F,-.01F,.1F},{0,0,.038F},{0,0,.046F},{0,0,.1F}}}; + +inline constexpr skeleton_spec smplx22_spec{"smplx22", smplx22_names, smplx22_parents, smplx22_offsets, {2,1}, {7,8,20,21}}; +inline constexpr skeleton_spec soma30_spec{"soma30", soma30_names, soma30_parents, soma30_offsets, {26,22}, {24,28,13,19}}; +inline constexpr skeleton_spec g1skel34_spec{"g1skel34", g1skel34_names, g1skel34_parents, g1skel34_offsets, {8,1}, {6,13,24,32}}; + +inline constexpr const skeleton_spec *find_skeleton(std::string_view key) noexcept { + if (key == smplx22_spec.key) return &smplx22_spec; + if (key == soma30_spec.key) return &soma30_spec; + if (key == g1skel34_spec.key) return &g1skel34_spec; + return nullptr; +} + +} // namespace kimodo::detail diff --git a/tests/generate_smoke.cpp b/tests/generate_smoke.cpp index 35e459e..59e01c5 100644 --- a/tests/generate_smoke.cpp +++ b/tests/generate_smoke.cpp @@ -1,4 +1,5 @@ #include #include #include -int main(int argc,char**argv){if(argc!=2)return 2;auto m=kimodo::model::load(argv[1]);if(!m){std::fprintf(stderr,"%s\n",m.error().c_str());return 1;}std::array e{};auto r=(*m)->generate_embedding(e,2,1,42,2.f,2.f);if(!r){std::fprintf(stderr,"%s\n",r.error().c_str());return 1;}if(r->frames!=2||r->joints!=22||r->root_positions.size()!=6||r->local_rotations_xyzw.size()!=176)return 1;return 0;} +#include +int main(int argc,char**argv){if(argc<2||argc>4)return 2;const unsigned joints=argc>=3?static_cast(std::stoul(argv[2])):22;auto m=kimodo::model::load(argv[1]);if(!m){std::fprintf(stderr,"%s\n",m.error().c_str());return 1;}std::array e{};auto r=(*m)->generate_embedding(e,2,1,42,2.f,2.f);if(!r){std::fprintf(stderr,"%s\n",r.error().c_str());return 1;}if(r->frames!=2||r->joints!=joints||r->root_positions.size()!=6||r->local_rotations_xyzw.size()!=2*joints*4)return 1;if(argc==4){std::ofstream out(argv[3],std::ios::binary);out.write(reinterpret_cast(r->root_positions.data()),static_cast(r->root_positions.size()*sizeof(float)));out.write(reinterpret_cast(r->local_rotations_xyzw.data()),static_cast(r->local_rotations_xyzw.size()*sizeof(float)));if(!out)return 1;}return 0;}