init: repo scaffolding, deps, prompts, plan

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
cmoyates 2026-03-01 04:48:51 -03:30
commit 897242d200
No known key found for this signature in database
GPG Key ID: F65E08480FDC22D2
27 changed files with 1919 additions and 0 deletions

16
.gitignore vendored Normal file
View File

@ -0,0 +1,16 @@
# Python-generated files
__pycache__/
*.py[oc]
build/
dist/
wheels/
*.egg-info
# Virtual environments
.venv
# Model checkpoints (large binary files)
checkpoints/
# Large generated fixtures (golden .npz kept small; regenerate with dump script)
reference/fixtures/*.npz

1
.python-version Normal file
View File

@ -0,0 +1 @@
3.12

41
CLAUDE.md Normal file
View File

@ -0,0 +1,41 @@
# CLAUDE.md — corridorkey-mlx
MLX inference port of CorridorKey for Apple Silicon.
## Architecture
- Input: 4ch (RGB + coarse alpha hint)
- Backbone: Hiera (timm, features_only=True) → 4 multiscale features
- Decoder heads: alpha (1ch) + foreground (3ch), upsampled to full res
- Refiner: CNN over RGB + coarse preds (7ch) → additive delta logits → sigmoid
## Repo layout
- `src/corridorkey_mlx/` — main package
- `model/` — MLX model definitions
- `convert/` — PyTorch→MLX weight conversion
- `inference/` — inference pipeline
- `io/` — image loading, saving, preprocessing
- `utils/` — shared helpers, layout transforms
- `scripts/` — CLI tools (dump reference, compare, bench)
- `prompts/` — phased port instructions
- `reference/` — PyTorch reference harness outputs
- `tests/` — parity and unit tests
## Conventions
- Python 3.12+, uv for deps
- ruff for lint/format, mypy for types, pytest for tests
- MLX uses NHWC — centralize layout transforms in `utils/`
- All non-trivial changes need a validation path
- Inference only — no training code
- Preserve PyTorch behavior before optimizing
## Commands
```bash
uv run pytest # run tests
uv run ruff check . # lint
uv run ruff format . # format
uv run mypy src/ # type check
```

101
README.md Normal file
View File

@ -0,0 +1,101 @@
# corridorkey-mlx
MLX inference port of [CorridorKey](https://github.com/nikopueringer/CorridorKey) for Apple Silicon.
## Architecture
```
RGB image + coarse alpha hint (4ch)
┌──────────────────┐
│ Hiera backbone │ (timm, features_only)
│ → 4 multiscale │
│ feature maps │
└──────────────────┘
┌────┴────┐
▼ ▼
┌───────┐ ┌───────┐
│ Alpha │ │ FG │
│ head │ │ head │
│ (1ch) │ │ (3ch) │
└───────┘ └───────┘
│ │
└────┬────┘
┌──────────────────┐
│ CNN Refiner │ RGB + coarse preds (7ch)
│ → delta logits │ → sigmoid
└──────────────────┘
final alpha + fg
```
## Phased Roadmap
| Phase | Scope | Status |
|-------|-------|--------|
| 1 | PyTorch reference harness + fixture dump | **In progress** |
| 2 | MLX decoder/refiner blocks + parity tests | Not started |
| 3 | Checkpoint conversion (PyTorch → MLX) | Not started |
| 4 | Full inference pipeline | Not started |
| 5 | Optimization + benchmarking | Not started |
See `prompts/` for detailed phase instructions.
## Setup
```bash
uv sync --group dev
```
For PyTorch reference work:
```bash
uv sync --group reference
```
## Development
```bash
uv run pytest # tests
uv run ruff check . # lint
uv run ruff format . # format
uv run mypy src/ # type check
```
## Reference Fixtures
Phase 1 generates golden reference tensors from PyTorch for MLX parity testing.
**Format:** single `reference/fixtures/golden.npz` (numpy compressed archive)
**Generate:**
```bash
uv run --group reference python scripts/dump_pytorch_reference.py \
--checkpoint checkpoints/CorridorKey_v1.0.pth
```
**Contents (all float32, NCHW, batch=1, img_size=512):**
| Key | Shape | Description |
|-----|-------|-------------|
| `input` | (1, 4, 512, 512) | Random input (seed=42) |
| `encoder_feature_0` | (1, 112, 128, 128) | Backbone stride-4 |
| `encoder_feature_1` | (1, 224, 64, 64) | Backbone stride-8 |
| `encoder_feature_2` | (1, 448, 32, 32) | Backbone stride-16 |
| `encoder_feature_3` | (1, 896, 16, 16) | Backbone stride-32 |
| `alpha_logits` | (1, 1, 128, 128) | Alpha decoder output (H/4) |
| `fg_logits` | (1, 3, 128, 128) | FG decoder output (H/4) |
| `alpha_logits_up` | (1, 1, 512, 512) | Alpha logits upsampled |
| `fg_logits_up` | (1, 3, 512, 512) | FG logits upsampled |
| `alpha_coarse` | (1, 1, 512, 512) | sigmoid(alpha_logits_up) |
| `fg_coarse` | (1, 3, 512, 512) | sigmoid(fg_logits_up) |
| `delta_logits` | (1, 4, 512, 512) | Refiner output (10x scaled) |
| `alpha_final` | (1, 1, 512, 512) | Final alpha prediction |
| `fg_final` | (1, 3, 512, 512) | Final FG prediction |
## Current Status
Phase 1 in progress — reference harness and fixture dump implemented.

View File

@ -0,0 +1,322 @@
---
title: "feat: CorridorKey MLX Inference Port"
type: feat
date: 2026-03-01
---
# CorridorKey MLX Inference Port
## Overview
Port CorridorKey's inference pipeline from PyTorch to MLX for native Apple Silicon execution. The port is staged across 5 phases, each with its own parity gate against PyTorch reference outputs. No training code. No UI.
## Problem Statement
CorridorKey (alpha matting model) runs on PyTorch. On Apple Silicon, PyTorch uses MPS which is slower and less memory-efficient than MLX's native Metal backend. A clean MLX port enables fast local inference without PyTorch overhead.
## Architecture
```
RGB + coarse alpha hint (4ch)
|
v
+------------------+
| Hiera backbone | timm, features_only=True
| -> 4 multiscale | feature maps
+------------------+
|
+----+----+
v v
+-------+ +-------+
| Alpha | | FG |
| head | | head |
| (1ch) | | (3ch) |
+-------+ +-------+
| |
+----+----+
v
+------------------+
| CNN Refiner | RGB + coarse preds (7ch)
| -> delta logits | -> sigmoid
+------------------+
|
v
final alpha + fg
```
**Key model components:**
- **Backbone:** Hiera (hierarchical vision transformer) -- 4 multiscale feature outputs
- **Decoder heads:** Two parallel heads (alpha 1ch, foreground 3ch) consuming backbone features
- **CNN Refiner:** Takes RGB + coarse predictions (7ch), predicts additive delta logits, final sigmoid
**MLX-specific considerations:**
- MLX uses NHWC layout natively vs PyTorch's NCHW
- Conv weight transpose needed: `(O,I,H,W)` -> `(O,H,W,I)`
- GroupNorm behavior must match PyTorch exactly for parity
- First conv is patched to 4 input channels (RGB + alpha hint)
## Implementation Phases
### Phase 1: PyTorch Reference Harness
**Prompt:** `prompts/phase1-backbone.md`
**Goal:** Deterministic reference pipeline that dumps intermediate tensors for staged MLX parity testing.
**Deliverables:**
- [x] `scripts/dump_pytorch_reference.py` -- loads checkpoint via state_dict, runs forward pass, saves intermediates
- [x] `reference/fixtures/` -- sample inputs and golden outputs
- [x] `tests/test_reference_fixtures.py` -- validates fixture shapes and dtypes
**Tensors to dump:**
1. 4 backbone feature maps (multiscale)
2. Alpha coarse logits
3. FG coarse logits
4. Alpha coarse probs
5. FG coarse probs
6. Refiner delta logits
7. Final alpha
8. Final FG
**Files touched:**
| File | Action |
|------|--------|
| `scripts/dump_pytorch_reference.py` | implement |
| `reference/fixtures/*.npz` or `*.safetensors` | generate |
| `tests/test_reference_fixtures.py` | implement (unskip) |
| `README.md` | update fixture format docs |
**Constraints:**
- Load via `state_dict`, not pickle
- Deterministic (fixed seed, mode)
- One tiny golden example checked in
- Print shape report via rich
**Definition of done:**
- Fixture files exist with all 8+ tensor groups
- Shape contract tests pass
- Script is idempotent
---
### Phase 2: MLX Decoder and Refiner Blocks
**Prompt:** `prompts/phase2-mlx-blocks.md`
**Goal:** MLX implementations of non-backbone blocks with parity tests against Phase 1 fixtures.
**Components to implement:**
- [ ] `MLP` -- feedforward block
- [ ] `DecoderHead` -- consumes backbone features, produces coarse predictions
- [ ] `RefinerBlock` -- single refiner stage
- [ ] `CNNRefinerModule` -- full refiner consuming 7ch input
**Files touched:**
| File | Action |
|------|--------|
| `src/corridorkey_mlx/model/decoder.py` | implement MLP + DecoderHead |
| `src/corridorkey_mlx/model/refiner.py` | implement RefinerBlock + CNNRefinerModule |
| `src/corridorkey_mlx/utils/layout.py` | implement NCHW to NHWC transforms |
| `tests/test_decoder_parity.py` | implement (unskip) |
| `tests/test_refiner_parity.py` | implement (unskip) |
**Constraints:**
- NHWC throughout, layout conversions only in `utils/layout.py`
- GroupNorm must match PyTorch behavior exactly
- Parity tests load saved backbone features (from Phase 1 fixtures) as input
- Report max abs error and mean abs error
**Definition of done:**
- Decoder parity test passes (max abs err < 1e-5)
- Refiner parity test passes (max abs err < 1e-5)
- Modules wired into partial model path for test usage
---
### Phase 3: Checkpoint Conversion
**Prompt:** `prompts/phase3-conversion.md`
**Goal:** Robust conversion pipeline from PyTorch checkpoint to MLX-compatible weights.
**Deliverables:**
- [ ] `src/corridorkey_mlx/convert/converter.py` -- key mapping + weight transforms
- [ ] Conversion diagnostic report (source key -> dest key, shapes, transform applied)
- [ ] Output as safetensors
**Files touched:**
| File | Action |
|------|--------|
| `src/corridorkey_mlx/convert/converter.py` | implement |
| `src/corridorkey_mlx/convert/__init__.py` | export converter |
| `tests/test_conversion.py` | implement (unskip) |
| `scripts/convert_weights.py` | create (CLI wrapper) |
**Key mapping concerns:**
- Explicit key-by-key mapping, no regex guessing
- Conv weights: `(O,I,H,W)` -> `(O,H,W,I)`
- Patched 4-channel first conv must be preserved exactly
- No silent fallbacks -- all mismatches are errors
**Diagnostic output per key:**
```
source_key -> dest_key | src_shape -> dst_shape | transform
```
**Definition of done:**
- Converter script exists and runs
- Mapping is explicit and auditable
- Shape validation passes for all completed modules
- No orphan keys (every source key maps or is explicitly skipped with reason)
---
### Phase 4: Full Inference Pipeline
**Prompt:** `prompts/phase4-inference-pipeline.md` (not yet written)
**Goal:** End-to-end inference matching PyTorch output.
**Deliverables:**
- [ ] `src/corridorkey_mlx/model/backbone.py` -- Hiera MLX port
- [ ] `src/corridorkey_mlx/model/corridorkey.py` -- full model composition
- [ ] `src/corridorkey_mlx/inference/pipeline.py` -- load, preprocess, forward, postprocess, save
- [ ] `src/corridorkey_mlx/io/image.py` -- PIL-based image I/O + preprocessing
- [ ] End-to-end parity test against PyTorch golden output
**Files touched:**
| File | Action |
|------|--------|
| `src/corridorkey_mlx/model/backbone.py` | implement Hiera MLX |
| `src/corridorkey_mlx/model/corridorkey.py` | implement full model |
| `src/corridorkey_mlx/inference/pipeline.py` | implement |
| `src/corridorkey_mlx/io/image.py` | implement |
| `tests/test_e2e_parity.py` | create |
| `main.py` | wire CLI via typer |
**Definition of done:**
- `uv run python main.py --image input.png --output output.png` produces correct result
- E2e parity test passes within tolerance
- Memory usage is reasonable (no unnecessary copies)
---
### Phase 5: Optimization and Benchmarking
**Prompt:** `prompts/phase5-optimization.md` (not yet written)
**Goal:** Production-quality performance on Apple Silicon.
**Deliverables:**
- [ ] `scripts/bench_mlx.py` -- latency, throughput, memory reporting
- [ ] `scripts/compare_reference.py` -- side-by-side output comparison
- [ ] Performance optimizations (compile, memory layout, batching)
**Potential optimizations:**
- `mx.compile()` on hot paths
- Memory-efficient attention if Hiera benefits
- Avoid unnecessary `mx.eval()` calls (lazy graph)
- Optimal image preprocessing (avoid numpy round-trips)
**Definition of done:**
- Benchmark script reports latency + peak memory
- Performance is competitive with PyTorch MPS on same hardware
- No correctness regression (parity tests still pass)
---
## Acceptance Criteria
### Functional Requirements
- [ ] All 5 phases complete with passing parity gates
- [ ] Single-image inference CLI works end-to-end
- [ ] Output visually matches PyTorch reference
### Quality Gates (per phase)
- [ ] Parity tests pass within documented tolerance
- [ ] No skipped tests without explicit phase rationale
- [ ] ruff + mypy clean
- [ ] Fixtures are deterministic and reproducible
## Dependencies and Prerequisites
| Dependency | Phase | Notes |
|-----------|-------|-------|
| Original CorridorKey checkpoint | 1 | needed for reference dump |
| Original CorridorKey source code | 1 | needed to understand architecture |
| PyTorch + timm (reference group) | 1 | `uv sync --group reference` |
| Phase 1 fixtures | 2, 3 | parity test inputs |
| Phase 2 + 3 complete | 4 | blocks + weights needed for full model |
## Risk Analysis
| Risk | Impact | Mitigation |
|------|--------|------------|
| Hiera has no existing MLX port | High -- Phase 4 blocker | Research early; may need full rewrite |
| GroupNorm numerical differences | Medium -- parity failures | Test tolerance tuning; document acceptable drift |
| 4ch first conv patching | Low -- conversion bug | Explicit test in Phase 3 |
| MLX API changes | Low -- version pinned | Pin `mlx>=0.31.0` in pyproject |
## Current Repo State
```
src/corridorkey_mlx/
__init__.py [exists]
model/
backbone.py [placeholder]
decoder.py [placeholder]
refiner.py [placeholder]
corridorkey.py [placeholder]
convert/
converter.py [placeholder]
inference/
pipeline.py [placeholder]
io/
image.py [placeholder]
utils/
layout.py [placeholder]
scripts/
dump_pytorch_reference.py [placeholder]
compare_reference.py [placeholder]
bench_mlx.py [placeholder]
tests/
test_import.py [passes]
test_reference_fixtures.py [skipped - Phase 1]
test_decoder_parity.py [skipped - Phase 2]
test_refiner_parity.py [skipped - Phase 2]
test_conversion.py [skipped - Phase 3]
prompts/
phase1-backbone.md [written]
phase2-mlx-blocks.md [written]
phase3-conversion.md [written]
phase4-inference-pipeline.md [empty]
phase5-optimization.md [empty]
```
## Open Questions
- Hiera port strategy: full rewrite vs adapting existing MLX vision transformer code?
- Acceptable e2e parity tolerance? (likely max abs < 1e-3 due to float32 Metal vs CUDA differences)
- Where is the original CorridorKey checkpoint hosted? (needed for Phase 1)
- Original CorridorKey source -- is it the nicehash/CorridorKey repo or a fork?
## References
- Original repo: nicehash/CorridorKey on GitHub
- MLX: ml-explore/mlx on GitHub
- timm Hiera: `timm.create_model("hiera_...", features_only=True)`
- Phase prompts: `prompts/phase{1-5}-*.md`

6
main.py Normal file
View File

@ -0,0 +1,6 @@
def main():
print("Hello from corridorkey-mlx!")
if __name__ == "__main__":
main()

View File

@ -0,0 +1,39 @@
Work only on the PyTorch reference harness.
Goal:
Create a deterministic reference pipeline that loads the original CorridorKey checkpoint and dumps intermediate tensors needed for staged MLX parity.
Deliverables:
- scripts/dump_pytorch_reference.py
- reference/fixtures/ sample inputs and outputs
- tests that validate fixture generation shape contracts
- README updates describing the fixture format
Requirements:
- Load the model via state_dict, not entire-model pickle semantics.
- Save:
- 4 backbone feature maps
- alpha coarse logits
- fg coarse logits
- alpha coarse probs
- fg coarse probs
- delta logits
- final alpha
- final fg
- Make fixture generation deterministic where practical.
- Keep one tiny golden example checked in.
- Print a concise shape report.
Do not:
- start MLX implementation
- refactor unrelated files
- add training code
Before editing:
- inspect the original CorridorKey model code carefully
- summarize exact tensors that will be dumped
- define file naming and serialization format first

View File

@ -0,0 +1,29 @@
Work only on the MLX implementations of the custom non-backbone blocks.
Goal:
Implement MLX versions of:
- MLP
- DecoderHead
- RefinerBlock
- CNNRefinerModule
Requirements:
- Use MLX idioms and explicit NHWC handling.
- Centralize tensor layout transforms in one utility module.
- Use pytorch-compatible GroupNorm behavior where needed for parity.
- Write parity tests that use saved PyTorch backbone features and saved coarse predictions.
- Report max abs error and mean abs error in test output or helper scripts.
Do not:
- port Hiera yet
- optimize prematurely
- spread layout conversions across many files
Definition of done:
- decoder parity test exists
- refiner parity test exists
- modules are wired into a partial model path for test usage

View File

@ -0,0 +1,31 @@
Work only on checkpoint conversion.
Goal:
Create a robust conversion pipeline from the PyTorch CorridorKey checkpoint to MLX-compatible weights.
Requirements:
- Inspect state_dict keys and map them explicitly.
- Convert conv weights from PyTorch layout to MLX layout.
- Preserve the patched 4-channel first conv behavior exactly.
- Write converter diagnostics:
- source key
- destination key
- source shape
- destination shape
- transform applied
- Save output as safetensors or npz.
- Validate load with MLX strict loading wherever possible.
Do not:
- attempt full end-to-end model parity yet if Hiera is incomplete
- hide key mismatches
- use silent fallbacks
Definition of done:
- converter script exists
- mapping file exists
- shape validation passes for completed modules
- conversion report is readable and auditable

View File

View File

51
pyproject.toml Normal file
View File

@ -0,0 +1,51 @@
[project]
name = "corridorkey-mlx"
version = "0.1.0"
description = "MLX inference port of CorridorKey for Apple Silicon"
readme = "README.md"
requires-python = ">=3.12"
dependencies = [
"mlx>=0.31.0",
"numpy>=2.4.2",
"pillow>=12.1.1",
"pydantic>=2.12.5",
"rich>=14.3.3",
"safetensors>=0.7.0",
"typer>=0.24.1",
]
[dependency-groups]
dev = [
"mypy>=1.19.1",
"pytest>=9.0.2",
"pytest-xdist>=3.8.0",
"ruff>=0.15.4",
]
reference = [
"timm>=1.0.25",
"torch>=2.10.0",
"torchvision>=0.25.0",
]
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"
[tool.hatch.build.targets.wheel]
packages = ["src/corridorkey_mlx"]
[tool.pytest.ini_options]
testpaths = ["tests"]
[tool.ruff]
target-version = "py312"
line-length = 99
src = ["src"]
[tool.ruff.lint]
select = ["E", "F", "I", "UP", "B", "SIM", "TCH"]
[tool.mypy]
python_version = "3.12"
strict = true
mypy_path = "src"

View File

@ -0,0 +1,3 @@
"""CorridorKey MLX — inference port for Apple Silicon."""
__version__ = "0.1.0"

View File

@ -0,0 +1 @@
"""PyTorch to MLX weight conversion utilities."""

View File

@ -0,0 +1,9 @@
"""PyTorch → MLX weight converter (not yet implemented).
Responsibilities:
- Map state_dict keys between PyTorch and MLX naming
- Transpose conv weights from NCHW NHWC
- Handle patched 4-channel first conv
- Emit diagnostic report (src key, dst key, shapes, transform)
- Save as safetensors
"""

View File

@ -0,0 +1 @@
"""Inference pipeline for CorridorKey MLX."""

View File

@ -0,0 +1,4 @@
"""Inference pipeline (not yet implemented).
Orchestrates: load image preprocess model forward postprocess save.
"""

View File

@ -0,0 +1 @@
"""Image I/O, preprocessing, and postprocessing."""

View File

@ -0,0 +1 @@
"""Image loading, saving, and preprocessing (not yet implemented)."""

View File

@ -0,0 +1 @@
"""Model definitions for CorridorKey MLX."""

View File

@ -0,0 +1,5 @@
"""Hiera backbone — MLX port (not yet implemented).
Original: timm Hiera with features_only=True.
Emits 4 multiscale feature maps.
"""

View File

@ -0,0 +1,4 @@
"""Top-level CorridorKey model — MLX port (not yet implemented).
Composes backbone + decoder heads + refiner into full pipeline.
"""

View File

@ -0,0 +1,5 @@
"""Decoder heads — MLX port (not yet implemented).
Two heads: alpha (1ch) and foreground (3ch).
Consume multiscale backbone features, upsample to full resolution.
"""

View File

@ -0,0 +1,5 @@
"""CNN refiner — MLX port (not yet implemented).
Input: RGB + coarse predictions (7ch total).
Output: additive delta logits final sigmoid.
"""

View File

@ -0,0 +1 @@
"""Shared utilities and tensor layout helpers."""

View File

@ -0,0 +1,4 @@
"""Tensor layout conversion utilities (not yet implemented).
Centralizes NCHW NHWC transforms. All layout conversions go through here.
"""

1237
uv.lock generated Normal file

File diff suppressed because it is too large Load Diff