hoopvec: what CV tracking costs downstream, measured

hoopvec reconstructs player-and-ball tracking from ordinary NBA broadcast video, trains a play-embedding model on the trajectories for similarity retrieval, and wraps both in a measured, reproducible eval harness. The research question is how much downstream analytics — play retrieval — degrade when computed on reconstructed tracking versus ground-truth tracking, and which perception errors matter most.

Tracking itself works. HOTA on SportsMOT basketball-val climbs 0.301 → 0.473 → 0.525 across ByteTrack, BoT-SORT, a detector fine-tune to 0.987 mAP@50, and a training-to-inference resolution fix, each isolated by single-variable ablation. The tracking is a means; the findings are the point.

The cost concentrates in one stage

Corrupting ground-truth tracks one error class at a time shows positional jitter and dropout cost almost nothing; ID-switches dominate. At a realistic budget the trained encoder falls to recall@1 0.68 against a zero-parameter hand-feature floor of 0.99 — the learned model is more fragile than the baseline under association error. Folding in the measured re-ID error drops it to 0.27: jersey coverage on real tracker output collapses from 0.73 to 0.37 purely from fragmentation (949 track-ids for 150 players), so roughly four of every ten players per possession land in arbitrary slots. It was confirmed three ways, including an end-to-end wire on real tracker output at floor recall@1 0.80. The lever to trust off broadcast video is the association stage, not sub-pixel registration.

Invariance and crop-robustness are entangled

Making the encoder order-robust, so it survives association error, collapses temporal-crop recall from 0.94 to about 0.45. This holds whether the invariance comes from augmentation or from an exactly permutation-invariant architecture — two independent routes pay the same crop cost, so the cost tracks the invariance itself, not the method used to get it.

The headline number is not what it looks like

The retrieval encoder’s recall@1 of 0.62 → 0.98 measures whether a possession retrieves an augmented copy of itself: instance-level invariance, not play similarity. On an actual play-type axis the self-supervised encoder scores 0.51 against a 0.50 random baseline — essentially chance. Changing only the training objective to supervised-contrastive, holding architecture, augmentation, split, and seed fixed, lifts held-out-game semantic precision@5 to 0.942. The task and the eval are sound; the self-supervised objective was the limitation.

The honest map

Some of this runs real end to end, some is simulated, and some is gated behind data or hardware — stated rather than blurred. The degradation study’s per-stage error budgets are a proxy; the real end-to-end number, 0.80, is the anchor, and it was a harder hit than the proxy implied. The clearest honesty note is the one full broadcast clip that runs end to end: its retrieved neighbors are not meaningful, because a hand-clicked homography and arbitrary player slots make the degradation study’s prediction empirical. A clean-looking top-5 there would have been less honest than reporting that it doesn’t work yet.

Serving, as a measured menu

Profiling turned the 9.9-fps baseline into a frontier and showed the deployed 1280px config was strictly dominated: 640px gives higher mAP and 2.6× the throughput, because the detector was fine-tuned at 640 and served at 1280. Re-benching the deployed path on the new default lifts throughput to 21.5 fps, and a fine-tuned yolov8n sits on the frontier at 44.7 fps.

What I’d do next

Association is the bottleneck, so a stronger tracker or real re-ID is the highest-leverage perception work — worth more than any detector or homography gain. Real play-type labels plus the supervised objective would turn the retrieval core into a product. And a broadcast court-ground-truth set is the one hard data blocker between the arena-only homography, which reaches 16px on held-out arenas, and a working top-down view.

The code and the committed eval JSON are in the repository.