Player tracking + play retrieval
hoopvec
July 2026 – present
Rebuilds player-and-ball tracking from ordinary NBA broadcast video, trains a play-embedding model for similarity retrieval, and wraps both in a measured, reproducible eval harness. The research question is how much downstream analytics degrade on reconstructed tracking versus ground truth — and which perception errors actually matter.
Results
| detection mAP@50 | 0.987fine-tuned yolov8m |
|---|---|
| tracking HOTA | 0.301 → 0.525 |
| retrieval recall@1 (instance) | 0.62 → 0.98 |
| retrieval precision@5 (semantic) | 0.942vs 0.51 self-supervised ≈ random |
| play retrieval under ID-switches | 0.99 → 0.68 → 0.27 |
| serving throughput | 9.9 → 21.5 fpsyolov8n on the frontier at 44.7 fps |
The finding
The learned encoder is more fragile than a zero-parameter baseline under association error
Positional jitter and dropout cost almost nothing; ID-switches dominate. At a realistic budget the trained encoder falls to recall@1 0.68 against a hand-feature floor of 0.99, and folding in measured re-ID error drops it to 0.27 — below the floor. That is the honest result, confirmed three ways, not a bug: the lever to trust off broadcast video is the association stage, not sub-pixel registration.
Honest limitations
- The 0.98 recall@1 headline is instance-invariance — a possession retrieving an augmented copy of itself — not semantic play similarity. On an actual play-type axis the self-supervised encoder scores 0.51 against a 0.50 random baseline.
- Court homography reaches 16px on held-out arenas but does not transfer to broadcast video. That is a data limit, not a tuning one.
- One full broadcast clip runs end to end, but its retrieved neighbors are not meaningful: hand-clicked homography and arbitrary player slots make the degradation study's prediction empirical. A clean-looking top-5 there would have been less honest.
What got built
- Rebuilt player tracking from NBA broadcast video, lifting HOTA from 0.301 to 0.525 on SportsMOT basketball-val through ByteTrack, BoT-SORT, a fine-tuned yolov8m detector at 0.987 mAP50, and a training-to-inference resolution fix, each isolated by single-variable ablation.
- Traced the cost of computing analytics on tracked video to one stage, player association, by corrupting ground-truth tracks one error class at a time: positional noise cost nothing, while ID-switches dropped play-retrieval recall@1 from 0.99 to 0.68, and to 0.27 under measured re-ID error.
- Showed the retrieval encoder's 0.98 recall@1 measured whether a possession matched an augmented copy of itself, not play similarity, scoring 0.51 against a 0.50 random baseline on play type, then lifted held-out-game precision@5 to 0.942 by changing only the training objective.
- Profiled the detection and tracking pipeline at 9.9 fps and found the deployed 1280px config strictly dominated by 640px, which delivered higher mAP at 2.6× the throughput, with a fine-tuned yolov8n reaching 44.7 fps on the Pareto frontier.
Stack
- PyTorch
- Ultralytics YOLOv8
- ByteTrack
- BoT-SORT
- FAISS
- FastAPI
- ONNX Runtime
- TrackEval