1. Why Standard Bounding-Box + Color Clustering Is Not Enough
Across Basketball, Football (Soccer), Handball, and US Football, relying solely on 2D bounding boxes and pixel color clustering introduces four classic issues:
- Bench & Sideline Contamination: Substitutes, coaches, or staff standing near the boundary line wear team colors and get falsely projected onto the active 2D radar.
- Jumping & Occluded Players: Using the bottom-center of a bounding box as a player's floor position projects jumping players (layups, headers, jump-shots) several meters further away than their true takeoff spot.
- Lighting & Jersey Ambiguities: Stadium shadows, motion blur, or dark/light kits can cause pure RGB/LAB clustering to swap a player's team.
- Missing Keypoint Models: Specialized ONNX keypoint detectors can fail on tight camera zooms or sports without a dedicated keypoint model (e.g., Handball or US Football).
vlm_priors=true does not trigger a second VLM call. It enriches the JSON schema of the primary detection pass with semantic player attributes and court calibration landmarks.
2. The Four Pillars of VLM Probabilistic Priors
A. Bayesian Team & Goalkeeper Classification
For each player, the VLM outputs a prior team_id ∈ {0, 1} based on holistic visual context (jersey, shorts, socks, helmet, and formation).
- Fused with unsupervised CIE-LAB & HSV torso color clustering to resolve shadow/lighting ambiguities.
- Goalkeepers (
T0-GK/T1-GK): In Football and Handball, goalkeepers wear distinct kits. Their team is resolved from their 2D goal-area position, defending team proximity, and VLM team polarity.
B. Active vs. Bench Role Filtering (role_status)
Every detected player is tagged with a role status:
active_on_court: Active player participating in play.active_inbound: Player standing just outside the boundary line performing a throw-in or inbound pass (kept active on the radar).bench_off_court: Substitute on the bench or sideline staff — labeled(Bench)on the image and excluded from the 2D radar.
C. Biomechanical Ground-Contact Estimation
To project a 2D bounding box onto the 2D court plane (Z = 0), we need the pixel where the player touches the floor:
- The VLM predicts
action_posture(standing_running,jumping_shoot,defensive_stance) andground_contact_status(visible_on_floor,airborne,truncated_bottom_edge). - It also predicts
ground_point_norm: the floor contact point directly beneath the player's center of mass.
D. Court Calibration Landmarks & RANSAC Homography
In the court_calibration block, the VLM identifies:
visible_court_side(left,right, orfull).keypoints: Floor-level line intersections (penalty box corners, free-throw line corners, 6m/9m handball arcs, NFL yard-line/hash intersections).court_zone_hintper player to regularize steep camera perspective.
3. Hybrid RANSAC 2D Homography Pipeline
Uses the specialized ONNX court keypoint model first when available (≥ 4 RANSAC inliers), or falls back to VLM court landmarks + goal bbox anchors.
Computes the 3×3 homography matrix H with outlier rejection against official court dimensions in centimeters.
Projects each player's and goalkeeper's floor contact point (x_px, y_px) to metric court coordinates (X_cm, Y_cm).
Applies sport rules (e.g. max 5v5 on court in basketball, IHF 6m goal-area rules in handball, max 1 goalkeeper per goal half).
4. Summary Comparison
| Feature | Disabled (vlm_priors=false) |
Enabled (vlm_priors=true — Default) |
|---|---|---|
Team Assignment (T0 / T1) |
KMeans color clustering only | Hybrid CIE-LAB/HSV clustering + VLM Bayesian team prior |
| Bench / Sideline Filtering | Geometric court bounds only | Semantic bench_off_court & active_inbound filtering |
| Jumping / Truncated Players | Bottom-center of 2D bounding box | Posture-compensated floor contact projection |
| 2D Court Homography | ONNX keypoint model only (perspective fallback otherwise) | ONNX keypoints (priority) + VLM RANSAC landmarks + Goal anchors |