Real-Time Game Video Analysis and Overlay System

I’m hiring an experienced developer to build a **real-time computer-vision system for game video analysis and interactive overlays**, initially supporting NBA 2K27, Call of Duty, Fortnite, and Rainbow Six Siege. The system will analyze a live gameplay stream externally using computer vision rather than direct game-memory access. I want a reusable CV platform with game-specific profiles, real-time detection/tracking, visual overlays, radar-style visualization, FOV filtering, timing analysis, prediction, smoothing, and detailed performance telemetry. CORE PIPELINE Game / PS5 / PC ↓ Remote Play / Capture Card ↓ Helios / Video Capture ↓ Live Frame ↓ CVPython / GCVWorker ↓ YOLO / Roboflow / Custom CV ↓ Detection + Classification ↓ Tracking / Filtering / FOV / Prediction ↓ Game-Specific Analysis ↓ ESP-Style Overlay / Radar / Meter Analysis ↓ GUI + Telemetry The critical requirement is that the software processes the **actual live video stream**. I do not want simulated detections, placeholder numbers, or a GUI that only reports that the system is running. NBA 2K27 METER ANALYSIS One of the main components is a CV system that can recognize and analyze the NBA 2K27 shooting meter from live gameplay. It should be able to: • Detect when the meter appears • Locate the meter • Track its position/state frame by frame • Identify the relevant release/green region • Detect timing/state transitions • Handle different meter presentations where possible • Handle different resolutions/scaling • Provide live timing telemetry • Record detection results for testing The architecture should allow additional meter styles and visual configurations to be added later. SHOOTER VISUAL ANALYSIS For Call of Duty, Fortnite, Rainbow Six Siege, and future supported games, the CV platform should support: • Player detection • Head detection • Bounding boxes • Target markers • Radar-style visualization • FOV visualization • Object/class detection • Target tracking • Target prioritization • Prediction • Smoothing • Confidence filtering The exact classes and behavior should be controlled through game profiles and model configuration. ESP-STYLE OVERLAY The overlay should visualize actual CV results in real time. Possible information includes: • Player boxes • Head points • Object markers • Confidence • Tracking state • FOV boundaries • Target/crosshair relationship • Other model-provided information The rendering system should remain lightweight and update independently from inference whenever possible. RADAR Radar should use the same underlying detection and tracking data rather than running a duplicate detector. It should provide a simplified 2D representation of detected objects/players and allow different coordinate interpretations for different games. TRACKING Detection determines where an object is in one frame. Tracking determines whether detections across multiple frames represent the same object. The tracker should reduce: • Jitter • Flickering • Target switching • Temporary detection loss • Unstable positions Tracking parameters should be configurable and independent from the detection model. FOV / FILTERING The system should support runtime-adjustable FOV and filtering. Useful parameters include: • FOV size • Confidence threshold • Detection classes • Target priority • Head/body priority • Tracking persistence • Prediction • Smoothing FOV should determine which detections are considered relevant around the screen center. PREDICTION / SMOOTHING Prediction may estimate future object position from recent movement. Example: Frame 1: X=800 Frame 2: X=810 Frame 3: X=820 Estimated next position ≈ X=830. Prediction must be configurable. Smoothing should reduce visual jitter while allowing the user to balance stability versus responsiveness. REAL-TIME FRAME PROCESSING A major requirement is a **latest-frame architecture**. For example, if the capture stream is 60 FPS but inference is 30 FPS, the application should not process an ever-growing queue of old frames. Preferred design: New frame arrives → replace previous unprocessed frame → process newest frame → discard stale frame This keeps the analysis synchronized with current gameplay. Measure: • Capture FPS • Inference FPS • Tracking FPS • GUI FPS • Inference time • Frame age • Dropped frames • Processing latency • Detection confidence At 60 FPS the frame interval is approximately 16.67 ms; at 120 FPS it is approximately 8.33 ms. LATENCY The system should measure actual CV latency instead of displaying arbitrary values. The developer should distinguish CV processing latency from the total video/input path, which may include: Controller → Console → Remote Play/network → Encoding → Decoding → Capture → CV → Tracking → Overlay → Display I want real measured telemetry for each stage where practical. HELIOS / CVPYTHON / GCVWORKER Expected integration: Helios Frame → CVPython → GCVWorker → CV Processing → Model → Results The developer must verify that the worker is genuinely receiving and processing live frames. The application should detect and report: • CV runtime startup failure • GCVWorker failure • Capture disconnect • Invalid frame • Model initialization failure • Inference stall • Missing dependency • Network/API failure where applicable YOLO / ROBOFLOW Support YOLO and/or Roboflow models/workflows. Possible classes include: • Player • Head • Body • NPC • Vehicle • Meter • Game-specific objects Game-specific models should be supported where they provide better results. For Roboflow Workflows, use the actual Workflow definition and output schema rather than assuming field names or response formats. Pipeline: Live Frame → Preprocess → Model/Workflow → Parse → Filter → Track → Game-Specific Analysis → Visualization IMAGE PROCESSING Support: Capture → ROI/crop → Resize → Color conversion → Normalization → Inference → Postprocessing ROI should be configurable where it improves performance. Input sizes such as 640×640 and 960×960 should be benchmarked for the target hardware. GPU