This paper presents a method to generate gameplay highlight reels by finetuning the X-CLIP multimodal model on an in-house FPS game dataset, achieving over 90% event detection accuracy and demonstrating transfer learning, while optimizing deployment through quantization.
Multimodal Data, Classification, Transfer Learning, Fine-tuning, Efficiency
Vignesh Edithal, Le Zhang, Ilia Blank, Imran Junejo
AMD
Generated by grok-3
Background Problem
The paper addresses the challenge of automatically generating highlight reels from gameplay videos to enhance gamers’ ability to share engaging content on social media, particularly in the fast-growing E-sports and Twitch communities. Traditional methods for highlight detection, such as game engine integration or OCR-based techniques, are costly and lack generalization across different games and languages. The key problem solved is the development of a scalable, data-efficient solution for detecting interesting events in gameplay videos without requiring per-game engineering, thus saving time for gamers and increasing audience engagement.
Method
The core method involves finetuning a multimodal video understanding model, X-CLIP, which combines video and text encoders to classify one-second gameplay clips into specific event types (e.g., Kill, Death) or background events. The process includes: (1) creating an in-house dataset of annotated gameplay videos from five first-person shooter (FPS) games, (2) preprocessing video clips by extracting and normalizing frames, (3) finetuning X-CLIP’s video encoder using game-specific text prompts (e.g., ‘CSGO. Kill. Player in front of the gun falls down’) to improve classification, and (4) applying post-training quantization (PTQ) using ONNX tools to reduce model size and inference time for deployment. The method leverages natural language supervision for generalization across games and uses cross-frame attention mechanisms in X-CLIP to model spatio-temporal dependencies efficiently.
Experiment
The experiments utilized an in-house dataset of 110 GB of gameplay videos from five FPS games (CS:GO, Valorant, PUBG, Fortnite, Overwatch 2), annotated for seven event types plus background events, split 80%-20% for training and testing. The setup aimed to evaluate event detection accuracy and generalization, with additional testing on an unseen game (Apex Legends) using a sliding window approach. Results showed the finetuned X-CLIP model achieved over 94.3% accuracy on the test set across all games, with strong performance on popular games like CS:GO and Valorant (95%), though lower on low-resource games like Fortnite (87.6%). Transfer learning was evident as low-resource games benefited from training with high-resource games. Post-quantization, accuracy slightly dropped to 92.9%, with a more significant decline for low-resource games. Runtime performance indicated a feasible 13 FPS drop during gameplay with minimal resource impact. However, the dataset imbalance and limited calibration data for quantization may affect real-world robustness, and the sliding window method for Apex Legends might overestimate accuracy. The results largely match expectations for high-resource games but highlight generalization challenges for underrepresented data.
Further Thoughts
The use of natural language supervision in X-CLIP for gameplay event detection opens intriguing avenues for cross-domain applications, such as adapting similar multimodal models for real-time sports highlight generation or surveillance event detection, where visual and contextual cues are critical. However, the heavy reliance on visual data without integrating audio modalities, as noted in the paper’s future work, limits the model’s ability to disambiguate complex events (e.g., distinguishing who scored a kill in CS:GO). Exploring joint audio-visual models could significantly enhance performance, drawing inspiration from recent works in multimodal learning for video understanding. Additionally, the dataset imbalance issue suggests a need for active learning strategies to prioritize data collection for underrepresented games or events, potentially improving generalization. Finally, the ethical implications of automated highlight generation, such as privacy concerns with shared gameplay footage or bias in event selection (e.g., favoring kills over strategic plays), warrant further investigation, especially as such tools become integrated into consumer software.