Computer Vision Data Extraction for Building AI Gameplay Context Agent

🧩 Overview

The project focuses on developing an AI agent capable of analyzing gameplay footage and providing real-time, context-aware recommendations based on the current in-game scenario. To achieve this, I designed and implemented a robust data science pipeline that integrates computer vision techniques to extract player status and item information from visual inputs. This visual data is then combined with textual game logs to create a comprehensive understanding of gameplay dynamics.

Key Machine Learning Techniques Involved:

1. Optical Character Recognition (OCR) for in-game text and status extraction
2. YOLO (You Only Look Once) for real-time object detection and tracking
3. Large Language Models (LLMs), specifically LLaMA 3.2, for semantic understanding and recommendation generation
4. Multi-source data aggregation and alignment to ensure consistent, structured input for downstream processing

📡 Results and Demo Video

AI Context Video Demo

AI_Context_Agent_Demo.mp4

A demo video was produced to showcase the system in action, featuring real-time captions and contextual descriptions generated by the LLaMA 3.2 model as the gameplay footage progresses. The synchronized output highlights the AI agent's ability to interpret game states and provide timely insights aligned with in-game events.

OCR and Yolo Detection Demo

Detection.Demo.mp4

🧠 Background

The game being used for the research and development of the proposed solution is Open Arena Quake 3. An open sourced first person shooting game that provides openly accessible gameplay files and logs, which records essential game events and metrics. Heres a brief description of how the game is being played:

• Player is on their own and must battle against other players or enemy bots.
• The target of a match is to score highest amongst all opponents.
• In the game, players can pick up different items, including weapon, health points, and armour points to aid them in killing enemies and sustaining themselves in battle.

🛠 Problem Statement

The provided textual log file records the major game events that have taken place during the game, including kill, item pickup, and new game round events. However, it does not provide context on what the player saw and their corresponding status in the game.This means the AI agent built by only using the inherent game log will not be able to analyze player’s decision making and provide insights due to the missing of essential game information.

Despite the huge missing information, it is still possible to fill the gap and develop the game context agent.

• Player status and vision data is clearly shown on the in-game screen, it is entirely feasible to extract the related data using computer vision methods.
• Player status data: Numbers indicating player's health points, armour points, and number of ammunition.
• Player vision data: What the player saw in the in-game environment, including enemy, weapon, and armour etc.

🚀 Solution Overview

As mentioned in the problem statement that player status and vision data is shown on the player's screen, this means that these information can be extracted by applying character recognition and object detection techniques.

The figure above shows a solution design diagram for the AI system proposed in this project. The system can be divided into the two sections data processing backend and Streamlit frontend. The following shows the steps of how this system is being operated.

1. Data Processing Pipeline

• When Open Arena is being played, game frame generation and log generation script runs in the backend to extraction the game images and raw log data
• Game frames are being annotated, then passed into a trained Yolo model and OCR model for extracting object detection data (eg. items, enemies) and player status data (eg. health, armour points). The two computer vision extracted data are then being cleaned to align its format with the final dataset. On the other hand, the raw textual log is being cleaned for the same purpose.
• The three cleaned datasets are being aggregated into a final dataset to be passed into Llama inference

2. Data Analytic Backend

• The annotated images are being trained in the Yolo
• Feature extraction will be done by passing data into the trained Yolo model and OCR model, insights will be saved in the form of CSV
• The large language model is being hosted in the backend, available for any query

3. Streamlit Frontend

• The client can view the detection results and their visualizations for Yolo and OCR respectively
• When the client queries the application for a summary of playing performance and further recommendations, Llama-3.2 will query the upload context data and provide relevant information and advice to the player

Initial Data Extraction

Below shows the initial data extraction results for Yolo and OCR.

Yolo

Yolo extracts the following information:

1. Detection Timestamp
2. Object type: Enemy, armour, ammmo, weapon, health
3. Object Confidence
4. Object Position

OCR

OCR extracts the following information:

1. Detection Timestamp
2. Health, armour, and ammo value
3. Confidence score for health, armour and ammo

Data Aggregation

One common attribute that is shared between log, OCR, and Yolo extracted data is timestamp. As a result, these three data are jointed together by their timestamp, arragning according to chronological order.
The figure above shows the aggregated dataset, which listed out the different information that happened in that second below:

1. Kill Event
2. Death Event
3. Score Event
4. Items Pickup Event
5. Items Detected Event
6. Player Status

The final aggregated dataset is fitted into the LLaMA 3.2 model, aligned with gameplay footage to produce contextual descriptions of the gameplay environment in real time.