SceneSense: Multi-Style Video Captioning Agent (Track 2)

SceneSense is a lightning-fast, token-efficient Video Captioning agent designed specifically for the AMD Developer Hackathon: ACT II.

Recognizing that traditional multi-stage video pipelines are too slow and prone to context loss, SceneSense utilizes a cutting-edge single-stage Vision-Language Model (VLM) architecture to guarantee perfect style alignment and speed.

Architecture & How It Works

1. Robust Frame Sampling: Uses OpenCV to extract exactly 10 evenly spaced frames from a video clip. It resizes and filters them down to the 4 most representative frames, entirely avoiding the instability of threshold-based scene detection.
2. Direct Visual Styling: Bypasses intermediate text summaries and feeds the frames directly into the VLM (powered by Fireworks AI) alongside distinct persona prompts.
3. Strict Output Guardrails: Uses persona-driven prompts and a strict <caption_output> XML constraint to ensure perfect output validation. A regex parser strips away any conversational fluff, ensuring the evaluator gets 100% clean data every time.

Getting Started

Prerequisites

• Docker installed on your machine
• A Fireworks AI API Key

Build and Run with Docker (Recommended for Judges)

1. Clone the repository:

git clone https://github.com/kirankigi5/SceneSense.git
cd SceneSense

2. Build the container:

docker build --platform linux/amd64 -t scenesense:latest .

3. Run the Captioning Agent: To run the agent in competition harness mode (reading from tasks.json and outputting to results.json):

docker run --rm \
  -e FIREWORKS_API_KEY="your_api_key_here" \
  -v "$(pwd)/tasks.json:/app/tasks.json" \
  -v "$(pwd)/results.json:/app/results.json" \
  -v "$(pwd):/app/video_files" \
  scenesense:latest python -m app.main --tasks-path /app/tasks.json --results-path /app/results.json

(Ensure that any local video files referenced in tasks.json are in the directory you mount to /app/video_files, or use absolute URLs).

Running Locally (Without Docker)

# Setup virtual environment
python -m venv .venv
source .venv/bin/activate

# Install dependencies
pip install -r requirements.txt
pip install -e .

# Run a quick local test
export FIREWORKS_API_KEY="your_api_key"
python -m app.main mock_video.mp4