ST-TrackFormer: Multiple Object Tracking Performance Improvement Using Spatial-Temporal Attention

Se-Eun Lee, Se-Hoon Jung, and Chun-Bo Sim, Sunchon National University

Journal of Korea Multimedia Society, Vol. 27, No. 11, November 2024

DOI: 10.9717/kmms.2024.27.11.1227

This repository is the official implementation of ST-TrackFormer. The method extends TrackFormer with spatial and temporal attention modules for multi-object tracking.

Overview

ST-TrackFormer applies spatial attention in the Transformer encoder and temporal attention in the encoder and decoder.

• Spatial attention models relationships within each frame.
• Temporal attention models information across consecutive frames.

MOT17 results reported in the paper:

Method	Recall	MOTA	IDF1
TrackFormer	70.5	67.7	68.2
ST-TrackFormer (Ours)	71.5	68.5	69.1

Contents

• Installation
• Data Preparation
• Pretrained Models
• Training
• Evaluation
• Results
• Citation
• Acknowledgements

Installation

1. Clone this repository:

   git clone git@github.com:se2un/ST-TrackFormer.git
   cd ST-TrackFormer

2. Install Python dependencies:

   pip install -r requirements.txt

3. Install PyTorch 1.8+ and torchvision from pytorch.org.

4. Install pycocotools:

   pip install -U 'git+https://github.com/timmeinhardt/cocoapi.git#subdirectory=PythonAPI'

5. Build the MultiScaleDeformableAttention CUDA operator:

   python src/trackformer/models/ops/setup.py build --build-base=src/trackformer/models/ops/ install

Data Preparation

Run the following commands from the repository root. The conversion script creates COCO-format annotations under each dataset directory.

MOT17 (required)

cd data
wget https://motchallenge.net/data/MOT17.zip
unzip MOT17.zip
cd ..
python src/generate_coco_from_mot.py

MOT16 (optional)

cd data
wget https://motchallenge.net/data/MOT16.zip
unzip MOT16.zip
cd ..
python src/generate_coco_from_mot.py --mot16

MOT20 (optional)

cd data
wget https://motchallenge.net/data/MOT20.zip
unzip MOT20.zip
cd ..
python src/generate_coco_from_mot.py --mot20

The expected dataset paths are:

data/
├── MOT17/
│   ├── train/
│   ├── test/
│   └── annotations/
├── MOT16/                      # optional
│   ├── train/
│   ├── test/
│   └── annotations/
└── MOT20/                      # optional
    ├── train/
    ├── test/
    └── annotations/

Pretrained Models

Download the base Deformable DETR pretrained model (required to initialize training):

cd models
wget https://vision.in.tum.de/webshare/u/meinhard/trackformer_models_v1.zip
unzip trackformer_models_v1.zip

This extracts r50_deformable_detr_plus_iterative_bbox_refinement-checkpoint_hidden_dim_288.pth, which is used as the starting checkpoint for MOT17 training.

ST-TrackFormer checkpoints are not included in this repository due to file size.

Training

ST-TrackFormer is trained directly on MOT17, initialized from the base Deformable DETR checkpoint above.

MOT17 Training

torchrun --nproc_per_node=4 --use_env src/train.py with \
    mot17 \
    deformable \
    multi_frame \
    tracking \
    full_res \
    output_dir=models/my_model_mot17

Training configuration is defined in cfgs/train_mot17.yaml and cfgs/train_full_res.yaml. Key hyperparameters:

Parameter	Value
Backbone	ResNet-50
Hidden dim	288
Batch size	1 (×4 GPUs)
Epochs	50
Learning rate	2e-4
LR drop	10
Image max size	650

Evaluation

Run evaluation commands from the repository root. MOT17-TRAIN-FRCNN selects the MOT17 training sequences with the public FRCNN detections.

For second-half validation, evaluate frames in the normalized range [0.5, 1.0]:

python src/track.py with \
    obj_detect_checkpoint_file=models/my_model_mot17/checkpoint_best_MOTA.pth \
    dataset_name=MOT17-TRAIN-FRCNN \
    output_dir=models/my_model_mot17/track_eval_half \
    frame_range.start=0.5 \
    frame_range.end=1.0

For evaluation on the complete MOT17 training sequences, omit the frame-range overrides:

python src/track.py with \
    obj_detect_checkpoint_file=models/my_model_mot17/checkpoint_best_MOTA.pth \
    dataset_name=MOT17-TRAIN-FRCNN \
    output_dir=models/my_model_mot17/track_eval

Tracking configuration can be adjusted in cfgs/track.yaml.

Results

The following tables report the values presented in the paper for MOT16, MOT17, and MOT20. The tables do not include test-server submissions.

MOT16

Method	Recall	MT	PT	FN	MOTA	IDF1
TrackFormer	74.30	50.00	49.79	50.72	71.40	78.10
ST-TrackFormer	75.10	50.00	50.21	49.28	72.40	77.70

MOT17

Method	Recall	MT	PT	FN	MOTA	IDF1
TrackFormer	70.50	49.19	49.81	50.88	67.70	68.20
ST-TrackFormer	71.50	50.81	50.19	49.12	68.50	69.10

MOT20

Method	Recall	MT	PT	FN	MOTA	IDF1
TrackFormer	78.50	48.62	51.59	52.10	76.70	70.50
ST-TrackFormer	80.20	51.38	48.41	47.90	78.20	71.80

Ablation Study (MOT17)

Comparison of different Spatial-Temporal Attention combinations:

Case	Encoder SA	Encoder TA	Decoder SA	Decoder TA	MOTA	IDF1
1 (Proposed)	✓	✓	-	✓	68.5	69.1
2	✓	-	-	✓	67.3	68.0
3	-	✓	-	✓	67.7	67.5
4	✓	✓	-	-	67.7	68.9
5	✓	-	-	-	67.3	68.0
6	-	✓	-	-	67.6	69.8
7	-	-	-	✓	68.0	69.5
8	-	-	-	-	67.7	68.2

Qualitative Results

The left image shows TrackFormer output and the right image shows ST-TrackFormer output on MOT16.

Citation

Please cite the following paper when using this implementation:

@article{lee2024sttrackformer,
    title={ST-TrackFormer: Multiple Object Tracking Performance Improvement Using Spatial-Temporal Attention},
    author={Se-Eun Lee and Se-Hoon Jung and Chun-Bo Sim},
    journal={Journal of Korea Multimedia Society},
    volume={27},
    number={11},
    pages={1227--1237},
    year={2024},
    doi={10.9717/kmms.2024.27.11.1227}
}

This codebase builds upon TrackFormer. Please also cite the original work:

@InProceedings{meinhardt2021trackformer,
    title={TrackFormer: Multi-Object Tracking with Transformers},
    author={Tim Meinhardt and Alexander Kirillov and Laura Leal-Taixe and Christoph Feichtenhofer},
    booktitle={CVPR},
    year={2022}
}

Acknowledgements

This work was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. RS-2024-00407739) and by the Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (IITP-2024-2020-0-01489).