EMG-CrossFormer: Multimodal Hand Gesture Recognition

Official repository for the paper:

Multimodal Surface EMG Hand Gesture Recognition Using Query-Based Transformers for Prosthetic Control (under peer review — preprint)

EMG-CrossFormer

EMG-CrossFormer is a multimodal model for hand gesture decoding. Representations ($\mathbf{z}_{\text{enc}_m}$) from an arbitrary number of unimodal encoder backbones are progressively fused through cascaded transformer decoder layers into a shared representation ($\mathbf{z}_{\text{fuse}}$). Learnable hand movement queries then decode $\mathbf{z}_{\text{fuse}}$ into $\mathbf{z}_{\text{dec}}$, from which a shared FFN produces the final gesture predictions.

Installation

The code has only been tested with a Linux environment. To set up all the required dependencies, please follow the instructions below:

• Clone the repository:

  git clone https://github.com/deepPNClab/emg-crossformer.git
  cd emg-crossformer

• (Recommended) Create and activate a conda environment:

  conda create --name emgcrossformer python=3.11
  conda activate emgcrossformer

• Install the dependencies:

  pip install -r requirements.txt

• Install extra dependencies for notebooks:

  pip install -r requirements-extra.txt

Data Preparation

Training data are openly available from NinaPro (DB2, DB3, DB7) and Harvard Dataverse (DB10).

NinaPro (DB2, DB3, DB7) — databases can be directly downloaded with wget. Example for DB3:

seq 1 11 | xargs -n 1 -P 4 -I {} \
  wget -c https://ninapro.hevs.ch/files/db3_Preproc/s{}_0.zip -P .

Adapt the subject count and file name pattern for DB2 and DB7.

Harvard Dataverse (DB10) — download .mat files only:

curl -s "https://dataverse.harvard.edu/api/datasets/:persistentId?persistentId=doi:10.7910/DVN/1Z3IOM" \
  | jq -r '.data.latestVersion.files[] | select(.dataFile.filename | endswith(".mat")) | "\(.dataFile.filename) \(.dataFile.id)"' \
  | while read -r name id; do
      echo "Downloading $name"
      curl -L -o "$name" "https://dataverse.harvard.edu/api/access/datafile/$id"
    done

Once downloaded, unzip and organize files with the following structure:

DB_X/
└── DBX_sY/
    ├── SY_E1_A1.mat
    ├── SY_E2_A1.mat  # if available
    └── SY_E3_A1.mat  # if available

Finally, convert .mat files to .pickle format for faster loading using convert_mat_to_pickle.py. You should get:

DB_X/
└── DBX_sY/
    ├── SY_E1_A1.pickle
    ├── SY_E2_A1.pickle  # if available
    └── SY_E3_A1.pickle  # if available

Training

Training is configured via YAML files in the configs folder:

File	Purpose
run.yaml	Main session config: database, model, device, and paths to other config files
paths.yaml	Filesystem paths used during training
model.yaml	Model architecture and initialization hyperparameters
augmentation.yaml	Data augmentation pipeline
dataloading.yaml	Data preloading and preprocessing settings
train.yaml	Training hyperparameters (learning rate, epochs, etc.)

Once configured, start training from the project root:

python3 main.py

Notebooks

Notebooks in the notebooks folder can be used to reproduce the figures presented in the paper and run statistical analyses to compare models.

Extending with New Models

Adding a new model is straightforward:

1. Implement your model and place the script in src/models
2. Add its initialization hyperparameters to configs/model.yaml
3. Set the model class name in configs/run.yaml
4. Run main.py

License

Code is released under the MIT License.

Contributing

See CONTRIBUTING.md and the Code of Conduct.

Citing

The paper is currently under peer review. A preprint is available here.