Skip to content
BranchesTags

Repository files navigation

🦑 OctoNet Toolbox 🦑

The Ultimate Multi-Modal Human Activity Understanding Toolkit

Project Page GitHub License

🎯 Revolutionary Multi-Modal Dataset for Human Activity Understanding

Comprehensive sensor fusion • State-of-the-art benchmarks • Ready-to-use visualization tools

OctoNet Dataset Overview - Multi-modal sensor data visualization

🚀 What's Inside This Toolbox

✨ Comprehensive OctoNet Toolkit ✨

This powerful toolbox provides everything you need to work with the OctoNet dataset:

🎨 Visualization Suite

  • Interactive dataset exploration tools
  • Multi-modal data visualization capabilities

⚡ Benchmark Framework

  • Reproducible benchmark implementations
  • Benchmark results recordings

🎯 Ready to dive into multi-modal human activity recognition? Let's get started!

🎨 Part 1: Dataset Visualization & Exploration 🎨

Interactive Multi-Modal Data Analysis Suite

💡 💻 Recommended Environment: Run the code in Python Jupyter Notebook demo.ipynb for the best interactive experience!

📁 Dataset Structure Overview

🗂️ Complete Directory Layout:

./dataset
├── mocap_csv_final          # Data: Final motion capture data in CSV format.
├── mocap_pose               # Data: Final motion capture data in npy format.
├── node_1                   # Data: Data related to multi-modal sensor node 1.
├── node_2                   # Data: Data related to multi-modal sensor node 2.
├── node_3                   # Data: Data related to multi-modal sensor node 3.
├── node_4                   # Data: Data related to multi-modal sensor node 4.
├── node_5                   # Data: Data related to multi-modal sensor node 5.
├── imu                      # Data: Inertial measurement unit data.
├── vayyar_pickle            # Data: vayyar mmWave radar data.
└── cut_manual.csv           # Manually curated data cuts.

📊 Dataset Metadata & Statistics

🔍 **📋 Click to View Complete OctoNet Dataset Metadata**

📝 Key Information:

📌 Important Notes:

  • 👥 Gender Classification: Male (M) and Female (F) participants
  • 🏃 Activity Types: PA&F indicates subjects performed both Programmed Aerobics and Freestyle activities
  • ⭐ Special Marking: Asterisk (*) denotes subjects who performed only Programmed Aerobics (no Freestyle)
  • 🏠 Scene Mapping: Scene 1: IDs 1-99, Scene 2: IDs 101-199, Scene 3: IDs 201-299
User (Gender) Exp ID Scene 1: Activity IDs Scene 1: PA&F Scene 2: Activity IDs Scene 2: PA&F Scene 3: Activity IDs Scene 3: PA&F
1 (M) 1, 11, 101, 201 all 62 ✓ 1–23 1–23, 57–62 ✓*
2 (M) 2, 12, 102, 112, 202 all 62 ✓ 9–29 ✓ 9–29
3 (M) 3, 13, 113, 213 all 62 ✓ ✓ ✓
4 (F) 4, 14, 104, 114, 204 all 62 ✓ 30–56 ✓ 30–56
5 (M) 5, 15, 115, 215 all 62 ✓ ✓ ✓
6 (F) 6, 16 all 62 ✓
7 (M) 7, 17, 117, 217 all 62 ✓ ✓ ✓
8 (M) 8, 18, 108, 118 all 62 ✓ 24–62 ✓ 24–62
9 (M) 9 all 62
10 (M) 10, 20, 120, 220 all 62 ✓ ✓ ✓
11 (F) 21 ✓
12 (M) 22 ✓
13 (F) 23 ✓
14 (M) 24 ✓
15 (F) 25 ✓
16 (F) 26 ✓
17 (F) 27 ✓
18 (F) 28 ✓
19 (F) 29 ✓
20 (F) 30, 230 ✓ ✓
21 (M) 31 ✓
22 (M) 32 ✓
23 (F) 33 ✓
24 (M) 34 ✓
25 (M) 35 ✓
26 (M) 36 ✓
27 (M) 37 ✓
28 (F) 38 ✓
29 (F) 39 ✓
30 (M) 40 ✓
31 (M) 41 ✓
32 (F) 42 ✓
33 (F) 43 ✓
34 (F) 44 ✓
35 (M) 45 ✓
36 (M) 46 ✓
37 (M) 47 ✓
38 (F) 48 ✓
39 (F) 49 ✓
40 (M) 111, 211 1–8 ✓ 1–8 ✓
41 (F) 121, 221 ✓ ✓

⚙️ Environment Setup & Installation ⚙️

Quick Start Guide for OctoNet Development Environment

🐍 Step 1: Create Conda Environment

🔧 Automated Environment Creation:

# Create the OctoNet environment from the provided specification
conda env create -f environment.yaml

# Install additional Python packages
pip install -r requirements.txt

# Activate the environment (uncomment when ready)
# conda activate octonet

💡 💻 Alternative: You can skip the conda environment creation if you're using an existing Python environment with compatible packages.

🚀 Step 2: Launch Jupyter Notebook

📓 Start Interactive Development:

# Launch Jupyter Notebook with the OctoNet environment
jupyter notebook demo.ipynb

🎯 Ready to explore the OctoNet dataset interactively!

📊 Sample Data Selection & Configuration 📊

Intelligent Dataset Loading with Flexible Configuration

🔧 Core Dataset Loading Function

📁 Smart Dataset Management:

In dataset_loader.py, we provide a powerful get_dataset function that enables flexible dataset loading with comprehensive configuration options:

def get_dataset(config, dataset_path="", mocap_downsample_num = None) -> OctonetDataset:
    """
    Args:
        config: config file
        dataset_path: path to the dataset
        mocap_downsample_num: number of downsample for mocap data, could be shadowed by config['mocap_downsample_num']
    Returns:
        OctonetDataset: a dataset object
    """
    ...

⚙️ Complete Configuration Template

🔧 Full Dataset Configuration Options:

config = {
    'exp_list': [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 101, 102, 104, 108, 111, 112, 113, 114, 115, 117, 118, 120, 121, 201, 202, 204, 208, 211, 213, 215, 217, 220, 221, 230]
    'activity_list': ['sit', 'walk', 'bow', 'sleep', 'dance', 'jog', 'falldown', 'jump', 'jumpingjack', 'thunmbup'
        'squat', 'lunge', 'turn', 'pushup', 'legraise', 'airdrum', 'boxing', 'shakehead',
        'answerphone', 'eat', 'drink', 'wipeface', 'pickup', 'jumprope', 'moppingfloor',
        'brushhair', 'bicepcurl', 'playphone', 'brushteeth', 'type', 'thumbup',
        'makeoksign', 'makevictorysign', 'drawcircleclockwise', 'drawcirclecounterclockwise',
        'stopsign', 'pullhandin','pushhandaway', 'handwave', 'sweep', 'clap', 'slide',
        'drawzigzag', 'dodge', 'bowling', 'liftupahand', 'tap', 'spreadandpinch', 'drawtriangle',
        'sneeze', 'cough', 'stagger', 'yawn', 'blownose', 'stretchoneself', 'touchface',
        'handshake', 'hug', 'pushsomeone', 'kicksomeone', 'punchsomeone', 'conversation', 'gym', 'freestyle'],  # Specify which activities to filter
    'node_id': [1, 2, 3, 4, 5], 
    'segmentation_flag': True, # whether to include segmentation in the dataset
    'modality': [ 'mmWave', 'IRA', 'uwb', 'ToF', 'polar', 'wifi', 'depthCamera', 'seekThermal','acoustic', 'imu', 'vayyar', 'mocap'] # depthCamera is RGB-D camera
}

🎯 Custom Configuration Example

🔍 Targeted Dataset Selection:

To select a specific subset of the dataset, you can customize the configuration. Here's a practical example:

config = {
    'exp_list': [1],  # select exp 1
    'activity_list': ['dance'],  # select activity 'dance'
    'node_id': [1, 2, 3, 4, 5],  # select all nodes
    'segmentation_flag': True, # data is segmented
    'modality': [ 'mmWave', 'IRA', 'uwb', 'ToF', 'polar', 'wifi', 'depthCamera', 'seekThermal','acoustic', 'imu', 'vayyar', 'mocap'], # select all modalities
    # 'modality': ['polar', 'depthCamera'], # select polar and depthCamera modalities
    # 'mocap_downsample_num': 6 # downsample the mocap data to 6 frames per second
}

💡 📝 Smart Loading: get_dataset intelligently includes all available data that matches your configuration criteria!

🎨 Interactive Visualization Suite

📊 Multi-Modal Data Visualization:

The comprehensive visualization code is provided in demo.ipynb. This interactive notebook will automatically generate:

  • 📈 Figures and plots for data analysis
  • 🎬 Video outputs for temporal data visualization
  • 📁 Organized results in the vis_output folder
# Sample configuration and usage
dataset_path = "dataset"
data_config = {
    'exp_list': [1],  # Specify which experiments to filter
    'activity_list': ['dance'],  
    'node_id': [1, 2, 3, 4, 5], 
    'segmentation_flag': True,
    'modality': [ 'mmWave', 'IRA', 'uwb', 'ToF', 'polar', 'wifi', 'depthCamera', 'seekThermal','acoustic', 'imu', 'vayyar', 'mocap'],
    # 'modality': ['polar', 'depthCamera'],
    # 'mocap_downsample_num': 6
}

# Get the DataLoader
dataset = get_dataset(data_config, dataset_path)
dataloader = get_dataloader(dataset, batch_size=1, shuffle=False, config=data_config)

for batch in dataloader:
    dump_seekthermal_frames_as_png(
        batch, 
        output_dir="validation_seekthermal"
    )
    visualize_seekthermal_and_rgb_mosaic_batch_discard_excess(
        batch,
        output_dir='seekthermal_rgb_mosaic_videos',
        fps_out=8.80
    )
    visualize_3_depth_3_rgb_mosaic_batch_discard_excess(
        batch,
        output_dir='depth_rgb_mosaic_discard',
        fps_out=10
    )
    visualize_4wifi_time_subcarrier_with_camera(
        batch,
        output_dir='wifi_rgb_mosaic_videos',
        fps_out=10.0,
        BW="40MHz"
    )
    visualize_ira_and_rgb_mosaic_batch_downsample_cam(
        batch,
        output_dir='ira_rgb_mosaic_videos',
        fps_out=6.91
    )
    visualize_mocap_and_rgb_mosaic_batch_downsample_mocap(
        batch,
        output_dir='mocap_rgb_mosaic_videos',
        fps_out=10
    )
    visualize_tof_and_rgb_mosaic_batch_downsample_tof(
        batch,
        output_dir='tof_rgb_mosaic_videos',
        fps_out=7.32
    )
    visualize_fmcw_and_rgb_mosaic_batch_raw_fixed_axes(
        batch,
        output_dir='fmcw_rgb_mosaic',
        fps_out=8.81
    )
    visualize_vayyar_txrx_only_and_camera(
        batch,
        output_dir="vayyar_rgb_mosaic",
        fps_out=10.0
    )
    visualize_acoustic_2node_melspectrogram_and_rgb(
        batch,
        output_dir="acoustic_melspec_plus_rgb",
        fps_out=10.0
    )
    visualize_polar_and_camera_batch(
        batch,
        output_dir="polar_hr_plus_rgb",
        fps_out=10.0,
        y_domain=None
    )
    visualize_imu_four_rows_no_zscore(
        batch,
        output_dir="imu_time_features_plus_rgb",
        fps_out=10.0
    )
    visualize_uwb_and_rgb_in_same_row_with_box(
        batch,
        output_dir="uwb_rgb_same_row_with_box",
        fps_out=10.0
    )
    break

⚡ Part 2: Benchmark & Reproducible Results ⚡

State-of-the-Art Performance Evaluation Framework

🏆 Comprehensive benchmarking suite for the OctoNet dataset with reproducible results and performance comparisons.

🚀 Step 1: Environment Setup

📁 Navigate to Benchmark Directory:

# Change to the benchmark directory
cd OctonetBenchmark

🐍 Python Environment Requirements:

📋 Recommended: Python 3.9–3.11 with our pre-configured Conda environment

🔧 Automated Environment Creation:

# Create the benchmark environment from the provided specification
conda env create -f environment.yml

# Activate the environment (name defined in environment.yml)
conda activate octo

# (Optional) Update existing environment after changes
conda env update -f environment.yml --prune

💡 Important Notes:

🎮 GPU Support: The environment.yml includes GPU-enabled PyTorch and CUDA libraries. For CPU-only setups, remove CUDA-related packages (pytorch-cuda, cudnn, cuda-*) and the nvidia channel.

⚡ Performance Tip: If conda solver struggles, try using mamba as a faster drop-in replacement for conda.

📥 Step 2: Dataset Download

🌐 Official Dataset Sources:

📊 Primary Source: Hugging Face Dataset
💻 Code Repository: GitHub Repository

⚡ Automated Download Process:

The dataset provider offers an automated script that downloads 16 chunks, merges, and extracts them seamlessly.

💾 Storage Requirements: ~1.5TB peak disk space during download and extraction

# One-command automated download and extraction
bash -c "$(wget -qO- https://huggingface.co/datasets/hku-aiot/OctoNet/resolve/main/download_octonet.sh)"

📁 Expected Dataset Structure:

After download and extraction, ensure you have the following structure under a directory named dataset (you can choose a different parent path):

./dataset
├── mocap_csv_final          # Data: Final motion capture data in CSV format.
├── mocap_pose               # Data: Final motion capture data in npy format.
├── node_1                   # Data: Data related to multi-modal sensor node 1.
├── node_2                   # Data: Data related to multi-modal sensor node 2.
├── node_3                   # Data: Data related to multi-modal sensor node 3.
├── node_4                   # Data: Data related to multi-modal sensor node 4.
├── node_5                   # Data: Data related to multi-modal sensor node 5.
├── imu                      # Data: Inertial measurement unit data.
├── vayyar_pickle            # Data: vayyar mmWave radar data.
└── cut_manual.csv           # Manually curated data cuts.

🔧 Step 3: Dataset Preparation

📦 Benchmark-Specific Setup:

Rename the dataset directory to octonet and place the dataset helper script Octonet.py inside it. You should end up with:

./octonet
├── mocap_csv_final          # Data: Final motion capture data in CSV format.
├── mocap_pose               # Data: Final motion capture data in npy format.
├── node_1                   # Data: Data related to multi-modal sensor node 1.
├── node_2                   # Data: Data related to multi-modal sensor node 2.
├── node_3                   # Data: Data related to multi-modal sensor node 3.
├── node_4                   # Data: Data related to multi-modal sensor node 4.
├── node_5                   # Data: Data related to multi-modal sensor node 5.
├── imu                      # Data: Inertial measurement unit data.
├── vayyar_pickle            # Data: vayyar mmWave radar data.
├── Octonet.py               # script: contains the dataset PyTorch functions and dataloader
└── cut_manual.csv           # Manually curated data cuts.

⚠️ Important Setup Notes:

📁 Folder Naming: This repository imports dataset utilities via from octonet.Octonet import get_dataset, custom_collate. The folder name must be exactly octonet.

📍 Custom Paths: If you downloaded to a different location, set dataset_path in your chosen configuration file to the absolute path of your octonet folder.

⚙️ Step 4: Configuration Selection

🎯 Choose Your Benchmark Configuration:

To reproduce the paper results, select the corresponding configuration YAML listed in the results table below. Example configurations already included:

📋 Available Configurations:

  • Configurations/acoustic_denesnet121_10.yaml
  • Configurations/ira_rf_net_pose.yaml

💡 Usage Tip: Pass the config name without the .yaml suffix to the --config_file flag.

🚀 Step 5: Execute Benchmarks

⚡ Running Your Selected Configuration:

🎯 Training + Testing (Full Pipeline):

python main.py --config_file acoustic_denesnet121_10 --cuda_index 0 --mode 0

🧪 Testing Only (Pre-trained Model):

python main.py --config_file acoustic_denesnet121_10 --cuda_index 0 --mode 1 --pretrained_model /absolute/path/to/weights.pth

🔄 Fine-tuning + Testing:

python main.py --config_file ira_rf_net_pose --cuda_index 0 --mode 2 --pretrained_model /absolute/path/to/weights.pth

💾 Model Saving Configuration:

🔧 To save trained model weights: Change model_save_enable: False to model_save_enable: True in your selected configuration file. Control output paths via trained_model_folder, log_folder, and tensorboard_folder in the same YAML.

📊 Reproducible Results & Performance Tables 📊

Complete Benchmark Results with Configurations, Runs, and Logs

📁 Additional Resources:

💾 Large Log Files: Download the logs folder and place it in the same directory as the 'Configurations' folder.
🔗 Download Link: SharePoint Logs Repository

🏃 Human Activity Recognition Results 🏃

Multi-Modal Performance Comparison

Modality Protocol ResNet 10/62 DenseNet 10/62 Swin-T 10/62 RFNet 10/62
RGB ID 91.5 (±2.6) Config
Log
Run / 93.4 (±0.9) Config
Log
Run		 93.2 (±2.3) Config
Log
Run / 91.2 (±1.0) Config
Log
Run		 94.9 (±2.0) Config
Log
Run / 93.1 (±0.9) Config
Log
Run		 89.7 (±2.8) Config
Log
Run / 60.9 (±1.8) Config
Log
Run
CU 46.0 (±3.4) Log / 12.3 (±0.9) Log 68.2 (±3.2) Log / 24.7 (±1.2) Log 37.0 (±3.3) Log / 7.7 (±0.7) Log 45.0 (±3.4) Log / 9.2 (±0.8) Log
CS 14.9 (±3.0) Log / 4.1 (±0.7) Log 33.3 (±4.0) Log / 11.3 (±1.1) Log 12.1 (±2.8) Log / 1.7 (±0.4) Log 13.5 (±2.9) Log / 3.1 (±0.6) Log
Depth ID 89.7 (±2.8) Config
Log
Run / 86.6 (±1.2) Config
Log
Run		 90.6 (±2.7) Config
Log
Run / 83.2 (±1.3) Config
Log
Run		 86.3 (±3.2) Config
Log
Run / 81.7 (±1.4) Config
Log
Run		 87.2 (±3.1) Config
Log
Run / 40.0 (±1.8) Config
Log
Run
CU 41.2 (±3.4) Log / 11.1 (±0.9) Log 64.9 (±3.3) Log / 27.3 (±1.2) Log 46.0 (±3.4) Log / 14.4 (±1.0) Log 45.0 (±3.4) Log / 11.2 (±0.9) Log
CS 17.7 (±3.2) Log / 3.9 (±0.7) Log 22.7 (±3.5) Log / 12.2 (±1.1) Log 23.4 (±3.6) Log / 4.3 (±0.7) Log 28.4 (±3.8) Log / 4.8 (±0.7) Log
ToF ID 86.8 (±3.1) Config
Log
Run / 70.3 (±1.6) Config
Log
Run		 N/A 82.6 (±3.5) Config
Log
Run / 51.8 (±1.8) Config
Log
Run		 89.3 (±2.8) Config
Log
Run / 75.9 (±1.5) Config
Log
Run
CU 44.5 (±3.4) Log / 11.8 (±0.9) Log N/A 46.4 (±3.4) Log / 15.3 (±1.0) Log 78.7 (±2.8) Log / 28.3 (±1.2) Log
CS 25.5 (±3.7) Log / 8.0 (±0.9) Log N/A 22.7 (±3.5) Log / 4.7 (±0.7) Log 44.7 (±4.2) Log / 18.6 (±1.3) Log
Thermal ID 90.1 (±2.7) Config
Log
Run / 85.0 (±1.3) Config
Log
Run		 91.7 (±2.5) Config
Log
Run / 85.4 (±1.3) Config
Log
Run		 85.1 (±3.2) Config
Log
Run / 79.2 (±1.5) Config
Log
Run		 47.1 (±4.6) Config
Log
Run / 28.6 (±1.6) Config
Log
Run
CU 50.2 (±3.5) Log / 25.7 (±1.2) Log 64.5 (±3.4) Log / 32.5 (±1.3) Log 46.8 (±3.5) Log / 15.6 (±1.0) Log 15.3 (±2.5) Log / 1.0 (±0.3) Log
CS 36.9 (±4.1) Log / 13.4 (±1.2) Log 44.0 (±4.2) Log / 21.0 (±1.4) Log 36.2 (±4.1) Log / 10.1 (±1.0) Log 17.7 (±3.2) Log / 2.1 (±0.5) Log
IRA ID 25.6 (±4.0) Config
Log
Run / 1.8 (±0.5) Config
Log
Run		 N/A 14.0 (±3.2) Config
Log
Run / 3.7 (±0.7) Config
Log
Run		 19.0 (±3.6) Config
Log
Run / 4.2 (±0.7) Config
Log
Run
CU 19.9 (±2.8) Log / 2.6 (±0.4) Log N/A 22.3 (±2.9) Log / 2.8 (±0.4) Log 21.8 (±2.8) Log / 3.2 (±0.5) Log
CS 18.4 (±3.3) Log / 0.8 (±0.3) Log N/A 20.6 (±3.4) Log / 3.8 (±0.6) Log 21.3 (±3.5) Log / 2.7 (±0.6) Log
FMCW ID 39.3 (±4.5) Config
Log
Run / 24.0 (±1.6) Config
Log
Run		 74.4 (±4.1) Config
Log
Run / 46.3 (±1.8) Config
Log
Run		 36.8 (±4.5) Config
Log
Run / 5.0 (±0.8) Config
Log
Run		 38.5 (±4.5) Config
Log
Run / 12.6 (±1.2) Config
Log
Run
CU 27.0 (±3.1) Log / 8.9 (±0.8) Log 44.1 (±3.4) Log / 16.1 (±1.0) Log 24.2 (±3.0) Log / 4.4 (±0.6) Log 26.5 (±3.0) Log / 7.2 (±0.7) Log
CS 26.0 (±4.3) Log / 5.3 (±1.0) Log 14.4 (±3.5) Log / 7.5 (±1.2) Log 14.4 (±3.5) Log / 3.6 (±0.8) Log 26.0 (±4.3) Log / 4.3 (±0.9) Log
SFCW ID 30.6 (±4.2) Config
Log
Run / 9.0 (±1.0) Config
Log
Run		 59.5 (±4.5) Config
Log
Run / 13.0 (±1.2) Config
Log
Run		 26.4 (±4.0) Config
Log
Run / 0.9 (±0.3) Config
Log
Run		 28.1 (±4.1) Config
Log
Run / 5.1 (±0.8) Config
Log
Run
CU 12.3 (±2.3) Log / 1.6 (±0.3) Log 4.3 (±1.4) Log / 1.2 (±0.3) Log 7.6 (±1.8) Log / 1.6 (±0.3) Log 13.3 (±2.3) Log / 2.2 (±0.4) Log
CS 11.3 (±2.7) Log / 2.5 (±0.5) Log 15.6 (±3.1) Log / 1.5 (±0.4) Log 7.8 (±2.3) Log / 1.6 (±0.4) Log 17.0 (±3.2) Log / 1.5 (±0.4) Log
UWB ID 98.3 (±1.2) Config
Log
Run / 93.8 (±0.9) Config
Log
Run		 88.4 (±2.9) Config
Log
Run / 80.1 (±1.4) Config
Log
Run		 100.0 (±0.0) Config
Log
Run / 90.4 (±1.1) Config
Log
Run		 94.2 (±2.1) Config
Log
Run / 75.8 (±1.5) Config
Log
Run
CU 62.6 (±3.3) Log / 21.5 (±1.1) Log 59.7 (±3.4) Log / 27.4 (±1.2) Log 17.1 (±2.6) Log / 2.7 (±0.4) Log 64.5 (±3.3) Log / 13.5 (±0.9) Log
CS 27.0 (±3.7) Log / 6.7 (±0.8) Log 20.6 (±3.4) Log / 6.3 (±0.8) Log 21.3 (±3.5) Log / 2.4 (±0.5) Log 12.1 (±2.8) Log / 1.7 (±0.4) Log
WiFi ID 93.3 (±2.3) Config
Log
Run / 91.1 (±1.0) Config
Log
Run		 90.8 (±2.6) Config
Log
Run / 91.0 (±1.0) Config
Log
Run		 91.7 (±2.5) Config
Log
Run / 92.3 (±1.0) Config
Log
Run		 81.7 (±3.5) Config
Log
Run / 60.5 (±1.8) Config
Log
Run
CU 13.3 (±2.3) Log / 3.4 (±0.5) Log 11.4 (±2.2) Log / 4.8 (±0.6) Log 12.3 (±2.3) Log / 2.3 (±0.4) Log 19.9 (±2.8) Log / 4.3 (±0.6) Log
CS 19.1 (±3.3) Log / 2.4 (±0.5) Log 11.3 (±2.7) Log / 1.9 (±0.5) Log 13.5 (±2.9) Log / 2.8 (±0.6) Log 11.3 (±2.7) Log / 1.1 (±0.4) Log
Acoustic ID 40.8 (±4.5) Config
Log
Run / 45.5 (±1.8) Config
Log
Run		 60.0 (±4.5) Config
Log
Run / 54.6 (±1.8) Config
Log
Run		 36.7 (±4.4) Config
Log
Run / 32.1 (±1.7) Config
Log
Run		 29.2 (±4.2) Config
Log
Run / 19.1 (±1.4) Config
Log
Run
CU 37.0 (±3.3) Log / 19.9 (±1.1) Log 42.7 (±3.4) Log / 16.4 (±1.0) Log 27.5 (±3.1) Log / 8.4 (±0.8) Log 20.4 (±2.8) Log / 7.1 (±0.7) Log
CS 26.2 (±3.7) Log / 9.3 (±1.0) Log 25.5 (±3.7) Log / 8.7 (±1.0) Log 12.8 (±2.8) Log / 1.9 (±0.5) Log 13.5 (±2.9) Log / 5.1 (±0.7) Log
IMU ID 96.6 (±1.7) Config
Log
Run / 96.5 (±0.7) Config
Log
Run		 97.4 (±1.5) Config
Log
Run / 95.7 (±0.7) Config
Log
Run		 98.3 (±1.2) Config
Log
Run / 95.7 (±0.7) Config
Log
Run		 94.0 (±2.2) Config
Log
Run / 35.8 (±1.8) Config
Log
Run
CU 73.5 (±3.0) Log / 43.9 (±1.4) Log 74.4 (±3.0) Log / 34.6 (±1.3) Log 82.9 (±2.6) Log / 40.8 (±1.3) Log 66.4 (±3.3) Log / 13.8 (±0.9) Log
CS 62.4 (±4.1) Log / 43.1 (±1.7) Log 62.4 (±4.1) Log / 31.5 (±1.6) Log 47.5 (±4.2) Log / 34.4 (±1.6) Log 54.6 (±4.2) Log / 12.4 (±1.1) Log

🧍 Human Pose Estimation Results 🧍

Multi-Modal Pose Estimation Performance

Modality Protocol ResNet DenseNet Swin-T RFNet
RGB ID 133.3 (±4.4) Config
Log
Run		 147.2 (±5.1) Config
Log
Run		 269.6 (±6.2) Config
Log
Run		 162.8 (±4.6) Config
Log
Run
CU 199.8 (±4.2) Log 204.8 (±4.8) Log 286.2 (±5.7) Log1 Log2 Log3 Log4 223.7 (±4.5) Log
CS 473.9 (±5.0) Log 524.6 (±4.3) Log 273.0 (±7.0) Log 331.8 (±6.5) Log
Depth ID 131.4 (±4.5) Config
Log
Run		 147.4 (±4.6) Config
Log
Run		 248.2 (±6.5) Config
Log
Run		 194.8 (±5.7) Config
Log
Run
CU 197.1 (±4.6) Log 212.5 (±4.6) Log 256.2 (±5.8) Log 230.7 (±5.3) Log
CS 363.6 (±5.6) Log 436.4 (±5.3) Log 305.1 (±7.0) Log 444.2 (±5.8) Log
ToF ID 152.5 (±5.2) Config
Log
Run		 N/A 252.1 (±6.0) Config
Log
Run		 162.2 (±5.0) Config
Log
Run
CU 205.7 (±5.0) Log N/A 257.3 (±5.7) Log 193.7 (±4.8) Log
CS 361.2 (±5.4) Log N/A 303.9 (±7.0) Log 363.7 (±4.8) Log
Thermal ID 142.8 (±4.7) Config
Log
Run		 147.0 (±4.9) Config
Log
Run		 259.9 (±5.9) Config
Log
Run		 254.3 (±6.1) Config
Log
Run
CU 216.9 (±4.4) Log 222.4 (±4.7) Log 259.9 (±5.8) Log 308.2 (±5.7) Log
CS 308.8 (±5.9) Log 325.4 (±5.3) Log 313.3 (±6.8) Log 403.4 (±7.0) Log
IRA ID 244.4 (±6.8) Config
Log
Run		 N/A 261.1 (±6.4) Config
Log
Run		 265.1 (±6.7) Config
Log
Run
CU 373.3 (±5.8) Log N/A 261.1 (±5.9) Log 299.1 (±5.8) Log
CS 398.8 (±7.2) Log N/A 313.0 (±6.8) Log 313.4 (±7.3) Log
FMCW ID 198.5 (±5.7) Config
Log
Run		 185.4 (±5.4) Config
Log
Run		 272.5 (±7.3) Config
Log
Run		 220.9 (±6.0) Config
Log
Run
CU 244.0 (±4.9) Log 236.8 (±4.7) Log 263.0 (±6.0) Log 272.0 (±5.1) Log
CS 369.4 (±10.6) Log 389.8 (±9.8) Log 338.8 (±10.1) Log 328.3 (±10.2) Log
SFCW ID 206.7 (±6.2) Config
Log
Run		 202.9 (±6.2) Config
Log
Run		 264.2 (±6.4) Config
Log
Run		 270.7 (±6.6) Config
Log
Run
CU 314.6 (±5.4) Log 334.4 (±5.4) Log 259.1 (±5.9) Log 408.1 (±23.4) Log
CS 352.2 (±7.0) Log 408.9 (±7.0) Log 339.7 (±8.9) Log 392.7 (±10.3) Log
UWB ID 142.4 (±4.8) Config
Log
Run		 158.0 (±5.2) Config
Log
Run		 260.5 (±6.1) Config
Log
Run		 159.5 (±5.0) Config
Log
Run
CU 241.2 (±4.8) Log 239.0 (±4.7) Log 261.3 (±5.8) Log 241.6 (±4.6) Log
CS 310.0 (±6.5) Log 327.6 (±6.6) Log 312.5 (±6.8) Log 295.8 (±6.8) Log
Wi-Fi ID 147.3 (±4.7) Config
Log
Run		 147.4 (±4.9) Config
Log
Run		 262.2 (±6.0) Config
Log
Run		 186.8 (±5.3) Config
Log
Run
CU 270.4 (±5.6) Log 267.8 (±5.8) Log 256.3 (±5.8) Log 274.2 (±5.6) Log
CS 399.4 (±5.7) Log 322.1 (±6.8) Log 312.8 (±6.8) Log 400.9 (±8.3) Log
Acoustic ID 258.8 (±6.9) Config
Log
Run		 256.8 (±6.7) Config
Log
Run		 271.2 (±6.7) Config
Log
Run		 243.6 (±6.8) Config
Log
Run
CU 304.1 (±5.8) Log 312.8 (±5.8) Log 260.6 (±5.8) Log 291.8 (±5.6) Log
CS 367.2 (±6.8) Log 441.4 (±6.9) Log 312.0 (±6.8) Log 323.3 (±7.2) Log
IMU ID 147.9 (±5.0) Config
Log
Run		 159.3 (±5.5) Config
Log
Run		 251.6 (±6.4) Config
Log
Run		 180.9 (±5.3) Config
Log
Run
CU 252.9 (±4.9) Log 274.3 (±5.0) Log 259.9 (±5.9) Log 266.3 (±5.0) Log
CS 289.7 (±6.8) Log 324.0 (±6.7) Log 310.8 (±6.9) Log 328.4 (±6.9) Log

📄 License & Citation 📄

📜 License

⚖️ Open Source License:

This project is licensed under the GPL-3.0 License. See the LICENSE file for complete details.

📚 Citation

🔬 Academic Recognition:

If you find this work useful in your research, please cite our paper:

📝 Citation: Coming soon...

🎉 Thank you for using OctoNet! 🎉

Empowering multi-modal human activity understanding through comprehensive sensor fusion

About

A Large-Scale Multi-Modal Dataset for Human Activity Understanding Grounded in Motion-Captured 3D Pose Labels

aiot-lab.github.io/OctoNet/

Resources

Readme
GPL-3.0 license
Activity
Custom properties

Stars

28 stars

Watchers

2 watching

Forks

0 forks
Report repository

Releases

Packages

Used by

Contributors

Languages