TACME

Task-Aware Cross-Modal Experts for Disjoint-Corpus
Multimodal Behavioral Understanding

One unified model for emotion, apparent personality, and ambivalence recognition across independently annotated multimodal corpora.

Elena Ryumina · Alexandr Axyonov · Dmitry Ryumin
_{HSE University · St. Petersburg Federal Research Center of the Russian Academy of Sciences (SPC RAS)}

Findings of the Association for Computational Linguistics: EMNLP

Overview · Results · Installation · Data · Citation

Abstract

Multimodal behavioral analysis is typically studied using separate task-specific corpora. Practical applications, however, would benefit from unified models that can support several behavioral predictions despite fragmented supervision. We study this problem as disjoint-corpus multimodal multi-task learning across emotion recognition on CMU-MOSEI, apparent Big Five personality assessment on ChaLearn FIv2, and ambivalence recognition on BAH. We propose Task-Aware Cross-Modal Experts (TACME), a multimodal fusion model with a shared bank of directed cross-modal Transformer experts and task-specific expert gates. Each expert models one ordered modality interaction, while task-specific gates select different expert mixtures for different behavioral tasks. We further adapt feature- and prediction-level flow-matching regularization to this disjoint-corpus behavioral setting and propose a Progress-Aware Gradient Balancing (PAGB) multi-task learning strategy for heterogeneous task optimization. Across three independent corpora and six task-specific metrics, TACME with PAGB and feature- and prediction-level flow-matching regularization achieves the best average rank of 1.00 among the compared multimodal fusion models. Although single-task TACME is stronger on the source-corpus metrics, the multi-task TACME model improves zero-shot unweighted average recall (UAR) on the unseen MELD test set from 28.9% to 35.9%, demonstrating improved generalization to unseen data.

Method Overview

TACME maps three independently annotated corpora to a unified four-stream modality interface:

• T — Transcript: verbal content.
• D — Description: a VLLM-generated textual description of visible behavior.
• V — Visual signal: visual behavior and scene information.
• A — Acoustic signal: acoustic and prosodic behavior.

Each variable-length stream is summarized by its mean and standard deviation over the sequence dimension, producing two fixed statistical tokens. With four modalities, a shared bank of 12 directed cross-modal Transformer experts models all ordered modality pairs without self-pairs. Task-specific gates then learn separate expert mixtures for emotion, apparent personality, and ambivalence recognition.

Because the corpora are disjoint, a label-availability mask activates only the task loss with available ground-truth labels for each sample. The model uses cross-entropy losses for emotion and ambivalence recognition and mean absolute error for personality assessment. PAGB updates task weights from relative validation progress, recent validation degradation, and EMA-smoothed gradient norms.

The proposed system combines five components:

• two-token Mean + STD statistical aggregation;
• directed cross-modal experts;
• task-aware expert weighting;
• feature- and prediction-level flow-matching (FM) regularization;
• Progress-Aware Gradient Balancing (PAGB).

TACME_v1 implements equal expert weighting, while TACME_v2 implements task-aware expert weighting.

Main Results

The main controlled comparisons use random seed 42. Checkpoints are selected by average rank over the six validation metrics; lower average rank indicates a better cross-task trade-off. The reported multi-task configuration uses TACME_v2, Mean + STD aggregation, feature- and prediction-level FM, and PAGB.

Corpus	Metric 1	Metric 2
CMU-MOSEI	mMF1: 62.68	mWAcc: 63.29
FIv2	mACC: 91.71	CCC: 69.89
BAH	MF1: 71.15	UAR: 72.07

This configuration achieves an average rank of 1.00 among the compared multimodal fusion models. In the separate comparison of multi-task learning strategies, PAGB obtains the best average rank of 2.33, followed by CAGrad with 2.67.

Task-aware expert weights

Emotion and personality primarily select visual-query experts, while ambivalence places more weight on acoustic- and transcript-query experts. The gates therefore learn task-dependent directed interactions instead of one shared fusion pattern.

Single-task and multi-task trade-off

The following TACME results are reported as mean ± standard deviation over five runs with different random seeds:

Setting	CMU-MOSEI mMF1	CMU-MOSEI mWAcc	FIv2 mACC	FIv2 CCC	BAH MF1	BAH UAR
Single-task TACME	67.82 ± 0.37	68.17 ± 0.42	92.59 ± 0.18	77.47 ± 1.17	78.05 ± 0.62	78.63 ± 0.78
Multi-task TACME	62.54 ± 0.26	63.32 ± 0.24	91.76 ± 0.05	70.14 ± 0.78	70.29 ± 0.90	71.71 ± 0.53
Δ multi-task − single-task	−5.28	−4.85	−0.83	−7.33	−7.76	−6.92

Under comparable corpus protocols, single-task TACME outperforms the task-specific systems considered in the paper on their reported metrics. It also outperforms multi-task TACME on all six source-corpus metrics. Single-task TACME is therefore preferable when performance on one source corpus is the primary objective.

In contrast, multi-task TACME reduces corpus specialization and transfers better to unseen data. In a zero-shot evaluation on the 2,610-utterance MELD test set, which is not used for training or model selection, it improves UAR from 28.9% to 35.9%:

Model	Neutral	Anger	Disgust	Fear	Happiness	Sadness	Surprise	UAR
Single-task TACME	33.1	6.4	16.2	0.0	91.3	24.5	31.0	28.9
Multi-task TACME	36.1	7.5	50.0	6.0	65.7	36.5	49.8	35.9

These results show a trade-off: single-task training gives stronger source-corpus performance, whereas multi-task training gives lower corpus specialization and better zero-shot generalization.

The paper additionally reports controlled comparisons of aggregation strategies and FM configurations, modality ablations, PAGB component ablations, corpus-identification diagnostics, task-dependent input attention, MTL loss convergence, PAGB training dynamics, task-output correlations, computational cost, a VLLM-description audit, and demographic subgroup diagnostics.

Repository Contents

• Paper-evaluated model variants: TACME_v1 and TACME_v2.
• Baseline fusion models: UniGraph2, DPDF-LQ, HME, and ComP.
• Multi-task learning strategies: Equal, Learned, NTKMTL, FairGrad, PCGrad, CAGrad, IMGrad, and PAGB.
• Feature- and prediction-level flow-matching regularization.
• Shared training pipeline for disjoint corpora with missing task labels.
• Separate VLLM-based behavior-description generation script.

The core runtime code is under src/. Analysis notebooks and plotting scripts are not required for model training.

Installation

python -m venv .venv
.\.venv\Scripts\activate
python -m pip install --upgrade pip
python -m pip install -r requirements.txt

The current source requires Python 3.10 or newer. The default requirements target the CUDA 12.4 PyTorch wheels. Adjust requirements.txt if your CUDA or PyTorch setup differs.

Data

This code expects locally downloaded datasets. Corpus files are not included in the repository.

Corpus	Task	Train	Dev	Test	Targets	Metrics
CMU-MOSEI	Emotion recognition	16,274	1,861	4,653	Neutral and six non-neutral emotions	mMF1, mWAcc
ChaLearn First Impressions v2	Apparent personality assessment	6,000	2,000	2,000	Five continuous Big Five scores	mACC, CCC
BAH	Ambivalence recognition	776	124	525	Binary ambivalence label	MF1, UAR

All three corpora are English-language and use the original train, development, and test splits. For CMU-MOSEI, the original six emotion-intensity annotations are converted to a seven-dimensional soft label over neutral and the six non-neutral emotions. The neutral coordinate is excluded from the reported mMF1 and mWAcc metrics. BAH is distributed under a proprietary license.

Configure each corpus separately in config.toml under [datasets.cmu_mosei], [datasets.fiv2], and [datasets.bah]. The expected directory structure is:

DATASET_ROOT/
  train_<configured_suffix>.csv
  dev_<configured_suffix>.csv
  test_<configured_suffix>.csv
  video/
    train/
    dev/
    test/
  audio/
    train/
    dev/
    test/

The default CMU-MOSEI and FIv2 paths use {split}_full_with_description.csv. The default BAH path uses {split}_full_with_description_with_qwen_instruct.csv. The CSV files must contain the task labels and text fields used by the configured modalities. The VLLM-generated description column is selected with:

[dataloader]
text_description_column = "text_llm"

Behavior Description Generation

Spoken transcripts used in the experiments were obtained with Whisper. The separate VLLM_description.py helper generates the behavioral-description stream before training; neither transcript generation nor behavioral-description generation is part of the main training pipeline.

The helper uses Qwen/Qwen3-VL-4B-Instruct and the exact neutral prompt reported in the paper. The prompt asks the model to describe only observable behavior and relevant scene context, and explicitly prohibits inference of emotions, personality traits, hesitation, confidence, intentions, mental states, or other hidden attributes. Generated descriptions are used only as an auxiliary textual modality, not as labels.

The generation settings used in the paper are:

model_name = "Qwen/Qwen3-VL-4B-Instruct"
video_fps = 8.0
max_frames = 16
max_new_tokens = 100
do_sample = False

Install the optional dependencies:

python -m pip install -U transformers accelerate "qwen-vl-utils[decord]"

Edit the Config block in VLLM_description.py and run:

python VLLM_description.py

The script writes a CSV with the following columns:

video_name,description

Merge or rename description into the dataset column selected by text_description_column before training.

The paper also audits the descriptions for direct target-label leakage, mental-state inference, and demographic or stereotypical language. No direct emotion, personality, or ambivalence labels are exposed in the inspected examples. The audit identifies one mild behavioral interpretation and two gendered person references, with no stereotypical claims observed. A systematic prompt study is outside the scope of the work.

Feature Extractors

The experimental pipeline uses the following pretrained extractors:

• Audio: EmoWav2Vec2.
• Transcript text: EmoRoBERTa.
• Behavioral descriptions: the same shared EmoRoBERTa.
• Visual signal: CLIP ViT-B/32.

The paper does not compare alternative pretrained feature extractors; it uses the same preprocessing, feature cache, data splits, and evaluation scripts for all fusion models and MTL strategies.

Main Configuration

The main experiment is controlled by config.toml.

Important fields:

• model_name: model architecture, e.g. TACME_v2.
• active_modalities: audio, text, scene, and behavior streams.
• active_tasks: emotion, personality, and ah (ambivalence).
• average_features: feature aggregation mode; mean_std_seq is used in the main TACME configuration.
• flow_matching_mode: off, features, heads, or heads+features. Here, features is feature-level FM and heads is prediction-level FM.
• flow_matching_weight: auxiliary FM loss weight.
• loss_weighting: equal or learned in the paper comparisons; the runtime also accepts fixed.
• mtl_method: off, ntkmtl, fairgrad, pcgrad, cagrad, imgrad, or pagb.
• pagb_*: PAGB hyperparameters and component-ablation flags.

The default config corresponds to the main multi-task TACME setting:

[train.model]
model_name = "TACME_v2"
hidden_dim = 128
num_transformer_heads = 8
tr_layers = 5
dropout = 0.15
flow_matching_mode = "heads+features"
flow_matching_steps = 4

[train.losses]
loss_weighting = "equal"
flow_matching_weight = 0.01
mtl_method = "pagb"
pagb_tau = 4.0
pagb_gamma = 0.5
pagb_min_weight = 0.5
pagb_max_weight = 2.0
pagb_ema = 0.5
pagb_grad_ema = 0.9
pagb_warmup_epochs = 1

[embeddings]
average_features = "mean_std_seq"

Running

Run the configured experiment:

python main.py

The pipeline builds feature caches under features/ and writes run outputs under results/.

To extract features without training:

[dataloader]
prepare_only = true

To run a single configuration:

[train.general]
search_type = "none"

To use search parameters from search_params.toml:

[train.general]
search_type = "exhaustive"

Ablations

Modality ablations are controlled through active_modalities:

[ablation]
active_modalities = ["audio", "text", "scene", "behavior"]

Task ablations, including single-task and pairwise dual-task training, are controlled through active_tasks:

[ablation]
active_tasks = ["emotion", "personality", "ah"]

Flow-matching ablations use:

[train.model]
flow_matching_mode = "off"  # "features", "heads", or "heads+features"

PAGB component ablations disable one component at a time:

[train.losses]
pagb_use_progress_lag = true
pagb_use_degradation = true
pagb_use_grad_correction = true
pagb_use_weight_smoothing = true
pagb_use_weight_clipping = true

For example, pagb_use_grad_correction = false gives the “w/o gradient correction” variant.

Reproducibility and Computational Cost

The main experiments use random seed 42. The reported single-task, pairwise dual-task, and full multi-task TACME configurations are evaluated over five random seeds. The paper reports batch size 32, up to 100 epochs, early-stopping patience 5, learning rate 1e-4, and weight decay 1e-5; the default repository configuration selects Adam from the evaluated optimizer candidates.

The computational-cost measurements in the paper use Python 3.8.20 and one NVIDIA Tesla V100-SXM2 GPU with 32 GB of memory; the current repository uses Python 3.10+ syntax. The task-aware TACME core has 3.88M parameters and occupies 15.04 MB. In the reported efficiency benchmark using raw aggregation and feature-level FM, the PAGB variant takes 12.9 minutes per epoch. These measurements exclude offline feature extraction and VLLM-based behavioral-description generation.

Scope and Limitations

Because each task is associated with a separate corpus, task identity remains coupled with corpus identity, and the model cannot be evaluated on all tasks for the same individuals. Mean + STD aggregation reduces computational cost but discards fine-grained temporal information. The directed expert bank grows as M(M−1), so larger modality sets may require sparse selection, parameter sharing, or pruning. Multi-task learning may also cause negative transfer when tasks share little behavioral structure.

TACME predictions of emotion, apparent personality, or ambivalence should not be treated as psychological diagnoses or used as the sole basis for high-stakes decisions. The paper reports diagnostic demographic subgroup results for FIv2 and BAH, but these results are not a definitive fairness assessment.

Citation

Preliminary citation metadata is provided below and in CITATION.cff. Proceedings pages, DOI, and other publication identifiers will be added after the official publication.

@inproceedings{ryumina2026tacme,
  title     = {{TACME}: Task-Aware Cross-Modal Experts for Disjoint-Corpus Multimodal Behavioral Understanding},
  author    = {Ryumina, Elena and Axyonov, Alexandr and Ryumin, Dmitry},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP},
  year      = {2026}
}

License

This repository is released under the MIT License. Dataset access and use remain subject to the terms of the respective corpus owners.