Multi-task Learning Challenge of The ABAW 4th Competition

Multi-task Learning Challenge of The ABAW 4th Competition

Introduction

4th Workshop and Competition on Affective Behavior Analysis in-the-wild (ABAW4) was held in conjunction witht the European Conference on Computer Vision (ECCV), 2022. There are 2 challenges in the competion: the multi-task learning challenge and learning from synthetic data challange.

I (CNU_Sclab Team) paticipated in the Multi-task Learning Challenge and achieved top 4 in the leaderboard. The problem is designing a machine learning model to jointly handle 3 affective behavior tasks: emotion recognition, valance-arousal estimation, and action unit detection. The detailed solution is described in my paper Multi-task Cross Attention Network in Facial Behavior Analysis. The paper is in proceeding of ECCV Workshop 2022.

I borrowed the SAM optimizer source code for Pytorch and refered to the ANFL of this paper

In addition, I used pre-train EfficientNet B0 on Facial Behavior Tasks of Savchenko in this project

Dependency

To use EfficientNet of Savchenko, we must use the exact version of timm:

pip install timm==0.4.5

Model Architecture

Pretrained Model

The pretrained model can be downloaded from this link.

s-Aff-Wild2 extracted feature: no augmented and augmented

This model was trained on the training set of the Aff-wild2 dataset, and evaluated on the validation set of the Aff-wild2 dataset.

The validation metrics are listed below:

PAU	PEX	PVA	PMTL
0.43	0.33	0.49	1.25