NAC-TCN: Temporal Convolutional Networks with Causal Dilated Neighborhood Attention for Emotion Understanding

Abstract

In the task of emotion recognition from videos, a key improvement has been to focus on emotions over time rather than a single frame. There are many architectures to address this task such as GRUs, LSTMs, Self-Attention, Transformers, and Temporal Convolutional Networks (TCNs). However, these methods suffer from high memory usage, large amounts of operations, or poor gradients. We propose a method known as Neighborhood Attention with Convolutions TCN (NAC-TCN) which incorporates the benefits of attention and Temporal Convolutional Networks while ensuring that causal relationships are understood which results in a reduction in computation and memory cost. We accomplish this by introducing a causal version of Dilated Neighborhood Attention while incorporating it with convolutions. Our model achieves comparable, better, or state-of-the-art performance over TCNs, TCAN, LSTMs, and GRUs while requiring fewer parameters on standard emotion recognition datasets. We publish our code online for easy reproducibility and use in other projects.

8 pages, presented at ICVIP 2023

References in corpus (8)

• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• On the difficulty of training Recurrent Neural Networks
• WaveNet: A Generative Model for Raw Audio
• Artificial Emotional Intelligence in Socially Assistive Robots for Older Adults: A Pilot Study
• Affect Analysis in-the-wild: Valence-Arousal, Expressions, Action Units and a Unified Framework
• Multi-modal Emotion Estimation for in-the-wild Videos
• ABAW: Valence-Arousal Estimation, Expression Recognition, Action Unit Detection & Multi-Task Learning Challenges
• Focus on Change: Mood Prediction by Learning Emotion Changes via Spatio-Temporal Attention