NAC-TCN: Temporal Convolutional Networks with Causal Dilated Neighborhood Attention for Emotion Understanding

Abstract

In the task of emotion recognition from videos, a key improvement has been to focus on emotions over time rather than a single frame. There are many architectures to address this task such as GRUs, LSTMs, Self-Attention, Transformers, and Temporal Convolutional Networks (TCNs). However, these methods suffer from high memory usage, large amounts of operations, or poor gradients. We propose a method known as Neighborhood Attention with Convolutions TCN (NAC-TCN) which incorporates the benefits of attention and Temporal Convolutional Networks while ensuring that causal relationships are understood which results in a reduction in computation and memory cost. We accomplish this by introducing a causal version of Dilated Neighborhood Attention while incorporating it with convolutions. Our model achieves comparable, better, or state-of-the-art performance over TCNs, TCAN, LSTMs, and GRUs while requiring fewer parameters on standard emotion recognition datasets. We publish our code online for easy reproducibility and use in other projects.

arXiv comment8 pages, presented at ICVIP 2023

References in corpus (19)

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling
• On the difficulty of training Recurrent Neural Networks
• WaveNet: A Generative Model for Raw Audio
• Visualizing the Loss Landscape of Neural Nets
• Residual Networks Behave Like Ensembles of Relatively Shallow Networks
• Deep Affect Prediction in-the-wild: Aff-Wild Database and Challenge, Deep Architectures, and Beyond
• Artificial Emotional Intelligence in Socially Assistive Robots for Older Adults: A Pilot Study
• Face Behavior a la carte: Expressions, Affect and Action Units in a Single Network
• Aff-Wild2: Extending the Aff-Wild Database for Affect Recognition
• Distribution Matching for Heterogeneous Multi-Task Learning: a Large-scale Face Study
• Affect Analysis in-the-wild: Valence-Arousal, Expressions, Action Units and a Unified Framework
• Trellis Networks for Sequence Modeling
• Temporal Convolutional Attention-based Network For Sequence Modeling
• Dilated Neighborhood Attention Transformer
• Multi-modal Emotion Estimation for in-the-wild Videos
• ABAW: Valence-Arousal Estimation, Expression Recognition, Action Unit Detection & Multi-Task Learning Challenges
• Focus on Change: Mood Prediction by Learning Emotion Changes via Spatio-Temporal Attention