SegCodeNet: Color-Coded Segmentation Masks for Activity Detection from Wearable Cameras

Abstract

Activity detection from first-person videos (FPV) captured using a wearable camera is an active research field with potential applications in many sectors, including healthcare, law enforcement, and rehabilitation. State-of-the-art methods use optical flow-based hybrid techniques that rely on features derived from the motion of objects from consecutive frames. In this work, we developed a two-stream network, the \emph{SegCodeNet}, that uses a network branch containing video-streams with color-coded semantic segmentation masks of relevant objects in addition to the original RGB video-stream. We also include a stream-wise attention gating that prioritizes between the two streams and a frame-wise attention module that prioritizes the video frames that contain relevant features. Experiments are conducted on an FPV dataset containing $18$ activity classes in office environments. In comparison to a single-stream network, the proposed two-stream method achieves an absolute improvement of $14.366\%$ and $10.324\%$ for averaged F1 score and accuracy, respectively, when average results are compared for three different frame sizes $224\times 224$, $112\times 112$, and $64\times 64$. The proposed method provides significant performance gains for lower-resolution images with absolute improvements of $17\%$ and $26\%$ in F1 score for input dimensions of $112\times 112$ and $64\times 64$, respectively. The best performance is achieved for a frame size of $224\times 224$ yielding an F1 score and accuracy of $90.176\%$ and $90.799\%$ which outperforms the state-of-the-art Inflated 3D ConvNet (I3D) \cite{carreira2017quo} method by an absolute margin of $4.529\%$ and $2.419\%$, respectively.

References in corpus (9)

• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• The Kinetics Human Action Video Dataset
• Wide Residual Networks
• One weird trick for parallelizing convolutional neural networks
• OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields
• The THUMOS Challenge on Action Recognition for Videos "in the Wild"
• A Short Note on the Kinetics-700-2020 Human Action Dataset
• A Short Note about Kinetics-600
• Different Approaches for Human Activity Recognition: A Survey