Dual-stream Network for Visual Recognition

Abstract

Transformers with remarkable global representation capacities achieve competitive results for visual tasks, but fail to consider high-level local pattern information in input images. In this paper, we present a generic Dual-stream Network (DS-Net) to fully explore the representation capacity of local and global pattern features for image classification. Our DS-Net can simultaneously calculate fine-grained and integrated features and efficiently fuse them. Specifically, we propose an Intra-scale Propagation module to process two different resolutions in each block and an Inter-Scale Alignment module to perform information interaction across features at dual scales. Besides, we also design a Dual-stream FPN (DS-FPN) to further enhance contextual information for downstream dense predictions. Without bells and whistles, the proposed DS-Net outperforms DeiT-Small by 2.4% in terms of top-1 accuracy on ImageNet-1k and achieves state-of-the-art performance over other Vision Transformers and ResNets. For object detection and instance segmentation, DS-Net-Small respectively outperforms ResNet-50 by 6.4% and 5.5% in terms of mAP on MSCOCO 2017, and surpasses the previous state-of-the-art scheme, which significantly demonstrates its potential to be a general backbone in vision tasks. The code will be released soon.

arXiv commentAccepted by NeurIPS 2021

References in corpus (16)

• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Language Models are Few-Shot Learners
• Transformer in Transformer
• Twins: Revisiting the Design of Spatial Attention in Vision Transformers
• Conditional Positional Encodings for Vision Transformers
• DeepViT: Towards Deeper Vision Transformer
• FreeAnchor: Learning to Match Anchors for Visual Object Detection
• LXMERT: Learning Cross-Modality Encoder Representations from Transformers
• CvT: Introducing Convolutions to Vision Transformers
• ResT: An Efficient Transformer for Visual Recognition
• CLIP-Adapter: Better Vision-Language Models with Feature Adapters
• ConTNet: Why not use convolution and transformer at the same time?
• Container: Context Aggregation Network
• ProTo: Program-Guided Transformer for Program-Guided Tasks
• iffDetector: Inference-aware Feature Filtering for Object Detection

Cited by in corpus (7)

• TCCT: Tightly-Coupled Convolutional Transformer on Time Series Forecasting
• Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling
• CLIP-Adapter: Better Vision-Language Models with Feature Adapters
• DSPoint: Dual-scale Point Cloud Recognition with High-frequency Fusion
• TANet: A new Paradigm for Global Face Super-resolution via Transformer-CNN Aggregation Network
• Learning Visual Abstract Reasoning through Dual-Stream Networks
• Boon: A Neural Search Engine for Cross-Modal Information Retrieval