Feature Pyramid Transformer

Abstract

Feature interactions across space and scales underpin modern visual recognition systems because they introduce beneficial visual contexts. Conventionally, spatial contexts are passively hidden in the CNN's increasing receptive fields or actively encoded by non-local convolution. Yet, the non-local spatial interactions are not across scales, and thus they fail to capture the non-local contexts of objects (or parts) residing in different scales. To this end, we propose a fully active feature interaction across both space and scales, called Feature Pyramid Transformer (FPT). It transforms any feature pyramid into another feature pyramid of the same size but with richer contexts, by using three specially designed transformers in self-level, top-down, and bottom-up interaction fashion. FPT serves as a generic visual backbone with fair computational overhead. We conduct extensive experiments in both instance-level (i.e., object detection and instance segmentation) and pixel-level segmentation tasks, using various backbones and head networks, and observe consistent improvement over all the baselines and the state-of-the-art methods.

Published at the European Conference on Computer Vision, 2020

References in corpus (5)

• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Rethinking Atrous Convolution for Semantic Image Segmentation
• High-Resolution Representations for Labeling Pixels and Regions
• OCNet: Object Context Network for Scene Parsing
• DropBlock: A regularization method for convolutional networks

Cited by in corpus (4)

• TSA-Net: Tube Self-Attention Network for Action Quality Assessment
• MOTR: End-to-End Multiple-Object Tracking with Transformer
• PointFlow: Flowing Semantics Through Points for Aerial Image Segmentation
• Multi-Scale Feature Aggregation by Cross-Scale Pixel-to-Region Relation Operation for Semantic Segmentation