2.3k citations · 3.5k across the 28 of their papers we have counts for
28 papers
What Makes for Good Tokenizers in Vision Transformer?
Shengju Qian, Yi Zhu, Wenbo Li +2
The architecture of transformers, which recently witness booming applications in vision tasks, has pivoted against the widespread convolutional paradigm. Relying on the tokenizatio…
MixGen: A New Multi-Modal Data Augmentation
Xiaoshuai Hao, Yi Zhu, Srikar Appalaraju +4
Data augmentation is a necessity to enhance data efficiency in deep learning. For vision-language pre-training, data is only augmented either for images or for text in previous wor…
BigDetection: A Large-scale Benchmark for Improved Object Detector Pre-training
Likun Cai, Zhi Zhang, Yi Zhu +3
Multiple datasets and open challenges for object detection have been introduced in recent years. To build more general and powerful object detection systems, in this paper, we cons…
Benchmarking Multimodal AutoML for Tabular Data with Text Fields
Xingjian Shi, Jonas Mueller, Nick Erickson +2
We consider the use of automated supervised learning systems for data tables that not only contain numeric/categorical columns, but one or more text fields as well. Here we assembl…
Blending Anti-Aliasing into Vision Transformer
Shengju Qian, Hao Shao, Yi Zhu +2
The transformer architectures, based on self-attention mechanism and convolution-free design, recently found superior performance and booming applications in computer vision. Howev…
Distiller: A Systematic Study of Model Distillation Methods in Natural Language Processing
Haoyu He, Xingjian Shi, Jonas Mueller +3
We aim to identify how different components in the KD pipeline affect the resulting performance and how much the optimal KD pipeline varies across different datasets/tasks, such as…
Progressive Coordinate Transforms for Monocular 3D Object Detection
Li Wang, Li Zhang, Yi Zhu +4
Recognizing and localizing objects in the 3D space is a crucial ability for an AI agent to perceive its surrounding environment. While significant progress has been achieved with e…
Video Contrastive Learning with Global Context
Haofei Kuang, Yi Zhu, Zhi Zhang +5
Contrastive learning has revolutionized self-supervised image representation learning field, and recently been adapted to video domain. One of the greatest advantages of contrastiv…
A Unified Efficient Pyramid Transformer for Semantic Segmentation
Fangrui Zhu, Yi Zhu, Li Zhang +3
Semantic segmentation is a challenging problem due to difficulties in modeling context in complex scenes and class confusions along boundaries. Most literature either focuses on co…
CrossNorm and SelfNorm for Generalization under Distribution Shifts
Zhiqiang Tang, Yunhe Gao, Yi Zhu +3
Traditional normalization techniques (e.g., Batch Normalization and Instance Normalization) generally and simplistically assume that training and test data follow the same distribu…
A Comprehensive Study of Deep Video Action Recognition
Yi Zhu, Xinyu Li, Chunhui Liu +7
Video action recognition is one of the representative tasks for video understanding. Over the last decade, we have witnessed great advancements in video action recognition thanks t…
FeatGraph: A Flexible and Efficient Backend for Graph Neural Network Systems
Yuwei Hu, Zihao Ye, Minjie Wang +6
Graph neural networks (GNNs) are gaining increasing popularity as a promising approach to machine learning on graphs. Unlike traditional graph workloads where each vertex/edge is a…
CSER: Communication-efficient SGD with Error Reset
Cong Xie, Shuai Zheng, Oluwasanmi Koyejo +3
The scalability of Distributed Stochastic Gradient Descent (SGD) is today limited by communication bottlenecks. We propose a novel SGD variant: Communication-efficient SGD with Err…
Accelerated Large Batch Optimization of BERT Pretraining in 54 minutes
Shuai Zheng, Haibin Lin, Sheng Zha +1
BERT has recently attracted a lot of attention in natural language understanding (NLU) and achieved state-of-the-art results in various NLU tasks. However, its success requires lar…
Nimble: Efficiently Compiling Dynamic Neural Networks for Model Inference
Haichen Shen, Jared Roesch, Zhi Chen +6
Modern deep neural networks increasingly make use of features such as dynamic control flow, data structures and dynamic tensor shapes. Existing deep learning systems focus on optim…
Improving Semantic Segmentation via Self-Training
Yi Zhu, Zhongyue Zhang, Chongruo Wu +6
Deep learning usually achieves the best results with complete supervision. In the case of semantic segmentation, this means that large amounts of pixelwise annotations are required…
ResNeSt: Split-Attention Networks
Hang Zhang, Chongruo Wu, Zhongyue Zhang +9
It is well known that featuremap attention and multi-path representation are important for visual recognition. In this paper, we present a modularized architecture, which applies t…
AutoGluon-Tabular: Robust and Accurate AutoML for Structured Data
Nick Erickson, Jonas Mueller, Alexander Shirkov +4
We introduce AutoGluon-Tabular, an open-source AutoML framework that requires only a single line of Python to train highly accurate machine learning models on an unprocessed tabula…
A Unified Optimization Approach for CNN Model Inference on Integrated GPUs
Leyuan Wang, Zhi Chen, Yizhi Liu +4
Modern deep learning applications urge to push the model inference taking place at the edge devices for multiple reasons such as achieving shorter latency, relieving the burden of…
Dynamic Mini-batch SGD for Elastic Distributed Training: Learning in the Limbo of Resources
Haibin Lin, Hang Zhang, Yifei Ma +4
With an increasing demand for training powers for deep learning algorithms and the rapid growth of computation resources in data centers, it is desirable to dynamically schedule di…
Language Models with Transformers
Chenguang Wang, Mu Li, Alexander J. Smola
The Transformer architecture is superior to RNN-based models in computational efficiency. Recently, GPT and BERT demonstrate the efficacy of Transformer models on various NLP tasks…
Bag of Freebies for Training Object Detection Neural Networks
Zhi Zhang, Tong He, Hang Zhang +3
Training heuristics greatly improve various image classification model accuracies~\cite{he2018bag}. Object detection models, however, have more complex neural network structures an…
Optimizing CNN Model Inference on CPUs
Yizhi Liu, Yao Wang, Ruofei Yu +3
The popularity of Convolutional Neural Network (CNN) models and the ubiquity of CPUs imply that better performance of CNN model inference on CPUs can deliver significant gain to a…
Revise Saturated Activation Functions
Bing Xu, Ruitong Huang, Mu Li
In this paper, we revise two commonly used saturated functions, the logistic sigmoid and the hyperbolic tangent (tanh). We point out that, besides the well-known non-zero centered…