262 citations · 1.9k across the 56 of their papers we have counts for
56 papers
Emu: Generative Pretraining in Multimodality
Quan Sun, Qiying Yu, Yufeng Cui +7
We present Emu, a Transformer-based multimodal foundation model, which can seamlessly generate images and texts in multimodal context. This omnivore model can take in any single-mo…
SVIT: Scaling up Visual Instruction Tuning
Bo Zhao, Boya Wu, Muyang He +1
Thanks to the emerging of foundation models, the large language and vision models are integrated to acquire the multimodal ability of visual captioning, question answering, etc. Al…
Pushing the Limits of 3D Shape Generation at Scale
Yu Wang, Xuelin Qian, Jingyang Huo +3
We present a significant breakthrough in 3D shape generation by scaling it to unprecedented dimensions. Through the adaptation of the Auto-Regressive model and the utilization of l…
Large-scale Dataset Pruning with Dynamic Uncertainty
Muyang He, Shuo Yang, Tiejun Huang +1
The state of the art of many learning tasks, e.g., image classification, is advanced by collecting larger datasets and then training larger models on them. As the outcome, the incr…
SegGPT: Segmenting Everything In Context
Xinlong Wang, Xiaosong Zhang, Yue Cao +3
We present SegGPT, a generalist model for segmenting everything in context. We unify various segmentation tasks into a generalist in-context learning framework that accommodates di…
EVA-02: A Visual Representation for Neon Genesis
Yuxin Fang, Quan Sun, Xinggang Wang +3
We launch EVA-02, a next-generation Transformer-based visual representation pre-trained to reconstruct strong and robust language-aligned vision features via masked image modeling.…
Reinforcement Learning Friendly Vision-Language Model for Minecraft
Haobin Jiang, Junpeng Yue, Hao Luo +2
One of the essential missions in the AI research community is to build an autonomous embodied agent that can achieve high-level performance across a wide spectrum of tasks. However…
Hard-aware Instance Adaptive Self-training for Unsupervised Cross-domain Semantic Segmentation
Chuang Zhu, Kebin Liu, Wenqi Tang +3
The divergence between labeled training data and unlabeled testing data is a significant challenge for recent deep learning models. Unsupervised domain adaptation (UDA) attempts to…
EVA: Exploring the Limits of Masked Visual Representation Learning at Scale
Yuxin Fang, Wen Wang, Binhui Xie +6
We launch EVA, a vision-centric foundation model to explore the limits of visual representation at scale using only publicly accessible data. EVA is a vanilla ViT pre-trained to re…
Images Speak in Images: A Generalist Painter for In-Context Visual Learning
Xinlong Wang, Wen Wang, Yue Cao +2
In-context learning, as a new paradigm in NLP, allows the model to rapidly adapt to various tasks with only a handful of prompts and examples. But in computer vision, the difficult…
Temporal Up-Sampling for Asynchronous Events
Xijie Xiang, Lin Zhu, Jianing Li +2
The event camera is a novel bio-inspired vision sensor. When the brightness change exceeds the preset threshold, the sensor generates events asynchronously. The number of valid eve…
Annotation Efficient Person Re-Identification with Diverse Cluster-Based Pair Selection
Lantian Xue, Yixiong Zou, Peixi Peng +2
Person Re-identification (Re-ID) has attracted great attention due to its promising real-world applications. However, in practice, it is always costly to annotate the training data…
1000x Faster Camera and Machine Vision with Ordinary Devices
Tiejun Huang, Yajing Zheng, Zhaofei Yu +13
In digital cameras, we find a major limitation: the image and video form inherited from a film camera obstructs it from capturing the rapidly changing photonic world. Here, we pres…
A Robust Visual Sampling Model Inspired by Receptive Field
Liwen Hu, Lei Ma, Dawei Weng +1
Spike camera mimicking the retina fovea can report per-pixel luminance intensity accumulation by firing spikes. As a bio-inspired vision sensor with high temporal resolution, it ha…
Event-based Video Reconstruction via Potential-assisted Spiking Neural Network
Lin Zhu, Xiao Wang, Yi Chang +3
Neuromorphic vision sensor is a new bio-inspired imaging paradigm that reports asynchronous, continuously per-pixel brightness changes called `events' with high temporal resolution…
Deep Reinforcement Learning with Spiking Q-learning
Ding Chen, Peixi Peng, Tiejun Huang +1
With the help of special neuromorphic hardware, spiking neural networks (SNNs) are expected to realize artificial intelligence (AI) with less energy consumption. It provides a prom…
Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modeling
Xumin Yu, Lulu Tang, Yongming Rao +3
We present Point-BERT, a new paradigm for learning Transformers to generalize the concept of BERT to 3D point cloud. Inspired by BERT, we devise a Masked Point Modeling (MPM) task…
Optical Flow Estimation for Spiking Camera
Liwen Hu, Rui Zhao, Ziluo Ding +4
As a bio-inspired sensor with high temporal resolution, the spiking camera has an enormous potential in real applications, especially for motion estimation in high-speed scenes. Ho…
Spatio-Temporal Recurrent Networks for Event-Based Optical Flow Estimation
Ziluo Ding, Rui Zhao, Jiyuan Zhang +4
Event camera has offered promising alternative for visual perception, especially in high speed and high dynamic range scenes. Recently, many deep learning methods have shown great…
Pruning of Deep Spiking Neural Networks through Gradient Rewiring
Yanqi Chen, Zhaofei Yu, Wei Fang +2
Spiking Neural Networks (SNNs) have been attached great importance due to their biological plausibility and high energy-efficiency on neuromorphic chips. As these chips are usually…
Optimal ANN-SNN Conversion for Fast and Accurate Inference in Deep Spiking Neural Networks
Jianhao Ding, Zhaofei Yu, Yonghong Tian +1
Spiking Neural Networks (SNNs), as bio-inspired energy-efficient neural networks, have attracted great attentions from researchers and industry. The most efficient way to train dee…
Spike Camera and Its Coding Methods
Siwei Dong, Tiejun Huang, Yonghong Tian
This paper introduces a spike camera with a distinct video capture scheme and proposes two methods of decoding the spike stream for texture reconstruction. The spike camera capture…
Deep Residual Learning in Spiking Neural Networks
Wei Fang, Zhaofei Yu, Yanqi Chen +3
Deep Spiking Neural Networks (SNNs) present optimization difficulties for gradient-based approaches due to discrete binary activation and complex spatial-temporal dynamics. Conside…
Learning Open Set Network with Discriminative Reciprocal Points
Guangyao Chen, Limeng Qiao, Yemin Shi +5
Open set recognition is an emerging research area that aims to simultaneously classify samples from predefined classes and identify the rest as 'unknown'. In this process, one of t…