1.1k citations · 3.9k across the 29 of their papers we have counts for
32 papers
Trace as State: Reasoning Traces as Conditional States for Long-Context Transformers
Xu Zou, Jie Tang
Transformers process information causally, but long-context reasoning may depend on task state discovered only later. We formalize this mismatch through conditional state update ta…
CogView2: Faster and Better Text-to-Image Generation via Hierarchical Transformers
Ming Ding, Wendi Zheng, Wenyi Hong +1
The development of the transformer-based text-to-image models are impeded by its slow generation and complexity for high-resolution images. In this work, we put forward a solution…
Are we really making much progress? Revisiting, benchmarking, and refining heterogeneous graph neural networks
Qingsong Lv, Ming Ding, Qiang Liu +7
Heterogeneous graph neural networks (HGNNs) have been blossoming in recent years, but the unique data processing and evaluation setups used by each work obstruct a full understandi…
Graph Robustness Benchmark: Benchmarking the Adversarial Robustness of Graph Machine Learning
Qinkai Zheng, Xu Zou, Yuxiao Dong +5
Adversarial attacks on graphs have posed a major threat to the robustness of graph machine learning (GML) models. Naturally, there is an ever-escalating arms race between attackers…
SketchNE: Embedding Billion-Scale Networks Accurately in One Hour
Yuyang Xie, Yuxiao Dong, Jiezhong Qiu +3
We study large-scale network embedding with the goal of generating high-quality embeddings for networks with more than 1 billion vertices and 100 billion edges. Recent attempts Lig…
P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks
Xiao Liu, Kaixuan Ji, Yicheng Fu +4
Prompt tuning, which only tunes continuous prompts with a frozen language model, substantially reduces per-task storage and memory usage at training. However, in the context of NLU…
Zero-Shot Information Extraction as a Unified Text-to-Triple Translation
Chenguang Wang, Xiao Liu, Zui Chen +3
We cast a suite of information extraction tasks into a text-to-triple translation framework. Instead of solving each task relying on task-specific datasets and models, we formalize…
Modeling Protein Using Large-scale Pretrain Language Model
Yijia Xiao, Jiezhong Qiu, Ziang Li +2
Protein is linked to almost every life process. Therefore, analyzing the biological structure and property of protein sequences is critical to the exploration of life, as well as d…
Multi-task curriculum learning in a complex, visual, hard-exploration domain: Minecraft
Ingmar Kanitscheider, Joost Huizinga, David Farhi +9
An important challenge in reinforcement learning is training agents that can solve a wide variety of tasks. If tasks depend on each other (e.g. needing to learn to walk before lear…
A Generalizable Approach to Learning Optimizers
Diogo Almeida, Clemens Winter, Jie Tang +1
A core issue with learning to optimize neural networks has been the lack of generalization to real world problems. To address this, we describe a system designed from a generalizat…
A Self-supervised Method for Entity Alignment
Xiao Liu, Haoyun Hong, Xinghao Wang +4
Entity alignment, aiming to identify equivalent entities across different knowledge graphs (KGs), is a fundamental problem for constructing large-scale KGs. Over the course of its…
Pre-Trained Models: Past, Present and Future
Xu Han, Zhengyan Zhang, Ning Ding +21
Large-scale pre-trained models (PTMs) such as BERT and GPT have recently achieved great success and become a milestone in the field of artificial intelligence (AI). Owing to sophis…
TDGIA:Effective Injection Attacks on Graph Neural Networks
Xu Zou, Qinkai Zheng, Yuxiao Dong +4
Graph Neural Networks (GNNs) have achieved promising performance in various real-world applications. However, recent studies have shown that GNNs are vulnerable to adversarial atta…
CogView: Mastering Text-to-Image Generation via Transformers
Ming Ding, Zhuoyi Yang, Wenyi Hong +8
Text-to-Image generation in the general domain has long been an open problem, which requires both a powerful generative model and cross-modal understanding. We propose CogView, a 4…
FastMoE: A Fast Mixture-of-Expert Training System
Jiaao He, Jiezhong Qiu, Aohan Zeng +3
Mixture-of-Expert (MoE) presents a strong potential in enlarging the size of language model to trillions of parameters. However, training trillion-scale MoE requires algorithm and…
Controllable Generation from Pre-trained Language Models via Inverse Prompting
Xu Zou, Da Yin, Qingyang Zhong +4
Large-scale pre-trained language models have demonstrated strong capabilities of generating realistic text. However, it remains challenging to control the generation results. Previ…
GPT Understands, Too
Xiao Liu, Yanan Zheng, Zhengxiao Du +4
Prompting a pretrained language model with natural language patterns has been proved effective for natural language understanding (NLU). However, our preliminary study reveals that…
GLM: General Language Model Pretraining with Autoregressive Blank Infilling
Zhengxiao Du, Yujie Qian, Xiao Liu +4
There have been various types of pretraining architectures including autoencoding models (e.g., BERT), autoregressive models (e.g., GPT), and encoder-decoder models (e.g., T5). How…
CODE: Contrastive Pre-training with Adversarial Fine-tuning for Zero-shot Expert Linking
Bo Chen, Jing Zhang, Xiaokang Zhang +6
Expert finding, a popular service provided by many online websites such as Expertise Finder, LinkedIn, and AMiner, is beneficial to seeking candidate qualifications, consultants, a…
CPM: A Large-scale Generative Chinese Pre-trained Language Model
Zhengyan Zhang, Xu Han, Hao Zhou +22
Pre-trained Language Models (PLMs) have proven to be beneficial for various downstream NLP tasks. Recently, GPT-3, with 175 billion parameters and 570GB training data, drew a lot o…
AIM 2020 Challenge on Efficient Super-Resolution: Methods and Results
Kai Zhang, Martin Danelljan, Yawei Li +75
This paper reviews the AIM 2020 challenge on efficient single image super-resolution with focus on the proposed solutions and results. The challenge task was to super-resolve an in…
A Matrix Chernoff Bound for Markov Chains and Its Application to Co-occurrence Matrices
Jiezhong Qiu, Chi Wang, Ben Liao +2
We prove a Chernoff-type bound for sums of matrix-valued random variables sampled via a regular (aperiodic and irreducible) finite Markov chain. Specially, consider a random walk o…
A framework for constructing a huge name disambiguation dataset: algorithms, visualization and human collaboration
Zhuoyue Xiao, Yutao Zhang, Bo Chen +2
We present a manually-labeled Author Name Disambiguation(AND) Dataset called WhoisWho, which consists of 399,255 documents and 45,187 distinct authors with 421 ambiguous author nam…
Self-supervised Learning: Generative or Contrastive
Xiao Liu, Fanjin Zhang, Zhenyu Hou +4
Deep supervised learning has achieved great success in the last decade. However, its deficiencies of dependence on manual labels and vulnerability to attacks have driven people to…