1.9k citations · 10.4k across the 198 of their papers we have counts for
199 papers
The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits
Shuming Ma, Hongyu Wang, Lingxiao Ma +7
Recent research, such as BitNet, is paving the way for a new era of 1-bit Large Language Models (LLMs). In this work, we introduce a 1-bit LLM variant, namely BitNet b1.58, in whic…
Calibrating LLM-Based Evaluator
Yuxuan Liu, Tianchi Yang, Shaohan Huang +6
Recent advancements in large language models (LLMs) on language modeling and emergent capabilities make them a promising reference-free evaluator of natural language generation qua…
Large Language Model for Science: A Study on P vs. NP
Qingxiu Dong, Li Dong, Ke Xu +4
In this work, we use large language models (LLMs) to augment and accelerate research on the P versus NP problem, one of the most important open problems in theoretical computer sci…
KOSMOS-2.5: A Multimodal Literate Model
Tengchao Lv, Yupan Huang, Jingye Chen +13
The automatic reading of text-intensive images represents a significant advancement toward achieving Artificial General Intelligence (AGI). In this paper we present KOSMOS-2.5, a m…
PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Training
Dawei Zhu, Nan Yang, Liang Wang +4
Large Language Models (LLMs) are trained with a pre-defined context length, restricting their use in scenarios requiring long inputs. Previous efforts for adapting LLMs to a longer…
Adapting Large Language Models to Domains via Reading Comprehension
Daixuan Cheng, Shaohan Huang, Furu Wei
We explore how continued pre-training on domain-specific corpora influences large language models, revealing that training on the raw corpora endows the model with domain knowledge…
Retentive Network: A Successor to Transformer for Large Language Models
Yutao Sun, Li Dong, Shaohan Huang +5
In this work, we propose Retentive Network (RetNet) as a foundation architecture for large language models, simultaneously achieving training parallelism, low-cost inference, and g…
WavMark: Watermarking for Audio Generation
Guangyu Chen, Yu Wu, Shujie Liu +3
Recent breakthroughs in zero-shot voice synthesis have enabled imitating a speaker's voice using just a few seconds of recording while maintaining a high level of realism. Alongsid…
LongNet: Scaling Transformers to 1,000,000,000 Tokens
Jiayu Ding, Shuming Ma, Li Dong +5
Scaling sequence length has become a critical demand in the era of large language models. However, existing methods struggle with either computational complexity or model expressiv…
Kosmos-2: Grounding Multimodal Large Language Models to the World
Zhiliang Peng, Wenhui Wang, Li Dong +4
We introduce Kosmos-2, a Multimodal Large Language Model (MLLM), enabling new capabilities of perceiving object descriptions (e.g., bounding boxes) and grounding text to the visual…
Learning to Retrieve In-Context Examples for Large Language Models
Liang Wang, Nan Yang, Furu Wei
Large language models (LLMs) have demonstrated their ability to learn in-context, allowing them to perform various tasks based on a few input-output examples. However, the effectiv…
In-context Autoencoder for Context Compression in a Large Language Model
Tao Ge, Jing Hu, Lei Wang +3
We propose the In-context Autoencoder (ICAE), leveraging the power of a large language model (LLM) to compress a long context into short compact memory slots that can be directly c…
Unleashing the Emergent Cognitive Synergy in Large Language Models: A Task-Solving Agent through Multi-Persona Self-Collaboration
Zhenhailong Wang, Shaoguang Mao, Wenshan Wu +3
Human intelligence thrives on cognitive synergy, where collaboration among different minds yield superior outcomes compared to isolated individuals. In this work, we propose Solo P…
Augmenting Language Models with Long-Term Memory
Weizhi Wang, Li Dong, Hao Cheng +4
Existing large language models (LLMs) can only afford fix-sized inputs due to the input length limit, preventing them from utilizing rich long-context information from past inputs.…
On the Off-Target Problem of Zero-Shot Multilingual Neural Machine Translation
Liang Chen, Shuming Ma, Dongdong Zhang +2
While multilingual neural machine translation has achieved great success, it suffers from the off-target issue, where the translation is in the wrong language. This problem is more…
Learning to Rank in Generative Retrieval
Yongqi Li, Nan Yang, Liang Wang +2
Generative retrieval stands out as a promising new paradigm in text retrieval that aims to generate identifier strings of relevant passages as the retrieval target. This generative…
MiniLLM: On-Policy Distillation of Large Language Models
Yuxian Gu, Li Dong, Furu Wei +1
Knowledge Distillation (KD) is a promising technique for reducing the high computational demand of large language models (LLMs). However, previous KD methods are primarily applied…
Multiview Identifiers Enhanced Generative Retrieval
Yongqi Li, Nan Yang, Liang Wang +2
Instead of simply matching a query to pre-existing passages, generative retrieval generates identifier strings of passages as the retrieval target. At a cost, the identifier must b…
VioLA: Unified Codec Language Models for Speech Recognition, Synthesis, and Translation
Tianrui Wang, Long Zhou, Ziqiang Zhang +6
Recent research shows a big convergence in model architecture, training objectives, and inference methods across various tasks for different modalities. In this paper, we propose V…
One-stop Training of Multiple Capacity Models
Lan Jiang, Haoyang Huang, Dongdong Zhang +2
Training models with varying capacities can be advantageous for deploying them in different scenarios. While high-capacity models offer better performance, low-capacity models requ…
Dual-Alignment Pre-training for Cross-lingual Sentence Embedding
Ziheng Li, Shaohan Huang, Zihan Zhang +7
Recent studies have shown that dual encoder models trained with the sentence-level translation ranking task are effective methods for cross-lingual sentence embedding. However, our…
Pre-Training to Learn in Context
Yuxian Gu, Li Dong, Furu Wei +1
In-context learning, where pre-trained language models learn to perform tasks from task examples and instructions in their contexts, has attracted much attention in the NLP communi…
Pre-training Language Model as a Multi-perspective Course Learner
Beiduo Chen, Shaohan Huang, Zihan Zhang +6
ELECTRA, the generator-discriminator pre-training framework, has achieved impressive semantic construction capability among various downstream tasks. Despite the convincing perform…
TextDiffuser: Diffusion Models as Text Painters
Jingye Chen, Yupan Huang, Tengchao Lv +3
Diffusion models have gained increasing attention for their impressive generation abilities but currently struggle with rendering accurate and coherent text. To address this issue,…