71 citations · 217 across the 26 of their papers we have counts for
27 papers
Hierarchical Sparse Attention Done Right: Toward Infinite Context Modeling
Xiang Hu, Xinyu Wei, Hao Gu +10
Scaling modern large language models (LLMs) to long contexts is limited by the quadratic computation cost, and poor length extrapolation of dense attention. Chunk-wise sparse atten…
RePo: Language Models with Context Re-Positioning
Huayang Li, Tianyu Zhao, Deng Cai +1
In-context learning is fundamental to modern Large Language Models (LLMs); however, prevailing architectures impose a rigid and fixed contextual structure by assigning linear or co…
SeqPE: Transformer with Sequential Position Encoding
Huayang Li, Yahui Liu, Hongyu Sun +5
Since self-attention layers in Transformers are permutation invariant by design, positional encodings must be explicitly incorporated to enable spatial understanding. However, fixe…
ALR: A Retrieve-then-Reason Framework for Long-context Question Answering
Huayang Li, Pat Verga, Priyanka Sen +5
The context window of large language models (LLMs) has been extended significantly in recent years. However, while the context length that the LLM can process has grown, the capabi…
On the Transformations across Reward Model, Parameter Update, and In-Context Prompt
Deng Cai, Huayang Li, Tingchen Fu +11
Despite the general capabilities of pre-trained large language models (LLMs), they still need further adaptation to better serve practical applications. In this paper, we demonstra…
M3T: A New Benchmark Dataset for Multi-Modal Document-Level Machine Translation
Benjamin Hsu, Xiaoyu Liu, Huayang Li +6
Document translation poses a challenge for Neural Machine Translation (NMT) systems. Most document-level NMT systems rely on meticulously curated sentence-level parallel data, assu…
Cross-lingual Contextualized Phrase Retrieval
Huayang Li, Deng Cai, Zhi Qu +4
Phrase-level dense retrieval has shown many appealing characteristics in downstream NLP tasks by leveraging the fine-grained information that phrases offer. In our work, we propose…
Inferflow: an Efficient and Highly Configurable Inference Engine for Large Language Models
Shuming Shi, Enbo Zhao, Deng Cai +3
We present Inferflow, an efficient and highly configurable inference engine for large language models (LLMs). With Inferflow, users can serve most of the common transformer models…
GPT4Video: A Unified Multimodal Large Language Model for lnstruction-Followed Understanding and Safety-Aware Generation
Zhanyu Wang, Longyue Wang, Zhen Zhao +7
While the recent advances in Multimodal Large Language Models (MLLMs) constitute a significant leap forward in the field, these models are predominantly confined to the realm of in…
Repetition In Repetition Out: Towards Understanding Neural Text Degeneration from the Data Perspective
Huayang Li, Tian Lan, Zihao Fu +5
There are a number of diverging hypotheses about the neural text degeneration problem, i.e., generating repetitive and dull loops, which makes this problem both interesting and con…
TextBind: Multi-turn Interleaved Multimodal Instruction-following in the Wild
Huayang Li, Siheng Li, Deng Cai +5
Large language models with instruction-following abilities have revolutionized the field of artificial intelligence. These models show exceptional generalizability to tackle variou…
PandaGPT: One Model To Instruction-Follow Them All
Yixuan Su, Tian Lan, Huayang Li +3
We present PandaGPT, an approach to emPower large lANguage moDels with visual and Auditory instruction-following capabilities. Our pilot experiments show that PandaGPT can perform…
A Frustratingly Simple Decoding Method for Neural Text Generation
Haoran Yang, Deng Cai, Huayang Li +3
We introduce a frustratingly simple, super efficient and surprisingly effective decoding method, which we call Frustratingly Simple Decoding (FSD), for neural text generation. The…
-gram Is Back: Residual Learning of Neural Text Generation with -gram Language Model
Huayang Li, Deng Cai, Jin Xu +1
-gram language models (LM) have been largely superseded by neural LMs as the latter exhibits better performance. However, we find that -gram models can achieve satisfactory p…
Learning to Break the Loop: Analyzing and Mitigating Repetitions for Neural Text Generation
Jin Xu, Xiaojiang Liu, Jianhao Yan +3
While large-scale neural language models, such as GPT2 and BART, have achieved impressive results on various text generation tasks, they tend to get stuck in undesirable sentence-l…
Investigating Data Variance in Evaluations of Automatic Machine Translation Metrics
Jiannan Xiang, Huayang Li, Yahui Liu +4
Current practices in metric evaluation focus on one single dataset, e.g., Newstest dataset in each year's WMT Metrics Shared Task. However, in this paper, we qualitatively and quan…
Visualizing the Relationship Between Encoded Linguistic Information and Task Performance
Jiannan Xiang, Huayang Li, Defu Lian +3
Probing is popular to analyze whether linguistic information can be captured by a well-trained deep neural model, but it is hard to answer how the change of the encoded linguistic…
A Survey on Retrieval-Augmented Text Generation
Huayang Li, Yixuan Su, Deng Cai +2
Recently, retrieval-augmented text generation attracted increasing attention of the computational linguistics community. Compared with conventional generation models, retrieval-aug…
Neural Machine Translation with Monolingual Translation Memory
Deng Cai, Yan Wang, Huayang Li +2
Prior work has proved that Translation memory (TM) can boost the performance of Neural Machine Translation (NMT). In contrast to existing work that uses bilingual corpus as TM and…
GWLAN: General Word-Level AutocompletioN for Computer-Aided Translation
Huayang Li, Lemao Liu, Guoping Huang +1
Computer-aided translation (CAT), the use of software to assist a human translator in the translation process, has been proven to be useful in enhancing the productivity of human t…
Data Augmentation for Text Generation Without Any Augmented Data
Wei Bi, Huayang Li, Jiacheng Huang
Data augmentation is an effective way to improve the performance of many neural text generation models. However, current data augmentation methods need to define or choose proper d…
TranSmart: A Practical Interactive Machine Translation System
Guoping Huang, Lemao Liu, Xing Wang +5
Automatic machine translation is super efficient to produce translations yet their quality is not guaranteed. This technique report introduces TranSmart, a practical human-machine…
Assessing Dialogue Systems with Distribution Distances
Jiannan Xiang, Yahui Liu, Deng Cai +3
An important aspect of developing dialogue systems is how to evaluate and compare the performance of different systems. Existing automatic evaluation metrics are based on turn-leve…
On the Branching Bias of Syntax Extracted from Pre-trained Language Models
Huayang Li, Lemao Liu, Guoping Huang +1
Many efforts have been devoted to extracting constituency trees from pre-trained language models, often proceeding in two stages: feature definition and parsing. However, this kind…