189 citations · 196 across the 22 of their papers we have counts for
18 papers · 1 filter
Elastic Attention: Test-time Adaptive Sparsity Ratios for Efficient Transformers
Zecheng Tang, Quantong Qiu, Yi Yang +6
The quadratic complexity of standard attention mechanisms poses a significant scalability bottleneck for large language models (LLMs) in long-context scenarios. While hybrid attent…
MemoryRewardBench: Benchmarking Reward Models for Long-Term Memory Management in Large Language Models
Zecheng Tang, Baibei Ji, Ruoxi Sun +7
Existing works increasingly adopt memory-centric mechanisms to process long contexts in a segment manner, and effective memory management is one of the key capabilities that enable…
LongRM: Revealing and Unlocking the Context Boundary of Reward Modeling
Zecheng Tang, Baibei Ji, Quantong Qiu +4
Reward model (RM) plays a pivotal role in aligning large language model (LLM) with human preferences. As real-world applications increasingly involve long history trajectories, e.g…
Revisiting Long-context Modeling from Context Denoising Perspective
Zecheng Tang, Baibei Ji, Juntao Li +3
Long-context models (LCMs) have demonstrated great potential in processing long sequences, facilitating many real-world applications. The success of LCMs can be attributed to their…
LOOM-Scope: a comprehensive and efficient LOng-cOntext Model evaluation framework
Zecheng Tang, Haitian Wang, Quantong Qiu +5
Long-context processing has become a fundamental capability for large language models~(LLMs). To assess model's long-context performance, numerous long-context evaluation benchmark…
Revealing and Mitigating Over-Attention in Knowledge Editing
Pinzheng Wang, Zecheng Tang, Keyan Zhou +3
Large Language Models have demonstrated superior performance across a wide range of tasks, but they still exhibit undesirable errors due to incorrect knowledge learned from the tra…
LOGO -- Long cOntext aliGnment via efficient preference Optimization
Zecheng Tang, Zechen Sun, Juntao Li +2
Long-context models(LCMs) have shown great potential in processing long input sequences(even more than 100M tokens) conveniently and effectively. With significant progress, recent…
Revealing and Mitigating the Local Pattern Shortcuts of Mamba
Wangjie You, Zecheng Tang, Juntao Li +2
Large language models (LLMs) have advanced significantly due to the attention mechanism, but their quadratic complexity and linear memory demands limit their performance on long-co…
L-CiteEval: Do Long-Context Models Truly Leverage Context for Responding?
Zecheng Tang, Keyan Zhou, Juntao Li +3
Long-context models (LCMs) have made remarkable strides in recent years, offering users great convenience for handling tasks that involve long context, such as document summarizati…
MemLong: Memory-Augmented Retrieval for Long Text Modeling
Weijie Liu, Zecheng Tang, Juntao Li +2
Recent advancements in Large Language Models (LLMs) have yielded remarkable success across diverse fields. However, handling long contexts remains a significant challenge for LLMs…
OpenBA-V2: Reaching 77.3% High Compression Ratio with Fast Multi-Stage Pruning
Dan Qiao, Yi Su, Pinzheng Wang +18
Large Language Models (LLMs) have played an important role in many fields due to their powerful capabilities.However, their massive number of parameters leads to high deployment re…
Rethinking Negative Instances for Generative Named Entity Recognition
Yuyang Ding, Juntao Li, Pinzheng Wang +3
Large Language Models (LLMs) have demonstrated impressive capabilities for generalizing in unseen tasks. In the Named Entity Recognition (NER) task, recent advancements have seen t…
Beyond Hard Samples: Robust and Effective Grammatical Error Correction with Cycle Self-Augmenting
Zecheng Tang, Kaifeng Qi, Juntao Li +1
Recent studies have revealed that grammatical error correction methods in the sequence-to-sequence paradigm are vulnerable to adversarial attack, and simply utilizing adversarial e…
OpenBA: An Open-sourced 15B Bilingual Asymmetric seq2seq Model Pre-trained from Scratch
Juntao Li, Zecheng Tang, Yuyang Ding +9
Large language models (LLMs) with billions of parameters have demonstrated outstanding performance on various natural language processing tasks. This report presents OpenBA, an ope…
CMD: a framework for Context-aware Model self-Detoxification
Zecheng Tang, Keyan Zhou, Juntao Li +5
Text detoxification aims to minimize the risk of language models producing toxic content. Existing detoxification methods of directly constraining the model output or further train…
Can Diffusion Model Achieve Better Performance in Text Generation? Bridging the Gap between Training and Inference!
Zecheng Tang, Pinzheng Wang, Keyan Zhou +3
Diffusion models have been successfully adapted to text generation tasks by mapping the discrete text into the continuous space. However, there exist nonnegligible gaps between tra…
Improving Temporal Generalization of Pre-trained Language Models with Lexical Semantic Change
Zhaochen Su, Zecheng Tang, Xinyan Guan +3
Recent research has revealed that neural language models at scale suffer from poor temporal generalization capability, i.e., the language model pre-trained on static data from past…
Chinese grammatical error correction based on knowledge distillation
Peng Xia, Yuechi Zhou, Ziyan Zhang +2
In view of the poor robustness of existing Chinese grammatical error correction models on attack test sets and large model parameters, this paper uses the method of knowledge disti…