18 citations · 40 across the 34 of their papers we have counts for
34 papers
CoMeT: Collaborative Memory Transformer for Efficient Long Context Modeling
Runsong Zhao, Shilei Liu, Jiwei Tang +8
The quadratic complexity and indefinitely growing key-value (KV) cache of standard Transformers pose a major barrier to long-context processing. To overcome this, we introduce the…
Causal Autoregressive Diffusion Language Model
Junhao Ruan, Bei Li, Yongjing Yin +6
In this work, we propose Causal Autoregressive Diffusion (CARD), a novel framework that unifies the training efficiency of ARMs with the high-throughput inference of diffusion mode…
SpanNorm: Reconciling Training Stability and Performance in Deep Transformers
Chao Wang, Bei Li, Jiaqi Zhang +8
The success of Large Language Models (LLMs) hinges on the stable training of deep Transformer architectures. A critical design choice is the placement of normalization layers, lead…
NiuTrans.LMT: Toward Inclusive and Scalable Multilingual Machine Translation with LLMs
Yingfeng Luo, Ziqiang Xu, Yuxuan Ouyang +9
Large language models have significantly advanced Multilingual Machine Translation (MMT), yet scaling to many languages while keeping quality robust across directions remains chall…
LaTeXTrans: Structured LaTeX Translation with Multi-Agent Coordination
Ziming Zhu, Chenglong Wang, Haosong Xv +8
Despite the remarkable progress of modern machine translation (MT) systems on general-domain texts, translating structured LaTeX-formatted documents remains a significant challenge…
GRAM: A Generative Foundation Reward Model for Reward Generalization
Chenglong Wang, Yang Gan, Yifu Huo +8
In aligning large language models (LLMs), reward models have played an important role, but are standardly trained as discriminative models and rely only on labeled human preference…
Dissecting Long-Chain-of-Thought Reasoning Models: An Empirical Study
Yongyu Mu, Jiali Zeng, Bei Li +5
Despite recent progress in training long-chain-of-thought reasoning models via scaling reinforcement learning (RL), its underlying training dynamics remain poorly understood, and s…
Beyond Decoder-only: Large Language Models Can be Good Encoders for Machine Translation
Yingfeng Luo, Tong Zheng, Yongyu Mu +8
The field of neural machine translation (NMT) has changed with the advent of large language models (LLMs). Much of the recent emphasis in natural language processing (NLP) has been…
Optimizing Speech Multi-View Feature Fusion through Conditional Computation
Weiqiao Shan, Yuhao Zhang, Yuchen Han +7
Recent advancements have highlighted the efficacy of self-supervised learning (SSL) features in various speech-related tasks, providing lightweight and versatile multi-view speech…
Boosting Text-To-Image Generation via Multilingual Prompting in Large Multimodal Models
Yongyu Mu, Hengyu Li, Junxin Wang +7
Previous work on augmenting large multimodal models (LMMs) for text-to-image (T2I) generation has focused on enriching the input space of in-context learning (ICL). This includes p…
SLAM: Towards Efficient Multilingual Reasoning via Selective Language Alignment
Yuchun Fan, Yongyu Mu, Yilin Wang +7
Despite the significant improvements achieved by large language models (LLMs) in English reasoning tasks, these models continue to struggle with multilingual reasoning. Recent stud…
Foundations of Large Language Models
Tong Xiao, Jingbo Zhu
This is a book about large language models. As indicated by the title, it primarily focuses on foundational concepts rather than comprehensive coverage of all cutting-edge technolo…
Early Exit Is a Natural Capability in Transformer-based Models: An Empirical Study on Early Exit without Joint Optimization
Weiqiao Shan, Long Meng, Tong Zheng +5
Large language models (LLMs) exhibit exceptional performance across various downstream tasks. However, they encounter limitations due to slow inference speeds stemming from their e…
Predictor-Corrector Enhanced Transformers with Exponential Moving Average Coefficient Learning
Bei Li, Tong Zheng, Rui Wang +8
Residual networks, as discrete approximations of Ordinary Differential Equations (ODEs), have inspired significant advancements in neural network design, including multistep method…
Forgetting Curve: A Reliable Method for Evaluating Memorization Capability for Long-context Models
Xinyu Liu, Runsong Zhao, Pengcheng Huang +5
Numerous recent works target to extend effective context length for language models and various methods, tasks and benchmarks exist to measure model's effective memorization length…
LRHP: Learning Representations for Human Preferences via Preference Pairs
Chenglong Wang, Yang Gan, Yifu Huo +7
To improve human-preference alignment training, current research has developed numerous preference datasets consisting of preference pairs labeled as "preferred" or "dispreferred".…
SpMis: An Investigation of Synthetic Spoken Misinformation Detection
Peizhuo Liu, Li Wang, Renqiang He +6
In recent years, speech generation technology has advanced rapidly, fueled by generative models and large-scale training techniques. While these developments have enabled the produ…
A Modular-based Strategy for Mitigating Gradient Conflicts in Simultaneous Speech Translation
Xiaoqian Liu, Yangfan Du, Jianjin Wang +5
Simultaneous Speech Translation (SimulST) involves generating target language text while continuously processing streaming speech input, presenting significant real-time challenges…
Position IDs Matter: An Enhanced Position Layout for Efficient Context Compression in Large Language Models
Runsong Zhao, Xin Liu, Xinyu Liu +4
Using special tokens (e.g., gist, memory, or compressed tokens) to compress context information is a common practice for large language models (LLMs). However, existing approaches…
NDP: Next Distribution Prediction as a More Broad Target
Junhao Ruan, Abudukeyumu Abudula, Xinyu Liu +7
Large language models (LLMs) trained on next-token prediction (NTP) paradigm have demonstrated powerful capabilities. However, the existing NTP paradigm contains several limitation…
RoVRM: A Robust Visual Reward Model Optimized via Auxiliary Textual Preference Data
Chenglong Wang, Yang Gan, Yifu Huo +9
Large vision-language models (LVLMs) often fail to align with human preferences, leading to issues like generating misleading content without proper visual context (also known as h…
Cross-layer Attention Sharing for Pre-trained Large Language Models
Yongyu Mu, Yuzhang Wu, Yuchun Fan +9
To enhance the efficiency of the attention mechanism within large language models (LLMs), previous works primarily compress the KV cache or group attention heads, while largely ove…
Translate-and-Revise: Boosting Large Language Models for Constrained Translation
Pengcheng Huang, Yongyu Mu, Yuzhang Wu +4
Imposing constraints on machine translation systems presents a challenging issue because these systems are not trained to make use of constraints in generating adequate, fluent tra…
Revisiting Interpolation Augmentation for Speech-to-Text Generation
Chen Xu, Jie Wang, Xiaoqian Liu +6
Speech-to-text (S2T) generation systems frequently face challenges in low-resource scenarios, primarily due to the lack of extensive labeled datasets. One emerging solution is cons…