18 citations · 40 across the 34 of their papers we have counts for
29 papers · 1 filter
Causal Autoregressive Diffusion Language Model
Junhao Ruan, Bei Li, Yongjing Yin +6
In this work, we propose Causal Autoregressive Diffusion (CARD), a novel framework that unifies the training efficiency of ARMs with the high-throughput inference of diffusion mode…
SpanNorm: Reconciling Training Stability and Performance in Deep Transformers
Chao Wang, Bei Li, Jiaqi Zhang +8
The success of Large Language Models (LLMs) hinges on the stable training of deep Transformer architectures. A critical design choice is the placement of normalization layers, lead…
NiuTrans.LMT: Toward Inclusive and Scalable Multilingual Machine Translation with LLMs
Yingfeng Luo, Ziqiang Xu, Yuxuan Ouyang +9
Large language models have significantly advanced Multilingual Machine Translation (MMT), yet scaling to many languages while keeping quality robust across directions remains chall…
LaTeXTrans: Structured LaTeX Translation with Multi-Agent Coordination
Ziming Zhu, Chenglong Wang, Haosong Xv +8
Despite the remarkable progress of modern machine translation (MT) systems on general-domain texts, translating structured LaTeX-formatted documents remains a significant challenge…
GRAM: A Generative Foundation Reward Model for Reward Generalization
Chenglong Wang, Yang Gan, Yifu Huo +8
In aligning large language models (LLMs), reward models have played an important role, but are standardly trained as discriminative models and rely only on labeled human preference…
Beyond Decoder-only: Large Language Models Can be Good Encoders for Machine Translation
Yingfeng Luo, Tong Zheng, Yongyu Mu +8
The field of neural machine translation (NMT) has changed with the advent of large language models (LLMs). Much of the recent emphasis in natural language processing (NLP) has been…
Boosting Text-To-Image Generation via Multilingual Prompting in Large Multimodal Models
Yongyu Mu, Hengyu Li, Junxin Wang +7
Previous work on augmenting large multimodal models (LMMs) for text-to-image (T2I) generation has focused on enriching the input space of in-context learning (ICL). This includes p…
SLAM: Towards Efficient Multilingual Reasoning via Selective Language Alignment
Yuchun Fan, Yongyu Mu, Yilin Wang +7
Despite the significant improvements achieved by large language models (LLMs) in English reasoning tasks, these models continue to struggle with multilingual reasoning. Recent stud…
Foundations of Large Language Models
Tong Xiao, Jingbo Zhu
This is a book about large language models. As indicated by the title, it primarily focuses on foundational concepts rather than comprehensive coverage of all cutting-edge technolo…
Early Exit Is a Natural Capability in Transformer-based Models: An Empirical Study on Early Exit without Joint Optimization
Weiqiao Shan, Long Meng, Tong Zheng +5
Large language models (LLMs) exhibit exceptional performance across various downstream tasks. However, they encounter limitations due to slow inference speeds stemming from their e…
Predictor-Corrector Enhanced Transformers with Exponential Moving Average Coefficient Learning
Bei Li, Tong Zheng, Rui Wang +8
Residual networks, as discrete approximations of Ordinary Differential Equations (ODEs), have inspired significant advancements in neural network design, including multistep method…
Forgetting Curve: A Reliable Method for Evaluating Memorization Capability for Long-context Models
Xinyu Liu, Runsong Zhao, Pengcheng Huang +5
Numerous recent works target to extend effective context length for language models and various methods, tasks and benchmarks exist to measure model's effective memorization length…
LRHP: Learning Representations for Human Preferences via Preference Pairs
Chenglong Wang, Yang Gan, Yifu Huo +7
To improve human-preference alignment training, current research has developed numerous preference datasets consisting of preference pairs labeled as "preferred" or "dispreferred".…
SpMis: An Investigation of Synthetic Spoken Misinformation Detection
Peizhuo Liu, Li Wang, Renqiang He +6
In recent years, speech generation technology has advanced rapidly, fueled by generative models and large-scale training techniques. While these developments have enabled the produ…
A Modular-based Strategy for Mitigating Gradient Conflicts in Simultaneous Speech Translation
Xiaoqian Liu, Yangfan Du, Jianjin Wang +5
Simultaneous Speech Translation (SimulST) involves generating target language text while continuously processing streaming speech input, presenting significant real-time challenges…
Position IDs Matter: An Enhanced Position Layout for Efficient Context Compression in Large Language Models
Runsong Zhao, Xin Liu, Xinyu Liu +4
Using special tokens (e.g., gist, memory, or compressed tokens) to compress context information is a common practice for large language models (LLMs). However, existing approaches…
NDP: Next Distribution Prediction as a More Broad Target
Junhao Ruan, Abudukeyumu Abudula, Xinyu Liu +7
Large language models (LLMs) trained on next-token prediction (NTP) paradigm have demonstrated powerful capabilities. However, the existing NTP paradigm contains several limitation…
Cross-layer Attention Sharing for Pre-trained Large Language Models
Yongyu Mu, Yuzhang Wu, Yuchun Fan +9
To enhance the efficiency of the attention mechanism within large language models (LLMs), previous works primarily compress the KV cache or group attention heads, while largely ove…
Translate-and-Revise: Boosting Large Language Models for Constrained Translation
Pengcheng Huang, Yongyu Mu, Yuzhang Wu +4
Imposing constraints on machine translation systems presents a challenging issue because these systems are not trained to make use of constraints in generating adequate, fluent tra…
Revisiting Interpolation Augmentation for Speech-to-Text Generation
Chen Xu, Jie Wang, Xiaoqian Liu +6
Speech-to-text (S2T) generation systems frequently face challenges in low-resource scenarios, primarily due to the lack of extensive labeled datasets. One emerging solution is cons…
Hybrid Alignment Training for Large Language Models
Chenglong Wang, Hang Zhou, Kaiyan Chang +5
Alignment training is crucial for enabling large language models (LLMs) to cater to human intentions and preferences. It is typically performed based on two stages with different o…
Efficient Prompting Methods for Large Language Models: A Survey
Kaiyan Chang, Songcheng Xu, Chenglong Wang +4
Prompting is a mainstream paradigm for adapting large language models to specific natural language processing tasks without modifying internal parameters. Therefore, detailed suppl…
Prior Constraints-based Reward Model Training for Aligning Large Language Models
Hang Zhou, Chenglong Wang, Yimin Hu +3
Reinforcement learning with human feedback for aligning large language models (LLMs) trains a reward model typically using ranking loss with comparison pairs.However, the training…
Revealing the Parallel Multilingual Learning within Large Language Models
Yongyu Mu, Peinan Feng, Zhiquan Cao +8
In this study, we reveal an in-context learning (ICL) capability of multilingual large language models (LLMs): by translating the input to several languages, we provide Parallel In…