163 citations · 476 across the 23 of their papers we have counts for
23 papers
Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAs
Microsoft, :, Abdelrahman Abouelenin +73
We introduce Phi-4-Mini and Phi-4-Multimodal, compact yet highly capable language and multimodal models. Phi-4-Mini is a 3.8-billion-parameter language model trained on high-qualit…
Repository Structure-Aware Training Makes SLMs Better Issue Resolver
Zexiong Ma, Shengnan An, Zeqi Lin +2
Language models have been applied to various software development tasks, but the performance varies according to the scale of the models. Large Language Models (LLMs) outperform Sm…
STAND-Guard: A Small Task-Adaptive Content Moderation Model
Minjia Wang, Pingping Lin, Siqi Cai +5
Content moderation, the process of reviewing and monitoring the safety of generated content, is important for development of welcoming online platforms and responsible large langua…
GRIN: GRadient-INformed MoE
Liyuan Liu, Young Jin Kim, Shuohang Wang +14
Mixture-of-Experts (MoE) models scale more effectively than dense models due to sparse computation through expert routing, selectively activating only a small subset of expert modu…
Make Your LLM Fully Utilize the Context
Shengnan An, Zexiong Ma, Zeqi Lin +2
While many contemporary large language models (LLMs) can process lengthy input, they still struggle to fully utilize information within the long context, known as the lost-in-the-m…
Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
Marah Abdin, Jyoti Aneja, Hany Awadalla +126
We introduce phi-3-mini, a 3.8 billion parameter language model trained on 3.3 trillion tokens, whose overall performance, as measured by both academic benchmarks and internal test…
Compositional API Recommendation for Library-Oriented Code Generation
Zexiong Ma, Shengnan An, Bing Xie +1
Large language models (LLMs) have achieved exceptional performance in code generation. However, the performance remains unsatisfactory in generating library-oriented code, especial…
Learning From Mistakes Makes LLM Better Reasoner
Shengnan An, Zexiong Ma, Zeqi Lin +3
Large language models (LLMs) recently exhibited remarkable reasoning capabilities on solving math problems. To further improve their reasoning capabilities, this work explores whet…
How Do In-Context Examples Affect Compositional Generalization?
Shengnan An, Zeqi Lin, Qiang Fu +4
Compositional generalization--understanding unseen combinations of seen primitives--is an essential reasoning capability in human intelligence. The AI community mainly studies this…
Skill-Based Few-Shot Selection for In-Context Learning
Shengnan An, Bo Zhou, Zeqi Lin +5
In-context learning is the paradigm that adapts large language models to downstream tasks by providing a few examples. Few-shot selection -- selecting appropriate examples for each…
Does Deep Learning Learn to Abstract? A Systematic Probing Framework
Shengnan An, Zeqi Lin, Bei Chen +3
Abstraction is a desirable capability for deep learning models, which means to induce abstract concepts from concrete instances and flexibly apply them beyond the learning context.…
When Language Model Meets Private Library
Daoguang Zan, Bei Chen, Zeqi Lin +3
With the rapid development of pre-training techniques, a number of language models have been pre-trained on large-scale code corpora and perform well in code generation. In this pa…
CodeT: Code Generation with Generated Tests
Bei Chen, Fengji Zhang, Anh Nguyen +4
The task of generating code solutions for a given programming problem can benefit from the use of pre-trained language models such as Codex, which can produce multiple diverse samp…
CERT: Continual Pre-Training on Sketches for Library-Oriented Code Generation
Daoguang Zan, Bei Chen, Dejian Yang +6
Code generation is a longstanding challenge, aiming to generate a code snippet based on a natural language description. Usually, expensive text-code paired data is essential for tr…
Making Large Language Models Better Reasoners with Step-Aware Verifier
Yifei Li, Zeqi Lin, Shizhuo Zhang +4
Few-shot learning is a challenging task that requires language models to generalize from limited examples. Large language models like GPT-3 and PaLM have made impressive progress i…
Input-Tuning: Adapting Unfamiliar Inputs to Frozen Pretrained Models
Shengnan An, Yifei Li, Zeqi Lin +6
Recently the prompt-tuning paradigm has attracted significant attention. By only tuning continuous prompts with a frozen pre-trained language model (PLM), prompt-tuning takes a ste…
Reasoning Like Program Executors
Xinyu Pi, Qian Liu, Bei Chen +6
Reasoning over natural language is a long-standing goal for the research community. However, studies have shown that existing language models are inadequate in reasoning. To addres…
Learning Algebraic Recombination for Compositional Generalization
Chenyao Liu, Shengnan An, Zeqi Lin +6
Neural sequence models exhibit limited compositional generalization ability in semantic parsing tasks. Compositional generalization requires algebraic recombination, i.e., dynamica…
TAPEX: Table Pre-training via Learning a Neural SQL Executor
Qian Liu, Bei Chen, Jiaqi Guo +4
Recent progress in language model pre-training has achieved a great success via leveraging large-scale unstructured textual data. However, it is still a challenge to apply pre-trai…
Iterative Utterance Segmentation for Neural Semantic Parsing
Yinuo Guo, Zeqi Lin, Jian-Guang Lou +1
Neural semantic parsers usually fail to parse long and complex utterances into correct meaning representations, due to the lack of exploiting the principle of compositionality. To…
Revisiting Iterative Back-Translation from the Perspective of Compositional Generalization
Yinuo Guo, Hualei Zhu, Zeqi Lin +3
Human intelligence exhibits compositional generalization (i.e., the capacity to understand and produce unseen combinations of seen components), but current neural seq2seq models la…
Hierarchical Poset Decoding for Compositional Generalization in Language
Yinuo Guo, Zeqi Lin, Jian-Guang Lou +1
We formalize human language understanding as a structured prediction task where the output is a partially ordered set (poset). Current encoder-decoder architectures do not take the…
Compositional Generalization by Learning Analytical Expressions
Qian Liu, Shengnan An, Jian-Guang Lou +6
Compositional generalization is a basic and essential intellective capability of human beings, which allows us to recombine known parts readily. However, existing neural network ba…