5 citations · 27 across the 26 of their papers we have counts for
27 papers
Drop-Then-Recovery: How Redundant Are Vision-Language-Action Models?
Guoheng Sun, Kaixi Feng, Shwai He +8
Vision-Language-Action (VLA) models enable instruction-driven robotic manipulation, but they inherit oversized language backbones from pretrained VLMs whose capacity far exceeds wh…
MoEless: Efficient MoE LLM Serving with Serverless Experts
Hanfei Yu, Bei Ouyang, Shwai He +2
Large Language Models (LLMs) increasingly adopt Mixture-of-Experts (MoE) architectures to scale efficiently under stringent resource constraints. However, MoE's sparse activation c…
Demystifying When Pruning Works via Representation Hierarchies
Shwai He, Guoheng Sun, Haichao Zhang +2
Network pruning, which removes less important parameters or architectures, is often expected to improve efficiency while preserving performance. However, this expectation does not…
Making Large Language Models Efficient Dense Retrievers
Yibin Lei, Shwai He, Ang Li +1
Recent work has shown that directly fine-tuning large language models (LLMs) for dense retrieval yields strong performance, but their substantial parameter counts make them computa…
Understanding and Harnessing Sparsity in Unified Multimodal Models
Shwai He, Chaorui Deng, Ang Li +1
Large multimodal models have achieved remarkable progress in both understanding and generation. Recent efforts pursue unified multimodal models that integrate heterogeneous compone…
DualSparse-MoE: Coordinating Tensor/Neuron-Level Sparsity with Expert Partition and Reconstruction
Weilin Cai, Le Qin, Shwai He +3
Mixture of Experts (MoE) has become a mainstream architecture for building Large Language Models (LLMs) by reducing per-token computation while enabling model scaling. It can be vi…
CogniPair: From LLM Chatbots to Conscious AI Agents -- GNWT-Based Multi-Agent Digital Twins for Social Pairing -- Dating & Hiring Applications
Wanghao Ye, Sihan Chen, Yiting Wang +18
Current large language model (LLM) agents lack authentic human psychological processes necessary for genuine digital twins and social AI applications. To address this limitation, w…
CoIn: Counting the Invisible Reasoning Tokens in Commercial Opaque LLM APIs
Guoheng Sun, Ziyao Wang, Bowei Tian +7
As post-training techniques evolve, large language models (LLMs) are increasingly augmented with structured multi-step reasoning abilities, often optimized through reinforcement le…
SymRTLO: Enhancing RTL Code Optimization with LLMs and Neuron-Inspired Symbolic Reasoning
Yiting Wang, Wanghao Ye, Ping Guo +11
Optimizing Register Transfer Level (RTL) code is crucial for improving the power, performance, and area (PPA) of digital circuits in the early stages of synthesis. Manual rewriting…
Capacity-Aware Inference: Mitigating the Straggler Effect in Mixture of Experts
Shwai He, Weilin Cai, Jiayi Huang +1
The Mixture of Experts (MoE) is an effective architecture for scaling large language models by leveraging sparse expert activation to balance performance and efficiency. However, u…
Router-Tuning: A Simple and Effective Approach for Enabling Dynamic-Depth in Transformers
Shwai He, Tao Ge, Guoheng Sun +3
Traditional transformer models often allocate a fixed amount of computational resources to every input token, leading to inefficient and unnecessary computation. To address this, t…
What Matters in Transformers? Not All Attention is Needed
Shwai He, Guoheng Sun, Zheyu Shen +1
While scaling Transformer-based large language models (LLMs) has demonstrated promising performance across various tasks, it also introduces redundant architectures, posing efficie…
Loki: Low-rank Keys for Efficient Sparse Attention
Prajwal Singhania, Siddharth Singh, Shwai He +2
Inference on large language models (LLMs) can be expensive in terms of the compute and memory costs involved, especially when long sequence lengths are used. In particular, the sel…
Towards Efficient Mixture of Experts: A Holistic Study of Compression Techniques
Shwai He, Daize Dong, Liang Ding +1
Scaling large language models has driven remarkable advancements across various domains, yet the continual increase in model size presents significant challenges for real-world dep…
Rethinking Pruning for Vision-Language Models: Strategies for Effective Sparsity and Performance Restoration
Shwai He, Ang Li, Tianlong Chen
Vision-Language Models (VLMs) integrate information from multiple modalities and have shown remarkable success across various tasks. However, deploying large-scale VLMs in resource…
Reformatted Alignment
Run-Ze Fan, Xuefeng Li, Haoyang Zou +5
The quality of finetuning data is crucial for aligning large language models (LLMs) with human values. Current methods to improve data quality are either labor-intensive or prone t…
Selective Reflection-Tuning: Student-Selected Data Recycling for LLM Instruction-Tuning
Ming Li, Lichang Chen, Jiuhai Chen +3
Instruction tuning is critical to large language models (LLMs) for achieving better instruction following and task adaptation capabilities but its success heavily relies on the tra…
Superfiltering: Weak-to-Strong Data Filtering for Fast Instruction-Tuning
Ming Li, Yong Zhang, Shwai He +5
Instruction tuning is critical to improve LLMs but usually suffers from low-quality and redundant data. Data filtering for instruction tuning has proved important in improving both…
Merging Experts into One: Improving Computational Efficiency of Mixture of Experts
Shwai He, Run-Ze Fan, Liang Ding +3
Scaling the size of language models usually leads to remarkable advancements in NLP tasks. But it often comes with a price of growing computational cost. Although a sparse Mixture…
Reflection-Tuning: Data Recycling Improves LLM Instruction-Tuning
Ming Li, Lichang Chen, Jiuhai Chen +4
Recent advancements in Large Language Models (LLMs) have expanded the horizons of natural language understanding and generation. Notably, the output control and alignment with the…
Accurate Prediction of Antibody Function and Structure Using Bio-Inspired Antibody Language Model
Hongtai Jing, Zhengtao Gao, Sheng Xu +7
In recent decades, antibodies have emerged as indispensable therapeutics for combating diseases, particularly viral infections. However, their development has been hindered by limi…
MerA: Merging Pretrained Adapters For Few-Shot Learning
Shwai He, Run-Ze Fan, Liang Ding +3
Adapter tuning, which updates only a few parameters, has become a mainstream method for fine-tuning pretrained language models to downstream tasks. However, it often yields subpar…
SparseAdapter: An Easy Approach for Improving the Parameter-Efficiency of Adapters
Shwai He, Liang Ding, Daize Dong +2
Adapter Tuning, which freezes the pretrained language models (PLMs) and only fine-tunes a few extra modules, becomes an appealing efficient alternative to the full model fine-tunin…
PAD-Net: An Efficient Framework for Dynamic Networks
Shwai He, Liang Ding, Daize Dong +3
Dynamic networks, e.g., Dynamic Convolution (DY-Conv) and the Mixture of Experts (MoE), have been extensively explored as they can considerably improve the model's representation p…