6 papers · 1 filter
When Is Rank-1 Steering Cheap? Geometry, Granularity, and Budgeted Search
John T. Robertson, Jianing Zhu, Haris Vikalo +1
Activation steering offers a lightweight way to control LLMs without retraining, but its effectiveness varies sharply across concepts. Prior work often reads this variability as ev…
Position: Weight Space Should Be a First-Class Generative AI Modality
Zhangyang Wang, Peihao Wang, Kai Wang
Neural network checkpoints have quietly become a large-scale data resource: millions of trained weight vectors now exist, each encoding task-, domain-, and architecture-specific kn…
ResearchGPT: Benchmarking and Training LLMs for End-to-End Computer Science Research Workflows
Penghao Wang, Yuhao Zhou, Mengxuan Wu +12
As large language models (LLMs) advance, the ultimate vision for their role in science is emerging: we could build an AI collaborator to effectively assist human beings throughout…
Data Efficient Any Transformer-to-Mamba Distillation via Attention Bridge
Penghao Wang, Yuhao Zhou, Mengxuan Wu +3
State-space models (SSMs) have emerged as promising alternatives to Transformers for sequence modeling. However, training competitive SSMs from scratch remains computationally inte…
Drag-and-Drop LLMs: Zero-Shot Prompt-to-Weights
Zhiyuan Liang, Dongwen Tang, Yuhao Zhou +11
Modern Parameter-Efficient Fine-Tuning (PEFT) methods such as low-rank adaptation (LoRA) reduce the cost of customizing large language models (LLMs), yet still require a separate o…
Make Optimization Once and for All with Fine-grained Guidance
Mingjia Shi, Ruihan Lin, Xuxi Chen +8
Learning to Optimize (L2O) enhances optimization efficiency with integrated neural networks. L2O paradigms achieve great outcomes, e.g., refitting optimizer, generating unseen solu…