9 papers
Praxist: From Experimental Artifacts to Solution Lineages
Jin Li, Ahmed Murtadha, Zhiyu Wang +14
Autonomous R&D agents now write, run, and improve executable artifacts under automated evaluation-but largely as laboratory instruments: shown on curated benchmarks, with gains tha…
Hidden Decoding at Scale: Latent Computation Scaling for Large Language Models
Aiwei Liu, Cheng Shi, Chuhan Wu +44
Scaling Large Language Models (LLMs) has been driven mainly by enlarging the Transformer backbone, but for an already-strong model this requires another round of costly pretraining…
HRM-Text: Efficient Pretraining Beyond Scaling
Guan Wang, Changling Liu, Chenyu Wang +6
The current pretraining paradigm for large language models relies on massive compute and internet-scale raw text, creating a significant barrier to foundational research. In contra…
Sample Complexity of Transfer Learning: An Optimal Transport Approach
Haoyang Cao, Xin Guo, Wenpin Tang +1
Transfer learning is an essential technique for many machine learning/AI models of complex structures such as large language models and generative AI. The essence of transfer learn…
Why Attend to Everything? Focus is the Key
Hengshuai Yao, Xing Chen, Ahmed Murtadha +8
Standard attention scales quadratically with sequence length. Efficient attention methods reduce this O(n^2) cost, but when retrofitted into pretrained models, they often degrade p…
GAIN: Multiplicative Modulation for Domain Adaptation
Hengshuai Yao, Xing Chen, Ahmed Murtadha +1
Adapting LLMs to new domains causes forgetting because standard methods (e.g., full fine-tuning, LoRA) inject new directions into the weight space. We show that forgetting is gover…
Thin Keys, Full Values: Reducing KV Cache via Low-Dimensional Attention Selection
Hengshuai Yao, Xing Chen, Ahmed Murtadha +1
Standard Transformer attention uses identical dimensionality for queries, keys, and values, yet these components serve different roles: queries and keys produce scalar attention we…
Learning Optimal Prompt Ensemble for Multi-source Visual Prompt Transfer
Enming Zhang, Liwen Cao, Yanru Wu +2
Prompt tuning has emerged as a lightweight strategy for adapting foundation models to downstream tasks, particularly for resource-constrained systems. As pre-trained prompts become…
TMT: Cross-domain Semantic Segmentation with Region-adaptive Transferability Estimation
Enming Zhang, Zhengyu Li, Yanru Wu +5
Recent advances in Vision Transformers (ViTs) have significantly advanced semantic segmentation performance. However, their adaptation to new target domains remains challenged by d…