9 papers
JIVEAdapter: A Multi-Task Additive Low-Rank Adapter via Joint and Individual Variation Explained (JIVE)
Sara Abdali, Pashmina Cameron
Parameter-efficient fine-tuning adapts pretrained models at a fraction of the cost of full fine-tuning, yet most low-rank adapters are single-task and represent each weight update…
AttSVD:Prompt-Adaptive Low-Rank KV Cache Compression via Attention-Guided SVD
Sara Abdali, Jongwoo Ko, Pashmina Cameron
The key-value (KV) cache of autoregressive transformers grows linearly with context length and dominates memory at long context. Most training-free remedies evict low-importance to…
Eliciting Weak-to-Strong Generalization with On-Policy Reverse Distillation
Youngrok Park, Sangmin Bae, Hojung Jung +6
Weak-to-strong generalization asks whether stronger models can learn from weaker supervisors and surpass them. This is particularly important for successive model generations and m…
Sliding-window beats linear attention
Alexia Jolicoeur-Martineau, Rhea Sanjay Sukthanker, Pashmina Cameron +1
Due to the nature of quadratic attention, Large Language Models (LLMs) consume a lot of memory and energy: every new token costs more than the previous one, and its keys and values…
Scaling Reasoning Efficiently via Relaxed On-Policy Distillation
Jongwoo Ko, Sara Abdali, Young Jin Kim +2
On-policy distillation (OPD) is a promising approach for transferring reasoning capabilities to capacity-constrained models, yet sampled-token OPD often suffers from entropy collap…
StableQAT: Stable Quantization-Aware Training at Ultra-Low Bitwidths
Tianyi Chen, Sihan Chen, Xiaoyi Qu +5
Quantization-aware training (QAT) is essential for deploying large models under strict memory and latency constraints, yet achieving stable and robust optimization at ultra-low bit…
CUA-Skill: Develop Skills for Computer Using Agent
Tianyi Chen, Yinheng Li, Michael Solodko +12
Computer-Using Agents (CUAs) aim to autonomously operate computer systems to complete real-world tasks. However, existing agentic systems remain difficult to scale and lag behind h…
AppSelectBench: Application-Level Tool Selection Benchmark
Tianyi Chen, Michael Solodko, Sen Wang +14
Computer Using Agents (CUAs) are increasingly equipped with external tools, enabling them to perform complex and realistic tasks. For CUAs to operate effectively, application selec…
Self-reflecting Large Language Models: A Hegelian Dialectical Approach
Sara Abdali, Michael Solodko, Can Goksen +4
In this paper, we introduce a self-reflection framework for Large Language Models (LLMs) grounded in the Hegelian Dialectic, a philosophical method in which an initial proposition…