7 citations · 9 across the 6 of their papers we have counts for
5 papers · 1 filter
The Ephemeral Cost of Switching Optimizers Across LLM Training Stages
Xinyu Zhao, Davis Wertheimer, Ashutosh Sharma +2
Recent work suggests that switching optimizers between pretraining and subsequent training stages incurs a performance cost, but systematic investigation of this phenomenon is lack…
A Self-Pruning Transformer: Extreme KV-Cache Compression with Universal Attention
Davis Wertheimer, Haochen Shen, Ahan Gupta +6
The large KV-cache size of modern LLMs creates a barrier to efficient deployment. Recent work has explored replacing attention layers' RoPE positional embeddings with alternative d…
Frayed RoPE and Long Inputs: A Geometric Perspective
Davis Wertheimer, Aozhong Zhang, Derrick Liu +2
Rotary Positional Embedding (RoPE) is a widely adopted technique for encoding position in language models, which, while effective, causes performance breakdown when input length ex…
FlashSinkhorn: IO-Aware Entropic Optimal Transport on GPU
Felix X. -F. Ye, Xingjie Li, An Yu +3
Entropic optimal transport (EOT) via Sinkhorn iterations is widely used in modern machine learning, yet GPU solvers remain inefficient at scale. Tensorized implementations suffer q…
SudokuSens: Enhancing Deep Learning Robustness for IoT Sensing Applications using a Generative Approach
Tianshi Wang, Jinyang Li, Ruijie Wang +7
This paper introduces SudokuSens, a generative framework for automated generation of training data in machine-learning-based Internet-of-Things (IoT) applications, such that the ge…