5 citations · 10 across the 10 of their papers we have counts for
13 papers
Compatibility Matters: A Framework for Token-Level Teacher Selection in On-Policy Distillation
Stanley Wei, Jiin Woo, Kaan Ozkara +1
On-policy distillation (OPD) has recently been introduced as a promising paradigm for continual learning, where a student policy can be supervised at the token-level. Typically, OP…
How to Train Your Little Dragon: Teaching a Small Language Model to Construct and Refine GPU Optimization Scripts
Annabelle Sujun Tang, Wei Tang, Songyao Jiang +7
Compiler transformations interact, making GPU kernel optimization a search for combinations that preserve the computation while improving performance. We introduce HOW TO TRAIN YOU…
Optimizing Large Language Models with Chained LMOs
Sungyoon Kim, Kaan Ozkara, Youngsuk Park
Muon has motivated a growing family of optimizers that compose multiple matrix normalizations, but these methods remain fragmented and lack a unified perspective. We introduce chai…
Directional Alignment Mitigates Reward Hacking in Reinforcement Learning for Language Models
Wenlong Deng, Jiaji Huang, Kaan Ozkara +4
Reward hacking arises when a model improves a proxy reward by exploiting shortcuts rather than solving the intended task. We study this failure mode through the geometry of reinfor…
MuonBP: Faster Muon via Block-Periodic Orthogonalization
Ahmed Khaled, Kaan Ozkara, Tao Yu +2
Gradient orthogonalization is a simple strategy that shows great utility in speeding up gradient descent. The Muon optimizer (Jordan, Jin, et al., 2024) combines gradient orthogona…
MEL: Multi-level Ensemble Learning for Resource-Constrained Environments
Krishna Praneet Gudipaty, Walid A. Hanafy, Kaan Ozkara +4
AI inference at the edge is becoming increasingly common for low-latency services. However, edge environments are power- and resource-constrained, and susceptible to failures. Conv…
SPIRE: Conditional Personalization for Federated Diffusion Generative Models
Kaan Ozkara, Ruida Zhou, Suhas Diggavi
Recent advances in diffusion models have revolutionized generative AI, but their sheer size makes on device personalization, and thus effective federated learning (FL), infeasible.…
Stochastic Rounding for LLM Training: Theory and Practice
Kaan Ozkara, Tao Yu, Youngsuk Park
As the parameters of Large Language Models (LLMs) have scaled to hundreds of billions, the demand for efficient training methods -- balancing faster computation and reduced memory…
ADEPT: Hierarchical Bayes Approach to Personalized Federated Unsupervised Learning
Kaan Ozkara, Bruce Huang, Ruida Zhou +1
Statistical heterogeneity of clients' local data is an important characteristic in federated learning, motivating personalized algorithms tailored to the local data statistics. Tho…
MADA: Meta-Adaptive Optimizers through hyper-gradient Descent
Kaan Ozkara, Can Karakus, Parameswaran Raman +4
Following the introduction of Adam, several novel adaptive optimizers for deep learning have been proposed. These optimizers typically excel in some tasks but may not outperform Ad…
A Generative Framework for Personalized Learning and Estimation: Theory, Algorithms, and Privacy
Kaan Ozkara, Antonious M. Girgis, Deepesh Data +1
A distinguishing characteristic of federated learning is that the (local) client data could have statistical heterogeneity. This heterogeneity has motivated the design of personali…
QuPeD: Quantized Personalization via Distillation with Applications to Federated Learning
Kaan Ozkara, Navjot Singh, Deepesh Data +1
Traditionally, federated learning (FL) aims to train a single global model while collaboratively using multiple clients and a server. Two natural challenges that FL algorithms face…
QuPeL: Quantized Personalization with Applications to Federated Learning
Kaan Ozkara, Navjot Singh, Deepesh Data +1
Traditionally, federated learning (FL) aims to train a single global model while collaboratively using multiple clients and a server. Two natural challenges that FL algorithms face…