72 citations · 330 across the 48 of their papers we have counts for
25 papers · 1 filter
When Less Data Favors Smaller Teachers: Rethinking Teacher Capacity and Data Selection for Knowledge Distillation
Minjae Park, Taesun Yeom, Jaeho Lee
Data pruning reduces the training cost of knowledge distillation (KD). However, the preferred teacher capacity changes with the data budget: smaller teachers can outperform larger…
KV-COBRA: KV Cache Compression via Co-Optimized Bit-Rank Allocation
Sihyeon Ha, Jaeho Lee, Yo-Seb Jeon
What limits KV-cache compression at extreme bit-rates? We argue that it is not the choice of compression scheme, but how its budget is allocated across attention heads. Existing me…
Over-Alignment vs Over-Fitting: The Role of Feature Learning Strength in Generalization
Taesun Yeom, Taehyeok Ha, Jaeho Lee
Feature learning strength (FLS), i.e., the inverse of the effective output scaling of a model, plays a critical role in shaping the optimization dynamics of neural nets. While its…
Neural Weight Compression for Language Models
Jegwang Ryu, Minkyu Kim, Seungjun Shin +3
Efficient compression of language model weights is increasingly critical as model scale and deployment grow. Yet, most existing methods rely on handcrafted transforms and heuristic…
Activation Quantization of Vision Encoders Needs Prefixing Registers
Seunghyeon Kim, Taesun Yeom, Jinho Kim +3
Large pretrained vision encoders are central to multimodal intelligence, powering applications from on-device vision processing to vision-language models. Since these applications…
On the Internal Representations of Graph Metanetworks
Taesun Yeom, Jaeho Lee
Weight space learning is an emerging paradigm in the deep learning community. The primary goal of weight space learning is to extract informative features from a set of parameters…
Fast Training of Sinusoidal Neural Fields via Scaling Initialization
Taesun Yeom, Sangyoon Lee, Jaeho Lee
Neural fields are an emerging paradigm that represent data as continuous functions parameterized by neural networks. Despite many advantages, neural fields often have a high traini…
Prefixing Attention Sinks can Mitigate Activation Outliers for Large Language Model Quantization
Seungwoo Son, Wonpyo Park, Woohyun Han +2
Despite recent advances in LLM quantization, activation quantization remains to be challenging due to the activation outliers. Conventional remedies, e.g., mixing precisions for di…
Hybrid Neural Representations for Spherical Data
Hyomin Kim, Yunhui Jang, Jaeho Lee +1
In this paper, we study hybrid neural representations for spherical data, a domain of increasing relevance in scientific research. In particular, our work focuses on weather and cl…
In Search of a Data Transformation That Accelerates Neural Field Training
Junwon Seo, Sangyoon Lee, Kwang In Kim +1
Neural field is an emerging paradigm in data representation that trains a neural network to approximate the given signal. A key obstacle that prevents its widespread adoption is th…
Debiased Distillation by Transplanting the Last Layer
Jiwoon Lee, Jaeho Lee
Deep models are susceptible to learning spurious correlations, even during the post-processing. We take a closer look at the knowledge distillation -- a popular post-processing tec…
The Role of Masking for Efficient Supervised Knowledge Distillation of Vision Transformers
Seungwoo Son, Jegwang Ryu, Namhoon Lee +1
Knowledge distillation is an effective method for training lightweight vision models. However, acquiring teacher supervision for training samples is often costly, especially from l…
Learning Large-scale Neural Fields via Context Pruned Meta-Learning
Jihoon Tack, Subin Kim, Sihyun Yu +3
We introduce an efficient optimization-based meta-learning technique for large-scale neural field training by realizing significant memory savings through automated online context…
Discovering and Mitigating Visual Biases through Keyword Explanation
Younghyun Kim, Sangwoo Mo, Minkyu Kim +3
Addressing biases in computer vision models is crucial for real-world AI deployments. However, mitigating visual biases is challenging due to their unexplainable nature, often iden…
Meta-Learning with Self-Improving Momentum Target
Jihoon Tack, Jongjin Park, Hankook Lee +2
The idea of using a separately trained target model (or teacher) to improve the performance of the student model has been increasingly popular in various machine learning domains,…
Few-Shot Unlearning by Model Inversion
Youngsik Yoon, Jinhwan Nam, Hyojeong Yun +3
We consider a practical scenario of machine unlearning to erase a target dataset, which causes unexpected behavior from the trained model. The target dataset is often assumed to be…
Spread Spurious Attribute: Improving Worst-group Accuracy with Spurious Attribute Estimation
Junhyun Nam, Jaehyung Kim, Jaeho Lee +1
The paradigm of worst-group loss minimization has shown its promise in avoiding to learn spurious correlations, but requires costly additional supervision on spurious attributes. T…
CoL: Contrastive Continual Learning
Hyuntak Cha, Jaeho Lee, Jinwoo Shin
Recent breakthroughs in self-supervised learning show that such algorithms learn visual representations that can be transferred better to unseen tasks than joint-training methods r…
MASKER: Masked Keyword Regularization for Reliable Text Classification
Seung Jun Moon, Sangwoo Mo, Kimin Lee +2
Pre-trained language models have achieved state-of-the-art accuracies on various text classification tasks, e.g., sentiment analysis, natural language inference, and semantic textu…
Provable Memorization via Deep Neural Networks using Sub-linear Parameters
Sejun Park, Jaeho Lee, Chulhee Yun +1
It is known that parameters are sufficient for neural networks to memorize arbitrary input-label pairs. By exploiting depth, we show that parameters suffice…
Layer-adaptive sparsity for the Magnitude-based Pruning
Jaeho Lee, Sejun Park, Sangwoo Mo +2
Recent discoveries on neural network pruning reveal that, with a carefully chosen layerwise sparsity, a simple magnitude-based pruning achieves state-of-the-art tradeoff between sp…
Learning from Failure: Training Debiased Classifier from Biased Classifier
Junhyun Nam, Hyuntak Cha, Sungsoo Ahn +2
Neural networks often learn to make predictions that overly rely on spurious correlation existing in the dataset, which causes the model to be biased. While previous work tackles t…
Minimum Width for Universal Approximation
Sejun Park, Chulhee Yun, Jaeho Lee +1
The universal approximation property of width-bounded networks has been studied as a dual of classical universal approximation results on depth-bounded networks. However, the criti…
Lookahead: A Far-Sighted Alternative of Magnitude-based Pruning
Sejun Park, Jaeho Lee, Sangwoo Mo +1
Magnitude-based pruning is one of the simplest methods for pruning neural networks. Despite its simplicity, magnitude-based pruning and its variants demonstrated remarkable perform…