7 citations · 48 across the 69 of their papers we have counts for
16 papers · 1 filter
Fractional Power-of-Two Quantization for Efficient and Effective Multiplier-Free LLM Inference
Sunghyun Wee, Geunjae Choi, Hyeonjin Kim +2
Large Language Models (LLMs) demand efficient inference, where the multiply-accumulate (MAC) operations in linear projections dominate compute and energy cost. Post-training quanti…
MoARa: Module-Aware Rank Allocation and Structure-Preserving Decomposition for Low-Rank LLM Pre-training
Keunyoung Kim, Nojun Kwak
Low-rank gradient projection reduces the optimizer-state memory cost of large language model (LLM) pretraining, but the steps and wall-clock time needed to reach a target quality r…
Zero-Shot Quantization for Object Detectors using Off-the-Shelf Generative Models
Hyunho Lee, Kyomin Hwang, Hyeonjin Kim +3
With an increasing number of Object Detection (OD) models being deployed on edge devices, Zero-Shot Quantization for OD (ZSQ-OD) aims to quantize these models when access to the or…
Toward Structural Multimodal Representations: Specialization, Selection, and Sparsification via Mixture-of-Experts
Hahyeon Choi, Nojun Kwak
We propose S3 (Specialization, Selection, Sparsification), a framework that rethinks multimodal learning through a structural perspective. Instead of encoding all signals into a fi…
Deep Edge Filter: Return of the Human-Crafted Layer in Deep Learning
Dongkwan Lee, Junhoo Lee, Nojun Kwak
We introduce the Deep Edge Filter, a novel approach that applies high-pass filtering to deep neural network features to improve model generalizability. Our method is motivated by o…
The Role of Teacher Calibration in Knowledge Distillation
Suyoung Kim, Seonguk Park, Junhoo Lee +1
Knowledge Distillation (KD) has emerged as an effective model compression technique in deep learning, enabling the transfer of knowledge from a large teacher model to a compact stu…
Understanding Differential Transformer Unchains Pretrained Self-Attentions
Chaerin Kong, Jiho Jang, Nojun Kwak
Differential Transformer has recently gained significant attention for its impressive empirical performance, often attributed to its ability to perform noise canceled attention. Ho…
Practical Dataset Distillation Based on Deep Support Vectors
Hyunho Lee, Junhoo Lee, Nojun Kwak
Conventional dataset distillation requires significant computational resources and assumes access to the entire dataset, an assumption impractical as it presumes all data resides o…
Mitigating the Bias in the Model for Continual Test-Time Adaptation
Inseop Chung, Kyomin Hwang, Jayeon Yoo +1
Continual Test-Time Adaptation (CTA) is a challenging task that aims to adapt a source pre-trained model to continually changing target domains. In the CTA setting, a model does no…
Deep Support Vectors
Junhoo Lee, Hyunho Lee, Kyomin Hwang +1
Deep learning has achieved tremendous success. However, unlike SVMs, which provide direct decision criteria and can be trained with a small dataset, it still has significant weakne…
Any-Way Meta Learning
Junhoo Lee, Yearim Kim, Hyunho Lee +1
Although meta-learning seems promising performance in the realm of rapid adaptability, it is constrained by fixed cardinality. When faced with tasks of varying cardinalities that w…
SHOT: Suppressing the Hessian along the Optimization Trajectory for Gradient-Based Meta-Learning
JunHoo Lee, Jayeon Yoo, Nojun Kwak
In this paper, we hypothesize that gradient-based meta-learning (GBML) implicitly suppresses the Hessian along the optimization trajectory in the inner loop. Based on this hypothes…
Unifying Vision-Language Representation Space with Single-tower Transformer
Jiho Jang, Chaerin Kong, Donghyeon Jeon +2
Contrastive learning is a form of distance learning that aims to learn invariant features from two related representations. In this paper, we explore the bold hypothesis that an im…
Backdoor Attacks in Federated Learning by Rare Embeddings and Gradient Ensembling
KiYoon Yoo, Nojun Kwak
Recent advances in federated learning have demonstrated its promising capability to learn on decentralized datasets. However, a considerable amount of work has raised concerns due…
Dynamic Collective Intelligence Learning: Finding Efficient Sparse Model via Refined Gradients for Pruned Weights
Jangho Kim, Jayeon Yoo, Yeji Song +2
With the growth of deep neural networks (DNN), the number of DNN parameters has drastically increased. This makes DNN models hard to be deployed on resource-limited embedded system…
StackNet: Stacking Parameters for Continual learning
Jangho Kim, Jeesoo Kim, Nojun Kwak
Training a neural network for a classification task typically assumes that the data to train are given from the beginning. However, in the real world, additional data accumulate gr…