8 citations · 20 across the 13 of their papers we have counts for
14 papers
Token-Scaled Logit Distillation for Ternary Weight Generative Language Models
Minsoo Kim, Sihwa Lee, Janghwan Lee +4
Generative Language Models (GLMs) have shown impressive performance in tasks such as text generation, understanding, and reasoning. However, the large model size poses challenges f…
Teacher Intervention: Improving Convergence of Quantization Aware Training for Ultra-Low Precision Transformers
Minsoo Kim, Kyuhong Shim, Seongmin Park +2
Pre-trained Transformer models such as BERT have shown great success in a wide range of applications, but at the cost of substantial increases in model complexity. Quantization-awa…
Sleep Model -- A Sequence Model for Predicting the Next Sleep Stage
Iksoo Choi, Wonyong Sung
As sleep disorders are becoming more prevalent there is an urgent need to classify sleep stages in a less disturbing way.In particular, sleep-stage classification using simple sens…
Exploring Attention Map Reuse for Efficient Transformer Neural Networks
Kyuhong Shim, Jungwook Choi, Wonyong Sung
Transformer-based deep neural networks have achieved great success in various sequence applications due to their powerful ability to model long-range dependency. The key module of…
Macro-block dropout for improved regularization in training end-to-end speech recognition models
Chanwoo Kim, Sathish Indurti, Jinhwan Park +1
This paper proposes a new regularization algorithm referred to as macro-block dropout. The overfitting issue has been a difficult problem in training large neural network models. T…
A Comparison of Transformer, Convolutional, and Recurrent Neural Networks on Phoneme Recognition
Kyuhong Shim, Wonyong Sung
Phoneme recognition is a very important part of speech recognition that requires the ability to extract phonetic features from multiple frames. In this paper, we compare and analyz…
Korean Tokenization for Beam Search Rescoring in Speech Recognition
Kyuhong Shim, Hyewon Bae, Wonyong Sung
The performance of automatic speech recognition (ASR) models can be greatly improved by proper beam-search decoding with external language model (LM). There has been an increasing…
Similarity and Content-based Phonetic Self Attention for Speech Recognition
Kyuhong Shim, Wonyong Sung
Transformer-based speech recognition models have achieved great success due to the self-attention (SA) mechanism that utilizes every frame in the feature extraction process. Especi…
Layer-wise Pruning of Transformer Attention Heads for Efficient Language Modeling
Kyuhong Shim, Iksoo Choi, Wonyong Sung +1
While Transformer-based models have shown impressive language modeling performance, the large computation cost is often prohibitive for practical use. Attention head pruning, which…
Stochastic Precision Ensemble: Self-Knowledge Distillation for Quantized Deep Neural Networks
Yoonho Boo, Sungho Shin, Jungwook Choi +1
The quantization of deep neural networks (QDNNs) has been actively studied for deployment in edge devices. Recent studies employ the knowledge distillation (KD) method to improve t…
S-SGD: Symmetrical Stochastic Gradient Descent with Weight Noise Injection for Reaching Flat Minima
Wonyong Sung, Iksoo Choi, Jinhwan Park +2
The stochastic gradient descent (SGD) method is most widely used for deep neural network (DNN) training. However, the method does not always converge to a flat minimum of the loss…
Quantized Neural Networks: Characterization and Holistic Optimization
Yoonho Boo, Sungho Shin, Wonyong Sung
Quantized deep neural networks (QDNNs) are necessary for low-power, high throughput, and embedded applications. Previous studies mostly focused on developing optimization methods f…
SQWA: Stochastic Quantized Weight Averaging for Improving the Generalization Capability of Low-Precision Deep Neural Networks
Sungho Shin, Yoonho Boo, Wonyong Sung
Designing a deep neural network (DNN) with good generalization capability is a complex process especially when the weights are severely quantized. Model averaging is a promising ap…
Generative Knowledge Transfer for Neural Language Models
Sungho Shin, Kyuyeon Hwang, Wonyong Sung
In this paper, we propose a generative knowledge transfer technique that trains an RNN based language model (student network) using text and output probabilities generated from a p…