293 citations · 1k across the 72 of their papers we have counts for
5 papers · 1 filter
Relative Positional Encoding for Transformers with Linear Complexity
Antoine Liutkus, Ondřej Cífka, Shih-Lun Wu +3
Recent advances in Transformer models allow for unprecedented sequence lengths, due to linear space and time complexity. In the meantime, relative positional encoding (RPE) was pro…
On Output Activation Functions for Adversarial Losses: A Theoretical Analysis via Variational Divergence Minimization and An Empirical Study on MNIST Classification
Hao-Wen Dong, Yi-Hsuan Yang
Recent years have seen adversarial losses been applied to many fields. Their applications extend beyond the originally proposed generative modeling to conditional generative and di…
Training Generative Adversarial Networks with Binary Neurons by End-to-end Backpropagation
Hao-Wen Dong, Yi-Hsuan Yang
We propose the BinaryGAN, a novel generative adversarial network (GAN) that uses binary neurons at the output layer of the generator. We employ the sigmoid-adjusted straight-throug…
Backpropagation with N-D Vector-Valued Neurons Using Arbitrary Bilinear Products
Zhe-Cheng Fan, Tak-Shing T. Chan, Yi-Hsuan Yang +1
Vector-valued neural learning has emerged as a promising direction in deep learning recently. Traditionally, training data for neural networks (NNs) are formulated as a vector of s…
Convolutional Generative Adversarial Networks with Binary Neurons for Polyphonic Music Generation
Hao-Wen Dong, Yi-Hsuan Yang
It has been shown recently that deep convolutional generative adversarial networks (GANs) can learn to generate music in the form of piano-rolls, which represent music by binary-va…