163 citations · 304 across the 12 of their papers we have counts for
4 papers · 1 filter
Zero-th Order Algorithm for Softmax Attention Optimization
Yichuan Deng, Zhihang Li, Sridhar Mahadevan +1
Large language models (LLMs) have brought about significant transformations in human society. Among the crucial computations in LLMs, the softmax unit holds great importance. Its h…
Attention Scheme Inspired Softmax Regression
Yichuan Deng, Zhihang Li, Zhao Song
Large language models (LLMs) have made transformed changes for human society. One of the key computation in LLMs is the softmax unit. This operation is important in LLMs because it…
Solving Regularized Exp, Cosh and Sinh Regression Problems
Zhihang Li, Zhao Song, Tianyi Zhou
In modern machine learning, attention computation is a fundamental task for training large language models such as Transformer, GPT-4 and ChatGPT. In this work, we study exponentia…
IntroVAE: Introspective Variational Autoencoders for Photographic Image Synthesis
Huaibo Huang, Zhihang Li, Ran He +2
We present a novel introspective variational autoencoder (IntroVAE) model for synthesizing high-resolution photographic images. IntroVAE is capable of self-evaluating the quality o…