4 papers
Dynamic Relational Priming Improves Transformer in Multivariate Time Series
Hunjae Lee, Corey Clark
Standard attention mechanisms in transformers employ static token representations that remain unchanged across all pair-wise computations in each layer. This limits their represent…
On the Runway Cascade of Transformers for Language Modeling
Hunjae Lee, Corey Clark
In decoder-only (causal) transformers, the computation graph created by causal masking routes information through both direct-path attention and indirect paths formed by intermedia…
Understanding the Failure Modes of Transformers through the Lens of Graph Neural Networks
Hunjae Lee
Transformers and more specifically decoder-only transformers dominate modern LLM architectures. While they have shown to work exceptionally well, they are not without issues, resul…
Transformer Modeling for Both Scalability and Performance in Multivariate Time Series
Hunjae Lee, Corey Clark
Variable count is among the main scalability bottlenecks for transformer modeling in multivariate time series (MTS) data. On top of this, a growing consensus in the field points to…