11 citations · 32 across the 13 of their papers we have counts for
13 papers
Macroeconomic Forecasting and Machine Learning
Ta-Chung Chi, Ting-Han Fan, Raffaele M. Ghigliazza +3
We forecast the full conditional distribution of macroeconomic outcomes by systematically integrating three key principles: using high-dimensional data with appropriate regularizat…
Attention Alignment and Flexible Positional Embeddings Improve Transformer Length Extrapolation
Ta-Chung Chi, Ting-Han Fan, Alexander I. Rudnicky
An ideal length-extrapolatable Transformer language model can handle sequences longer than the training length without any fine-tuning. Such long-context utilization capability rel…
Advancing Regular Language Reasoning in Linear Recurrent Neural Networks
Ting-Han Fan, Ta-Chung Chi, Alexander I. Rudnicky
In recent studies, linear recurrent neural networks (LRNNs) have achieved Transformer-level performance in natural language and long-range modeling, while offering rapid parallel t…
Latent Positional Information is in the Self-Attention Variance of Transformer Language Models Without Positional Embeddings
Ta-Chung Chi, Ting-Han Fan, Li-Wei Chen +2
The use of positional embeddings in transformer language models is widely accepted. However, recent research has called into question the necessity of such embeddings. We further e…
Transformer Working Memory Enables Regular Language Reasoning and Natural Language Length Extrapolation
Ta-Chung Chi, Ting-Han Fan, Alexander I. Rudnicky +1
Unlike recurrent models, conventional wisdom has it that Transformers cannot perfectly model regular languages. Inspired by the notion of working memory, we propose a new Transform…
Dissecting Transformer Length Extrapolation via the Lens of Receptive Field Analysis
Ta-Chung Chi, Ting-Han Fan, Alexander I. Rudnicky +1
Length extrapolation permits training a transformer language model on short sequences that preserves perplexities when tested on substantially longer sequences. A relative position…
Training Discrete Deep Generative Models via Gapped Straight-Through Estimator
Ting-Han Fan, Ta-Chung Chi, Alexander I. Rudnicky +1
While deep generative models have succeeded in image processing, natural language processing, and reinforcement learning, training that involves discrete random variables remains c…
KERPLE: Kernelized Relative Positional Embedding for Length Extrapolation
Ta-Chung Chi, Ting-Han Fan, Peter J. Ramadge +1
Relative positional embeddings (RPE) have received considerable attention since RPEs effectively model the relative distance among tokens and enable length extrapolation. We propos…
Explaining Off-Policy Actor-Critic From A Bias-Variance Perspective
Ting-Han Fan, Peter J. Ramadge
Off-policy Actor-Critic algorithms have demonstrated phenomenal experimental performance but still require better explanations. To this end, we show its policy evaluation error on…
Soft Actor-Critic With Integer Actions
Ting-Han Fan, Yubo Wang
Reinforcement learning is well-studied under discrete actions. Integer actions setting is popular in the industry yet still challenging due to its high dimensionality. To this end,…
PowerGym: A Reinforcement Learning Environment for Volt-Var Control in Power Distribution Systems
Ting-Han Fan, Xian Yeow Lee, Yubo Wang
We introduce PowerGym, an open-source reinforcement learning environment for Volt-Var control in power distribution systems. Following OpenAI Gym APIs, PowerGym targets minimizing…
A Contraction Approach to Model-based Reinforcement Learning
Ting-Han Fan, Peter J. Ramadge
Despite its experimental success, Model-based Reinforcement Learning still lacks a complete theoretical understanding. To this end, we analyze the error in the cumulative reward us…
Model Imitation for Model-Based Reinforcement Learning
Yueh-Hua Wu, Ting-Han Fan, Peter J. Ramadge +1
Model-based reinforcement learning (MBRL) aims to learn a dynamic model to reduce the number of interactions with real-world environments. However, due to estimation error, rollout…