Feed-Forward Networks with Attention Can Solve Some Long-Term Memory Problems

Colin Raffel, Daniel P. W. Ellis

We propose a simplified model of attention which is applicable to feed-forward neural networks and demonstrate that the resulting model can solve the synthetic "addition" and "mult…