E-BATCH: Energy-Efficient and High-Throughput RNN Batching

Franyell Silfa, Jose Maria Arnau, Antonio Gonzalez

Recurrent Neural Network (RNN) inference exhibits low hardware utilization due to the strict data dependencies across time-steps. Batching multiple requests can increase throughput…