LittleBird: Efficient Faster & Longer Transformer for Question Answering

Minchul Lee, Kijong Han, Myeong Cheol Shin

BERT has shown a lot of sucess in a wide variety of NLP tasks. But it has a limitation dealing with long inputs due to its attention mechanism. Longformer, ETC and BigBird addresse…