1.1k citations · 3.2k across the 12 of their papers we have counts for
12 papers · 1 filter
Characteristics of Harmful Text: Towards Rigorous Benchmarking of Language Models
Maribeth Rauh, John Mellor, Jonathan Uesato +9
Large language models produce human-like text that drive a growing number of applications. However, recent literature and, increasingly, real world observations, have demonstrated…
Training Compute-Optimal Large Language Models
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch +19
We investigate the optimal model size and number of tokens for training a transformer language model under a given compute budget. We find that current large language models are si…
Scaling Language Models: Methods, Analysis & Insights from Training Gopher
Jack W. Rae, Sebastian Borgeaud, Trevor Cai +77
Language modelling provides a step towards intelligent communication systems by harnessing large repositories of written human knowledge to better predict and understand the world.…
Challenges in Detoxifying Language Models
Johannes Welbl, Amelia Glaese, Jonathan Uesato +7
Large language models (LM) generate remarkably fluent text and can be efficiently adapted across NLP tasks. Measuring and guaranteeing the quality of generated text in terms of saf…
Undersensitivity in Neural Reading Comprehension
Johannes Welbl, Pasquale Minervini, Max Bartolo +2
Current reading comprehension models generalise well to in-distribution test sets, yet perform poorly on adversarially selected inputs. Most prior work on adversarial inputs studie…
Beat the AI: Investigating Adversarial Human Annotation for Reading Comprehension
Max Bartolo, Alastair Roberts, Johannes Welbl +2
Innovations in annotation methodology have been a catalyst for Reading Comprehension (RC) datasets and models. One recent trend to challenge current RC models is to involve a model…
Reducing Sentiment Bias in Language Models via Counterfactual Evaluation
Po-Sen Huang, Huan Zhang, Ray Jiang +6
Advances in language modeling architectures and the availability of large text corpora have driven progress in automatic text generation. While this results in models capable of ge…
Achieving Verified Robustness to Symbol Substitutions via Interval Bound Propagation
Po-Sen Huang, Robert Stanforth, Johannes Welbl +5
Neural networks are part of many contemporary NLP systems, yet their empirical successes come at the price of vulnerability to adversarial attacks. Previous work has used adversari…
Jack the Reader - A Machine Reading Framework
Dirk Weissenborn, Pasquale Minervini, Tim Dettmers +8
Many Machine Reading and Natural Language Understanding tasks require reading supporting text in order to answer questions. For example, in Question Answering, the supporting text…
Constructing Datasets for Multi-hop Reading Comprehension Across Documents
Johannes Welbl, Pontus Stenetorp, Sebastian Riedel
Most Reading Comprehension methods limit themselves to queries which can be answered using a single sentence, paragraph, or document. Enabling models to combine disjoint pieces of…
Frustratingly Short Attention Spans in Neural Language Modeling
Michał Daniluk, Tim Rocktäschel, Johannes Welbl +1
Neural language models predict the next token using a latent representation of the immediate token history. Recently, various methods for augmenting neural language models with an…
A Factorization Machine Framework for Testing Bigram Embeddings in Knowledgebase Completion
Johannes Welbl, Guillaume Bouchard, Sebastian Riedel
Embedding-based Knowledge Base Completion models have so far mostly combined distributed representations of individual entities or relations to compute truth scores of missing link…