23 citations · 78 across the 20 of their papers we have counts for
24 papers
Cracks in the Foundation: Seemingly Minor Architectural Choices Impact Long Context Extension
Amanda Bertsch, Luca Soldaini, Matthew R. Gormley +4
One might imagine that architectural variations within the dense transformer paradigm have a limited effect on accuracy. However, we demonstrate that this is not the case in the lo…
Oolong: Evaluating Long Context Reasoning and Aggregation Capabilities
Amanda Bertsch, Adithya Pratapa, Teruko Mitamura +2
As model context lengths continue to grow, concerns about whether models effectively use the full context length have persisted. While several carefully designed long-context evalu…
A Taxonomy for Data Contamination in Large Language Models
Medha Palavalli, Amanda Bertsch, Matthew R. Gormley
Large language models pretrained on extensive web corpora demonstrate remarkable performance across a wide range of downstream tasks. However, a growing concern is data contaminati…
In-Context Learning with Long-Context Models: An In-Depth Exploration
Amanda Bertsch, Maor Ivgi, Emily Xiao +4
As model context lengths continue to increase, the number of demonstrations that can be provided in-context approaches the size of entire training datasets. We study the behavior o…
Learning Mutually Informed Representations for Characters and Subwords
Yilin Wang, Xinyi Hu, Matthew R. Gormley
Most pretrained language models rely on subword tokenization, which processes text as a sequence of subword tokens. However, different granularities of text, such as characters, su…
It's MBR All the Way Down: Modern Generation Techniques Through the Lens of Minimum Bayes Risk
Amanda Bertsch, Alex Xie, Graham Neubig +1
Minimum Bayes Risk (MBR) decoding is a method for choosing the outputs of a machine learning system based not on the output with the highest probability, but the output with the lo…
MDACE: MIMIC Documents Annotated with Code Evidence
Hua Cheng, Rana Jafari, April Russell +4
We introduce a dataset for evidence/rationale extraction on an extreme multi-label classification task over long medical documents. One such task is Computer-Assisted Coding (CAC)…
SummQA at MEDIQA-Chat 2023:In-Context Learning with GPT-4 for Medical Summarization
Yash Mathur, Sanketh Rangreji, Raghav Kapoor +3
Medical dialogue summarization is challenging due to the unstructured nature of medical conversations, the use of medical terminology in gold summaries, and the need to identify ke…
Unlimiformer: Long-Range Transformers with Unlimited Length Input
Amanda Bertsch, Uri Alon, Graham Neubig +1
Since the proposal of transformers, these models have been limited to bounded input lengths, because of their need to attend to every token in the input. In this work, we propose U…
Revisiting text decomposition methods for NLI-based factuality scoring of summaries
John Glover, Federico Fancellu, Vasudevan Jagannathan +2
Scoring the factuality of a generated summary involves measuring the degree to which a target text contains factual information using the input document as support. Given the simil…
AdaFocal: Calibration-aware Adaptive Focal Loss
Arindam Ghosh, Thomas Schaaf, Matthew R. Gormley
Much recent work has been devoted to the problem of ensuring that a neural network's confidence scores match the true probability of being correct, i.e. the calibration problem. Of…
He Said, She Said: Style Transfer for Shifting the Perspective of Dialogues
Amanda Bertsch, Graham Neubig, Matthew R. Gormley
In this work, we define a new style transfer task: perspective shift, which reframes a dialogue from informal first person to a formal third person rephrasing of the text. This tas…
On Efficiently Acquiring Annotations for Multilingual Models
Joel Ruben Antony Moniz, Barun Patra, Matthew R. Gormley
When tasked with supporting multiple languages for a given problem, two approaches have arisen: training a model for each language with the annotation budget divided equally among…
Leveraging Pretrained Models for Automatic Summarization of Doctor-Patient Conversations
Longxiang Zhang, Renato Negrinho, Arindam Ghosh +4
Fine-tuning pretrained models for automatically summarizing doctor-patient conversation transcripts presents many challenges: limited training data, significant domain shift, long…
Comparative Error Analysis in Neural and Finite-state Models for Unsupervised Character-level Transduction
Maria Ryskina, Eduard Hovy, Taylor Berg-Kirkpatrick +1
Traditionally, character-level transduction problems have been solved with finite-state models designed to encode structural and linguistic knowledge of the underlying process, whe…
An Empirical Investigation of Beam-Aware Training in Supertagging
Renato Negrinho, Matthew R. Gormley, Geoffrey J. Gordon
Structured prediction is often approached by training a locally normalized model with maximum likelihood and decoding approximately with beam search. This approach leads to mismatc…
Phonetic and Visual Priors for Decipherment of Informal Romanization
Maria Ryskina, Matthew R. Gormley, Taylor Berg-Kirkpatrick
Informal romanization is an idiosyncratic process used by humans in informal digital communication to encode non-Latin script languages into Latin character sets found on common ke…
Towards modular and programmable architecture search
Renato Negrinho, Darshan Patil, Nghia Le +3
Neural architecture search methods are able to find high performance deep learning architectures with minimal effort from an expert. However, current systems focus on specific use-…
Bilingual Lexicon Induction with Semi-supervision in Non-Isometric Embedding Spaces
Barun Patra, Joel Ruben Antony Moniz, Sarthak Garg +2
Recent work on bilingual lexicon induction (BLI) has frequently depended either on aligned bilingual lexicons or on distribution matching, often with an assumption about the isomet…
Learning Beam Search Policies via Imitation Learning
Renato Negrinho, Matthew R. Gormley, Geoffrey J. Gordon
Beam search is widely used for approximate decoding in structured prediction problems. Models often use a beam at test time but ignore its existence at train time, and therefore do…
Neural Factor Graph Models for Cross-lingual Morphological Tagging
Chaitanya Malaviya, Matthew R. Gormley, Graham Neubig
Morphological analysis involves predicting the syntactic traits of a word (e.g. {POS: Noun, Case: Acc, Gender: Fem}). Previous work in morphological tagging improves performance fo…
Embedding Lexical Features via Low-Rank Tensors
Mo Yu, Mark Dredze, Raman Arora +1
Modern NLP models rely heavily on engineered features, which often combine word and contextual information into complex lexical features. Such combination results in large numbers…
Approximation-Aware Dependency Parsing by Belief Propagation
Matthew R. Gormley, Mark Dredze, Jason Eisner
We show how to train the fast dependency parser of Smith and Eisner (2008) for improved accuracy. This parser can consider higher-order interactions among edges while retaining O(n…
Improved Relation Extraction with Feature-Rich Compositional Embedding Models
Matthew R. Gormley, Mo Yu, Mark Dredze
Compositional embedding models build a representation (or embedding) for a linguistic structure based on its component word embeddings. We propose a Feature-rich Compositional Embe…