Publications (14)
AlexaTM 20B: Few-Shot Learning Using a Large-Scale Multilingual Seq2Seq Model
Saleh Soltan, Shankar Ananthakrishnan, Jack FitzGerald +13
In this work, we demonstrate that multilingual large-scale sequence-to-sequence (seq2seq) models, pre-trained on a mixture of denoising and Causal Language Modeling (CLM) tasks, ar…
Measuring and Eliminating Refusals in Military Large Language Models
Jack FitzGerald, Dylan Bates, Aristotelis Lazaridis +17
Military Large Language Models (LLMs) must provide accurate information to the warfighter in time-critical and dangerous situations. However, today's LLMs are imbued with safety be…
Wanda++: Pruning Large Language Models via Regional Gradients
Yifan Yang, Kai Zhen, Bhavana Ganesh +11
Large Language Models (LLMs) pruning seeks to remove unimportant weights for inference speedup with minimal accuracy impact. However, existing methods often suffer from accuracy de…
The Massively Multilingual Natural Language Understanding 2022 (MMNLU-22) Workshop and Competition
Christopher Hench, Charith Peris, Jack FitzGerald +1
Despite recent progress in Natural Language Understanding (NLU), the creation of multilingual NLU systems remains a challenge. It is common to have NLU systems limited to a subset…
The Amazon Nova Family of Models: Technical Report and Model Card
Amazon AGI, Aaron Langford, Aayush Shah +783
We present Amazon Nova, a new generation of state-of-the-art foundation models that deliver frontier intelligence and industry-leading price performance. Amazon Nova Pro is a highl…
Alexa Teacher Model: Pretraining and Distilling Multi-Billion-Parameter Encoders for Natural Language Understanding Systems
Jack FitzGerald, Shankar Ananthakrishnan, Konstantine Arkoudas +38
We present results from a large-scale experiment on pretraining encoders with non-embedding parameter counts ranging from 700M to 9.3B, their subsequent distillation into smaller m…
Can Editing 1 Neuron Fix Repetition Loops in LLMs?
Aristotelis Lazaridis, Aman Sharma, Dylan Bates +3
Yes. Can it cure doom loops? Probably not. The Gemma 4 instruction-tuned models share a reproducible failure: on long factual enumeration prompts, such as listing every episode of…
QuAILoRA: Quantization-Aware Initialization for LoRA
Neal Lawton, Aishwarya Padmakumar, Judith Gaspers +4
QLoRA reduces the memory-cost of fine-tuning a large language model (LLM) with LoRA by quantizing the base LLM. However, quantization introduces quantization errors that negatively…
EDGE-OPD: Internalizing Privileged Context with Evidence Guided On-Policy Distillation
Aristotelis Lazaridis, Dylan Bates, Aman Sharma +3
On-Policy Distillation (OPD) has gained wide attraction as an LLM post-training paradigm due to its effectiveness in improving capabilities without introducing model distribution d…
MATTER: Memory-Augmented Transformer Using Heterogeneous Knowledge Sources
Dongkyu Lee, Chandana Satya Prakash, Jack FitzGerald +1
Leveraging external knowledge is crucial for achieving high performance in knowledge-intensive tasks, such as question answering. The retrieve-and-read approach is widely adopted f…
Controlling the Extraction of Memorized Data from Large Language Models via Prompt-Tuning
Mustafa Safa Ozdayi, Charith Peris, Jack FitzGerald +5
Large Language Models (LLMs) are known to memorize significant portions of their training data. Parts of this memorized content have been shown to be extractable by simply querying…
EdgeRunner 20B: Military Task Parity with GPT-5 while Running on the Edge
Jack FitzGerald, Aristotelis Lazaridis, Dylan Bates +17
We present EdgeRunner 20B, a fine-tuned version of gpt-oss-20b optimized for military tasks. EdgeRunner 20B was trained on 1.6M high-quality records curated from military documenta…
PHLoRA: data-free Post-hoc Low-Rank Adapter extraction from full-rank checkpoint
Bhoomit Vasani, Jack FitzGerald, Anjie Fang +1
We introduce PHLoRA (Pronounced "flora"). (Post-hoc LoRA), a simple yet powerful method to extract low-rank adaptation adapters from full-rank fine-tuned models without requiring a…
MASSIVE: A 1M-Example Multilingual Natural Language Understanding Dataset with 51 Typologically-Diverse Languages
Jack FitzGerald, Christopher Hench, Charith Peris +13
We present the MASSIVE dataset--Multilingual Amazon Slu resource package (SLURP) for Slot-filling, Intent classification, and Virtual assistant Evaluation. MASSIVE contains 1M real…