6 papers · 1 filter
Muse Glimmer 30B: An On-Device Agentic Model
Arash Ebrahim Abtahizadeh, Muna Aghamelu, Adil Ahmed +327
We introduce Muse Glimmer, an open-source 30B parameter causal language model with perception encoder, purpose-built for autonomous agentic tasks on consumer hardware. Muse Glimmer…
Bootstrapping Embeddings for Low Resource Languages
Merve Basoz, Andrew Horne, Mattia Opper
Embedding models are crucial to modern NLP. However, the creation of the most effective models relies on carefully constructed supervised finetuning data. For high resource languag…
Banyan: Improved Representation Learning with Explicit Structure
Mattia Opper, N. Siddharth
We present Banyan, a model that efficiently learns semantic representations by leveraging explicit hierarchical structure. While transformers excel at scale, they struggle in low-r…
Self-StrAE at SemEval-2024 Task 1: Making Self-Structuring AutoEncoders Learn More With Less
Mattia Opper, N. Siddharth
This paper presents two simple improvements to the Self-Structuring AutoEncoder (Self-StrAE). Firstly, we show that including reconstruction to the vocabulary as an auxiliary objec…
On the effect of curriculum learning with developmental data for grammar acquisition
Mattia Opper, J. Morrison, N. Siddharth
This work explores the degree to which grammar acquisition is driven by language `simplicity' and the source modality (speech vs. text) of data. Using BabyBERTa as a probe, we find…
StrAE: Autoencoding for Pre-Trained Embeddings using Explicit Structure
Mattia Opper, Victor Prokhorov, N. Siddharth
This work presents StrAE: a Structured Autoencoder framework that through strict adherence to explicit structure, and use of a novel contrastive objective over tree-structured repr…