Publications (30)
Leveraging LLMs for Synthesizing Training Data Across Many Languages in Multilingual Dense Retrieval
Nandan Thakur, Jianmo Ni, Gustavo Hernández Ãbrego +3
There has been limited success for dense retrieval models in multilingual retrieval, due to uneven and scarce training data available across multiple languages. Synthetic training…
Language-agnostic BERT Sentence Embedding
Fangxiaoyu Feng, Yinfei Yang, Daniel Cer +2
While BERT is an effective method for learning monolingual sentence embeddings for semantic similarity and embedding based transfer learning (Reimers and Gurevych, 2019), BERT base…
Improving Multilingual Sentence Embedding using Bi-directional Dual Encoder with Additive Margin Softmax
Yinfei Yang, Gustavo Hernandez Abrego, Steve Yuan +6
In this paper, we present an approach to learn multilingual sentence embeddings using a bi-directional dual-encoder with additive margin softmax. The embeddings are able to achieve…
Universal Sentence Encoder
Daniel Cer, Yinfei Yang, Sheng-yi Kong +10
We present models for encoding sentences into embedding vectors that specifically target transfer learning to other NLP tasks. The models are efficient and result in accurate perfo…
Gemini Embedding 2: A Native Multimodal Embedding Model from Gemini
Madhuri Shanbhogue, Zhe Li, Shanfeng Zhang +86
We introduce Gemini Embedding 2, a native multimodal embedding model that allows embedding video, audio, image, and text modalities in a unified representation space. We leverage t…
SeqGenSQL -- A Robust Sequence Generation Model for Structured Query Language
Ning Li, Bethany Keller, Mark Butler +1
We explore using T5 (Raffel et al. (2019)) to directly translate natural language questions into SQL statements. General purpose natural language that interfaces to information sto…
Crisscrossed Captions: Extended Intramodal and Intermodal Semantic Similarity Judgments for MS-COCO
Zarana Parekh, Jason Baldridge, Daniel Cer +2
By supporting multi-modal retrieval training and evaluation, image captioning datasets have spurred remarkable progress on representation learning. Unfortunately, datasets have lim…
Overcoming Catastrophic Forgetting in Zero-Shot Cross-Lingual Generation
Tu Vu, Aditya Barua, Brian Lester +3
In this paper, we explore the challenging problem of performing a generative task in a target language when labeled data is only available in English, using summarization as a case…
Gemma: Open Models Based on Gemini Research and Technology
Gemma Team, Thomas Mesnard, Cassidy Hardin +105
This work introduces Gemma, a family of lightweight, state-of-the art open models built from the research and technology used to create Gemini models. Gemma models demonstrate stro…
Effective Parallel Corpus Mining using Bilingual Sentence Embeddings
Mandy Guo, Qinlan Shen, Yinfei Yang +8
This paper presents an effective approach for parallel corpus mining using bilingual sentence embeddings. Our embedding models are trained to produce similar representations exclus…
Learning Semantic Textual Similarity from Conversations
Yinfei Yang, Steve Yuan, Daniel Cer +7
We present a novel approach to learn representations for sentence-level semantic similarity using conversational data. Our method trains an unsupervised model to predict conversati…
Neural Retrieval for Question Answering with Cross-Attention Supervised Data Augmentation
Yinfei Yang, Ning Jin, Kuo Lin +2
Neural models that independently project questions and answers into a shared embedding space allow for efficient continuous space retrieval from large corpora. Independently comput…
Gecko: Versatile Text Embeddings Distilled from Large Language Models
Jinhyuk Lee, Zhuyun Dai, Xiaoqi Ren +17
We present Gecko, a compact and versatile text embedding model. Gecko achieves strong retrieval performance by leveraging a key idea: distilling knowledge from large language model…
Universal Sentence Representation Learning with Conditional Masked Language Model
Ziyi Yang, Yinfei Yang, Daniel Cer +2
This paper presents a novel training method, Conditional Masked Language Modeling (CMLM), to effectively learn sentence representations on large scale unlabeled corpora. CMLM integ…
SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluation
Daniel Cer, Mona Diab, Eneko Agirre +2
Semantic Textual Similarity (STS) measures the meaning similarity of sentences. Applications include machine translation (MT), summarization, generation, question answering (QA), s…
A Simple and Effective Method To Eliminate the Self Language Bias in Multilingual Representations
Ziyi Yang, Yinfei Yang, Daniel Cer +1
Language agnostic and semantic-language information isolation is an emerging research direction for multilingual representations models. We explore this problem from a novel angle…
EmbeddingGemma: Powerful and Lightweight Text Representations
Henrique Schechter Vera, Sahil Dua, Biao Zhang +86
We introduce EmbeddingGemma, a new lightweight, open text embedding model based on the Gemma 3 language model family. Our innovative training recipe strategically captures knowledg…
ATEB: Evaluating and Improving Advanced NLP Tasks for Text Embedding Models
Simeng Han, Frank Palma Gomez, Tu Vu +6
Traditional text embedding benchmarks primarily evaluate embedding models' capabilities to capture semantic similarity. However, more advanced NLP tasks require a deeper understand…
Knowledge Prompts: Injecting World Knowledge into Language Models through Soft Prompts
Cicero Nogueira dos Santos, Zhe Dong, Daniel Cer +4
Soft prompts have been recently proposed as a tool for adapting large frozen language models (LMs) to new tasks. In this work, we repurpose soft prompts to the task of injecting wo…
CRISP: Clustering Multi-Vector Representations for Denoising and Pruning
João Veneroso, Rajesh Jayaram, Jinmeng Rao +3
Multi-vector models, such as ColBERT, are a significant advancement in neural information retrieval (IR), delivering state-of-the-art performance by representing queries and docume…
SPoT: Better Frozen Model Adaptation through Soft Prompt Transfer
Tu Vu, Brian Lester, Noah Constant +2
There has been growing interest in parameter-efficient methods to apply pre-trained language models to downstream tasks. Building on the Prompt Tuning approach of Lester et al. (20…
Learning Cross-Lingual Sentence Representations via a Multi-task Dual-Encoder Model
Muthuraman Chidambaram, Yinfei Yang, Daniel Cer +4
A significant roadblock in multilingual neural language modeling is the lack of labeled non-English data. One potential method for overcoming this issue is learning cross-lingual t…
Hierarchical Document Encoder for Parallel Corpus Mining
Mandy Guo, Yinfei Yang, Keith Stevens +5
We explore using multilingual document embeddings for nearest neighbor mining of parallel data. Three document-level representations are investigated: (i) document embeddings gener…
Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Models
Jianmo Ni, Gustavo Hernández Ãbrego, Noah Constant +4
We provide the first exploration of sentence embeddings from text-to-text transformers (T5). Sentence embeddings are broadly useful for language processing tasks. While T5 achieves…
Multilingual Universal Sentence Encoder for Semantic Retrieval
Yinfei Yang, Daniel Cer, Amin Ahmad +9
We introduce two pre-trained retrieval focused multilingual sentence encoding models, respectively based on the Transformer and CNN model architectures. The models embed text from…
Transforming LLMs into Cross-modal and Cross-lingual Retrieval Systems
Frank Palma Gomez, Ramon Sanabria, Yun-hsuan Sung +3
Large language models (LLMs) are trained on text-only data that go far beyond the languages with paired speech and text data. At the same time, Dual Encoder (DE) based retrieval sy…
NT5?! Training T5 to Perform Numerical Reasoning
Peng-Jian Yang, Ying Ting Chen, Yuechan Chen +1
Numerical reasoning over text (NRoT) presents unique challenges that are not well addressed by existing pre-training objectives. We explore five sequential training schedules that…
Gemini Embedding: Generalizable Embeddings from Gemini
Jinhyuk Lee, Feiyang Chen, Sahil Dua +44
In this report, we introduce Gemini Embedding, a state-of-the-art embedding model leveraging the power of Gemini, Google's most capable large language model. Capitalizing on Gemini…
ReQA: An Evaluation for End-to-End Answer Retrieval Models
Amin Ahmad, Noah Constant, Yinfei Yang +1
Popular QA benchmarks like SQuAD have driven progress on the task of identifying answer spans within a specific passage, with models now surpassing human performance. However, retr…
MultiReQA: A Cross-Domain Evaluation for Retrieval Question Answering Models
Mandy Guo, Yinfei Yang, Daniel Cer +2
Retrieval question answering (ReQA) is the task of retrieving a sentence-level answer to a question from an open corpus (Ahmad et al.,2019).This paper presents MultiReQA, anew mult…