297 citations · 657 across the 41 of their papers we have counts for
46 papers
PaliGemma: A versatile 3B VLM for transfer
Lucas Beyer, Andreas Steiner, André Susano Pinto +32
PaliGemma is an open Vision-Language Model (VLM) that is based on the SigLIP-So400m vision encoder and the Gemma-2B language model. It is trained to be a versatile and broadly know…
Best Practices and Lessons Learned on Synthetic Data
Ruibo Liu, Jerry Wei, Fangyu Liu +8
The success of AI models relies on the availability of large, diverse, and high-quality datasets, which can be challenging to obtain due to data scarcity, privacy concerns, and hig…
Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context
Gemini Team, Petko Georgiev, Ving Ian Lei +1132
In this report, we introduce the Gemini 1.5 family of models, representing the next generation of highly compute-efficient multimodal models capable of recalling and reasoning over…
POSQA: Probe the World Models of LLMs with Size Comparisons
Chang Shu, Jiuzhou Han, Fangyu Liu +2
Embodied language comprehension emphasizes that language understanding is not solely a matter of mental processing in the brain but also involves interactions with the physical and…
Sparkles: Unlocking Chats Across Multiple Images for Multimodal Instruction-Following Models
Yupan Huang, Zaiqiao Meng, Fangyu Liu +3
Large language models exhibit enhanced zero-shot performance on various tasks when fine-tuned with instruction-following data. Multimodal instruction-following models extend these…
Compositional Zero-Shot Domain Transfer with Text-to-Text Models
Fangyu Liu, Qianchu Liu, Shruthi Bannur +9
Label scarcity is a bottleneck for improving task performance in specialised domains. We propose a novel compositional transfer learning framework (DoT5 - domain compositional zero…
Reranking Overgenerated Responses for End-to-End Task-Oriented Dialogue Systems
Songbo Hu, Ivan Vulić, Fangyu Liu +1
End-to-end (E2E) task-oriented dialogue (ToD) systems are prone to fall into the so-called "likelihood trap", resulting in generated responses which are dull, repetitive, and often…
On-the-fly Denoising for Data Augmentation in Natural Language Understanding
Tianqing Fang, Wenxuan Zhou, Fangyu Liu +3
Data Augmentation (DA) is frequently used to provide additional training data without extra human annotation automatically. However, data augmentation may introduce noisy data that…
DePlot: One-shot visual language reasoning by plot-to-table translation
Fangyu Liu, Julian Martin Eisenschlos, Francesco Piccinno +7
Visual language such as charts and plots is ubiquitous in the human world. Comprehending plots and charts requires strong reasoning skills. Prior state-of-the-art (SOTA) models req…
MatCha: Enhancing Visual Language Pretraining with Math Reasoning and Chart Derendering
Fangyu Liu, Francesco Piccinno, Syrine Krichene +6
Visual language data such as plots, charts, and infographics are ubiquitous in the human world. However, state-of-the-art vision-language models do not perform well on these data.…
How to tackle an emerging topic? Combining strong and weak labels for Covid news NER
Aleksander Ficek, Fangyu Liu, Nigel Collier
Being able to train Named Entity Recognition (NER) models for emerging topics is crucial for many real-world applications especially in the medical domain where new topics are cont…
Improving Bilingual Lexicon Induction with Cross-Encoder Reranking
Yaoyiran Li, Fangyu Liu, Ivan Vulić +1
Bilingual lexicon induction (BLI) with limited bilingual supervision is a crucial yet challenging task in multilingual NLP. Current state-of-the-art BLI methods rely on the inducti…
Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understanding
Kenton Lee, Mandar Joshi, Iulia Turc +7
Visually-situated language is ubiquitous -- sources range from textbooks with diagrams to web pages with images and tables, to mobile apps with buttons and forms. Perhaps due to th…
Do ever larger octopi still amplify reporting biases? Evidence from judgments of typical colour
Fangyu Liu, Julian Martin Eisenschlos, Jeremy R. Cole +1
Language models (LMs) trained on raw texts have no direct access to the physical world. Gordon and Van Durme (2013) point out that LMs can thus suffer from reporting bias: texts ra…
WinoDict: Probing language models for in-context word acquisition
Julian Martin Eisenschlos, Jeremy R. Cole, Fangyu Liu +1
We introduce a new in-context learning paradigm to measure Large Language Models' (LLMs) ability to learn novel words during inference. In particular, we rewrite Winograd-style co-…
On Reality and the Limits of Language Data: Aligning LLMs with Human Norms
Nigel H. Collier, Fangyu Liu, Ehsan Shareghi
Recent advancements in Large Language Models (LLMs) harness linguistic associations in vast natural language data for practical applications. However, their ability to understand t…
TweetNLP: Cutting-Edge Natural Language Processing for Social Media
Jose Camacho-Collados, Kiamehr Rezaee, Talayeh Riahi +11
In this paper we present TweetNLP, an integrated platform for Natural Language Processing (NLP) in social media. TweetNLP supports a diverse set of NLP tasks, including generic foc…
Language Models Can See: Plugging Visual Controls in Text Generation
Yixuan Su, Tian Lan, Yahui Liu +5
Generative language models (LMs) such as GPT-2/3 can be prompted to generate text with remarkable quality. While they are designed for text-prompted generation, it remains an open…
Modality-Balanced Embedding for Video Retrieval
Xun Wang, Bingqing Ke, Xuanping Li +6
Video search has become the main routine for users to discover videos relevant to a text query on large short-video sharing platforms. During training a query-video bi-encoder mode…
Visual Spatial Reasoning
Fangyu Liu, Guy Emerson, Nigel Collier
Spatial relations are a basic part of human cognition. However, they are expressed in natural language in a variety of ways, and previous work has suggested that current vision-and…
Probing Cross-Lingual Lexical Knowledge from Multilingual Sentence Encoders
Ivan Vulić, Goran Glavaš, Fangyu Liu +3
Pretrained multilingual language models (LMs) can be successfully transformed into multilingual sentence encoders (SEs; e.g., LaBSE, xMPNet) via additional fine-tuning or model dis…
Improving Word Translation via Two-Stage Contrastive Learning
Yaoyiran Li, Fangyu Liu, Nigel Collier +2
Word translation or bilingual lexicon induction (BLI) is a key cross-lingual task, aiming to bridge the lexical gap between different languages. In this work, we propose a robust a…
Revisiting Parameter-Efficient Tuning: Are We Really There Yet?
Guanzheng Chen, Fangyu Liu, Zaiqiao Meng +1
Parameter-Efficient Tuning (PETuning) methods have been deemed by many as the new paradigm for using pretrained language models (PLMs). By tuning just a fraction amount of paramete…
IGLUE: A Benchmark for Transfer Learning across Modalities, Tasks, and Languages
Emanuele Bugliarello, Fangyu Liu, Jonas Pfeiffer +4
Reliable evaluation benchmarks designed for replicability and comprehensiveness have driven progress in machine learning. Due to the lack of a multilingual benchmark, however, visi…