297 citations · 825 across the 28 of their papers we have counts for
28 papers
Gemma: Open Models Based on Gemini Research and Technology
Gemma Team, Thomas Mesnard, Cassidy Hardin +105
This work introduces Gemma, a family of lightweight, state-of-the art open models built from the research and technology used to create Gemini models. Gemma models demonstrate stro…
Best Practices and Lessons Learned on Synthetic Data
Ruibo Liu, Jerry Wei, Fangyu Liu +8
The success of AI models relies on the availability of large, diverse, and high-quality datasets, which can be challenging to obtain due to data scarcity, privacy concerns, and hig…
Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context
Gemini Team, Petko Georgiev, Ving Ian Lei +1132
In this report, we introduce the Gemini 1.5 family of models, representing the next generation of highly compute-efficient multimodal models capable of recalling and reasoning over…
Higher Layers Need More LoRA Experts
Chongyang Gao, Kezhen Chen, Jinmeng Rao +7
Parameter-efficient tuning (PEFT) techniques like low-rank adaptation (LoRA) offer training efficiency on Large Language Models, but their impact on model performance remains limit…
Exploring Collaboration Mechanisms for LLM Agents: A Social Psychology View
Jintian Zhang, Xin Xu, Ningyu Zhang +3
As Natural Language Processing (NLP) systems are increasingly employed in intricate social environments, a pressing query emerges: Can these NLP systems mirror human-esque collabor…
On the Effectiveness of Speech Self-supervised Learning for Music
Yinghao Ma, Ruibin Yuan, Yizhi Li +12
Self-supervised learning (SSL) has shown promising results in various speech and natural language processing applications. However, its efficacy in music information retrieval (MIR…
MARBLE: Music Audio Representation Benchmark for Universal Evaluation
Ruibin Yuan, Yinghao Ma, Yizhi Li +22
In the era of extensive intersection between art and Artificial Intelligence (AI), such as image generation and fiction co-creation, AI for music remains relatively nascent, partic…
MERT: Acoustic Music Understanding Model with Large-Scale Self-supervised Training
Yizhi Li, Ruibin Yuan, Ge Zhang +17
Self-supervised learning (SSL) has recently emerged as a promising paradigm for training generalisable models on large-scale data in the fields of vision, text, and speech. Althoug…
Interactive Natural Language Processing
Zekun Wang, Ge Zhang, Kexin Yang +19
Interactive Natural Language Processing (iNLP) has emerged as a novel paradigm within the field of NLP, aimed at addressing limitations in existing frameworks while aligning with t…
Training Socially Aligned Language Models on Simulated Social Interactions
Ruibo Liu, Ruixin Yang, Chenyan Jia +5
Social alignment in AI systems aims to ensure that these models behave according to established societal values. However, unlike humans, who derive consensus on value judgments thr…
Chinese Open Instruction Generalist: A Preliminary Release
Ge Zhang, Yemin Shi, Ruibo Liu +9
Instruction tuning is widely recognized as a key technique for building generalist language models, which has attracted the attention of researchers and the public with the release…
Second Thoughts are Best: Learning to Re-Align With Human Values from Text Edits
Ruibo Liu, Chenyan Jia, Ge Zhang +3
We present Second Thought, a new learning paradigm that enables language models (LMs) to re-align with human values. By modeling the chain-of-edits between value-unaligned and valu…
MAP-Music2Vec: A Simple and Effective Baseline for Self-Supervised Music Audio Representation Learning
Yizhi Li, Ruibin Yuan, Ge Zhang +11
The deep learning community has witnessed an exponentially growing interest in self-supervised learning (SSL). However, it still remains unexplored how to build a framework for lea…
EnDex: Evaluation of Dialogue Engagingness at Scale
Guangxuan Xu, Ruibo Liu, Fabrice Harel-Canada +2
We propose EnDex, the first human-reaction based model to evaluate dialogue engagingness. EnDex is trained on 80k Reddit-based Engagement Dataset (RED) curated using a novel distan…
Mind's Eye: Grounded Language Model Reasoning through Simulation
Ruibo Liu, Jason Wei, Shixiang Shane Gu +5
Successful and effective communication between humans and AI relies on a shared experience of the world. By training solely on written text, current language models (LMs) miss the…
Non-Parallel Text Style Transfer with Self-Parallel Supervision
Ruibo Liu, Chongyang Gao, Chenyan Jia +2
The performance of existing text style transfer models is severely limited by the non-parallel datasets on which the models are trained. In non-parallel datasets, no direct mapping…
Knowledge Infused Decoding
Ruibo Liu, Guoqing Zheng, Shashank Gupta +5
Pre-trained language models (LMs) have been shown to memorize a substantial amount of knowledge from the pre-training corpora; however, they are still limited in recalling factuall…
Language Model Augmented Relevance Score
Ruibo Liu, Jason Wei, Soroush Vosoughi
Although automated metrics are commonly used to evaluate NLG systems, they often correlate poorly with human judgements. Newer metrics such as BERTScore have addressed many weaknes…
Modulating Language Models with Emotions
Ruibo Liu, Jason Wei, Chenyan Jia +1
Generating context-aware language that embodies diverse emotions is an important step towards building empathetic NLP systems. In this paper, we propose a formulation of modulated…
Embedding Heterogeneous Networks into Hyperbolic Space Without Meta-path
Lili Wang, Chongyang Gao, Chenghan Huang +3
Networks found in the real-world are numerous and varied. A common type of network is the heterogeneous network, where the nodes (and edges) can be of different types. Accordingly,…
Mitigating Political Bias in Language Models Through Reinforced Calibration
Ruibo Liu, Chenyan Jia, Jason Wei +3
Current large-scale language models can be politically biased as a result of the data they are trained on, potentially causing serious problems when they are deployed in real-world…
Political Depolarization of News Articles Using Attribute-aware Word Embeddings
Ruibo Liu, Lili Wang, Chenyan Jia +1
Political polarization in the US is on the rise. This polarization negatively affects the public sphere by contributing to the creation of ideological echo chambers. In this paper,…
Improvements and Extensions on Metaphor Detection
Weicheng Ma, Ruibo Liu, Lili Wang +1
Metaphors are ubiquitous in human language. The metaphor detection task (MD) aims at detecting and interpreting metaphors from written language, which is crucial in natural languag…
An Empirical Survey of Unsupervised Text Representation Methods on Twitter Data
Lili Wang, Chongyang Gao, Jason Wei +3
The field of NLP has seen unprecedented achievements in recent years. Most notably, with the advent of large-scale pre-trained Transformer-based language models, such as BERT, ther…