565 citations · 2.8k across the 120 of their papers we have counts for
56 papers · 1 filter
Omitted Variable Bias in Language Models Under Distribution Shift
Victoria Lin, Louis-Philippe Morency, Eli Ben-Michael
Despite their impressive performance on a wide variety of tasks, modern language models remain susceptible to distribution shifts, exhibiting brittle behavior when evaluated on dat…
IoT-LM: Large Multisensory Language Models for the Internet of Things
Shentong Mo, Russ Salakhutdinov, Louis-Philippe Morency +1
The Internet of Things (IoT) network integrating billions of smart physical devices embedded with sensors, software, and communication technologies is a critical and rapidly expand…
HEMM: Holistic Evaluation of Multimodal Foundation Models
Paul Pu Liang, Akshay Goindani, Talha Chafekar +4
Multimodal foundation models that can holistically process text alongside images, video, audio, and other sensory modalities are increasingly used in a variety of real-world applic…
Optimizing Language Models for Human Preferences is a Causal Inference Problem
Victoria Lin, Eli Ben-Michael, Louis-Philippe Morency
As large language models (LLMs) see greater use in academic and commercial settings, there is increasing interest in methods that allow language models to generate texts aligned wi…
Comparative Knowledge Distillation
Alex Wilf, Alex Tianyi Xu, Paul Pu Liang +3
In the era of large scale pretrained models, Knowledge Distillation (KD) serves an important role in transferring the wisdom of computationally heavy teacher models to lightweight,…
MultiIoT: Benchmarking Machine Learning for the Internet of Things
Shentong Mo, Louis-Philippe Morency, Russ Salakhutdinov +1
The next generation of machine learning systems must be adept at perceiving and interacting with the physical world through a diverse array of sensory channels. Commonly referred t…
MultiZoo & MultiBench: A Standardized Toolkit for Multimodal Deep Learning
Paul Pu Liang, Yiwei Lyu, Xiang Fan +4
Learning multimodal representations involves integrating information from multiple heterogeneous sources of data. In order to accelerate progress towards understudied modalities an…
Understanding Masked Autoencoders via Hierarchical Latent Variable Models
Lingjing Kong, Martin Q. Ma, Guangyi Chen +4
Masked autoencoder (MAE), a simple and effective self-supervised learning framework based on the reconstruction of masked image regions, has recently achieved prominent success in…
Neural Mixed Effects for Nonlinear Personalized Predictions
Torsten Wörtwein, Nicholas Allen, Lisa B. Sheeber +3
Personalized prediction is a machine learning approach that predicts a person's future observations based on their past labeled observations and is typically used for sequential ta…
Factorized Contrastive Learning: Going Beyond Multi-view Redundancy
Paul Pu Liang, Zihao Deng, Martin Ma +3
In a wide range of multimodal tasks, contrastive learning has become a particularly appealing approach since it can successfully learn representations from abundant unlabeled data…
Multimodal Learning Without Labeled Multimodal Data: Guarantees and Applications
Paul Pu Liang, Chun Kai Ling, Yun Cheng +6
In many machine learning systems that jointly learn from multiple modalities, a core research question is to understand the nature of multimodal interactions: how modalities combin…
Multimodal Fusion Interactions: A Study of Human and Automatic Quantification
Paul Pu Liang, Yun Cheng, Ruslan Salakhutdinov +1
In order to perform multimodal fusion of heterogeneous signals, we need to understand their interactions: how each modality individually provides information useful for a task and…
Difference-Masking: Choosing What to Mask in Continued Pretraining
Alex Wilf, Syeda Nahida Akter, Leena Mathur +5
The self-supervised objective of masking-and-predicting has led to promising performance gains on a variety of downstream tasks. However, while most approaches randomly mask tokens…
Counterfactual Augmentation for Multimodal Learning Under Presentation Bias
Victoria Lin, Louis-Philippe Morency, Dimitrios Dimitriadis +1
In real-world machine learning systems, labels are often derived from user behaviors that the system wishes to encourage. Over time, new models must be trained as new training exam…
Quantifying & Modeling Multimodal Interactions: An Information Decomposition Framework
Paul Pu Liang, Yun Cheng, Xiang Fan +9
The recent explosion of interest in multimodal applications has resulted in a wide selection of datasets and methods for representing and integrating information from different mod…
SeedBERT: Recovering Annotator Rating Distributions from an Aggregated Label
Aneesha Sampath, Victoria Lin, Louis-Philippe Morency
Many machine learning tasks -- particularly those in affective computing -- are inherently subjective. When asked to classify facial expressions or to rate an individual's attracti…
Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questions
Paul Pu Liang, Amir Zadeh, Louis-Philippe Morency
Multimodal machine learning is a vibrant multi-disciplinary research field that aims to design computer agents with intelligent capabilities such as understanding, reasoning, and l…
Face-to-Face Contrastive Learning for Social Intelligence Question-Answering
Alex Wilf, Martin Q. Ma, Paul Pu Liang +2
Creating artificial social intelligence - algorithms that can understand the nuances of multi-person interactions - is an exciting and emerging challenge in processing facial expre…
MultiViz: Towards Visualizing and Understanding Multimodal Models
Paul Pu Liang, Yiwei Lyu, Gunjan Chhablani +5
The promise of multimodal models for real-world applications has inspired research in visualizing and understanding their internal mechanics with the end goal of empowering stakeho…
Conditional Contrastive Learning with Kernel
Yao-Hung Hubert Tsai, Tianqin Li, Martin Q. Ma +4
Conditional contrastive learning frameworks consider the conditional sampling procedure that constructs positive or negative data pairs conditioned on specific variables. Fair cont…
DIME: Fine-grained Interpretations of Multimodal Models via Disentangled Local Explanations
Yiwei Lyu, Paul Pu Liang, Zihao Deng +2
The ability for a human to understand an Artificial Intelligence (AI) model's decision-making process is critical in enabling stakeholders to visualize model behavior, perform mode…
PACS: A Dataset for Physical Audiovisual CommonSense Reasoning
Samuel Yu, Peter Wu, Paul Pu Liang +2
In order for AI to be safely deployed in real-world scenarios such as hospitals, schools, and the workplace, it must be able to robustly reason about the physical world. Fundamenta…
High-Modality Multimodal Transformer: Quantifying Modality & Interaction Heterogeneity for High-Modality Representation Learning
Paul Pu Liang, Yiwei Lyu, Xiang Fan +6
Many real-world problems are inherently multimodal, from spoken language, gestures, and paralinguistics humans use to communicate, to force, proprioception, and visual sensors on r…
Learning Weakly-Supervised Contrastive Representations
Yao-Hung Hubert Tsai, Tianqin Li, Weixin Liu +3
We argue that a form of the valuable information provided by the auxiliary information is its implied data clustering information. For instance, considering hashtags as auxiliary i…