Learning Transferable Visual Models From Natural Language Supervision

Abstract

State-of-the-art computer vision systems are trained to predict a fixed set of predetermined object categories. This restricted form of supervision limits their generality and usability since additional labeled data is needed to specify any other visual concept. Learning directly from raw text about images is a promising alternative which leverages a much broader source of supervision. We demonstrate that the simple pre-training task of predicting which caption goes with which image is an efficient and scalable way to learn SOTA image representations from scratch on a dataset of 400 million (image, text) pairs collected from the internet. After pre-training, natural language is used to reference learned visual concepts (or describe new ones) enabling zero-shot transfer of the model to downstream tasks. We study the performance of this approach by benchmarking on over 30 different existing computer vision datasets, spanning tasks such as OCR, action recognition in videos, geo-localization, and many types of fine-grained object classification. The model transfers non-trivially to most tasks and is often competitive with a fully supervised baseline without the need for any dataset specific training. For instance, we match the accuracy of the original ResNet-50 on ImageNet zero-shot without needing to use any of the 1.28 million training examples it was trained on. We release our code and pre-trained model weights at https://github.com/OpenAI/CLIP.

References in corpus (66)

• Adam: A Method for Stochastic Optimization
• Scikit-learn: Machine Learning in Python
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Distributed Representations of Words and Phrases and their Compositionality
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• A Simple Framework for Contrastive Learning of Visual Representations
• Distributed Representations of Sentences and Documents
• EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• Bootstrap your own latent: A new approach to self-supervised Learning
• Gaussian Error Linear Units (GELUs)
• Language Models are Few-Shot Learners
• Shortcut Learning in Deep Neural Networks
• Improved Baselines with Momentum Contrastive Learning
• SGDR: Stochastic Gradient Descent with Warm Restarts
• YFCC100M: The New Data in Multimedia Research
• Scaling Laws for Neural Language Models
• Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings
• Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
• Mixed Precision Training
• HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips
• Data-Efficient Image Recognition with Contrastive Predictive Coding
• Deep Fragment Embeddings for Bidirectional Image Sentence Mapping
• VSE++: Improving Visual-Semantic Embeddings with Hard Negatives
• Contrastive Multiview Coding
• Training Deep Nets with Sublinear Memory Cost
• Semi-supervised Sequence Learning
• Big Self-Supervised Models are Strong Semi-Supervised Learners
• Underspecification Presents Challenges for Credibility in Modern Machine Learning
• Deep Learning Scaling is Predictable, Empirically
• PlaNet - Photo Geolocation with Convolutional Neural Networks
• Do ImageNet Classifiers Generalize to ImageNet?
• A Short Note on the Kinetics-700-2020 Human Action Dataset
• The Natural Language Decathlon: Multitask Learning as Question Answering
• Release Strategies and the Social Impacts of Language Models
• Large-Scale Adversarial Training for Vision-and-Language Representation Learning
• Contrastive Learning of Medical Visual Representations from Paired Images and Text
• Learning Robust Global Representations by Penalizing Local Predictive Power
• LXMERT: Learning Cross-Modality Encoder Representations from Transformers
• Self-Supervised MultiModal Versatile Networks
• On Empirical Comparisons of Optimizers for Deep Learning
• UNITER: UNiversal Image-TExt Representation Learning
• Natural Adversarial Examples
• Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination
• Measuring Robustness to Natural Distribution Shifts in Image Classification
• A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark
• Learning and Evaluating General Linguistic Intelligence
• ImageBERT: Cross-modal Pre-training with Large-scale Weak-supervised Image-Text Data
• The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes
• FairFace: Face Attribute Dataset for Balanced Race, Gender, and Age
• Making Pre-trained Language Models Better Few-shot Learners
• ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graph
• Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training
• Jukebox: A Generative Model for Music
• Language Understanding for Text-based Games Using Deep Reinforcement Learning
• Deep Structured Output Learning for Unconstrained Text Recognition
• Rethinking Few-Shot Image Classification: a Good Embedding Is All You Need?
• Generating Wikipedia by Summarizing Long Sequences
• A Multimodal Framework for the Detection of Hateful Memes
• ShapeWorld - A new test methodology for multimodal language understanding
• The Effect of Natural Distribution Shift on Question Answering Models
• TAP: Text-Aware Pre-training for Text-VQA and Text-Caption
• Learning Video Representations from Textual Web Supervision
• RareAct: A video dataset of unusual interactions
• Exposing and Correcting the Gender Bias in Image Captioning Datasets and Models

Cited by in corpus (425)

• A Survey on Visual Transformer
• Transformers in Vision: A Survey
• Learning to Prompt for Vision-Language Models
• On the Opportunities and Risks of Foundation Models
• Diffusion Models Beat GANs on Image Synthesis
• Evaluating Large Language Models Trained on Code
• Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
• Zero-Shot Text-to-Image Generation
• SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers
• Align before Fuse: Vision and Language Representation Learning with Momentum Distillation
• ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision
• Self-Supervised Representation Learning: Introduction, Advances and Challenges
• Deep Industrial Image Anomaly Detection: A Survey
• LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs
• SimVLM: Simple Visual Language Model Pretraining with Weak Supervision
• Generalized Out-of-Distribution Detection: A Survey
• Easily Accessible Text-to-Image Generation Amplifies Demographic Stereotypes at Large Scale
• Towards artificial general intelligence via a multimodal foundation model
• Open-vocabulary Object Detection via Vision and Language Knowledge Distillation
• VLP: A Survey on Vision-Language Pre-training
• Making the Most of Text Semantics to Improve Biomedical Vision--Language Processing
• Auditing large language models: a three-layered approach
• Decoding speech perception from non-invasive brain recordings
• TidyBot: Personalized Robot Assistance with Large Language Models
• FILIP: Fine-grained Interactive Language-Image Pre-Training
• Segment Anything Is Not Always Perfect: An Investigation of SAM on Different Real-world Applications
• WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learning
• You Only Look at One Sequence: Rethinking Transformer in Vision through Object Detection
• True Few-Shot Learning with Language Models
• CLIP-Mesh: Generating textured meshes from text using pretrained image-text models
• Unifying Vision-and-Language Tasks via Text Generation
• ActionCLIP: A New Paradigm for Video Action Recognition
• StyleNeRF: A Style-based 3D-Aware Generator for High-resolution Image Synthesis
• Multimodal datasets: misogyny, pornography, and malignant stereotypes
• How Much Can CLIP Benefit Vision-and-Language Tasks?
• Swin Transformer V2: Scaling Up Capacity and Resolution
• CLIP2Video: Mastering Video-Text Retrieval via Image CLIP
• Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigm
• Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling
• AdaCLIP: Adapting CLIP with Hybrid Learnable Prompts for Zero-Shot Anomaly Detection
• OCRBench: On the Hidden Mystery of OCR in Large Multimodal Models
• AI Art in Architecture
• CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval
• SuperAnimal pretrained pose estimation models for behavioral analysis
• Exploring the Limits of Out-of-Distribution Detection
• PromptCharm: Text-to-Image Generation through Multi-modal Prompting and Refinement
• MMRec: Simplifying Multimodal Recommendation
• CLIP-Art: Contrastive Pre-training for Fine-Grained Art Classification
• CLIPort: What and Where Pathways for Robotic Manipulation
• CycleMLP: A MLP-like Architecture for Dense Prediction
• Pretrained Transformers as Universal Computation Engines
• Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Concepts
• Generative Artificial Intelligence in Learning Analytics: Contextualising Opportunities and Challenges through the Learning Analytics Cycle
• PanGu-$\alpha$: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computation
• A Foundation Language-Image Model of the Retina (FLAIR): Encoding Expert Knowledge in Text Supervision
• CLIPDraw: Exploring Text-to-Drawing Synthesis through Language-Image Encoders
• VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modeling
• Unsolved Problems in ML Safety
• CPT: Colorful Prompt Tuning for Pre-trained Vision-Language Models
• Multimodal Few-Shot Learning with Frozen Language Models
• WenLan: Bridging Vision and Language by Large-Scale Multi-Modal Pre-Training
• Efficient Self-supervised Vision Transformers for Representation Learning
• Supervised Compression for Resource-Constrained Edge Computing Systems
• GODIVA: Generating Open-DomaIn Videos from nAtural Descriptions
• LLM-driven Multimodal Target Volume Contouring in Radiation Oncology
• XAIR: A Framework of Explainable AI in Augmented Reality
• A Simple Fix to Mahalanobis Distance for Improving Near-OOD Detection
• CLIP-Fields: Weakly Supervised Semantic Fields for Robotic Memory
• Exploring scalable medical image encoders beyond text supervision
• Learning from models beyond fine-tuning
• Improving Video-Text Retrieval by Multi-Stream Corpus Alignment and Dual Softmax Loss
• Towards a Holistic Framework for Multimodal Large Language Models in Three-dimensional Brain CT Report Generation
• Partial success in closing the gap between human and machine vision
• Bidirectional Generation of Structure and Properties Through a Single Molecular Foundation Model
• Enhancing Representation in Radiography-Reports Foundation Model: A Granular Alignment Algorithm Using Masked Contrastive Learning
• Open-Vocabulary SAM: Segment and Recognize Twenty-thousand Classes Interactively
• Multiscale Vision Transformers
• ROCOv2: Radiology Objects in COntext Version 2, an Updated Multimodal Image Dataset
• MERLOT: Multimodal Neural Script Knowledge Models
• RGB$\leftrightarrow$X: Image decomposition and synthesis using material- and lighting-aware diffusion models
• ImageBART: Bidirectional Context with Multinomial Diffusion for Autoregressive Image Synthesis
• Retrieval-Enhanced Machine Learning
• Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modeling
• LambdaNetworks: Modeling Long-Range Interactions Without Attention
• From large language models to multimodal AI: A scoping review on the potential of generative AI in medicine
• Underwater-Art: Expanding Information Perspectives With Text Templates For Underwater Acoustic Target Recognition
• What Makes Multi-modal Learning Better than Single (Provably)
• Subspace Regularizers for Few-Shot Class Incremental Learning
• Container: Context Aggregation Network
• Understanding Collaborative Practices and Tools of Professional UX Practitioners in Software Organizations
• Vector Quantized Diffusion Model for Text-to-Image Synthesis
• C$\cdot$ASE: Learning Conditional Adversarial Skill Embeddings for Physics-based Characters
• VALUE: A Multi-Task Benchmark for Video-and-Language Understanding Evaluation
• DenseCLIP: Language-Guided Dense Prediction with Context-Aware Prompting
• LipLearner: Customizable Silent Speech Interactions on Mobile Devices
• LAFITE: Towards Language-Free Training for Text-to-Image Generation
• Unifying Multimodal Transformer for Bi-directional Image and Text Generation
• Adversarial Retriever-Ranker for dense text retrieval
• Mobile Foundation Model as Firmware
• Evaluating CLIP: Towards Characterization of Broader Capabilities and Downstream Implications
• Exploring the Limits of Large Scale Pre-training
• Pre-trained Language Models in Biomedical Domain: A Systematic Survey
• TypeDance: Creating Semantic Typographic Logos from Image through Personalized Generation
• Lightweight Transformers for Clinical Natural Language Processing
• The Clever Hans Effect in Unsupervised Learning
• VIMPAC: Video Pre-Training via Masked Token Prediction and Contrastive Learning
• UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learning
• RGB-T Tracking Based on Mixed Attention
• The Future of Intelligent Healthcare: A Systematic Analysis and Discussion on the Integration and Impact of Robots Using Large Language Models for Healthcare
• How Does Fine-Tuning Impact Out-of-Distribution Detection for Vision-Language Models?
• Clinical Insights: A Comprehensive Review of Language Models in Medicine
• CLOOB: Modern Hopfield Networks with InfoLOOB Outperform CLIP
• RepsNet: Combining Vision with Language for Automated Medical Reports
• Quantum circuit synthesis with diffusion models
• UFO: A UniFied TransfOrmer for Vision-Language Representation Learning
• Neural Symbolic Regression that Scales
• DanceGen: Supporting Choreography Ideation and Prototyping with Generative AI
• When Robots Get Chatty: Grounding Multimodal Human-Robot Conversation and Collaboration
• Scaling Laws for Transfer
• Generating Procedural Materials from Text or Image Prompts
• Cross-Lingual Cross-Modal Retrieval with Noise-Robust Learning
• A Text-guided Protein Design Framework
• Predictive Coding: a Theoretical and Experimental Review
• CLIP-Lung: Textual Knowledge-Guided Lung Nodule Malignancy Prediction
• Deep Learning Technique for Human Parsing: A Survey and Outlook
• CrossA11y: Identifying Video Accessibility Issues via Cross-modal Grounding
• Sparse Visual Counterfactual Explanations in Image Space
• Diffusion-Based Co-Speech Gesture Generation Using Joint Text and Audio Representation
• VisionTasker: Mobile Task Automation Using Vision Based UI Understanding and LLM Task Planning
• GANzilla: User-Driven Direction Discovery in Generative Adversarial Networks
• LanguageRefer: Spatial-Language Model for 3D Visual Grounding
• Towards Open-World Text-Guided Face Image Generation and Manipulation
• QVHighlights: Detecting Moments and Highlights in Videos via Natural Language Queries
• Human-like object concept representations emerge naturally in multimodal large language models
• Jigsaw: Supporting Designers to Prototype Multimodal Applications by Chaining AI Foundation Models
• Algorithmic Ways of Seeing: Using Object Detection to Facilitate Art Exploration
• The Dawn of AI-Native EDA: Opportunities and Challenges of Large Circuit Models
• Significantly improving zero-shot X-ray pathology classification via fine-tuning pre-trained image-text encoders
• CLIP-ReIdent: Contrastive Training for Player Re-Identification
• The Evolution of Out-of-Distribution Robustness Throughout Fine-Tuning
• Audio Description Customization
• ELEV-VISION-SAM: Integrated Vision Language and Foundation Model for Automated Estimation of Building Lowest Floor Elevation
• Masked Vision-Language Transformer in Fashion
• PathM3: A Multimodal Multi-Task Multiple Instance Learning Framework for Whole Slide Image Classification and Captioning
• Multimodal Self-Supervised Learning of General Audio Representations
• VTG-GPT: Tuning-Free Zero-Shot Video Temporal Grounding with GPT
• SafeGen: Mitigating Sexually Explicit Content Generation in Text-to-Image Models
• Crossing the principle-practice gap in AI ethics with ethical problem-solving
• Poisoning and Backdooring Contrastive Learning
• Open-Vocabulary Animal Keypoint Detection with Semantic-feature Matching
• Breaking with Fixed Set Pathology Recognition through Report-Guided Contrastive Training
• Contrastive Attention for Automatic Chest X-ray Report Generation
• TexSliders: Diffusion-Based Texture Editing in CLIP Space
• ClipMatrix: Text-controlled Creation of 3D Textured Meshes
• FabGPT: An Efficient Large Multimodal Model for Complex Wafer Defect Knowledge Queries
• Zero-Shot Text-Guided Object Generation with Dream Fields
• Language Grounding with 3D Objects
• Cross-Modal Retrieval for Motion and Text via DropTriple Loss
• On the effectiveness of adversarial training against common corruptions
• MixNorm: Test-Time Adaptation Through Online Normalization Estimation
• Towards Unsupervised Domain Adaptation via Domain-Transformer
• Object-aware Contrastive Learning for Debiased Scene Representation
• Mind the Gap: Domain Gap Control for Single Shot Domain Adaptation for Generative Adversarial Networks
• Multi-Label Image Classification with Contrastive Learning
• Joint Learning of Localized Representations from Medical Images and Reports
• Objects in Semantic Topology
• Latent Diffusion Models with Image-Derived Annotations for Enhanced AI-Assisted Cancer Diagnosis in Histopathology
• GANravel: User-Driven Direction Disentanglement in Generative Adversarial Networks
• Hierarchical Matching and Reasoning for Multi-Query Image Retrieval
• CLIP-It! Language-Guided Video Summarization
• Situating the social issues of image generation models in the model life cycle: a sociotechnical approach
• RadEdit: stress-testing biomedical vision models via diffusion image editing
• Multi-modal Self-supervised Pre-training for Regulatory Genome Across Cell Types
• Morig: Motion-aware rigging of character meshes from point clouds
• Credible, Unreliable or Leaked?: Evidence Verification for Enhanced Automated Fact-checking
• MURAL: Multimodal, Multitask Retrieval Across Languages
• Look at What I'm Doing: Self-Supervised Spatial Grounding of Narrations in Instructional Videos
• Synthetic images aid the recognition of human-made art forgeries
• Modeling Text-visual Mutual Dependency for Multi-modal Dialog Generation
• Rethinking model prototyping through the MedMNIST+ dataset collection
• Soundify: Matching Sound Effects to Video
• Giving Robots a Voice: Human-in-the-Loop Voice Creation and open-ended Labeling
• Towards Interpretable Radiology Report Generation via Concept Bottlenecks using a Multi-Agentic RAG
• ClipCap: CLIP Prefix for Image Captioning
• Large-vocabulary forensic pathological analyses via prototypical cross-modal contrastive learning
• Relax: Composable Abstractions for End-to-End Dynamic Machine Learning
• Differentiable Physics: A Position Piece
• Data Augmentation in Earth Observation: A Diffusion Model Approach
• RetAssist: Facilitating Vocabulary Learners with Generative Images in Story Retelling Practices
• Tight Mutual Information Estimation With Contrastive Fenchel-Legendre Optimization
• Parameter-Efficient Fine-Tuning of Large Pretrained Models for Instance Segmentation Tasks
• PanoGen++: Domain-Adapted Text-Guided Panoramic Environment Generation for Vision-and-Language Navigation
• Who's in and who's out? A case study of multimodal CLIP-filtering in DataComp
• MedPromptX: Grounded Multimodal Prompting for Chest X-ray Diagnosis
• Collection Space Navigator: An Interactive Visualization Interface for Multidimensional Datasets
• Brain Tumor Synthetic Data Generation with Adaptive StyleGANs
• A Simple Long-Tailed Recognition Baseline via Vision-Language Model
• Temperature as Uncertainty in Contrastive Learning
• INTERN: A New Learning Paradigm Towards General Vision
• Vision Beyond Boundaries: An Initial Design Space of Domain-specific Large Vision Models in Human-robot Interaction
• Simpler, Faster, Stronger: Breaking The log-K Curse On Contrastive Learners With FlatNCE
• Towards Neuro-Symbolic Video Understanding
• A Pre-trained Sequential Recommendation Framework: Popularity Dynamics for Zero-shot Transfer
• AI Illustrator: Translating Raw Descriptions into Images by Prompt-based Cross-Modal Generation
• StreetSurfaceVis: a dataset of crowdsourced street-level imagery annotated by road surface type and quality
• Generative Art Using Neural Visual Grammars and Dual Encoders
• Cross-modal feature fusion for robust point cloud registration with ambiguous geometry
• FairyTailor: A Multimodal Generative Framework for Storytelling
• Unleashing the Potential of Pre-Trained Diffusion Models for Generalizable Person Re-Identification
• VIINTER: View Interpolation with Implicit Neural Representations of Images
• Generative AI-based Prompt Evolution Engineering Design Optimization With Vision-Language Model
• Exploring AI-assisted Ideation and Prototyping for Choreography
• Induce, Edit, Retrieve: Language Grounded Multimodal Schema for Instructional Video Retrieval
• Data-Efficient Pretraining via Contrastive Self-Supervision
• Generalized SAM: Efficient Fine-Tuning of SAM for Variable Input Image Sizes
• Synthetic Boost: Leveraging Synthetic Data for Enhanced Vision-Language Segmentation in Echocardiography
• Multimodal Approach for Harmonized System Code Prediction
• MaIL: A Unified Mask-Image-Language Trimodal Network for Referring Image Segmentation
• CUPID: Adaptive Curation of Pre-training Data for Video-and-Language Representation Learning
• SORNet: Spatial Object-Centric Representations for Sequential Manipulation
• Video-Text Pre-training with Learned Regions
• A CLIP-Enhanced Method for Video-Language Understanding
• Depth Any Canopy: Leveraging Depth Foundation Models for Canopy Height Estimation
• Web Archives Metadata Generation with GPT-4o: Challenges and Insights
• AI vs. Human Paintings? Deciphering Public Interactions and Perceptions towards AI-Generated Paintings on TikTok
• SniffyArt: The Dataset of Smelling Persons
• Evaluating the Fairness of Discriminative Foundation Models in Computer Vision
• Active Divergence with Generative Deep Learning -- A Survey and Taxonomy
• Cerberus: Attribute-based person re-identification using semantic IDs
• EfficientCLIP: Efficient Cross-Modal Pre-training by Ensemble Confident Learning and Language Modeling
• Large-Scale Zero-Shot Image Classification from Rich and Diverse Textual Descriptions
• Training-free Zero-shot Composed Image Retrieval via Weighted Modality Fusion and Similarity
• IDEA: Increasing Text Diversity via Online Multi-Label Recognition for Vision-Language Pre-training
• Self-supervised Semi-supervised Learning for Data Labeling and Quality Evaluation
• SEM-CLIP: Precise Few-Shot Learning for Nanoscale Defect Detection in Scanning Electron Microscope Image
• Delving into E-Commerce Product Retrieval with Vision-Language Pre-training
• VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding
• Collective Intelligence for Deep Learning: A Survey of Recent Developments
• Positive Sample Propagation along the Audio-Visual Event Line
• Unleashing Transformers: Parallel Token Prediction with Discrete Absorbing Diffusion for Fast High-Resolution Image Generation from Vector-Quantized Codes
• Semantically Guided Representation Learning For Action Anticipation
• LightningDOT: Pre-training Visual-Semantic Embeddings for Real-Time Image-Text Retrieval
• EBDM: Exemplar-guided Image Translation with Brownian-bridge Diffusion Models
• Towards General Purpose Vision Systems
• Robust Optimization for Multilingual Translation with Imbalanced Data
• Proof Artifact Co-training for Theorem Proving with Language Models
• VideoMap: Supporting Video Editing Exploration, Brainstorming, and Prototyping in the Latent Space
• Meme Sentiment Analysis Enhanced with Multimodal Spatial Encoding and Facial Embedding
• Embodied BERT: A Transformer Model for Embodied, Language-guided Visual Task Completion
• Adaptive Parametric Activation: Unifying and Generalising Activation Functions Across Tasks
• On the Expressive Power of Self-Attention Matrices
• Foundation Models for the Digital Twin Creation of Cyber-Physical Systems
• IMUDiffusion: A Diffusion Model for Multivariate Time Series Synthetisation for Inertial Motion Capturing Systems
• Multi-label Cluster Discrimination for Visual Representation Learning
• Tackling Copyright Issues in AI Image Generation Through Originality Estimation and Genericization
• Paint4Poem: A Dataset for Artistic Visualization of Classical Chinese Poems
• No One Representation to Rule Them All: Overlapping Features of Training Methods
• On the Value of Labeled Data and Symbolic Methods for Hidden Neuron Activation Analysis
• AlignZeg: Mitigating Objective Misalignment for Zero-shot Semantic Segmentation
• Don't waste SAM
• ReCLIP++: Learn to Rectify the Bias of CLIP for Unsupervised Semantic Segmentation
• Rethinking Person Re-Identification via Semantic-Based Pretraining
• Future Illiteracies -- Architectural Epistemology and Artificial Intelligence
• Generalized Decision Transformer for Offline Hindsight Information Matching
• SoK: How Robust is Image Classification Deep Neural Network Watermarking? (Extended Version)
• Lidar Panoptic Segmentation in an Open World
• BiPC: Bidirectional Probability Calibration for Unsupervised Domain Adaption
• MMInstruct: A High-Quality Multi-Modal Instruction Tuning Dataset with Extensive Diversity
• Comprehensive Dataset of Synthetic and Manipulated Overhead Imagery for Development and Evaluation of Forensic Tools
• Optimal Linear Subspace Search: Learning to Construct Fast and High-Quality Schedulers for Diffusion Models
• A New Perspective to Boost Performance Fairness for Medical Federated Learning
• Parsing Objects at a Finer Granularity: A Survey
• GAN Cocktail: mixing GANs without dataset access
• An Image is Worth More Than a Thousand Words: Towards Disentanglement in the Wild
• Gaze-Guided Graph Neural Network for Action Anticipation Conditioned on Intention
• User-Aware Prefix-Tuning is a Good Learner for Personalized Image Captioning
• Empowering Morphing Attack Detection using Interpretable Image-Text Foundation Model
• Segmentation in Style: Unsupervised Semantic Image Segmentation with Stylegan and CLIP
• 3D Object Detection and High-Resolution Traffic Parameters Extraction Using Low-Resolution LiDAR Data
• PEAR: Phrase-Based Hand-Object Interaction Anticipation
• Symbolic Behaviour in Artificial Intelligence
• Attention Guidance through Video Script: A Case Study of Object Focusing on 360° VR Video Tours
• Using Multiple Instance Learning to Build Multimodal Representations
• Robustness Tokens: Towards Adversarial Robustness of Transformers
• fruit-SALAD: A Style Aligned Artwork Dataset to reveal similarity perception in image embeddings
• CLIP4Caption ++: Multi-CLIP for Video Caption
• MV-MR: multi-views and multi-representations for self-supervised learning and knowledge distillation
• Scalable Clustering: Large Scale Unsupervised Learning of Gaussian Mixture Models with Outliers
• GazeFusion: Saliency-Guided Image Generation
• Visual Conceptual Blending with Large-scale Language and Vision Models
• Is Object Detection Necessary for Human-Object Interaction Recognition?
• FovEx: Human-Inspired Explanations for Vision Transformers and Convolutional Neural Networks
• Automating Generative Deep Learning for Artistic Purposes: Challenges and Opportunities
• Exploring the Transferability of a Foundation Model for Fundus Images: Application to Hypertensive Retinopathy
• Winning the ICCV'2021 VALUE Challenge: Task-aware Ensemble and Transfer Learning with Visual Concepts
• A Fine-Grained Analysis on Distribution Shift
• Boon: A Neural Search Engine for Cross-Modal Information Retrieval
• Billion-scale Similarity Search Using a Hybrid Indexing Approach with Advanced Filtering
• Cluster-based Video Summarization with Temporal Context Awareness
• Clarify: Improving Model Robustness With Natural Language Corrections
• Controllable Navigation Instruction Generation with Chain of Thought Prompting
• Centered Masking for Language-Image Pre-Training
• Improved Transformer for High-Resolution GANs
• Annotation-Free Human Sketch Quality Assessment
• BEAMetrics: A Benchmark for Language Generation Evaluation Evaluation
• Telling Creative Stories Using Generative Visual Aids
• Habits and goals in synergy: a variational Bayesian framework for behavior
• Auditing AI models for Verified Deployment under Semantic Specifications
• Connecting Language and Vision for Natural Language-Based Vehicle Retrieval
• Videogenic: Identifying Highlight Moments in Videos with Professional Photographs as a Prior
• Less is More: Sparse Sampling for Dense Reaction Predictions
• Rethinking Open-Set Object Detection: Issues, a New Formulation, and Taxonomy
• VidLanKD: Improving Language Understanding via Video-Distilled Knowledge Transfer
• Multi-label Iterated Learning for Image Classification with Label Ambiguity
• Qu-ANTI-zation: Exploiting Quantization Artifacts for Achieving Adversarial Outcomes
• EncoderMI: Membership Inference against Pre-trained Encoders in Contrastive Learning
• Evolving Evocative 2D Views of Generated 3D Objects
• Predict, Prevent, and Evaluate: Disentangled Text-Driven Image Manipulation Empowered by Pre-Trained Vision-Language Model
• Pointer Value Retrieval: A new benchmark for understanding the limits of neural network generalization
• Modern Hopfield Networks for Few- and Zero-Shot Reaction Template Prediction
• Pri3D: Can 3D Priors Help 2D Representation Learning?
• FoodMLLM-JP: Leveraging Multimodal Large Language Models for Japanese Recipe Generation
• Are CLIP features all you need for Universal Synthetic Image Origin Attribution?
• Intelligent Anomaly Detection for Lane Rendering Using Transformer with Self-Supervised Pre-Training and Customized Fine-Tuning
• ViSeRet: A simple yet effective approach to moment retrieval via fine-grained video segmentation
• MAP-Elites with Transverse Assessment for Multimodal Problems in Creative Domains
• Benchmarking histopathology foundation models in a multi-center dataset for skin cancer subtyping
• Structure-Enhanced Protein Instruction Tuning: Towards General-Purpose Protein Understanding with LLMs
• Scaling Law for Recommendation Models: Towards General-purpose User Representations
• Vision Language Models as Values Detectors
• Pretrained Encoders are All You Need
• Media Content Atlas: A Pipeline to Explore and Investigate Multidimensional Media Space using Multimodal LLMs
• Net2Brain: A Toolbox to compare artificial vision models with human brain responses
• Pseudo-Labeling Enhanced by Privileged Information and Its Application to In Situ Sequencing Images
• SOOD-ImageNet: a Large-Scale Dataset for Semantic Out-Of-Distribution Image Classification and Semantic Segmentation
• SituationalLLM: Proactive language models with scene awareness for dynamic, contextual task guidance
• Integrating Auxiliary Information in Self-supervised Learning
• Dynamic Exploration Graph: A Novel Approach for Efficient Nearest Neighbor Search in Evolving Multimedia Datasets
• Designing A Sustainable Marine Debris Clean-up Framework without Human Labels
• What Matters for Ad-hoc Video Search? A Large-scale Evaluation on TRECVID
• EVOQUER: Enhancing Temporal Grounding with Video-Pivoted BackQuery Generation
• Does Vision-and-Language Pretraining Improve Lexical Grounding?
• L-Verse: Bidirectional Generation Between Image and Text
• Investigating the diversity and stylization of contemporary user generated visual arts in the complexity entropy plane
• Data-Efficient Language-Supervised Zero-Shot Learning with Self-Distillation
• Level Up Your Tutorials: VLMs for Game Tutorials Quality Assessment
• StyleCLIPDraw: Coupling Content and Style in Text-to-Drawing Synthesis
• Annotating Satellite Images of Forests with Keywords from a Specialized Corpus in the Context of Change Detection
• Zero-Shot Information Extraction as a Unified Text-to-Triple Translation
• Fréchet Denoised Distance: Enhancing Plausibility Evaluation for Generated Designs with Denoising Autoencoder
• Why Do Better Loss Functions Lead to Less Transferable Features?
• 10 Security and Privacy Problems in Large Foundation Models
• Sparse MoEs meet Efficient Ensembles
• Improving Lossless Compression Rates via Monte Carlo Bits-Back Coding
• Controllable 3D Object Generation with Single Image Prompt
• Steering Large Text-to-Image Model for Abstract Art Synthesis: Preference-based Prompt Optimization and Visualization
• Trivial or impossible -- dichotomous data difficulty masks model differences (on ImageNet and beyond)
• Reading Isn't Believing: Adversarial Attacks On Multi-Modal Neurons
• A Retrospective of the Tutorial on Opportunities and Challenges of Online Deep Learning
• D2C: Diffusion-Denoising Models for Few-shot Conditional Generation
• eProduct: A Million-Scale Visual Search Benchmark to Address Product Recognition Challenges
• Will Multi-modal Data Improves Few-shot Learning?
• Planning Multimodal Exploratory Actions for Online Robot Attribute Learning
• Toward Foundation Models for Earth Monitoring: Proposal for a Climate Change Benchmark
• Personalizing Pre-trained Models
• CLOP: Video-and-Language Pre-Training with Knowledge Regularizations
• Contrastive Fine-tuning Improves Robustness for Neural Rankers
• Multi-modality fusion using canonical correlation analysis methods: Application in breast cancer survival prediction from histology and genomics
• An Overview on Generative AI at Scale with Edge-Cloud Computing
• One to Transfer All: A Universal Transfer Framework for Vision Foundation Model with Few Data
• EMID: An Emotional Aligned Dataset in Audio-Visual Modality
• Bridge Diffusion Model: Bridge Chinese Text-to-Image Diffusion Model with English Communities
• Achieving Human Parity on Visual Question Answering
• Combating Label Noise With A General Surrogate Model For Sample Selection
• Product1M: Towards Weakly Supervised Instance-Level Product Retrieval via Cross-modal Pretraining
• Learning Co-Speech Gesture Representations in Dialogue through Contrastive Learning: An Intrinsic Evaluation
• Can masking background and object reduce static bias for zero-shot action recognition?
• TPA3D: Triplane Attention for Fast Text-to-3D Generation
• Physically Interpretable Probabilistic Domain Characterization
• Characterizing Generalization under Out-Of-Distribution Shifts in Deep Metric Learning
• Self-Distilled Self-Supervised Representation Learning
• VL-LTR: Learning Class-wise Visual-Linguistic Representation for Long-Tailed Visual Recognition
• VisualSem: A High-quality Knowledge Graph for Vision and Language
• Cross-Modal Retrieval Augmentation for Multi-Modal Classification
• Inverse Problems Leveraging Pre-trained Contrastive Representations
• Cut the CARP: Fishing for zero-shot story evaluation
• Unsupervised Source Separation By Steering Pretrained Music Models
• Artistic Autonomy in AI Art
• Scaling Laws for the Few-Shot Adaptation of Pre-trained Image Classifiers
• CLIP4Caption: CLIP for Video Caption
• Embed Everything: A Method for Efficiently Co-Embedding Multi-Modal Spaces
• An animated picture says at least a thousand words: Selecting Gif-based Replies in Multimodal Dialog
• MOMENTA: A Multimodal Framework for Detecting Harmful Memes and Their Targets
• Few-Shot Intent Detection via Contrastive Pre-Training and Fine-Tuning
• Data Efficient Masked Language Modeling for Vision and Language
• Aligning Cross-lingual Sentence Representations with Dual Momentum Contrast
• Fine-Grained Chemical Entity Typing with Multimodal Knowledge Representation
• Language Models as Zero-shot Visual Semantic Learners
• Training Efficiency and Robustness in Deep Learning
• Ownership and Creativity in Generative Models
• Hypernymization of named entity-rich captions for grounding-based multi-modal pretraining
• MTFusion: Reconstructing Any 3D Object from Single Image Using Multi-word Textual Inversion
• POET: Prompt Offset Tuning for Continual Human Action Adaptation
• Spline-based Transformers
• Temporal Image Caption Retrieval Competition -- Description and Results
• Image-text matching for large-scale book collections
• Balancing Performance and Efficiency in Zero-shot Robotic Navigation
• Knowledge graphs for empirical concept retrieval
• DiffAugment: Diffusion based Long-Tailed Visual Relationship Recognition
• Using Saliency and Cropping to Improve Video Memorability
• Mutltimodal AI Companion for Interactive Fairytale Co-creation
• SpaceEdit: Learning a Unified Editing Space for Open-Domain Image Editing
• An implementation of the "Guess who?" game using CLIP
• ReGroup: Recursive Neural Networks for Hierarchical Grouping of Vector Graphic Primitives
• Discriminator Synthesis: On reusing the other half of Generative Adversarial Networks
• DIVA: Dataset Derivative of a Learning Task
• TransAug: Translate as Augmentation for Sentence Embeddings
• Human in the Loop for Machine Creativity
• Shared Visual Representations of Drawing for Communication: How do different biases affect human interpretability and intent?
• Learning to See by Looking at Noise
• Kernel Identification Through Transformers
• Survey of Visual-Semantic Embedding Methods for Zero-Shot Image Retrieval
• Axiomatic Explanations for Visual Search, Retrieval, and Similarity Learning
• What is Multimodality?
• Evo* 2020 -- Late-Breaking Abstracts Volume