257 citations · 437 across the 40 of their papers we have counts for
23 papers · 1 filter
Provable Training Data Identification for Large Language Models
Zhenlong Liu, Hao Zeng, Weiran Huang +1
Identifying training data of large-scale models is critical for copyright litigation, privacy auditing, and ensuring fair evaluation. However, existing works typically treat this t…
Information-Theoretic Perspectives on Optimizers
Zhiquan Tan, Weiran Huang
The interplay of optimizers and architectures in neural networks is complicated and hard to understand why some optimizers work better on some specific architectures. In this paper…
ATLAS: Adapter-Based Multi-Modal Continual Learning with a Two-Stage Learning Strategy
Hong Li, Zhiquan Tan, Xingyu Li +1
While vision-and-language models significantly advance in many fields, the challenge of continual learning is unsolved. Parameter-efficient modules like adapters and prompts presen…
Exploring Information-Theoretic Metrics Associated with Neural Collapse in Supervised Training
Kun Song, Zhiquan Tan, Bochao Zou +3
In this paper, we introduce matrix entropy as an analytical tool for studying supervised learning, investigating the information content of data representations and classification…
Unveiling the Dynamics of Information Interplay in Supervised Learning
Kun Song, Zhiquan Tan, Bochao Zou +2
In this paper, we use matrix information theory as an analytical tool to analyze the dynamics of the information interplay between data representations and classification head vect…
Provable Contrastive Continual Learning
Yichen Wen, Zhiquan Tan, Kaipeng Zheng +2
Continual learning requires learning incremental tasks with dynamic data distributions. So far, it has been observed that employing a combination of contrastive loss and distillati…
The Information of Large Language Model Geometry
Zhiquan Tan, Chenghai Li, Weiran Huang
This paper investigates the information encoded in the embeddings of large language models (LLMs). We conduct simulations to analyze the representation entropy and discover a power…
Diff-eRank: A Novel Rank-Based Metric for Evaluating Large Language Models
Lai Wei, Zhiquan Tan, Chenghai Li +2
Large Language Models (LLMs) have transformed natural language processing and extended their powerful capabilities to multi-modal domains. As LLMs continue to advance, it is crucia…
Understanding Grokking Through A Robustness Viewpoint
Zhiquan Tan, Weiran Huang
Recently, an interesting phenomenon called grokking has gained much attention, where generalization occurs long after the models have initially overfitted the training data. We try…
MM-LIMA: Less Is More for Alignment in Multi-Modal Datasets
Lai Wei, Xiaozhe Li, Zihao Jiang +2
Multimodal large language models are typically trained in two stages: first pre-training on image-text pairs, and then fine-tuning using supervised vision-language instruction data…
FILM: How can Few-Shot Image Classification Benefit from Pre-Trained Language Models?
Zihao Jiang, Yunkai Dang, Dong Pang +2
Few-shot learning aims to train models that can be generalized to novel classes with only a few samples. Recently, a line of works are proposed to enhance few-shot learning with ac…
Rethinking Weak Supervision in Helping Contrastive Learning
Jingyi Cui, Weiran Huang, Yifei Wang +1
Contrastive learning has shown outstanding performances in both supervised and unsupervised learning, and has recently been introduced to solve weakly supervised learning problems…
Matrix Information Theory for Self-Supervised Learning
Yifan Zhang, Zhiquan Tan, Jingqin Yang +2
The maximum entropy encoding framework provides a unified perspective for many non-contrastive learning methods like SimSiam, Barlow Twins, and MEC. Inspired by this framework, we…
ArCL: Enhancing Contrastive Learning with Augmentation-Robust Representations
Xuyang Zhao, Tianqi Du, Yisen Wang +2
Self-Supervised Learning (SSL) is a paradigm that leverages unlabeled data for model training. Empirical studies show that SSL can achieve promising performance in distribution shi…
When Noisy Labels Meet Long Tail Dilemmas: A Representation Calibration Method
Manyi Zhang, Xuyang Zhao, Jun Yao +2
Real-world large-scale datasets are both noisily labeled and class-imbalanced. The issues seriously hurt the generalization of trained models. It is hence significant to address th…
Your Contrastive Learning Is Secretly Doing Stochastic Neighbor Embedding
Tianyang Hu, Zhili Liu, Fengwei Zhou +2
Contrastive learning, especially self-supervised contrastive learning (SSCL), has achieved great success in extracting powerful features from unlabeled data. In this work, we contr…
Towards the Generalization of Contrastive Self-Supervised Learning
Weiran Huang, Mingyang Yi, Xuyang Zhao +1
Recently, self-supervised learning has attracted great attention, since it only requires unlabeled data for model training. Contrastive learning is one popular method for self-supe…
Can Pretext-Based Self-Supervised Learning Be Boosted by Downstream Data? A Theoretical Analysis
Jiaye Teng, Weiran Huang, Haowei He
Pretext-based self-supervised learning learns the semantic representation via a handcrafted pretext task over unlabeled data and then uses the learned representation for downstream…
New Interpretations of Normalization Methods in Deep Learning
Jiacheng Sun, Xiangyong Cao, Hanwen Liang +3
In recent years, a variety of normalization methods have been proposed to help train neural networks, such as batch normalization (BN), layer normalization (LN), weight normalizati…
Locally Differentially Private (Contextual) Bandits Learning
Kai Zheng, Tianle Cai, Weiran Huang +2
We study locally differentially private (LDP) bandits learning in this paper. First, we propose simple black-box reduction frameworks that can solve a large family of context-free…
Meta-Learning PAC-Bayes Priors in Model Averaging
Yimin Huang, Weiran Huang, Liang Li +1
Nowadays model uncertainty has become one of the most important problems in both academia and industry. In this paper, we mainly consider the scenario in which we have a common mod…
Community Exploration: From Offline Optimization to Online Learning
Xiaowei Chen, Weiran Huang, Wei Chen +1
We introduce the community exploration problem that has many real-world applications such as online advertising. In the problem, an explorer allocates limited budget to explore com…
Combinatorial Pure Exploration with Continuous and Separable Reward Functions and Its Applications (Extended Version)
Weiran Huang, Jungseul Ok, Liang Li +1
We study the Combinatorial Pure Exploration problem with Continuous and Separable reward functions (CPE-CS) in the stochastic multi-armed bandit setting. In a CPE-CS instance, we a…