12 citations · 37 across the 13 of their papers we have counts for
13 papers
To Cool or not to Cool? Temperature Network Meets Large Foundation Models via DRO
Zi-Hao Qiu, Siqi Guo, Mao Xu +3
The temperature parameter plays a profound role during training and/or inference with large foundation models (LFMs) such as large language models (LLMs) and CLIP models. Particula…
Efficient Methods for Non-stationary Online Learning
Peng Zhao, Yan-Feng Xie, Lijun Zhang +1
Non-stationary online learning has drawn much attention in recent years. In particular, dynamic regret and adaptive regret are proposed as two principled performance measures for o…
Learning Unnormalized Statistical Models via Compositional Optimization
Wei Jiang, Jiayu Qin, Lingyu Wu +3
Learning unnormalized statistical models (e.g., energy-based models) is computationally challenging due to the complexity of handling the partition function. To eschew this complex…
Blockwise Stochastic Variance-Reduced Methods with Parallel Speedup for Multi-Block Bilevel Optimization
Quanqi Hu, Zi-Hao Qiu, Zhishuai Guo +2
In this paper, we consider non-convex multi-block bilevel optimization (MBBO) problems, which involve lower level problems and have important applications in machine learn…
Not All Semantics are Created Equal: Contrastive Self-supervised Learning with Automatic Temperature Individualization
Zi-Hao Qiu, Quanqi Hu, Zhuoning Yuan +3
In this paper, we aim to optimize a contrastive loss with individualized temperatures in a principled and systematic manner for self-supervised learning. The common practice of usi…
Stochastic Approximation Approaches to Group Distributionally Robust Optimization and Beyond
Lijun Zhang, Haomin Bai, Peng Zhao +2
This paper investigates group distributionally robust optimization (GDRO) with the goal of learning a model that performs well over different distributions. First, we formulate…
Multi-block-Single-probe Variance Reduced Estimator for Coupled Compositional Optimization
Wei Jiang, Gang Li, Yibo Wang +2
Variance reduction techniques such as SPIDER/SARAH/STORM have been extensively studied to improve the convergence rates of stochastic non-convex optimization, which usually maintai…
Smoothed Online Convex Optimization Based on Discounted-Normal-Predictor
Lijun Zhang, Wei Jiang, Jinfeng Yi +1
In this paper, we investigate an online prediction strategy named as Discounted-Normal-Predictor (Kapralov and Panigrahy, 2010) for smoothed online convex optimization (SOCO), in w…
Provable Stochastic Optimization for Global Contrastive Learning: Small Batch Does Not Harm Performance
Zhuoning Yuan, Yuexin Wu, Zi-Hao Qiu +4
In this paper, we study contrastive learning from an optimization perspective, aiming to analyze and address a fundamental issue of existing contrastive learning methods that eithe…
Large-scale Stochastic Optimization of NDCG Surrogates for Deep Learning with Provable Convergence
Zi-Hao Qiu, Quanqi Hu, Yongjian Zhong +2
NDCG, namely Normalized Discounted Cumulative Gain, is a widely used ranking metric in information retrieval and machine learning. However, efficient and provable stochastic method…
Optimal Algorithms for Stochastic Multi-Level Compositional Optimization
Wei Jiang, Bokun Wang, Yibo Wang +2
In this paper, we investigate the problem of stochastic multi-level compositional optimization, where the objective function is a composition of multiple smooth but possibly non-co…
Adaptivity and Non-stationarity: Problem-dependent Dynamic Regret for Online Convex Optimization
Peng Zhao, Yu-Jie Zhang, Lijun Zhang +1
We investigate online convex optimization in non-stationary environments and choose dynamic regret as the performance measure, defined as the difference between cumulative loss inc…
Fast Rates of ERM and Stochastic Approximation: Adaptive to Error Bound Conditions
Mingrui Liu, Xiaoxuan Zhang, Lijun Zhang +2
Error bound conditions (EBC) are properties that characterize the growth of an objective function when a point is moved away from the optimal set. They have recently received incre…