48 citations · 141 across the 26 of their papers we have counts for
27 papers
Rethinking Routing Exploration in Modern Mixture-of-Experts
Bingcong Li, Hao Ma, Michael Muehlebach +2
Mixture-of-experts (MoE) models have become an important approach for scaling model parameters. Modern MoE architectures increasingly combine two design choices: fine-grained exper…
Low-Rank Adaptation Redux for Large Models
Bingcong Li, Yilang Zhang, Georgios B. Giannakis
Low-rank adaptation (LoRA) has emerged as the de facto standard for parameter-efficient fine-tuning (PEFT) of foundation models, enabling the adaptation of billion-parameter networ…
ANCRe: Adaptive Neural Connection Reassignment for Efficient Depth Scaling
Yilang Zhang, Bingcong Li, Niao He +1
Scaling network depth has been a central driver behind the success of modern foundation models, yet recent investigations suggest that deep layers are often underutilized. This pap…
VASSO: Variance Suppression for Sharpness-Aware Minimization
Bingcong Li, Yilang Zhang, Georgios B. Giannakis
Sharpness-aware minimization (SAM) has well-documented merits in enhancing generalization of deep neural network models. Accounting for sharpness in the loss function geometry, whe…
Learnable Loss Geometries with Mirror Descent for Scalable and Convergent Meta-Learning
Yilang Zhang, Bingcong Li, Georgios B. Giannakis
Utilizing task-invariant knowledge acquired from related tasks as prior information, meta-learning offers a principled approach to learning a new task with limited data records. Sa…
RefLoRA: Refactored Low-Rank Adaptation for Efficient Fine-Tuning of Large Models
Yilang Zhang, Bingcong Li, Georgios B. Giannakis
Low-Rank Adaptation (LoRA) lowers the computational and memory overhead of fine-tuning large models by updating a low-dimensional subspace of the pre-trained weight matrix. Albeit…
Preconditioned Sharpness-Aware Minimization: Unifying Analysis and a Novel Learning Algorithm
Yilang Zhang, Bingcong Li, Georgios B. Giannakis
Targeting solutions over `flat' regions of the loss landscape, sharpness-aware minimization (SAM) has emerged as a powerful tool to improve generalizability of deep neural network…
Meta-Learning with Versatile Loss Geometries for Fast Adaptation Using Mirror Descent
Yilang Zhang, Bingcong Li, Georgios B. Giannakis
Utilizing task-invariant prior knowledge extracted from related tasks, meta-learning is a principled framework that empowers learning a new task especially when data records are li…
Contractive error feedback for gradient compression
Bingcong Li, Shuai Zheng, Parameswaran Raman +2
On-device memory concerns in distributed deep learning have become severe due to (i) the growth of model size in multi-GPU training, and (ii) the wide adoption of deep neural netwo…
Enhancing Sharpness-Aware Optimization Through Variance Suppression
Bingcong Li, Georgios B. Giannakis
Sharpness-aware minimization (SAM) has well documented merits in enhancing generalization of deep neural networks, even without sizable data augmentation. Embracing the geometry of…
Conic Descent Redux for Memory-Efficient Optimization
Bingcong Li, Georgios B. Giannakis
Conic programming has well-documented merits in a gamut of signal processing and machine learning tasks. This contribution revisits a recently developed first-order conic descent (…
Scalable Bayesian Meta-Learning through Generalized Implicit Gradients
Yilang Zhang, Bingcong Li, Shijian Gao +1
Meta-learning owns unique effectiveness and swiftness in tackling emerging tasks with limited data. Its broad applicability is revealed by viewing it as a bi-level optimization pro…
Surrogate modeling for Bayesian optimization beyond a single Gaussian process
Qin Lu, Konstantinos D. Polyzos, Bingcong Li +1
Bayesian optimization (BO) has well-documented merits for optimizing black-box functions with an expensive evaluation cost. Such functions emerge in applications as diverse as hype…
Distributionally Robust Semi-Supervised Learning Over Graphs
Alireza Sadeghi, Meng Ma, Bingcong Li +1
Semi-supervised learning (SSL) over graph-structured data emerges in many network science applications. To efficiently manage learning over graphs, variants of graph neural network…
Heavy Ball Momentum for Conditional Gradient
Bingcong Li, Alireza Sadeghi, Georgios B. Giannakis
Conditional gradient, aka Frank Wolfe (FW) algorithms, have well-documented merits in machine learning and signal processing applications. Unlike projection-based methods, momentum…
PingAn-VCGroup's Solution for ICDAR 2021 Competition on Scientific Table Image Recognition to Latex
Yelin He, Xianbiao Qi, Jiaquan Ye +5
This paper presents our solution for the ICDAR 2021 Competition on Scientific Table Image Recognition to LaTeX. This competition has two sub-tasks: Table Structure Reconstruction (…
Enhancing Parameter-Free Frank Wolfe with an Extra Subproblem
Bingcong Li, Lingda Wang, Georgios B. Giannakis +1
Aiming at convex optimization under structural constraints, this work introduces and analyzes a variant of the Frank Wolfe (FW) algorithm termed ExtraFW. The distinct feature of Ex…
Adversarial Linear Contextual Bandits with Graph-Structured Side Observations
Lingda Wang, Bingcong Li, Huozhi Zhou +3
This paper studies the adversarial graphical contextual bandits, a variant of adversarial multi-armed bandits that leverage two categories of the most common side information: \emp…
How Does Momentum Help Frank Wolfe?
Bingcong Li, Mario Coutino, Georgios B. Giannakis +1
We unveil the connections between Frank Wolfe (FW) type algorithms and the momentum in Accelerated Gradient Methods (AGM). On the negative side, these connections illustrate why mo…
Adaptive Step Sizes in Variance Reduction via Regularization
Bingcong Li, Georgios B. Giannakis
The main goal of this work is equipping convex and nonconvex problems with Barzilai-Borwein (BB) step size. With the adaptivity of BB step sizes granted, they can fail when the obj…
Nearly Optimal Algorithms for Piecewise-Stationary Cascading Bandits
Lingda Wang, Huozhi Zhou, Bingcong Li +2
Cascading bandit (CB) is a popular model for web search and online advertising, where an agent aims to learn the most attractive items out of a ground set of size during th…
A Multistep Lyapunov Approach for Finite-Time Analysis of Biased Stochastic Approximation
Gang Wang, Bingcong Li, Georgios B. Giannakis
Motivated by the widespread use of temporal-difference (TD-) and Q-learning algorithms in reinforcement learning, this paper studies a class of biased stochastic approximation (SA)…
Almost Tune-Free Variance Reduction
Bingcong Li, Lingda Wang, Georgios B. Giannakis
The variance reduction class of algorithms including the representative ones, SVRG and SARAH, have well documented merits for empirical risk minimization problems. However, they re…
On the Convergence of SARAH and Beyond
Bingcong Li, Meng Ma, Georgios B. Giannakis
The main theme of this work is a unifying algorithm, \textbf{L}oop\textbf{L}ess \textbf{S}ARAH (L2S) for problems formulated as summation of individual loss functions. L2S broa…