4 citations · 11 across the 16 of their papers we have counts for
16 papers
Attention Mechanisms Through the Lens of Numerical Methods: Approximation Methods and Alternative Formulations
Michel Fabrice Serret, Alice Cortinovis, Yijun Dong +10
The attention mechanism is the computational core of modern Transformer architectures, but its quadratic complexity in the input sequence length is the bottleneck for large-scale i…
When does Chain-of-Thought Help: A Markovian Perspective
Zihan Wang, Yijun Dong, Qi Lei
Chain-of-Thought (CoT) prompting is a widely used inference-time technique for improving reasoning, yet its gains are uneven across tasks. We analyze when and why CoT helps by mode…
A Task-Centric Theory for Iterative Self-Improvement with Easy-to-Hard Curricula
Chenruo Liu, Yijun Dong, Yiqiu Shen +1
Iterative self-improvement fine-tunes an autoregressive large language model (LLM) on reward-verified outputs generated by the LLM itself. In contrast to the empirical success of s…
Randomized time stepping of nonlinearly parametrized solutions of evolution problems
Yijun Dong, Paul Schwerdtner, Benjamin Peherstorfer
The Dirac-Frenkel variational principle is a widely used building block for using nonlinear parametrizations in the context of model reduction and numerically solving partial diffe…
Does Weak-to-strong Generalization Happen under Spurious Correlations?
Chenruo Liu, Yijun Dong, Qi Lei
We initiate a unified theoretical and algorithmic study of a key problem in weak-to-strong (W2S) generalization: when fine-tuning a strong pre-trained student with pseudolabels fro…
Discrepancies are Virtue: Weak-to-Strong Generalization through Lens of Intrinsic Dimension
Yijun Dong, Yicheng Li, Yunai Li +2
Weak-to-strong (W2S) generalization is a type of finetuning (FT) where a strong (large) student model is trained on pseudo-labels generated by a weak teacher. Surprisingly, W2S FT…
Greedy Output Approximation: Towards Efficient Structured Pruning for LLMs Without Retraining
Jianwei Li, Yijun Dong, Qi Lei
To remove redundant components of large language models (LLMs) without incurring significant computational costs, this work focuses on single-shot pruning without a retraining phas…
Sketchy Moment Matching: Toward Fast and Provable Data Selection for Finetuning
Yijun Dong, Hoang Phan, Xiang Pan +1
We revisit data selection in a modern context of finetuning from a fundamental perspective. Extending the classical wisdom of variance minimization in low dimensions to high-dimens…
Adaptive Parallelizable Algorithms for Interpolative Decompositions via Partially Pivoted LU
Katherine J. Pearce, Chao Chen, Yijun Dong +1
Interpolative and CUR decompositions involve "natural bases" of row and column subsets, or skeletons, of a given matrix that approximately span its row and column spaces. These low…
Randomized Dimension Reduction with Statistical Guarantees
Yijun Dong
Large models and enormous data are essential driving forces of the unprecedented successes achieved by modern algorithms, especially in scientific computing and machine learning. N…
Robust Blockwise Random Pivoting: Fast and Accurate Adaptive Interpolative Decomposition
Yijun Dong, Chao Chen, Per-Gunnar Martinsson +1
The interpolative decomposition (ID) aims to construct a low-rank approximation formed by a basis consisting of row/column skeletons in the original matrix and a corresponding inte…
Cluster-aware Semi-supervised Learning: Relational Knowledge Distillation Provably Learns Clustering
Yijun Dong, Kevin Miller, Qi Lei +1
Despite the empirical success and practical significance of (relational) knowledge distillation that matches (the relations of) features between teacher and student models, the cor…
Efficient Bounds and Estimates for Canonical Angles in Randomized Subspace Approximations
Yijun Dong, Per-Gunnar Martinsson, Yuji Nakatsukasa
Randomized subspace approximation with "matrix sketching" is an effective approach for constructing approximate partial singular value decompositions (SVDs) of large matrices. The…
Adaptively Weighted Data Augmentation Consistency Regularization for Robust Optimization under Concept Shift
Yijun Dong, Yuege Xie, Rachel Ward
Concept shift is a prevailing problem in natural tasks like medical image segmentation where samples usually come from different subpopulations with variant correlations between fe…
Sample Efficiency of Data Augmentation Consistency Regularization
Shuo Yang, Yijun Dong, Rachel Ward +3
Data augmentation is popular in the training of large neural networks; currently, however, there is no clear theoretical comparison between different algorithmic choices on how to…
Simpler is better: A comparative study of randomized algorithms for computing the CUR decomposition
Yijun Dong, Per-Gunnar Martinsson
The CUR decomposition is a technique for low-rank approximation that selects small subsets of the columns and rows of a given matrix to use as bases for its column and rowspaces. I…