123 citations · 218 across the 19 of their papers we have counts for
21 papers
Empowering Hybrid Attention Models on NPUs
Yinyuan Zhang, Daliang Xu, Xiaolong Huang +4
Hybrid attention models have emerged as a crucial architecture for Large Language Models (LLMs) (e.g., the Qwen3.5 and Kimi series). Their memory and computational efficiency make…
Efficient Long-Horizon Learning for Learned Optimization
Xiaolong Huang, Benjamin Thérien, James Harrison +1
Learned optimization aims to improve upon hand-designed optimizers (e.g., Adam and Muon) by meta-learning small neural network optimizers over a distribution of tasks. While recent…
Neural Network Optimization Reimagined: Decoupled Techniques for Scratch and Fine-Tuning
Xin Ning, Qiankun Li, Xiaolong Huang +5
With the accumulation of resources in the era of big data and the rise of pre-trained models in deep learning, optimizing neural networks for various tasks often involves different…
GANORM: Lifespan Normative Modeling of EEG Network Topology based on Multinational Cross-Spectra
Shiang Hu, Xiaolong Huang, Yifan Hu +4
Charting the lifespan evolutionary trajectory of brain function serves as the normative standard for preventing mental disorders during brain development and aging. Although numero…
PyLO: Towards Accessible Learned Optimizers in PyTorch
Paul Janson, Benjamin Therien, Quentin Anthony +3
Learned optimizers have been an active research topic over the past decade, with increasing progress toward practical, general-purpose optimizers that can serve as drop-in replacem…
MuLoCo: Muon is a practical inner optimizer for DiLoCo
Benjamin Thérien, Xiaolong Huang, Aaron Defazio +2
DiLoCo is a powerful framework for training large language models (LLMs), enabling larger optimal batch sizes and increased accelerator utilization under networking constraints. Ho…
Scaling Laws of Synthetic Data for Language Models
Zeyu Qin, Qingxiu Dong, Xingxing Zhang +10
Large language models (LLMs) achieve strong performance across diverse tasks, largely driven by high-quality web data used in pre-training. However, recent studies indicate this da…
WildLong: Synthesizing Realistic Long-Context Instruction Data at Scale
Jiaxi Li, Xingxing Zhang, Xun Wang +6
Large language models (LLMs) with extended context windows enable tasks requiring extensive information integration but are limited by the scarcity of high-quality, diverse dataset…
Exploring the distribution of connectivity weights in resting-state EEG networks
Shiang Hu, Xiao Gong, Xiaolong Huang +2
The resting-state brain networks (RSNs) reflects the functional connectivity patterns between brain modules, providing essential foundations for decoding intrinsic neural informati…
Chain-of-Retrieval Augmented Generation
Liang Wang, Haonan Chen, Nan Yang +3
This paper introduces an approach for training o1-like RAG models that retrieve and reason over relevant information step by step before generating the final answer. Conventional R…
Bootstrap Your Own Context Length
Liang Wang, Nan Yang, Xingxing Zhang +2
We introduce a bootstrapping approach to train long-context language models by exploiting their short-context capabilities only. Our method utilizes a simple agent workflow to synt…
Synthetic Data (Almost) from Scratch: Generalized Instruction Tuning for Language Models
Haoran Li, Qingxiu Dong, Zhengyang Tang +17
We introduce Generalized Instruction Tuning (called GLAN), a general and scalable method for instruction tuning of Large Language Models (LLMs). Unlike prior work that relies on se…
Multilingual E5 Text Embeddings: A Technical Report
Liang Wang, Nan Yang, Xiaolong Huang +3
This technical report presents the training methodology and evaluation results of the open-source multilingual E5 text embedding models, released in mid-2023. Three embedding model…
One Step Learning, One Step Review
Xiaolong Huang, Qiankun Li, Xueran Li +1
Visual fine-tuning has garnered significant attention with the rise of pre-trained vision models. The current prevailing method, full fine-tuning, suffers from the issue of knowled…
Improving Text Embeddings with Large Language Models
Liang Wang, Nan Yang, Xiaolong Huang +3
In this paper, we introduce a novel and simple method for obtaining high-quality text embeddings using only synthetic data and less than 1k training steps. Unlike existing methods…
Large Search Model: Redefining Search Stack in the Era of LLMs
Liang Wang, Nan Yang, Xiaolong Huang +3
Modern search engines are built on a stack of different components, including query understanding, retrieval, multi-stage ranking, and question answering, among others. These compo…
Text Embeddings by Weakly-Supervised Contrastive Pre-training
Liang Wang, Nan Yang, Xiaolong Huang +5
This paper presents E5, a family of state-of-the-art text embeddings that transfer well to a wide range of tasks. The model is trained in a contrastive manner with weak supervision…
Effective and Efficient Query-aware Snippet Extraction for Web Search
Jingwei Yi, Fangzhao Wu, Chuhan Wu +4
Query-aware webpage snippet extraction is widely used in search engines to help users better understand the content of the returned webpages before clicking. Although important, it…
2nd Place Solution to Google Universal Image Embedding
Xiaolong Huang, Qiankun Li
Image representations are a critical building block of computer vision applications. This paper presents the 2nd place solution to the Google Universal Image Embedding Competition,…
LexMAE: Lexicon-Bottlenecked Pretraining for Large-Scale Retrieval
Tao Shen, Xiubo Geng, Chongyang Tao +5
In large-scale retrieval, the lexicon-weighting paradigm, learning weighted sparse representations in vocabulary space, has shown promising results with high quality and low latenc…
SimLM: Pre-training with Representation Bottleneck for Dense Passage Retrieval
Liang Wang, Nan Yang, Xiaolong Huang +5
In this paper, we propose SimLM (Similarity matching with Language Model pre-training), a simple yet effective pre-training method for dense passage retrieval. It employs a simple…