1 citations · 2 across the 5 of their papers we have counts for
6 papers
STAPLE: Static Sparse Upcycling of Pretrained Checkpoints via Embedding Modules
Ranajoy Sadhukhan, Euijin Hong, Sheng Cao +9
Sparse computation can scale the parametric capacity of LLMs without proportionally increasing active computation, but pretraining sparse architectures remains expensive and data-h…
STEM: Scaling Transformers with Embedding Modules
Ranajoy Sadhukhan, Sheng Cao, Harry Dong +5
Fine-grained sparsity promises higher parametric capacity without proportional per-token compute, but often suffers from training instability, load balancing, and communication ove…
Supracompetitive Pricing Under AI Monoculture
Shengyu Cao, Ming Hu
When competing sellers delegate pricing to a shared AI model, such as a large language model, correlated recommendations combined with performance-driven updates aggregating seller…
Param for Direct Weight Mixing: Post-Train Large Language Model at Zero Cost
Sheng Cao, Mingrui Wu, Karthik Prasad +2
The post-training phase of large language models is essential for enhancing capabilities such as instruction-following, reasoning, and alignment with human preferences. However, it…
LLM-Augmented Retrieval: Enhancing Retrieval Models Through Language Models and Doc-Level Embedding
Mingrui Wu, Sheng Cao
Recently embedding-based retrieval or dense retrieval have shown state of the art results, compared with traditional sparse or bag-of-words based approaches. This paper introduces…
VLSI-Inspired Methods for Student Learning Community Creation and Refinement
Sheng Lun Cao
COVID-19 significantly disrupted how educational contents are delivered in academic institutions, rapidly accelerating the adoption of online and blended learning. This thesis expl…