6 citations · 22 across the 17 of their papers we have counts for
6 papers · 1 filter
BOOST: BOttleneck-Optimized Scalable Training Framework for Low-Rank Large Language Models
Zhengyang Wang, Ziyue Liu, Ruijie Zhang +5
The scale of transformer model pre-training is constrained by the increasing computation and communication cost. Low-rank bottleneck architectures offer a promising solution to sig…
LaX: Boosting Low-Rank Training of Foundation Models via Latent Crossing
Ruijie Zhang, Ziyue Liu, Zhengyang Wang +1
Training foundation models such as ViTs and LLMs requires tremendous computing cost. Low-rank matrix or tensor factorization offers a parameter-efficient alternative, but often dow…
DeepOHeat-v1: Efficient Operator Learning for Fast and Trustworthy Thermal Simulation and Optimization in 3D-IC Design
Xinling Yu, Ziyue Liu, Hai Li +5
Thermal analysis is crucial in 3D-IC design due to increased power density and complex heat dissipation paths. Although operator learning frameworks such as DeepOHeat~\cite{liu2023…
QuZO: Quantized Zeroth-Order Fine-Tuning for Large Language Models
Jiajun Zhou, Yifan Yang, Kai Zhen +6
Language Models (LLMs) are often quantized to lower precision to reduce the memory cost and latency in inference. However, quantization often degrades model performance, thus fine-…
Scalable Back-Propagation-Free Training of Optical Physics-Informed Neural Networks
Yequan Zhao, Xinling Yu, Xian Xiao +6
Physics intelligence and digital twins often require rapid and repeated performance evaluation of various engineering systems (e.g. robots, autonomous vehicles, semiconductor chips…
CoLA: Compute-Efficient Pre-Training of LLMs via Low-Rank Activation
Ziyue Liu, Ruijie Zhang, Zhengyang Wang +7
The full-size MLPs and the projection layers in attention introduce tremendous model sizes of large language models (LLMs), consuming extensive computational resources in pre-train…