11 citations · 16 across the 24 of their papers we have counts for
6 papers · 1 filter
Transformer as Linear Expansion of Learngene
Shiyu Xia, Miaosen Zhang, Xu Yang +3
We propose expanding the shared Transformer module to produce and initialize Transformers of varying depths, enabling adaptation to diverse resource constraints. Drawing an analogy…
Building Variable-sized Models via Learngene Pool
Boyu Shi, Shiyu Xia, Xu Yang +3
Recently, Stitchable Neural Networks (SN-Net) is proposed to stitch some pre-trained networks for quickly building numerous networks with different complexity and performance trade…
Manipulating the Label Space for In-Context Classification
Haokun Chen, Xu Yang, Yuhang Huang +3
After pre-training by generating the next word conditional on previous words, the Language Model (LM) acquires the ability of In-Context Learning (ICL) that can learn a new task co…
FedDAT: An Approach for Foundation Model Finetuning in Multi-Modal Heterogeneous Federated Learning
Haokun Chen, Yao Zhang, Denis Krompass +2
Recently, foundation models have exhibited remarkable advancements in multi-modal learning. These models, equipped with millions (or billions) of parameters, typically require a su…
FedPop: Federated Population-based Hyperparameter Tuning
Haokun Chen, Denis Krompass, Jindong Gu +1
Federated Learning (FL) is a distributed machine learning (ML) paradigm, in which multiple clients collaboratively train ML models without centralizing their local data. Similar to…
Exploring Diverse In-Context Configurations for Image Captioning
Xu Yang, Yongliang Wu, Mingzhuo Yang +2
After discovering that Language Models (LMs) can be good in-context few-shot learners, numerous strategies have been proposed to optimize in-context sequence configurations. Recent…