6 papers
BERT-JEPA: Reorganizing CLS Embeddings for Language-Invariant Semantics
Taj Gillin, Adam Lalani, Kenneth Zhang +1
Joint Embedding Predictive Architectures (JEPA) are a novel self supervised training technique that have shown recent promise across domains. We introduce BERT-JEPA (BEPA), a train…
Dual Decomposition of Weights and Singular Value Low Rank Adaptation
Jialong Han, Si Zhang, Ke Zhang
Parameter-Efficient Fine-Tuning (PEFT) has emerged as a critical paradigm for adapting Large Language Models (LLMs) to downstream tasks, among which Low-rank Adaptation (LoRA) repr…
OSoRA: Output-Dimension and Singular-Value Initialized Low-Rank Adaptation
Jialong Han, Si Zhang, Ke Zhang
Fine-tuning Large Language Models (LLMs) has become increasingly challenging due to their massive scale and associated computational costs. Parameter-Efficient Fine-Tuning (PEFT) m…
Drama: Mamba-Enabled Model-Based Reinforcement Learning Is Sample and Parameter Efficient
Wenlong Wang, Ivana Dusparic, Yucheng Shi +2
Model-based reinforcement learning (RL) offers a solution to the data inefficiency that plagues most model-free RL algorithms. However, learning a robust world model often requires…
AGD: an Auto-switchable Optimizer using Stepwise Gradient Difference for Preconditioning Matrix
Yun Yue, Zhiling Ye, Jiadi Jiang +2
Adaptive optimizers, such as Adam, have achieved remarkable success in deep learning. A key component of these optimizers is the so-called preconditioning matrix, providing enhance…
MA^2: A Self-Supervised and Motion Augmenting Autoencoder for Gait-Based Automatic Disease Detection
Yiqun Liu, Ke Zhang, Yin Zhu
Ground reaction force (GRF) is the force exerted by the ground on a body in contact with it. GRF-based automatic disease detection (ADD) has become an emerging medical diagnosis me…