3 citations · 3 across the 11 of their papers we have counts for
6 papers · 1 filter
TapWeight: Reweighting Pretraining Objectives for Task-Adaptive Pretraining
Ruiyi Zhang, Sai Ashish Somayajula, Pengtao Xie
Large-scale general domain pretraining followed by downstream-specific finetuning has become a predominant paradigm in machine learning. However, discrepancies between the pretrain…
Improving the Language Understanding Capabilities of Large Language Models Using Reinforcement Learning
Bokai Hu, Sai Ashish Somayajula, Xin Pan +1
Instruction-fine-tuned large language models (LLMs) under 14B parameters continue to underperform on natural language understanding (NLU) tasks, often trailing smaller models like…
Generalizable and Stable Finetuning of Pretrained Language Models on Low-Resource Texts
Sai Ashish Somayajula, Youwei Liang, Abhishek Singh +2
Pretrained Language Models (PLMs) have advanced Natural Language Processing (NLP) tasks significantly, but finetuning PLMs on low-resource datasets poses significant challenges suc…
AutoLoRA: Automatically Tuning Matrix Ranks in Low-Rank Adaptation Based on Meta Learning
Ruiyi Zhang, Rushi Qiang, Sai Ashish Somayajula +1
Large-scale pretraining followed by task-specific finetuning has achieved great success in various NLP tasks. Since finetuning all parameters of large pretrained models poses subst…
Downstream Task Guided Masking Learning in Masked Autoencoders Using Multi-Level Optimization
Han Guo, Ramtin Hosseini, Ruiyi Zhang +4
Masked Autoencoder (MAE) is a notable method for self-supervised pretraining in visual representation learning. It operates by randomly masking image patches and reconstructing the…
Token-Specific Watermarking with Enhanced Detectability and Semantic Coherence for Large Language Models
Mingjia Huo, Sai Ashish Somayajula, Youwei Liang +3
Large language models generate high-quality responses with potential misinformation, underscoring the need for regulation by distinguishing AI-generated and human-written texts. Wa…