4 papers
MeRoTune: RoPE-Safe Merging with a Tunable Dial
Salman Faroz
When you merge two fine-tuned models from the same base checkpoint by simply averaging their weights, you implicitly assume their attention subspaces are still aligned. Recent work…
Domain-Adaptive Continued Pre-Training of Small Language Models
Salman Faroz
Continued pre-training of small language models offers a promising path for domain adaptation with limited computational resources. I've investigated this approach within education…
DisEmbed: Transforming Disease Understanding through Embeddings
Salman Faroz
The medical domain is vast and diverse, with many existing embedding models focused on general healthcare applications. However, these models often struggle to capture a deep under…
Sensitivity Analysis On Loss Landscape
Salman Faroz
Gradients can be employed for sensitivity analysis. Here, we leverage the advantages of the Loss Landscape to comprehend which independent variables impact the dependent variable.…