25 citations · 132 across the 68 of their papers we have counts for
52 papers · 1 filter
Topological Steering
Benoît Guérand, Tan Minh Nguyen
With the rapid rise of large language models (LLMs), controlling undesirable model behaviors has become increasingly important. Existing behavioral control methods typically interv…
MuonSSM: Orthogonalizing State Space Models for Sequence Modeling
Thai-Khanh Nguyen, Ngoc-Bich-Uyen Vo, Thieu N. Vo +2
State space models (SSMs) have emerged as efficient linear-time alternatives to attention for long-sequence modeling. However, existing SSMs often suffer from instability and memor…
Geometric and Stochastic Analysis of Discontinuities in Sparse Mixture-of-Experts
Tho Tran Huu, Huu-Tuan Nguyen, Thien-Hai Nguyen +4
Sparse Mixture-of-Experts (SMoE) architectures are now widely deployed in state-of-the-art language and vision models, where conditional routing allows scaling to very large networ…
Functional Equivalence in Attention: A Comprehensive Study with Applications to Linear Mode Connectivity
Viet-Hoang Tran, Vinh Khanh Bui, Van-Hoan Trinh +2
Neural network parameter spaces are inherently non-injective, as distinct parameter configurations can realize identical functions through functional equivalence. While this symmet…
Conservation Laws for Modern Neural Architectures
Viet-Hoang Tran, Vinh Khanh Bui, Tan Lai Ngoc +3
Understanding gradient descent dynamics is key to explaining the success of over-parameterized models, where implicit bias manifests through conservation laws in gradient flow. Whi…
High-Dimensional Random Projection for Activation Steering in Language Models
Minh-Hieu Pham, Bach Do, Laziz Abdullaev +2
Activation steering has emerged as a key methodology for controlling the behavior of large language models (LLMs). Existing difference-in-means based methods, however, are fundamen…
Quasi-Equivariant Metanetworks
Viet-Hoang Tran, An Nguyen, Benoît Guérand +2
Metanetworks are neural architectures designed to operate directly on pretrained weights to perform downstream tasks. However, the parameter space serves only as a proxy for the un…
Concept Heterogeneity-aware Representation Steering
Laziz U. Abdullaev, Noelle Y. L. Wong, Ryan T. Z. Lee +3
Representation steering offers a lightweight mechanism for controlling the behavior of large language models (LLMs) by intervening on internal activations at inference time. Most e…
Dynamical Properties of Tokens in Self-Attention and Effects of Positional Encoding
Duy-Tung Pham, An The Nguyen, Viet-Hoang Tran +4
This paper investigates the dynamical properties of tokens in pre-trained Transformer models and explores their application to improving Transformers. To this end, we analyze the d…
Selective Sinkhorn Routing for Improved Sparse Mixture of Experts
Duc Anh Nguyen, Huu Binh Ta, Nhuan Le Duc +2
Sparse Mixture-of-Experts (SMoE) models are scalable and computationally efficient, enabling large increases in model capacity with limited inference overhead. Existing SMoE method…
Angular Steering: Behavior Control via Rotation in Activation Space
Hieu M. Vu, Tan M. Nguyen
Controlling specific behaviors in large language models while preserving their general capabilities is a central challenge for safe and reliable artificial intelligence deployment.…
On Linear Mode Connectivity of Mixture-of-Experts Architectures
Viet-Hoang Tran, Van Hoan Trinh, Khanh Vinh Bui +1
Linear Mode Connectivity (LMC) is a notable phenomenon in the loss landscapes of neural networks, wherein independently trained models have been observed to be connected--up to per…
Modeling Expert Interactions in Sparse Mixture of Experts via Graph Structures
Minh-Khoi Nguyen-Nhat, Rachel S. Y. Teo, Laziz Abdullaev +3
Sparse Mixture of Experts (SMoE) has emerged as a promising solution to achieving unparalleled scalability in deep learning by decoupling model parameter count from computational c…
Expert Merging in Sparse Mixture of Experts with Nash Bargaining
Dung V. Nguyen, Anh T. Nguyen, Minh H. Nguyen +6
Existing expert merging strategies for Sparse Mixture of Experts (SMoE) typically rely on input-dependent or input-independent averaging of expert parameters, but often lack a prin…
Activation Steering with a Feedback Controller
Dung V. Nguyen, Hieu M. Vu, Nhi Y. Pham +2
Controlling the behaviors of large language models (LLM) is fundamental to their safety alignment and reliable deployment. However, existing steering methods are primarily driven b…
Tree-Sliced Wasserstein Distance with Nonlinear Projection
Thanh Tran, Viet-Hoang Tran, Thanh Chu +4
Tree-Sliced methods have recently emerged as an alternative to the traditional Sliced Wasserstein (SW) distance, replacing one-dimensional lines with tree-based metric spaces and i…
Promoting Ensemble Diversity with Interactive Bayesian Distributional Robustness for Fine-tuning Foundation Models
Ngoc-Quan Pham, Tuan Truong, Quyen Tran +3
We introduce Interactive Bayesian Distributional Robustness (IBDR), a novel Bayesian inference framework that allows modeling the interactions between particles, thereby enhancing…
Spherical Tree-Sliced Wasserstein Distance
Viet-Hoang Tran, Thanh T. Chu, Khoi N. M. Nguyen +3
Sliced Optimal Transport (OT) simplifies the OT problem in high-dimensional spaces by projecting supports of input measures onto one-dimensional lines and then exploiting the close…
Distance-Based Tree-Sliced Wasserstein Distance
Hoang V. Tran, Khoi N. M. Nguyen, Trang Pham +3
To overcome computational challenges of Optimal Transport (OT), several variants of Sliced Wasserstein (SW) has been developed in the literature. These approaches exploit the close…
CAMEx: Curvature-aware Merging of Experts
Dung V. Nguyen, Minh H. Nguyen, Luc Q. Nguyen +3
Existing methods for merging experts during model training and fine-tuning predominantly rely on Euclidean geometry, which assumes a flat parameter space. This assumption can limit…
Transformer Meets Twicing: Harnessing Unattended Residual Information
Laziz Abdullaev, Tan M. Nguyen
Transformer-based deep learning models have achieved state-of-the-art performance across numerous language and vision tasks. While the self-attention mechanism, a core component of…
Revisiting Kernel Attention with Correlated Gaussian Process Representation
Long Minh Bui, Tho Tran Huu, Duy Dinh +2
Transformers have increasingly become the de facto method to model sequential data with state-of-the-art performance. Due to its widespread use, being able to estimate and calibrat…
Tight Clusters Make Specialized Experts
Stefan K. Nielsen, Rachel S. Y. Teo, Laziz U. Abdullaev +1
Sparse Mixture-of-Experts (MoE) architectures have emerged as a promising approach to decoupling model capacity from computational cost. At the core of the MoE model is the router,…
An Attention-based Framework for Fair Contrastive Learning
Stefan K. Nielsen, Tan M. Nguyen
Contrastive learning has proven instrumental in learning unbiased representations of data, especially in complex environments characterized by high-cardinality and high-dimensional…