6 citations · 6 across the 22 of their papers we have counts for
23 papers
Clean: Second-order LLM Training at Linear Memory Cost via Nystr{ö}m Sketching
Beheshteh T. Rakhshan, Sahar Rajabi, Maziar Sargordi Shikai Fang +4
Training large language models (LLMs) entails a fundamental trade-off: memory-efficient optimizers such as Adam discard cross-parameter curvature, whereas full-curvature methods su…
Evaluation Awareness in Language Models: Representation, Verbalization, and Control
Farzaneh Heidari, Amin Memarian, Guillaume Rabusseau
Both capability and safety benchmarks rest upon the assumption that the behavior of language models undergoing a test is informative about their behavior in deployment. This assump…
Generative Learning for Quantum Measurement Design
Jun Dai, Olivier Nahman-Lévesque, Guillaume Rabusseau +2
Extracting quantum information from a quantum state is a fundamental task of quantum computation, often requiring the estimation of many non-commuting observables under a finite me…
From Expressivity to Sample Complexity: Narrow Teachers for Transformers via C-RASP
Michael Rizvi-Martel, Satwik Bhattamishra, Guillaume Rabusseau +1
A theoretical understanding of Transformers is crucial to better understand the capacities and limitations of large language models (LLMs). There is much work analyzing the express…
TN-SHAP-G: Graph-Structured Tensor Network Surrogates for Shapley Values and Interactions
Farzaneh Heidari, Guillaume Rabusseau
Shapley values are a widely used tool for attributing importance and interactions among input variables in black-box models, but their computation involves a function defined over…
Tensor Cookbook: Mastering Tensors through Diagrams
Beheshteh T. Rakhshan, Guillaume Rabusseau
High-dimensional data arise naturally in many areas of science and engineering, including machine learning, signal processing, computational physics, and statistics. Such data are…
Orth-Dion: Eliminating Geometric Mismatch in Distributed Low-Rank Spectral Optimization
Tatsuhiro Nakamori, Laura Gomezjurado Gonzalez, Ganesh Talluri +5
Low-rank gradient compression reduces communication in distributed training by representing updates with rank- factors. Dion is a recent method that approximates Muon, a spectra…
On the Role of Depth in the Expressivity of RNNs
Maude Lizaire, Michael Rizvi-Martel, Éric Dupuis +1
The benefits of depth in feedforward neural networks are well known: composing multiple layers of linear transformations with nonlinear activations enables complex computations. Wh…
The Illusion of Superposition? A Principled Analysis of Latent Thinking in Language Models
Michael Rizvi-Martel, Guillaume Rabusseau, Marius Mosbach
Latent reasoning via continuous chain-of-thoughts (Latent CoT) has emerged as a promising alternative to discrete CoT reasoning. Operating in continuous space increases expressivit…
Model Merging via Data-Free Covariance Estimation
Marawan Gamal Abdel Hameed, Derek Tam, Pascal Jr Tikeng Notsawo +2
Model merging provides a way of cheaply combining individual models to produce a model that inherits each individual's capabilities. While some merging methods can approach the per…
KQ-SVD: Compressing the KV Cache with Provable Guarantees on Attention Fidelity
Damien Lesens, Beheshteh T. Rakhshan, Guillaume Rabusseau
The Key-Value (KV) cache is central to the efficiency of transformer-based large language models (LLMs), storing previously computed vectors to accelerate inference. Yet, as sequen…
FlowQ-Net: A Generative Framework for Automated Quantum Circuit Design
Jun Dai, Michael Rizvi-Martel, Guillaume Rabusseau
Designing efficient quantum circuits is a central bottleneck to exploring the potential of quantum computing, particularly for noisy intermediate-scale quantum (NISQ) devices, wher…
Tractable Shapley Values and Interactions via Tensor Networks
Farzaneh Heidari, Chao Li, Guillaume Rabusseau
We show how to replace the O(2^n) coalition enumeration over n features behind Shapley values and Shapley-style interaction indices with a few-evaluation scheme on a tensor-network…
Benefits and Limitations of Communication in Multi-Agent Reasoning
Michael Rizvi-Martel, Satwik Bhattamishra, Neil Rathi +2
Chain-of-thought prompting has popularized step-by-step reasoning in large language models, yet model performance still degrades as problem complexity and context length grow. By d…
Efficient Probabilistic Tensor Networks
Marawan Gamal Abdel Hameed, Guillaume Rabusseau
Tensor networks (TNs) enable compact representations of large tensors through shared parameters. Their use in probabilistic modeling is particularly appealing, as probabilistic ten…
Numerical PDE solvers outperform neural PDE solvers
Patrick Chatain, Michael Rizvi-Martel, Guillaume Rabusseau +1
We present DeepFDM, a differentiable finite-difference framework for learning spatially varying coefficients in time-dependent partial differential equations (PDEs). By embedding a…
Grokking Beyond the Euclidean Norm of Model Parameters
Pascal Jr Tikeng Notsawo, Guillaume Dumas, Guillaume Rabusseau
Grokking refers to a delayed generalization following overfitting when optimizing artificial neural networks with gradient-based methods. In this work, we demonstrate that grokking…
GFlowNets for Hamiltonian decomposition in groups of compatible operators
Isaac L. Huidobro-Meezs, Jun Dai, Guillaume Rabusseau +1
Quantum computing presents a promising alternative for the direct simulation of quantum systems with the potential to explore chemical problems beyond the capabilities of classical…
ROSA: Random Subspace Adaptation for Efficient Fine-Tuning
Marawan Gamal Abdel Hameed, Aristides Milios, Siva Reddy +1
Model training requires significantly more memory, compared with inference. Parameter efficient fine-tuning (PEFT) methods provide a means of adapting large models to downstream ta…
A Tensor Decomposition Perspective on Second-order RNNs
Maude Lizaire, Michael Rizvi-Martel, Marawan Gamal Abdel Hameed +1
Second-order Recurrent Neural Networks (2RNNs) extend RNNs by leveraging second-order interactions for sequence modelling. These models are provably more expressive than their firs…
Efficient Leverage Score Sampling for Tensor Train Decomposition
Vivek Bharadwaj, Beheshteh T. Rakhshan, Osman Asif Malik +1
Tensor Train~(TT) decomposition is widely used in the machine learning and quantum physics communities as a popular tool to efficiently compress high-dimensional tensor data. In th…
High-Order Pooling for Graph Neural Networks with Tensor Decomposition
Chenqing Hua, Guillaume Rabusseau, Jian Tang
Graph Neural Networks (GNNs) are attracting growing attention due to their effectiveness and flexibility in modeling a variety of graph-structured data. Exiting GNN architectures u…
Rademacher Random Projections with Tensor Networks
Beheshteh T. Rakhshan, Guillaume Rabusseau
Random projection (RP) have recently emerged as popular techniques in the machine learning community for their ability in reducing the dimension of very high-dimensional tensors. F…