5 citations · 13 across the 13 of their papers we have counts for
13 papers
Exact Quantile Balancing and Load-Error Injection for Mixture-of-Experts
Pit Neitemeier, Jiaze Li, Alessio Serra +2
Mixture-of-Experts (MoE) training requires global load balance to prevent expert under-utilization and local balance for efficient expert-parallel execution. Existing distributed Q…
Good Pretraining, Bad SFT: Checkpoint Quality Across the Training Stack
Sohir Maskey, Philipp Scholl, Jonas Knupp +2
Language-model checkpoints are commonly selected by pretraining loss or benchmark scores, assuming that the highest-scoring checkpoint will remain the best starting point for subse…
Studying quantization trade-offs for efficient inference deployment in machine translation
Jim Zhao, Sohir Maskey, Koen Oostermeijer +2
Deploying large language models in realistic server environments poses challenges, as the system needs to provide high-quality responses with low latency. Quantization is a common…
The Price of Robustness: Stable Classifiers Need Overparameterization
Jonas von Berg, Adalbert Fono, Massimiliano Datres +2
The relationship between overparameterization, stability, and generalization remains incompletely understood in the setting of discontinuous classifiers. We address this gap by est…
1-Bit Wonder: Improving QAT Performance in the Low-Bit Regime through K-Means Quantization
Sohir Maskey, Constantin Eichenberg, Johannes Messner +1
Quantization-aware training (QAT) is an effective method to drastically reduce the memory footprint of LLMs while keeping performance degradation at an acceptable level. However, t…
Graph Representational Learning: When Does More Expressivity Hurt Generalization?
Sohir Maskey, Raffaele Paolino, Fabian Jogl +2
Graph Neural Networks (GNNs) are powerful tools for learning on structured data, yet the relationship between their expressivity and predictive performance remains unclear. We intr…
Graph Neural Networks for Enhancing Ensemble Forecasts of Extreme Rainfall
Christopher Bülte, Sohir Maskey, Philipp Scholl +2
Climate change is increasing the occurrence of extreme precipitation events, threatening infrastructure, agriculture, and public safety. Ensemble prediction systems provide probabi…
Generalization Bounds for Message Passing Networks on Mixture of Graphons
Sohir Maskey, Gitta Kutyniok, Ron Levie
We study the generalization capabilities of Message Passing Neural Networks (MPNNs), a prevalent class of Graph Neural Networks (GNN). We derive generalization bounds specifically…
Weisfeiler and Leman Go Loopy: A New Hierarchy for Graph Representational Learning
Raffaele Paolino, Sohir Maskey, Pascal Welke +1
We introduce -loopy Weisfeiler-Leman (-WL), a novel hierarchy of graph isomorphism tests and a corresponding GNN framework, -MPNN, that can count cycles up…
A Fractional Graph Laplacian Approach to Oversmoothing
Sohir Maskey, Raffaele Paolino, Aras Bacho +1
Graph neural networks (GNNs) have shown state-of-the-art performances in various applications. However, GNNs often struggle to capture long-range dependencies in graphs due to over…
Generalized Laplacian Positional Encoding for Graph Representation Learning
Sohir Maskey, Ali Parviz, Maximilian Thiessen +3
Graph neural networks (GNNs) are the primary tool for processing graph-structured data. Unfortunately, the most commonly used GNNs, called Message Passing Neural Networks (MPNNs) s…
Generalization Analysis of Message Passing Neural Networks on Large Random Graphs
Sohir Maskey, Ron Levie, Yunseok Lee +1
Message passing neural networks (MPNN) have seen a steep rise in popularity since their introduction as generalizations of convolutional neural networks to graph-structured data, a…
Transferability of Graph Neural Networks: an Extended Graphon Approach
Sohir Maskey, Ron Levie, Gitta Kutyniok
We study spectral graph convolutional neural networks (GCNNs), where filters are defined as continuous functions of the graph shift operator (GSO) through functional calculus. A sp…