181 citations · 702 across the 49 of their papers we have counts for
63 papers
DataComp-LM: In search of the next generation of training sets for language models
Jeffrey Li, Alex Fang, Georgios Smyrnis +56
We introduce DataComp for Language Models (DCLM), a testbed for controlled dataset experiments with the goal of improving language models. As part of DCLM, we provide a standardize…
Deep Learning for Accelerated and Robust MRI Reconstruction: a Review
Reinhard Heckel, Mathews Jacob, Akshay Chaudhari +2
Deep learning (DL) has recently emerged as a pivotal technology for enhancing magnetic resonance imaging (MRI), a critical tool in diagnostic radiology. This review paper provides…
Language models scale reliably with over-training and on downstream tasks
Samir Yitzhak Gadre, Georgios Smyrnis, Vaishaal Shankar +22
Scaling laws are useful guides for derisking expensive training runs, as they predict performance of large models using cheaper, small-scale experiments. However, there remain gaps…
Approximating Positive Homogeneous Functions with Scale Invariant Neural Networks
Stefan Bamberger, Reinhard Heckel, Felix Krahmer
We investigate to what extent it is possible to solve linear inverse problems with networks. Due to the scaling invariance arising from the linearity, an optimal reconstruct…
K-band: Self-supervised MRI Reconstruction via Stochastic Gradient Descent over K-space Subsets
Frederic Wang, Han Qi, Alfredo De Goyeneche +3
Although deep learning (DL) methods are powerful for solving inverse problems, their reliance on high-quality training data is a major hurdle. This is significant in high-dimension…
Learning Provably Robust Estimators for Inverse Problems via Jittering
Anselm Krainovic, Mahdi Soltanolkotabi, Reinhard Heckel
Deep neural networks provide excellent performance for inverse problems such as denoising. However, neural networks can be sensitive to adversarial or worst-case perturbations. Thi…
Analyzing the Sample Complexity of Self-Supervised Image Reconstruction Methods
Tobit Klug, Dogukan Atik, Reinhard Heckel
Supervised training of deep neural networks on pairs of clean image and noisy measurement achieves state-of-the-art performance for many image reconstruction tasks, but such traini…
Implicit Neural Networks with Fourier-Feature Inputs for Free-breathing Cardiac MRI Reconstruction
Johannes F. Kunz, Stefan Ruschke, Reinhard Heckel
Cardiac magnetic resonance imaging (MRI) requires reconstructing a real-time video of a beating heart from continuous highly under-sampled measurements. This task is challenging si…
Zero-Shot Noise2Noise: Efficient Image Denoising without any Data
Youssef Mansour, Reinhard Heckel
Recently, self-supervised neural networks have shown excellent image denoising performance. However, current dataset free methods are either computationally expensive, require a no…
Information-Theoretic Foundations of DNA Data Storage
Ilan Shomorony, Reinhard Heckel
Due to its longevity and enormous information density, DNA is an attractive medium for archival data storage. Thanks to rapid technological advances, DNA storage is becoming practi…
Monotonic Risk Relationships under Distribution Shifts for Regularized Risk Minimization
Daniel LeJeune, Jiayu Liu, Reinhard Heckel
Machine learning systems are often applied to data that is drawn from a different distribution than the training distribution. Recent work has shown that for a variety of classific…
Test-time Recalibration of Conformal Predictors Under Distribution Shift Based on Unlabeled Examples
Fatih Furkan Yilmaz, Reinhard Heckel
Modern image classifiers are very accurate, but the predictions come without uncertainty estimates. Conformal predictors provide uncertainty estimates by computing a set of classes…
Scaling Laws For Deep Learning Based Image Reconstruction
Tobit Klug, Reinhard Heckel
Deep neural networks trained end-to-end to map a measurement of a (noisy) image to a clean image perform excellent for a variety of linear inverse problems. Current methods are onl…
Regularization-wise double descent: Why it occurs and how to eliminate it
Fatih Furkan Yilmaz, Reinhard Heckel
The risk of overparameterized models, in particular deep neural networks, is often double-descent shaped as a function of the model size. Recently, it was shown that the risk as a…
Theoretical Perspectives on Deep Learning Methods in Inverse Problems
Jonathan Scarlett, Reinhard Heckel, Miguel R. D. Rodrigues +2
In recent years, there have been significant advances in the use of deep learning methods in inverse problems such as denoising, compressive sensing, inpainting, and super-resoluti…
Test-Time Training Can Close the Natural Distribution Shift Performance Gap in Deep Learning Based Compressed Sensing
Mohammad Zalbagi Darestani, Jiayu Liu, Reinhard Heckel
Deep learning based image reconstruction methods outperform traditional methods. However, neural networks suffer from a performance drop when applied to images from a different dis…
Image-to-Image MLP-mixer for Image Reconstruction
Youssef Mansour, Kang Lin, Reinhard Heckel
Neural networks are highly effective tools for image reconstruction problems such as denoising and compressive sensing. To date, neural networks for image reconstruction are almost…
Provable Continual Learning via Sketched Jacobian Approximations
Reinhard Heckel
An important problem in machine learning is the ability to learn tasks in a sequential manner. If trained with standard first-order methods most models forget previously learned ta…
Achieving the Capacity of a DNA Storage Channel with Linear Coding Schemes
Kel Levick, Reinhard Heckel, Ilan Shomorony
Due to the redundant nature of DNA synthesis and sequencing technologies, a basic model for a DNA storage system is a multi-draw "shuffling-sampling" channel. In this model, a rand…
Untrained Graph Neural Networks for Denoising
Samuel Rey, Santiago Segarra, Reinhard Heckel +1
A fundamental problem in signal processing is to denoise a signal. While there are many well-performing methods for denoising signals defined on regular supports, such as images de…
Interpolation can hurt robust generalization even when there is no noise
Konstantin Donhauser, Alexandru Ţifrea, Michael Aerni +2
Numerous recent works show that overparameterization implicitly reduces variance for min-norm interpolators and max-margin classifiers. These findings suggest that ridge regulariza…
Data augmentation for deep learning based accelerated MRI reconstruction with limited data
Zalan Fabian, Reinhard Heckel, Mahdi Soltanolkotabi
Deep neural networks have emerged as very successful tools for image restoration and reconstruction tasks. These networks are often trained end-to-end to directly reconstruct an im…
Measuring Robustness in Deep Learning Based Compressive Sensing
Mohammad Zalbagi Darestani, Akshay S. Chaudhari, Reinhard Heckel
Deep neural networks give state-of-the-art accuracy for reconstructing images from few and noisy measurements, a problem arising for example in accelerated magnetic resonance imagi…
Active Sampling Count Sketch (ASCS) for Online Sparse Estimation of a Trillion Scale Covariance Matrix
Zhenwei Dai, Aditya Desai, Reinhard Heckel +1
Estimating and storing the covariance (or correlation) matrix of high-dimensional data is computationally challenging because both memory and computational requirements scale quadr…