353 citations · 1.4k across the 27 of their papers we have counts for
19 papers · 1 filter
Characterizing and overcoming the greedy nature of learning in multi-modal deep neural networks
Nan Wu, Stanisław Jastrzębski, Kyunghyun Cho +1
We hypothesize that due to the greedy nature of learning in multi-modal deep neural networks, these models tend to rely on just one modality while under-fitting the other modalitie…
Relative Molecule Self-Attention Transformer
Łukasz Maziarka, Dawid Majchrowski, Tomasz Danel +5
Self-supervised learning holds promise to revolutionize molecule property prediction - a central task to drug discovery and many more industries - by enabling data efficient learni…
Catastrophic Fisher Explosion: Early Phase Fisher Matrix Impacts Generalization
Stanislaw Jastrzebski, Devansh Arpit, Oliver Astrand +6
The early phase of training a deep neural network has a dramatic effect on the local curvature of the loss function. For instance, using a small learning rate does not guarantee st…
RetroGNN: Approximating Retrosynthesis by Graph Neural Networks for De Novo Drug Design
Cheng-Hao Liu, Maksym Korablyov, Stanisław Jastrzębski +3
De novo molecule generation often results in chemically unfeasible molecules. A natural idea to mitigate this problem is to bias the search process towards more easily synthesizabl…
An artificial intelligence system for predicting the deterioration of COVID-19 patients in the emergency department
Farah E. Shamout, Yiqiu Shen, Nan Wu +17
During the coronavirus disease 2019 (COVID-19) pandemic, rapid and accurate triage of patients at the emergency department is critical to inform decision-making. We propose a data-…
Molecule Edit Graph Attention Network: Modeling Chemical Reactions as Sequences of Graph Edits
Mikołaj Sacha, Mikołaj Błaż, Piotr Byrski +5
The central challenge in automated synthesis planning is to be able to generate and predict outcomes of a diverse set of chemical reactions. In particular, in many cases, the most…
Understanding the robustness of deep neural network classifiers for breast cancer screening
Witold Oleszkiewicz, Taro Makino, Stanisław Jastrzębski +5
Deep neural networks (DNNs) show promise in breast cancer screening, but their robustness to input perturbations must be better understood before they can be clinically implemented…
The Break-Even Point on Optimization Trajectories of Deep Neural Networks
Stanislaw Jastrzebski, Maciej Szymczak, Stanislav Fort +4
The early phase of training of deep neural networks is critical for their final performance. In this work, we study how the hyperparameters of stochastic gradient descent (SGD) use…
Molecule Attention Transformer
Łukasz Maziarka, Tomasz Danel, Sławomir Mucha +3
Designing a single neural network architecture that performs competitively across a range of molecule property prediction tasks remains largely an open challenge, and its solution…
Large Scale Structure of Neural Network Loss Landscapes
Stanislav Fort, Stanislaw Jastrzebski
There are many surprising and perhaps counter-intuitive properties of optimization of deep neural networks. We propose and experimentally verify a unified phenomenological model of…
Split Batch Normalization: Improving Semi-Supervised Learning under Domain Shift
Michał Zając, Konrad Zolna, Stanisław Jastrzębski
Recent work has shown that using unlabeled data in semi-supervised learning is not always beneficial and can even hurt generalization, especially when there is a class mismatch bet…
Deep Neural Networks Improve Radiologists' Performance in Breast Cancer Screening
Nan Wu, Jason Phang, Jungkyu Park +29
We present a deep convolutional neural network for breast cancer screening exam classification, trained and evaluated on over 200,000 exams (over 1,000,000 images). Our network ach…
Non-linear ICA based on Cramer-Wold metric
Przemysław Spurek, Aleksandra Nowak, Jacek Tabor +2
Non-linear source separation is a challenging open problem with many applications. We extend a recently proposed Adversarial Non-linear ICA (ANICA) model, and introduce Cramer-Wold…
Parameter-Efficient Transfer Learning for NLP
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski +5
Fine-tuning large pre-trained models is an effective transfer mechanism in NLP. However, in the presence of many downstream tasks, fine-tuning is parameter inefficient: an entire n…
Stiffness: A New Perspective on Generalization in Neural Networks
Stanislav Fort, Paweł Krzysztof Nowak, Stanislaw Jastrzebski +1
In this paper we develop a new perspective on generalization of neural networks by proposing and investigating the concept of a neural network stiffness. We measure how stiff a net…
Neural Architecture Search Over a Graph Search Space
Stanisław Jastrzębski, Quentin de Laroussilhe, Mingxing Tan +3
Neural Architecture Search (NAS) enabled the discovery of state-of-the-art architectures in many domains. However, the success of NAS depends on the definition of the search space.…
Cramer-Wold AutoEncoder
Szymon Knop, Jacek Tabor, Przemysław Spurek +3
We propose a new generative model, Cramer-Wold Autoencoder (CWAE). Following WAE, we directly encourage normality of the latent space. Our paper uses also the recent idea from Slic…
Three Factors Influencing Minima in SGD
Stanisław Jastrzębski, Zachary Kenton, Devansh Arpit +4
We investigate the dynamical and convergent properties of stochastic gradient descent (SGD) applied to Deep Neural Networks (DNNs). Characterizing the relation between learning rat…
Learning to Compute Word Embeddings On the Fly
Dzmitry Bahdanau, Tom Bosc, Stanisław Jastrzębski +3
Words in natural language follow a Zipfian distribution whereby some words are frequent but most are rare. Learning representations for words in the "long tail" of this distributio…