66 citations · 418 across the 62 of their papers we have counts for
10 papers · 1 filter
MANCE: Manifold Aware Concept Erasure
Matan Avitan, Yoav Goldberg, Yanai Elazar
Concept erasure aims to remove a target concept from a representation while preserving the other information encoded in it. This is difficult because representations encode many co…
Training Large Scale Polynomial CNNs for E2E Inference over Homomorphic Encryption
Moran Baruch, Nir Drucker, Gilad Ezov +5
Training large-scale CNNs that during inference can be run under Homomorphic Encryption (HE) is challenging due to the need to use only polynomial operations. This limits HE-based…
Log-linear Guardedness and its Implications
Shauli Ravfogel, Yoav Goldberg, Ryan Cotterell
Methods for erasing human-interpretable concepts from neural representations that assume linearity have been found to be tractable and useful. However, the impact of this removal o…
Kernelized Concept Erasure
Shauli Ravfogel, Francisco Vargas, Yoav Goldberg +1
The representation space of neural models for textual data emerges in an unsupervised manner during training. Understanding how those representations encode human-interpretable con…
Linear Adversarial Concept Erasure
Shauli Ravfogel, Michael Twiton, Yoav Goldberg +1
Modern neural models trained on textual data rely on pre-trained representations that emerge without direct supervision. As these representations are increasingly being used in rea…
Thinking Like Transformers
Gail Weiss, Yoav Goldberg, Eran Yahav
What is the computational model behind a Transformer? Where recurrent neural networks have direct parallels in finite state machines, allowing clear discussion and thought around a…
BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models
Elad Ben-Zaken, Shauli Ravfogel, Yoav Goldberg
We introduce BitFit, a sparse-finetuning method where only the bias-terms of the model (or a subset of them) are being modified. We show that with small-to-medium training data, ap…
A simple geometric proof for the benefit of depth in ReLU networks
Asaf Amrami, Yoav Goldberg
We present a simple proof for the benefit of depth in multi-layer feedforward network with rectified activation ("depth separation"). Specifically we present a sequence of classifi…
Effects of Parameter Norm Growth During Transformer Training: Inductive Bias from Gradient Descent
William Merrill, Vivek Ramanujan, Yoav Goldberg +2
The capacity of neural networks like the widely adopted transformer is known to be very high. Evidence is emerging that they learn successfully due to inductive bias in the trainin…
Learning Deterministic Weighted Automata with Queries and Counterexamples
Gail Weiss, Yoav Goldberg, Eran Yahav
We present an algorithm for extraction of a probabilistic deterministic finite automaton (PDFA) from a given black-box language model, such as a recurrent neural network (RNN). The…