Gaussian Error Linear Units (GELUs)
arXiv:1606.08415
Abstract
We propose the Gaussian Error Linear Unit (GELU), a high-performing neural network activation function. The GELU activation function is , where the standard Gaussian cumulative distribution function. The GELU nonlinearity weights inputs by their value, rather than gates inputs by their sign as in ReLUs (). We perform an empirical evaluation of the GELU nonlinearity against the ReLU and ELU activations and find performance improvements across all considered computer vision, natural language processing, and speech tasks.
arXiv commentTrimmed version of 2016 draft
References in corpus (6)
- SGDR: Stochastic Gradient Descent with Warm Restarts
- Learning with Pseudo-Ensembles
- Zoneout: Regularizing RNNs by Randomly Preserving Hidden Activations
- Deep Residual Networks with Exponential Linear Unit
- Natural Neural Networks
- Adjusting for Dropout Variance in Batch Normalization and Weight Initialization
Cited by in corpus (434)
- Learning Transferable Visual Models From Natural Language Supervision
- EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
- A Survey on Visual Transformer
- Attention Mechanisms in Computer Vision: A Survey
- PVT v2: Improved Baselines with Pyramid Vision Transformer
- E(3)-Equivariant Graph Neural Networks for Data-Efficient and Accurate Interatomic Potentials
- MLP-Mixer: An all-MLP Architecture for Vision
- Vision Transformers for Single Image Dehazing
- Transformer in Transformer
- ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
- Unified Language Model Pre-training for Natural Language Understanding and Generation
- CoAtNet: Marrying Convolution and Attention for All Data Sizes
- A novel time-frequency Transformer based on self-attention mechanism and its application in fault diagnosis of rolling bearings
- EfficientDet: Scalable and Efficient Object Detection
- Learnable latent embeddings for joint behavioral and neural analysis
- SpookyNet: Learning Force Fields with Electronic Degrees of Freedom and Nonlocal Effects
- VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text
- Distributional Soft Actor-Critic: Off-Policy Reinforcement Learning for Addressing Value Estimation Errors
- Stiff-PINN: Physics-Informed Neural Network for Stiff Chemical Kinetics
- P2T: Pyramid Pooling Transformer for Scene Understanding
- MetaFormer Baselines for Vision
- Action Transformer: A Self-Attention Model for Short-Time Pose-Based Human Action Recognition
- nnFormer: Interleaved Transformer for Volumetric Segmentation
- Transformer-based Acoustic Modeling for Hybrid Speech Recognition
- High-Performance Large-Scale Image Recognition Without Normalization
- The Principles of Deep Learning Theory
- Integer Quantization for Deep Learning Inference: Principles and Empirical Evaluation
- Decoding speech perception from non-invasive brain recordings
- Dual Cross-Attention for Medical Image Segmentation
- Perceiver IO: A General Architecture for Structured Inputs & Outputs
- You Only Look at One Sequence: Rethinking Transformer in Vision through Object Detection
- Restormer: Efficient Transformer for High-Resolution Image Restoration
- FCN-Transformer Feature Fusion for Polyp Segmentation
- Language Modeling with Deep Transformers
- ST-GRAT: A Novel Spatio-temporal Graph Attention Network for Accurately Forecasting Dynamically Changing Road Speed
- CTCNet: A CNN-Transformer Cooperation Network for Face Image Super-Resolution
- Keyword Transformer: A Self-Attention Model for Keyword Spotting
- Uncovering the Limits of Adversarial Training against Norm-Bounded Adversarial Examples
- Physics-guided deep learning framework for predictive modeling of the Reynolds stress anisotropy
- CMU-Net: A Strong ConvMixer-based Medical Ultrasound Image Segmentation Network
- ERNIE: Enhanced Language Representation with Informative Entities
- Perceiver: General Perception with Iterative Attention
- Reliable extrapolation of deep neural operators informed by physics or sparse observations
- SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Training
- MLIC: Multi-Reference Entropy Model for Learned Image Compression
- Physics-Informed Neural Networks for Shell Structures
- Uformer: A General U-Shaped Transformer for Image Restoration
- Fixing Data Augmentation to Improve Adversarial Robustness
- When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentations
- What Are Bayesian Neural Network Posteriors Really Like?
- Mixout: Effective Regularization to Finetune Large-scale Pretrained Language Models
- Fine-tuning wav2vec2 for speaker recognition
- Aligning AI With Shared Human Values
- CycleMLP: A MLP-like Architecture for Dense Prediction
- StructBERT: Incorporating Language Structures into Pre-training for Deep Language Understanding
- Pretrained Transformers as Universal Computation Engines
- Smooth Adversarial Training
- Strategies for the Construction of Machine-Learning Potentials for Accurate and Efficient Atomic-Scale Simulations
- Transformer-based Spatial-Temporal Feature Learning for EEG Decoding
- VTGAN: Semi-supervised Retinal Image Synthesis and Disease Prediction using Vision Transformers
- A Critical Review of Physics-Informed Machine Learning Applications in Subsurface Energy Systems
- CharBERT: Character-aware Pre-trained Language Model
- Discovering Parametric Activation Functions
- Compacter: Efficient Low-Rank Hypercomplex Adapter Layers
- Ground state energy functional with Hartree-Fock efficiency and chemical accuracy
- HTR-VT: Handwritten Text Recognition with Vision Transformer
- Transformers in Single Object Tracking: An Experimental Survey
- Benchmarking Detection Transfer Learning with Vision Transformers
- Dive into Deep Learning
- Fourier Neural Operator Surrogate Model to Predict 3D Seismic Waves Propagation
- COOT: Cooperative Hierarchical Transformer for Video-Text Representation Learning
- Learning Dynamic Graph Representation of Brain Connectome with Spatio-Temporal Attention
- Hybrid Spectrogram and Waveform Source Separation
- HiCMAE: Hierarchical Contrastive Masked Autoencoder for Self-Supervised Audio-Visual Emotion Recognition
- EfficientPose: An efficient, accurate and scalable end-to-end 6D multi object pose estimation approach
- IA-RED: Interpretability-Aware Redundancy Reduction for Vision Transformers
- Actionable and Interpretable Fault Localization for Recurring Failures in Online Service Systems
- Large Kernel Distillation Network for Efficient Single Image Super-Resolution
- KPGT: Knowledge-Guided Pre-training of Graph Transformer for Molecular Property Prediction
- Music Demixing Challenge 2021
- Evolution TANN and the identification of internal variables and evolution equations in solid mechanics
- Combining Deep Reinforcement Learning and Search for Imperfect-Information Games
- 6D-ViT: Category-Level 6D Object Pose Estimation via Transformer-based Instance Representation Learning
- Guided Depth Map Super-resolution: A Survey
- LRT: An Efficient Low-Light Restoration Transformer for Dark Light Field Images
- CMT: Convolutional Neural Networks Meet Vision Transformers
- T-former: An Efficient Transformer for Image Inpainting
- RDP-Net: Region Detail Preserving Network for Change Detection
- Multiscale Vision Transformers
- An Image is Worth 16x16 Words, What is a Video Worth?
- Accurate emulator for the redshift-space power spectrum of dark matter halos and its application to galaxy power spectrum
- Self-attending RNN for Speech Enhancement to Improve Cross-corpus Generalization
- Deep Learning Methods for Partial Differential Equations and Related Parameter Identification Problems
- ReduNet: A White-box Deep Network from the Principle of Maximizing Rate Reduction
- Evolving Normalization-Activation Layers
- Depth-Wise Convolutions in Vision Transformers for Efficient Training on Small Datasets
- EmoNeXt: an Adapted ConvNeXt for Facial Emotion Recognition
- U-TILISE: A Sequence-to-sequence Model for Cloud Removal in Optical Satellite Time Series
- Graph-MLP: Node Classification without Message Passing in Graph
- "No, to the Right" -- Online Language Corrections for Robotic Manipulation via Shared Autonomy
- Applications of physics informed neural operators
- A neural operator-based surrogate solver for free-form electromagnetic inverse design
- Rapid Seismic Waveform Modeling and Inversion with Neural Operators
- Transformer Encoder with Multiscale Deep Learning for Pain Classification Using Physiological Signals
- Differentially Private Fine-tuning of Language Models
- Very Deep VAEs Generalize Autoregressive Models and Can Outperform Them on Images
- DeFT-AN: Dense Frequency-Time Attentive Network for Multichannel Speech Enhancement
- Towards an astronomical foundation model for stars with a Transformer-based model
- PPT Fusion: Pyramid Patch Transformerfor a Case Study in Image Fusion
- Only Train Once: A One-Shot Neural Network Training And Pruning Framework
- A Modified Batch Intrinsic Plasticity Method for Pre-training the Random Coefficients of Extreme Learning Machines
- ObjFormer: Learning Land-Cover Changes From Paired OSM Data and Optical High-Resolution Imagery via Object-Guided Transformer
- Mask and Reason: Pre-Training Knowledge Graph Transformers for Complex Logical Queries
- Evolutionary Preference Learning via Graph Nested GRU ODE for Session-based Recommendation
- Fast and Sample-Efficient Interatomic Neural Network Potentials for Molecules and Materials Based on Gaussian Moments
- The Cosmic Graph: Optimal Information Extraction from Large-Scale Structure using Catalogues
- Clinically-Inspired Multi-Agent Transformers for Disease Trajectory Forecasting from Multimodal Data
- Trex: Learning Execution Semantics from Micro-Traces for Binary Similarity
- RiemannONets: Interpretable Neural Operators for Riemann Problems
- TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data
- Parameter Efficient Multimodal Transformers for Video Representation Learning
- Economic Topology Optimization of District Heating Networks using a Pipe Penalization Approach
- An equivariant graph neural network for the elasticity tensors of all seven crystal systems
- CT-Net: Arbitrary-Shaped Text Detection via Contour Transformer
- SpectralFormer: Rethinking Hyperspectral Image Classification with Transformers
- LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention
- Machine learning the derivative discontinuity of density-functional theory
- MINN: Learning the dynamics of differential-algebraic equations and application to battery modeling
- Pay Attention to MLPs
- Concurrent ischemic lesion age estimation and segmentation of CT brain using a Transformer-based network
- Applications of Scientific Machine Learning for the Analysis of Functionally Graded Porous Beams
- MetaFormer Is Actually What You Need for Vision
- Multimodal Model with Text and Drug Embeddings for Adverse Drug Reaction Classification
- User Retention-oriented Recommendation with Decision Transformer
- Scaling Vision with Sparse Mixture of Experts
- Thermally Averaged Magnetic Anisotropy Tensors via Machine Learning Based on Gaussian Moments
- SVFAP: Self-supervised Video Facial Affect Perceiver
- S-MLP: Spatial-Shift MLP Architecture for Vision
- Hierarchical Vision Transformers for Cardiac Ejection Fraction Estimation
- FitVid: Overfitting in Pixel-Level Video Prediction
- NormFormer: Improved Transformer Pretraining with Extra Normalization
- Conditional Motion In-betweening
- Visual Relationship Detection with Visual-Linguistic Knowledge from Multimodal Representations
- On the Benefit of Width for Neural Networks: Disappearance of Bad Basins
- Simple Training Strategies and Model Scaling for Object Detection
- Pixel super-resolved virtual staining of label-free tissue using diffusion models
- Large language models for automated scholarly paper review: A survey
- Pretrained Transformers Improve Out-of-Distribution Robustness
- How fine can fine-tuning be? Learning efficient language models
- IGLUE: A Benchmark for Transfer Learning across Modalities, Tasks, and Languages
- Interaction-Aware Trajectory Planning for Autonomous Vehicles with Analytic Integration of Neural Networks into Model Predictive Control
- ConvMLP: Hierarchical Convolutional MLPs for Vision
- SRC-Net: Bi-Temporal Spatial Relationship Concerned Network for Change Detection
- Mapping Dark Matter in the Milky Way using Normalizing Flows and Gaia DR3
- Emerging Cross-lingual Structure in Pretrained Language Models
- SE(3)-equivariant prediction of molecular wavefunctions and electronic densities
- X-LXMERT: Paint, Caption and Answer Questions with Multi-Modal Transformers
- Learn molecular representations from large-scale unlabeled molecules for drug discovery
- Knowledge Fusion and Semantic Knowledge Ranking for Open Domain Question Answering
- Accelerating Simulation of Stiff Nonlinear Systems using Continuous-Time Echo State Networks
- Fast Dynamic 1D Simulation of Divertor Plasmas with Neural PDE Surrogates
- MCTNet: A Multi-Scale CNN-Transformer Network for Change Detection in Optical Remote Sensing Images
- Multi-head or Single-head? An Empirical Comparison for Transformer Training
- What Formal Languages Can Transformers Express? A Survey
- Magnetohydrodynamics with Physics Informed Neural Operators
- Self-Attentive Hawkes Processes
- Bolt: Bridging the Gap between Auto-tuners and Hardware-native Performance
- Characterizing signal propagation to close the performance gap in unnormalized ResNets
- KG-BART: Knowledge Graph-Augmented BART for Generative Commonsense Reasoning
- Comparing Machine Learning and Interpolation Methods for Loop-Level Calculations
- PALM: Pre-training an Autoencoding&Autoregressive Language Model for Context-conditioned Generation
- Random Features for Kernel Approximation: A Survey on Algorithms, Theory, and Beyond
- Feature Learning in Infinite-Width Neural Networks
- SF2Former: Amyotrophic Lateral Sclerosis Identification From Multi-center MRI Data Using Spatial and Frequency Fusion Transformer
- Associating Objects with Transformers for Video Object Segmentation
- HRPVT: High-Resolution Pyramid Vision Transformer for medium and small-scale human pose estimation
- TransEM:Residual Swin-Transformer based regularized PET image reconstruction
- Enhancing Few-shot Image Classification with Cosine Transformer
- DeTiME: Diffusion-Enhanced Topic Modeling using Encoder-decoder based LLM
- Automated Novelty Evaluation of Academic Paper: A Collaborative Approach Integrating Human and Large Language Model Knowledge
- Music Source Separation Based on a Lightweight Deep Learning Framework (DTTNET: DUAL-PATH TFC-TDF UNET)
- Visual Parser: Representing Part-whole Hierarchies with Transformers
- A Lorentz-Equivariant Transformer for All of the LHC
- Towards Crowdsourced Training of Large Neural Networks using Decentralized Mixture-of-Experts
- Efficient Transformers with Dynamic Token Pooling
- ASiT: Local-Global Audio Spectrogram vIsion Transformer for Event Classification
- Mixed Precision Low-bit Quantization of Neural Network Language Models for Speech Recognition
- Machine learning modeling of the atomic structure and physical properties of alkali and alkaline-earth aluminosilicate glasses and melts
- AutoFormer: Searching Transformers for Visual Recognition
- I-BERT: Integer-only BERT Quantization
- Revisiting Tensor Basis Neural Networks for Reynolds stress modeling: application to plane channel and square duct flows
- Pruning Self-attentions into Convolutional Layers in Single Path
- GNNBuilder: An Automated Framework for Generic Graph Neural Network Accelerator Generation, Simulation, and Optimization
- Transformer-Driven Inverse Problem Transform for Fast Blind Hyperspectral Image Dehazing
- On Explaining Your Explanations of BERT: An Empirical Study with Sequence Classification
- VARA-TTS: Non-Autoregressive Text-to-Speech Synthesis based on Very Deep VAE with Residual Attention
- OnDev-LCT: On-Device Lightweight Convolutional Transformers towards federated learning
- Rethinking Channel Dimensions for Efficient Model Design
- Hybrid Auxiliary Field Quantum Monte Carlo for Molecular Systems
- MusiCoder: A Universal Music-Acoustic Encoder Based on Transformers
- INet: Inter-Intra-slice Interpolation Network for Medical Slice Synthesis
- Continuous PDE Dynamics Forecasting with Implicit Neural Representations
- AReLU: Attention-based Rectified Linear Unit
- RikiNet: Reading Wikipedia Pages for Natural Question Answering
- Improving Generalization for Multimodal Fake News Detection
- GRAM: An Interpretable Approach for Graph Anomaly Detection using Gradient Attention Maps
- Galaxy clustering from the bottom up: A Streaming Model emulator I
- Dual-path Self-Attention RNN for Real-Time Speech Enhancement
- NeuBTF: Neural fields for BTF encoding and transfer
- A Parameter-Efficient Learning Approach to Arabic Dialect Identification with Pre-Trained General-Purpose Speech Model
- Categorical Normalizing Flows via Continuous Transformations
- Visual Transformers for Primates Classification and Covid Detection
- Fixed-Dimensional and Permutation Invariant State Representation of Autonomous Driving
- Searching for Efficient Multi-Stage Vision Transformers
- NILMFormer: Non-Intrusive Load Monitoring that Accounts for Non-Stationarity
- On the locality of local neural operator in learning fluid dynamics
- Zero-touch Continuous Network Slicing Control via Scalable Actor-Critic Learning
- MM-ALT: A Multimodal Automatic Lyric Transcription System
- Local neural operator for solving transient partial differential equations on varied domains
- The Low-Rank Simplicity Bias in Deep Networks
- A Multidimensional Graph Fourier Transformation Neural Network for Vehicle Trajectory Prediction
- Normalized Attention Without Probability Cage
- Forensic Analysis and Localization of Multiply Compressed MP3 Audio Using Transformers
- Counterfactual Data Augmentation using Locally Factored Dynamics
- DirectMultiStep: Direct Route Generation for Multistep Retrosynthesis
- LLIC: Large Receptive Field Transform Coding with Adaptive Weights for Learned Image Compression
- Thank you for Attention: A survey on Attention-based Artificial Neural Networks for Automatic Speech Recognition
- Code-switched inspired losses for generic spoken dialog representations
- Eformer: Edge Enhancement based Transformer for Medical Image Denoising
- Reverse Ordering Techniques for Attention-Based Channel Prediction
- A Multi-grained based Attention Network for Semi-supervised Sound Event Detection
- Knowledge Enhanced Attention for Robust Natural Language Inference
- Augmenting Decompiler Output with Learned Variable Names and Types
- Speech Separation with Pretrained Frontend to Minimize Domain Mismatch
- Can weight sharing outperform random architecture search? An investigation with TuNAS
- MLP Singer: Towards Rapid Parallel Korean Singing Voice Synthesis
- A Fast and Robust BERT-based Dialogue State Tracker for Schema-Guided Dialogue Dataset
- BayesFT: Bayesian Optimization for Fault Tolerant Neural Network Architecture
- Exploring Continuous Integrate-and-Fire for Adaptive Simultaneous Speech Translation
- Deep calibration of the quadratic rough Heston model
- Generative Pre-Training for Speech with Autoregressive Predictive Coding
- MixerGAN: An MLP-Based Architecture for Unpaired Image-to-Image Translation
- Deep neural networks approach to microbial colony detection -- a comparative analysis
- TransMed: Transformers Advance Multi-modal Medical Image Classification
- CATFace: Cross-Attribute-Guided Transformer with Self-Attention Distillation for Low-Quality Face Recognition
- Language Models are Good Translators
- High-dimensional reinforcement learning for optimization and control of ultracold quantum gases
- Variance-reduced Language Pretraining via a Mask Proposal Network
- A Continuous Convolutional Trainable Filter for Modelling Unstructured Data
- Training speaker recognition systems with limited data
- Reconstructing nodal pressures in water distribution systems with graph neural networks
- Biologically Inspired Oscillating Activation Functions Can Bridge the Performance Gap between Biological and Artificial Neurons
- Deep Learning in Diabetic Foot Ulcers Detection: A Comprehensive Evaluation
- NAS-VAD: Neural Architecture Search for Voice Activity Detection
- Activation Function Optimization Scheme for Image Classification
- Smooth activations and reproducibility in deep networks
- SAPAG: A Self-Adaptive Privacy Attack From Gradients
- Equivariant Neural Networks for Spin Dynamics Simulations of Itinerant Magnets
- Lightweight Convolutional Representations for On-Device Natural Language Processing
- Deep Learning is Singular, and That's Good
- NEU: A Meta-Algorithm for Universal UAP-Invariant Feature Representation
- Complexity Measures for Neural Networks with General Activation Functions Using Path-based Norms
- One4all User Representation for Recommender Systems in E-commerce
- GoonDAE: Denoising-Based Driver Assistance for Off-Road Teleoperation
- ScaleVLAD: Improving Multimodal Sentiment Analysis via Multi-Scale Fusion of Locally Descriptors
- A Novel Stochastic Transformer-based Approach for Post-Traumatic Stress Disorder Detection using Audio Recording of Clinical Interviews
- Is Supervised Syntactic Parsing Beneficial for Language Understanding? An Empirical Investigation
- Shifted Chunk Transformer for Spatio-Temporal Representational Learning
- Multichannel Orthogonal Transform-Based Perceptron Layers for Efficient ResNets
- Computational limits to the legibility of the imaged human brain
- A Qualitative Study of the Dynamic Behavior for Adaptive Gradient Algorithms
- Efficient Image Pre-Training with Siamese Cropped Masked Autoencoders
- Generative Model for Constructing Reaction Path from Initial to Final States
- Quizbowl: The Case for Incremental Question Answering
- Rethinking Implicit Neural Representations for Vision Learners
- Optimizing Inference Performance of Transformers on CPUs
- Trial by FIRE: Probing the dark matter density profile of dwarf galaxies with GraphNPE
- DAGN: Discourse-Aware Graph Network for Logical Reasoning
- Variable Name Recovery in Decompiled Binary Code using Constrained Masked Language Modeling
- Psi-GAN: A power-spectrum-informed generative adversarial network for the emulation of large-scale structure maps across cosmologies and redshifts
- VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding
- CoPR: Towards Accurate Visual Localization With Continuous Place-descriptor Regression
- Hire-MLP: Vision MLP via Hierarchical Rearrangement
- Supervising the Transfer of Reasoning Patterns in VQA
- Multi-Task Learning for Conversational Question Answering over a Large-Scale Knowledge Base
- Encoding Distributional Soft Actor-Critic for Autonomous Driving in Multi-lane Scenarios
- Embodied BERT: A Transformer Model for Embodied, Language-guided Visual Task Completion
- Link Prediction on N-ary Relational Facts: A Graph-based Approach
- Time-Space Transformers for Video Panoptic Segmentation
- RGB-D-Fusion: Image Conditioned Depth Diffusion of Humanoid Subjects
- Predictive Modeling: BIM Command Recommendation Based on Large-scale Usage Logs
- ADF & TransApp: A Transformer-Based Framework for Appliance Detection Using Smart Meter Consumption Series
- One Self-Configurable Model to Solve Many Abstract Visual Reasoning Problems
- Deep Representation Learning for Open Vocabulary Electroencephalography-to-Text Decoding
- Bayesian sparsification for deep neural networks with Bayesian model reduction
- Sens-BERT: Enabling Transferability and Re-calibration of Calibration Models for Low-cost Sensors under Reference Measurements Scarcity
- Self-supervised Learning of Rotation-invariant 3D Point Set Features using Transformer and its Self-distillation
- IDIAPers @ Causal News Corpus 2022: Efficient Causal Relation Identification Through a Prompt-based Few-shot Approach
- LaRa: Latents and Rays for Multi-Camera Bird's-Eye-View Semantic Segmentation
- Dealing with training and test segmentation mismatch: FBK@IWSLT2021
- Bilingual Alignment Pre-Training for Zero-Shot Cross-Lingual Transfer
- Tractable structured natural gradient descent using local parameterizations
- A Neural Network Perturbation Theory Based on the Born Series
- On the Universality of the Double Descent Peak in Ridgeless Regression
- Our Evaluation Metric Needs an Update to Encourage Generalization
- Language Modelling for Source Code with Transformer-XL
- Enriching Non-Autoregressive Transformer with Syntactic and SemanticStructures for Neural Machine Translation
- Probabilistic Software Modeling: A Data-driven Paradigm for Software Analysis
- Beyond Point Estimate: Inferring Ensemble Prediction Variation from Neuron Activation Strength in Recommender Systems
- Symmetrical Gaussian Error Linear Units (SGELUs)
- MOI-Mixer: Improving MLP-Mixer with Multi Order Interactions in Sequential Recommendation
- Know Your Limits: Uncertainty Estimation with ReLU Classifiers Fails at Reliable OOD Detection
- Using differentiable programming to obtain an energy and density-optimized exchange-correlation functional
- OkwuGbé: End-to-End Speech Recognition for Fon and Igbo
- XDA: Accurate, Robust Disassembly with Transfer Learning
- Exoplanet Transit Candidate Identification in TESS Full-Frame Images via a Transformer-Based Algorithm
- SGD-QA: Fast Schema-Guided Dialogue State Tracking for Unseen Services
- Mesa: A Memory-saving Training Framework for Transformers
- Extract then Distill: Efficient and Effective Task-Agnostic BERT Distillation
- Self-Attention Gazetteer Embeddings for Named-Entity Recognition
- Boundary Aware U-Net for Glacier Segmentation
- Improving Robustness using Generated Data
- Learning to Retrieve Entity-Aware Knowledge and Generate Responses with Copy Mechanism for Task-Oriented Dialogue Systems
- Unified Multi-modal Diagnostic Framework with Reconstruction Pre-training and Heterogeneity-combat Tuning
- BD-MSA: Body decouple VHR Remote Sensing Image Change Detection method guided by multi-scale feature information aggregation
- Evaluation of self-supervised pre-training for automatic infant movement classification using wearable movement sensors
- Style Equalization: Unsupervised Learning of Controllable Generative Sequence Models
- Pretraining the Noisy Channel Model for Task-Oriented Dialogue
- Demystifying BERT: Implications for Accelerator Design
- Multi-scale Transformer Language Models
- A Study on MIMO Channel Estimation by 2D and 3D Convolutional Neural Networks
- K-TanH: Efficient TanH For Deep Learning
- Generalisation of Cyberbullying Detection
- Hierarchical Autoencoder-based Lossy Compression for Large-scale High-resolution Scientific Data
- Three-body renormalization group limit cycles based on unsupervised feature learning
- "I'm Not Mad": Commonsense Implications of Negation and Contradiction
- Stochastic Super-resolution of Cosmological Simulations with Denoising Diffusion Models
- TAG: Gradient Attack on Transformer-based Language Models
- Salient Object Ranking with Position-Preserved Attention
- Polyphone Disambiguation in Mandarin Chinese with Semi-Supervised Learning
- PhytNet -- Tailored Convolutional Neural Networks for Custom Botanical Data
- Sisyphus: A Cautionary Tale of Using Low-Degree Polynomial Activations in Privacy-Preserving Deep Learning
- SparseDNN: Fast Sparse Deep Learning Inference on CPUs
- Undivided Attention: Are Intermediate Layers Necessary for BERT?
- Activated Gradients for Deep Neural Networks
- Improving ensemble extreme precipitation forecasts using generative artificial intelligence
- Vision Guided Generative Pre-trained Language Models for Multimodal Abstractive Summarization
- NeRV: Neural Representations for Videos
- Video Frame Interpolation Transformer
- Optimized spiking neurons classify images with high accuracy through temporal coding with two spikes
- Scope and Arbitration in Machine Learning Clinical EEG Classification
- Don't Fear Peculiar Activation Functions: EUAF and Beyond
- Sparse Attention with Linear Units
- TLCFuse: Temporal Multi-Modality Fusion Towards Occlusion-Aware Semantic Segmentation-Aided Motion Planning
- Stability and Generalization of Bilevel Programming in Hyperparameter Optimization
- TransMask: A Compact and Fast Speech Separation Model Based on Transformer
- Initializing ReLU networks in an expressive subspace of weights
- Unsupervised Explanation Generation for Machine Reading Comprehension
- TEASEL: A Transformer-Based Speech-Prefixed Language Model
- End-to-End Entity Detection with Proposer and Regressor
- q-Neurons: Neuron Activations based on Stochastic Jackson's Derivative Operators
- Reborn Mechanism: Rethinking the Negative Phase Information Flow in Convolutional Neural Network
- ERNIE-M: Enhanced Multilingual Representation by Aligning Cross-lingual Semantics with Monolingual Corpora
- Subquadratic Overparameterization for Shallow Neural Networks
- Learn-able parameter guided Activation Functions
- Multi-Scale Local-Temporal Similarity Fusion for Continuous Sign Language Recognition
- SMU: smooth activation function for deep networks using smoothing maximum technique
- PreSizE: Predicting Size in E-Commerce using Transformers
- Dual Interaction Network with Cross-Image Attention for Medical Image Segmentation
- Deep Feature Response Discriminative Calibration
- Efficient and Accurate Gradients for Neural SDEs
- Structured second-order methods via natural gradient descent
- RaftMLP: How Much Can Be Done Without Attention and with Less Spatial Locality?
- PointMixer: MLP-Mixer for Point Cloud Understanding
- Evaluating Graphical Perception Capabilities of Vision Transformers
- SMedBERT: A Knowledge-Enhanced Pre-trained Language Model with Structured Semantics for Medical Text Mining
- Soft Autoencoder and Its Wavelet Adaptation Interpretation
- Octa: Omissions and Conflicts in Target-Aspect Sentiment Analysis
- Discover the Mysteries of the Maya: Selected Contributions from the Machine Learning Challenge & The Discovery Challenge Workshop at ECML PKDD 2021
- Detecting Logical Relation In Contract Clauses
- Universal Performance Gap of Neural Quantum States Applied to the Hofstadter-Bose-Hubbard Model
- Simultaneous Speech Translation for Live Subtitling: from Delay to Display
- K-PLUG: Knowledge-injected Pre-trained Language Model for Natural Language Understanding and Generation in E-Commerce
- On the Bias Against Inductive Biases
- SFB-net for cardiac segmentation: Bridging the semantic gap with attention
- Verifying Quantized Neural Networks using SMT-Based Model Checking
- Scalable neural network-based blackbox optimization
- Towards Accurate Quantization and Pruning via Data-free Knowledge Transfer
- Parametric Rectified Power Sigmoid Units: Learning Nonlinear Neural Transfer Analytical Forms
- Randomized Stochastic Gradient Descent Ascent
- PairConnect: A Compute-Efficient MLP Alternative to Attention
- AMMASurv: Asymmetrical Multi-Modal Attention for Accurate Survival Analysis with Whole Slide Images and Gene Expression Data
- Regularized Flexible Activation Function Combinations for Deep Neural Networks
- Attention as Activation
- Computational homogenization for aerogel-like polydisperse open-porous materials using neural network--based surrogate models on the microscale
- Orthogonal-Padé Activation Functions: Trainable Activation functions for smooth and faster convergence in deep networks
- SocialTrans: A Deep Sequential Model with Social Information for Web-Scale Recommendation Systems
- Using mixup as regularization and tuning hyper-parameters for ResNets
- ISyNet: Convolutional Neural Networks design for AI accelerator
- MC-SSL0.0: Towards Multi-Concept Self-Supervised Learning
- Introducing the DOME Activation Functions
- Refine Neutrino Events Reconstruction with BEiT-3
- Implicit regularization of deep residual networks towards neural ODEs
- Sines, Transient, Noise Neural Modeling of Piano Notes
- Privacy-Preserving Public Release of Datasets for Support Vector Machine Classification
- Interpretable Prediction of Lymph Node Metastasis in Rectal Cancer MRI Using Variational Autoencoders
- Exploring the Early Universe with Deep Learning
- Resolution scaling governs DINOv3 transfer performance in chest radiograph classification
- Interpretable Deep Learning for Stock Returns: A Consensus-Bottleneck Asset Pricing Model
- jBOT: Semantic Jet Representation Clustering Emerges from Self-Distillation
- Sparsity-Probe: Analysis tool for Deep Learning Models
- Analysis of the Compaction Behavior of Textile Reinforcements in Low-Resolution In-Situ CT Scans via Machine-Learning and Descriptor-Based Methods
- ViFiT: Reconstructing Vision Trajectories from IMU and Wi-Fi Fine Time Measurements
- KST-Mixer: Kinematic Spatio-Temporal Data Mixer For Colon Shape Estimation
- CITIES: Contextual Inference of Tail-Item Embeddings for Sequential Recommendation
- AMU-EURANOVA at CASE 2021 Task 1: Assessing the stability of multilingual BERT
- Adversarial Generation and Encoding of Nested Texts
- Towards Automatic Evaluation of Dialog Systems: A Model-Free Off-Policy Evaluation Approach
- Multi-mode Transformer Transducer with Stochastic Future Context
- Differentiable Random Access Memory using Lattices
- Knowledge Transfer by Discriminative Pre-training for Academic Performance Prediction
- Multilingual Translation via Grafting Pre-trained Language Models
- EIS -- a family of activation functions combining Exponential, ISRU, and Softplus
- Commonsense Knowledge in Word Associations and ConceptNet
- Explainable Recommendations via Attentive Multi-Persona Collaborative Filtering
- PlueckerNet: Learn to Register 3D Line Reconstructions
- ComQA:Compositional Question Answering via Hierarchical Graph Neural Networks
- MaxVA: Fast Adaptation of Step Sizes by Maximizing Observed Variance of Gradients
- End-to-end Biomedical Entity Linking with Span-based Dictionary Matching
- Avoiding Kernel Fixed Points: Computing with ELU and GELU Infinite Networks
- Sckwoky at SHROOM-Visions: Character-Level Hallucination-Span Detection in Image-Conditioned VLMs via Dual-Prompted LoRA Decoders
- MIRA: Leveraging Multi-Intention Co-click Information in Web-scale Document Retrieval using Deep Neural Networks
- No Answer is Better Than Wrong Answer: A Reflection Model for Document Level Machine Reading Comprehension
- A Lightweight Graph Transformer Network for Human Mesh Reconstruction from 2D Human Pose