On the Properties of the Softmax Function with Application in Game Theory and Reinforcement Learning

Abstract

In this paper, we utilize results from convex analysis and monotone operator theory to derive additional properties of the softmax function that have not yet been covered in the existing literature. In particular, we show that the softmax function is the monotone gradient map of the log-sum-exp function. By exploiting this connection, we show that the inverse temperature parameter determines the Lipschitz and co-coercivity properties of the softmax function. We then demonstrate the usefulness of these properties through an application in game-theoretic reinforcement learning.

arXiv comment10 pages, 4 figures. Comments are welcome

References in corpus (5)

• From Softmax to Sparsemax: A Sparse Model of Attention and Multi-Label Classification
• Coupled Replicator Equations for the Dynamics of Learning in Multiagent Systems
• On the Dual Formulation of Boosting Algorithms
• Dynamics of Boltzmann Q-Learning in Two-Player Two-Action Games
• The Replicator Equation as an Inference Dynamic

Cited by in corpus (32)

• DeeperGCN: All You Need to Train Deeper GCNs
• LESSON: Multi-Label Adversarial False Data Injection Attack for Deep Learning Locational Detection
• Q-Learning based system for path planning with unmanned aerial vehicles swarms in obstacle environments
• Hopfield Networks is All You Need
• Model-Free Mean-Field Reinforcement Learning: Mean-Field MDP and Mean-Field Q-Learning
• Stochastic Optimization of Sorting Networks via Continuous Relaxations
• Formal Verification of Input-Output Mappings of Tree Ensembles
• Lipschitz Continuity in Model-based Reinforcement Learning
• On Controllable Sparse Alternatives to Softmax
• Model-Based Multi-Agent RL in Zero-Sum Markov Games with Near-Optimal Sample Complexity
• Towards Visual Distortion in Black-Box Attacks
• Voice2Series: Reprogramming Acoustic Models for Time Series Classification
• Monte Carlo Information Geometry: The dually flat case
• Joint User Association and Power Allocation in Heterogeneous Ultra Dense Network via Semi-Supervised Representation Learning
• Fast Transformers with Clustered Attention
• On the Role of Dataset Quality and Heterogeneity in Model Confidence
• Learning Mean-Field Games
• Reinforcement Learning with Function Approximation: From Linear to Nonlinear
• Know Your Limits: Uncertainty Estimation with ReLU Classifiers Fails at Reliable OOD Detection
• Feature Robust Optimal Transport for High-dimensional Data
• Distributed Evolution Strategies Using TPUs for Meta-Learning
• Class Anchor Clustering: a Loss for Distance-based Open Set Recognition
• Deep Learning for Power System Security Assessment
• Momentum Accelerates Evolutionary Dynamics
• On the Stability of Graph Convolutional Neural Networks under Edge Rewiring
• On the human-recognizability phenomenon of adversarially trained deep image classifiers
• Couplings for Multinomial Hamiltonian Monte Carlo
• MLANE: Meta-Learning Based Adaptive Network Embedding
• Graphmax for Text Generation
• Confidence Preservation Property in Knowledge Distillation Abstractions
• GL-Coarsener: A Graph representation learning framework to construct coarse grid hierarchy for AMG solvers
• SStaGCN: Simplified stacking based graph convolutional networks