How to Not Measure Disentanglement

Abstract

To evaluate disentangled representations several metrics have been proposed. However, theoretical guarantees for conventional metrics of disentanglement are missing. Moreover, conventional metrics do not have a consistent correlation with the outcomes of qualitative studies. In this paper we analyze metrics of disentanglement and their properties. We conclude that existing metrics of disentanglement were created to reflect different characteristics of disentanglement and do not satisfy two basic desirable properties: (1) assign a high score to representations that are disentangled according to the definition; and (2) assign a low score to representations that are entangled according to the definition. In addition, we propose a new metric of disentanglement and prove that it satisfies both of the properties.

References in corpus (6)

• Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations
• Disentangling by Factorising
• Towards a Definition of Disentangled Representations
• Understanding disentangling in $\beta$-VAE
• Semantic Jitter: Dense Supervision for Visual Comparisons via Synthetic Images
• A Preliminary Study of Disentanglement With Insights on the Inadequacy of Metrics