Descriptive vs. inferential community detection in networks: pitfalls, myths, and half-truths

Abstract

Community detection is one of the most important methodological fields of network science, and one which has attracted a significant amount of attention over the past decades. This area deals with the automated division of a network into fundamental building blocks, with the objective of providing a summary of its large-scale structure. Despite its importance and widespread adoption, there is a noticeable gap between what is arguably the state-of-the-art and the methods that are actually used in practice in a variety of fields. Here we attempt to address this discrepancy by dividing existing methods according to whether they have a "descriptive" or an "inferential" goal. While descriptive methods find patterns in networks based on context-dependent notions of community structure, inferential methods articulate generative models, and attempt to fit them to data. In this way, they are able to provide insights into the mechanisms of network formation, and separate structure from randomness in a manner supported by statistical evidence. We review how employing descriptive methods with inferential aims is riddled with pitfalls and misleading answers, and thus should be in general avoided. We argue that inferential methods are more typically aligned with clearer scientific questions, yield more robust results, and should be in many cases preferred. We attempt to dispel some myths and half-truths often believed when community detection is employed in practice, in an effort to improve both the use of such methods as well as the interpretation of their results.

arXiv comment57 pages, 18 figures

References in corpus (80)

• Fast unfolding of communities in large networks
• Community structure in social and biological networks
• Modularity and community structure in networks
• Community detection in graphs
• From Louvain to Leiden: guaranteeing well-connected communities
• Maps of random walks on complex networks reveal community structure
• Mixing patterns in networks
• Benchmark graphs for testing community detection algorithms
• Resolution limit in community detection
• Community detection algorithms: a comparative analysis
• Hierarchical structure and the prediction of missing links in networks
• Statistical Mechanics of Community Detection
• Stochastic blockmodels and community structure in networks
• Community detection in networks: A user guide
• The performance of modularity maximization in practical contexts
• Modularity from Fluctuations in Random Graphs and Complex Networks
• Consensus clustering in complex networks
• An information-theoretic framework for resolving community structure in complex networks
• Missing and spurious interactions and the reconstruction of complex networks
• Mixed membership stochastic blockmodels
• Modularity and community detection in bipartite networks
• Asymptotic analysis of the stochastic block model for modular networks and its algorithmic applications
• The phase transition in inhomogeneous random graphs
• Spectral clustering and the high-dimensional stochastic blockmodel
• Limits of modularity maximization in community detection
• The ground truth about metadata and community detection in networks
• Class of correlated random networks with hidden variables
• Community detection in networks with positive and negative links
• Analysis of the structure of complex networks at different resolution levels
• Statistical physics of inference: Thresholds and algorithms
• Spectral methods for network community detection and graph partitioning
• Random Walks, Markov Processes and the Multiscale Modular Organization of Complex Networks
• Laplacian Dynamics and Multiscale Modular Structure in Networks
• Phase transition in the detection of modules in sparse networks
• Structure and inference in annotated networks
• Community detection in networks: Modularity optimization and maximum likelihood are equivalent
• A Bayesian Approach to Network Modularity
• Modularity clustering is force-directed layout
• Hierarchical Block Structures and High-resolution Model Selection in Large Networks
• Parsimonious module inference in large networks
• Efficient Monte Carlo and greedy heuristic for the inference of stochastic block models
• Network structure from rich but noisy data
• Community Detection as an Inference Problem
• Identification of core-periphery structure in networks
• Nonparametric Bayesian inference of the microcanonical stochastic block model
• Scalable detection of statistically significant communities and hierarchies, using message-passing for modularity
• Efficiently inferring community structure in bipartite networks
• Evaluating Overfit and Underfit in Models of Network Community Structure
• The many facets of community detection in complex networks
• Stacking Models for Nearly Optimal Link Prediction in Complex Networks
• Network reconstruction and community detection from dynamics
• Community detection for correlation matrices
• Estimating the number of communities in a network
• Network histograms and universality of blockmodel approximation
• Bayesian stochastic blockmodeling
• When are networks truly modular?
• Model Selection for Degree-corrected Block Models
• Reconstructing networks with unknown and heterogeneous errors
• Statistical inference of assortative community structures
• Estimating the resolution limit of the map equation in community detection
• Clustering Drives Assortativity and Community Structure in Ensembles of Networks
• Consistencies and inconsistencies between model selection and link prediction in networks
• Consistency of community structure in complex networks
• Structural inference for uncertain networks
• Hierarchical multiresolution method to overcome the resolution limit in complex networks
• Descriptive vs. inferential community detection in networks: pitfalls, myths, and half-truths
• The Computer Science and Physics of Community Detection: Landscapes, Phase Transitions, and Hardness
• Community detection in networks without observing edges
• Merge-split Markov chain Monte Carlo for community detection
• Phase transitions in random Potts systems and the community detection problem: spin-glass type and dynamic perspectives
• Network structure, metadata and the prediction of missing nodes and annotations
• Revealing consensus and dissensus between network partitions
• Multilayer Network Science: from Cells to Societies
• Disentangling homophily, community structure and triadic closure in networks
• Reconstructing networks
• Universality of the stochastic block model
• Thermodynamics of Community Structure
• Representative community divisions of networks
• Gillespie algorithms for stochastic multiagent dynamics in populations and network
• Algorithmic detectability threshold of the stochastic block model

Cited by in corpus (9)

• Local dominance unveils clusters in networks
• Descriptive vs. inferential community detection in networks: pitfalls, myths, and half-truths
• Extreme rainfall propagation within Boreal Summer Intraseasonal Oscillation modulated by Pacific sea surface temperature
• Heuristic Modularity Maximization Algorithms for Community Detection Rarely Return an Optimal Partition or Anything Similar
• Random graphs and real networks with weak geometric coupling
• Quantifying the role of supernatural entities and the effect of missing data in Irish sagas
• P-TimeSync: A Precise Time Synchronization Simulation with Network Propagation Delays
• Kaleidoscopic reorganization of network communities across different scales
• Block-corrected Modularity for Community Detection