Boba: Authoring and Visualizing Multiverse Analyses

Abstract

Multiverse analysis is an approach to data analysis in which all "reasonable" analytic decisions are evaluated in parallel and interpreted collectively, in order to foster robustness and transparency. However, specifying a multiverse is demanding because analysts must manage myriad variants from a cross-product of analytic decisions, and the results require nuanced interpretation. We contribute Boba: an integrated domain-specific language (DSL) and visual analysis system for authoring and reviewing multiverse analyses. With the Boba DSL, analysts write the shared portion of analysis code only once, alongside local variations defining alternative decisions, from which the compiler generates a multiplex of scripts representing all possible analysis paths. The Boba Visualizer provides linked views of model results and the multiverse decision space to enable rapid, systematic assessment of consequential decisions and robustness, including sampling uncertainty and model fit. We demonstrate Boba's utility through two data analysis case studies, and reflect on challenges and design opportunities for multiverse analysis software.

submitted to IEEE Transactions on Visualization and Computer Graphics (Proc. VAST)

References in corpus (4)

• Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC
• Using stacking to average Bayesian predictive distributions
• Paths Explored, Paths Omitted, Paths Obscured: Decision Points & Selective Reporting in End-to-End Data Analysis
• Tea: A High-level Language and Runtime System for Automating Statistical Analysis

Cited by in corpus (6)

• Causalvis: Visualizations for Causal Inference
• CoPrompt: Supporting Prompt Sharing and Referring in Collaborative Natural Language Programming
• Tisane: Authoring Statistical Models via Formal Reasoning from Conceptual and Data Relationships
• Post-selection Inference in Multiverse Analysis (PIMA): an inferential framework based on the sign flipping score test
• MetaExplorer: Facilitating Reasoning with Epistemic Uncertainty in Meta-analysis
• Toward Human-Quantum Computer Interaction: Interface Techniques for Usable Quantum Computing