Hypersparse Neural Network Analysis of Large-Scale Internet Traffic

Abstract

The Internet is transforming our society, necessitating a quantitative understanding of Internet traffic. Our team collects and curates the largest publicly available Internet traffic data containing 50 billion packets. Utilizing a novel hypersparse neural network analysis of "video" streams of this traffic using 10,000 processors in the MIT SuperCloud reveals a new phenomena: the importance of otherwise unseen leaf nodes and isolated links in Internet traffic. Our neural network approach further shows that a two-parameter modified Zipf-Mandelbrot distribution accurately describes a wide variety of source/destination statistics on moving sample windows ranging from 100,000 to 100,000,000 packets over collections that span years and continents. The inferred model parameters distinguish different network streams and the model leaf parameter strongly correlates with the fraction of the traffic in different underlying network topologies. The hypersparse neural network pipeline is highly adaptable and different network statistics and training models can be incorporated with simple changes to the image filter functions.

arXiv comment11 pages, 10 figures, 3 tables, 60 citations; to appear in IEEE High Performance Extreme Computing (HPEC) 2019

References in corpus (10)

• Emergence of scaling in random networks
• Power-law distributions in empirical data
• The diameter of the world wide web
• Community Structure in Time-Dependent, Multiscale, and Multiplex Networks
• Beyond the Zipf-Mandelbrot law in quantitative linguistics
• Power-law distributions in binned empirical data
• Interactive Supercomputing on 40,000 Cores for Machine Learning and Data Analysis
• ${\ell }_0$-Motivated Low-Rank Sparse Subspace Clustering
• Long-Range Correlations and Memory in the Dynamics of Internet Interdomain Routing
• A Scale Invariant Approach for Sparse Signal Recovery

Cited by in corpus (11)

• Spatial Temporal Analysis of 40,000,000,000,000 Internet Darkspace Packets
• 75,000,000,000 Streaming Inserts/Second Using Hierarchical Hypersparse GraphBLAS Matrices
• GraphBLAS on the Edge: Anonymized High Performance Streaming of Network Traffic
• GraphChallenge.org Triangle Counting Performance
• Large Scale Enrichment and Statistical Cyber Characterization of Network Traffic (Enriquecimiento a gran escala y caracterización cibernética estadística del tráfico de red)
• Anonymized Network Sensing Graph Challenge
• Deployment of Real-Time Network Traffic Analysis using GraphBLAS Hypersparse Matrices and D4M Associative Arrays
• Temporal Correlation of Internet Observatories and Outposts
• Focusing and Calibration of Large Scale Network Sensors using GraphBLAS Anonymized Hypersparse Matrices
• Teaching Network Traffic Matrices in an Interactive Game Environment
• What is Normal? A Big Data Observational Science Model of Anonymized Internet Traffic