NetSage - an open privacy-aware network measurement, analysis, and visualization service

Principal Investigators

Jen Schopf (Lead PI at IU)
Sean Peisert
(Former Co-PI; Former Lead at LBNL and UC Davis → Security Advisor)
Andrew Lake (Current Lead at LBNL) Jason Leigh (Co-PI; Lead at UH)

LBNL Project Alumni:

Jon Dugan
Dipankar Dwivedi (Postdoc)
Anna Giannakou (Postdoc)
Monte Goode
Brian Tierney (ESnet Scientist 1988-2017) → retired
Chris Tracy

NetSage is an open privacy-aware network measurement, analysis, and visualization service designed to address the needs of today’s international networks. Modern science is increasingly data-driven and collaborative in nature, producing petabytes of data that can be shared by tens to thousands of scientists all over the world. The NSF-supported International Research Network Connection (IRNC) links have been essential to performing these science experiments.

Providing near real-time monitoring and visualization of international data transfers will help ensure that scientific workflows are operating at maximum efficiency. NetSage services provide an unprecedented combination of passive measurements, including SNMP data, flow data, and Zeek/Bro-based traffic analysis, as well as active measurements, mainly perfSONAR, and longitudinal network performance data visualization. User privacy is a significant concern in this project given the data flowing through the exchange points. NetSage addresses these concerns through the use of a privacy advisory board that will ensure the data gathering activities are conducted to meet all community standards. The proposed work is a partnership between Indiana University, University of California at Davis, Lawrence Berkeley National Laboratory, and University of Hawaii at Manoa. This uniquely strong team combines backgrounds in production international network support, networking measurement and prediction tools, network-intensive applications and data visualization.

This project is supported by the National Science Foundation’s  International Research Network Connections (IRNC) program.

Read more and demo the interface at the project web site.

Netflow Analysis and Prediction Source Code at GitHub

Publications resulting from this project:

Anna Giannakou, Dipankar Dwivedi, and Sean Peisert, “A Machine Learning Approach for Packet Loss Prediction in Science Flows,” Future Generation Computer Systems - Special Issue on Innovating the Network for Data Intensive Science - INDIS 2018, accepted 25 July, 2019.

Alberto Gonzalez, Jason Leigh, Sean Peisert, Brian Tierney, Andrew Lee, Jennifer M. Schopf, “Monitoring Big Data Transfers Over International Research Network Connections”, Proceedings of IEEE BigData Congress 2017, Honolulu, Hawaii, June 2017,

Alberto Gonzalez, Jason Leigh, Sean Peisert, Brian Tierney, Andrew Lee, Jennifer M. Schopf, “NetSage: Open Privacy-Aware Network Measurement, Analysis, And Visualization Service”, Proceedings of TNC16 Networking Conference, Prague, Czech Republic, June 2016,

Software resulting from this project:

Research Network Transfer Performance Predictor (netperf-predict)

More information is available on other Berkeley Lab R&D projects focusing on cybersecurity in general, as well as specifically on cybersecurity for scientific and high-performance computing.