Cybersecurity for Research Cyberinfrastructure and High-Performance Computing

Berkeley Lab Computing Sciences Research is an active participant in a number of projects in the arena of security for scientific, high-performance computing systems and high-bandwidth research and education networks. Research sponsors have typically included DOE’s ASCR program, and the National Science Foundation (NSF) SaTC program and OAC office, among others.

Berkeley Lab’s cybersecurity goals are to research, develop, evaluate, adapt, and integrate advanced security and privacy solutions that enable or improve scientific workflows that may otherwise not be possible due to real or perceived security restrictions that, using today’s solution, impose onerous usability and/or performance constraints, thereby hindering scientific progress.

Berkeley Lab has had a leadership role in security in research cyberinfrastructure environments for many years, including the development of the Zeek (Bro) Network Security Monitor, the 100G performance enhancements of Zeek (Bro), and Zeek (Bro)’s commercial spin-off, Corelight, Inc., as well as leading several DOE-sponsored activities related to defining a cybersecurity research program within the DOE Office of Science. Berkeley Lab is the lead institution of Trusted CI, the NSF Cybersecurity Center of Excellence.

Recent highlights of Berkeley Lab Computing Sciences’ cybersecurity R&D activities in research cyberinfrastructure include:

Leading studies into scientific data integrity, scientific data confidentiality, and software assurance in science, operational technology in science, and building security into NSF Major Facilities by design. ⇒ The latter is directly impacting design, construction, and operations of the California Coastal Research Vessel, the NSF Regional Class Research Vessels, U.S. Antarctic Program’s $1B icebreaker, and Ocean Observatory Initiatives’ replacement of hundreds of underwater autonomous vehicles.
Developed definitions and R&D roadmaps for hardware/software co-design of future HPC systems, high-throughput networks, and networked scientific instruments to build cybersecurity in by design. ⇒ Led directly to HPC cybersecurity elements of DOE funding solicitations and has been central to NIST HPC Security working group.
Development of secure computation architectures optimized for scientific computing to ensure trustworthiness of scientific data from the edge to the HPC center.
Development and application of differential privacy to power grid and vehicle mobility data and applications ⇒ The DOE Office of Cybersecurity, Energy Security, and Emergency Response (CESER) is seeking to deploy the former operationally and the latter has already enabled mobility research otherwise not possible due to data sharing restrictions.
Co-leading the Open Science Cyber Risk Profile (OSCRP) working group — an approach to help research cyberinfrastructure operators understand cyber risks. ⇒ Now a recommended reference in all NSF CICI solicitations since 2018 and the NSF Research Infrastructure Guide (RIG) (21-107, Dec. 2021).
Codification of the “Medical Science DMZ” — a “network design pattern” for enabling secure, high-volume, high-throughput transfer of sensitive data, such as data subject to HIPAA or CUI regulations. ⇒ Now used by companies and research institutions globally, including the NSF Global Research Platform.

DOE Cybersecurity Workshops and Reports

ASCR Cybersecurity for Scientific Computing Workshop, June 2–3, 2015 [report]

ASCR Cybersecurity Workshop, January 7–9, 2015 [report, news]

DOE Grassroots Cybersecurity Initiative, 2008–2010 [Frincke presentation, Catlett ASCAC presentation, report 1, report 2, report 3]

DOE Cybersecurity R&D Challenges for Open Science: Developing a Roadmap and Vision, January 24–26, 2007 [news, report]

Some recent news:

Cybersecurity Center of Excellence Receives Five-Year, $6M/Year Award From NSF [expanded announcement] — Oct. 3, 2024

Announcing publication of the Operational Technology Procurement Vendor Matrix — Dec. 15, 2023

Updates on Trusted CI’s Efforts in Cybersecurity by Design of NSF Academic Maritime Facilities — Jul. 24, 2023

SAVE THE DATE: Announcing the 2023 NSF Cybersecurity Summit, Oct 24-26, 2023 in Berkeley, CA — Mar. 21, 2023

Registration Open for 3rd HPC Security Workshop at NIST NCCoE — Feb. 3, 2023

Announcing the 2023 Trusted CI Annual Challenge: Building Security Into NSF Major Facilities By Design — Jan. 25, 2023

Sean Peisert, Publication of the Trusted CI Roadmap for Securing Operational Technology in NSF Scientific Research — Nov. 16, 2022.

Sean Peisert, Open Science Cyber Risk Profile (OSCRP) Updated with Science DMZ, Software Assurance, Operational Technology, and Cloud Computing Elements — Nov. 1, 2022

Findings of the 2022 Trusted CI Study on the Security of Operational Technology in NSF Scientific Research — July 15, 2022

Berkeley Lab’s Sean Peisert Tapped to Take on Deputy Director Role — June 28, 2022

Better Scientific Software (BSSw) Helps Promote Trusted CI Guide to Securing Scientific Software — May 13, 2022

Announcing the 2022 Trusted CI Annual Challenge on Scientific OT/CPS Security - Jan. 5, 2022

Older News

Key Representative Publications:

Andrew Adams, Dan Arnold, Jeannette Dopheide, Ryan Kiser, Mark Krenz, Drew Paine, Sean Peisert, Michael Simpson, and John Zage, “Trusted CI Operational Technology Procurement Vendor Matrix,” Dec. 14, 2023. DOI: 10.5281/zenodo.10257813

Jim Basney, Sean Peisert, Scott Russell, Kelli Shute, Bart Miller, and Kathy Benninger, “A Vision for Securing NSF’s Essential Scientific Cyberinfrastructure - Trusted CI Five-Year Strategic Plan (2024-2029),” August 1, 2023.

Ammar Haydari, Chen-Nee Chuah, Michael Zhang, Jane Macfarlane, and Sean Peisert, “Differentially Private Map Matching for Mobility Trajectories,” Proceedings of the 2022 Annual Computer Security Applications Conference (ACSAC), Austin, TX, December 5-9, 2022.

Andrew Adams, Emily K. Adams, Dan Gunter, Ryan Kiser, Mark Krenz, Sean Peisert, and John Zage. “Roadmap for Securing Operational Technology in NSF Scientific Research,” Trusted CI Report, November 16 2022.

Ayaz Akram, Venkatesh Akella, Sean Peisert, and Jason Lowe-Power, “SoK: Limitations of Confidential Computing via TEEs for High-Performance Compute Systems,” Proceedings of the 2022 IEEE International Symposium on Secure and Private Execution Environment Design (SEED), Sept. 26–27, 2022.

Andrew Adams, Kay Avila, Elisa Heymann, Mark Krenz, Jason R. Lee, Barton Miller, and Sean Peisert, “Guide to Securing Scientific Software,” Trusted CI Report, December 14, 2021.

Sean Peisert, “Trustworthy Scientific Computing,” Communications of the ACM (CACM), 64(5), pp. 18–21, May 2021.

Ayaz Akram, Anna Giannakou, Venkatesh Akella, Jason Lowe-Power, and Sean Peisert, “Performance Analysis of Scientific Computing Workloads on General Purpose TEEs,” Proceedings of the 35th IEEE International Parallel & Distributed Processing Sysmposium (IPDPS), May 17–21, 2021.

Sean Peisert, Eli Dart, William K. Barnett, James Cuff, Robert L. Grossman, Edward Balas, Ari Berman, Anurag Shankar, and Brian Tierney, “The Medical Science DMZ: An Network Design Pattern for Data-Intensive Medical Science”, Journal of the American Medical Informatics Association (JAMIA), 25,(3):267–274, March 2018.

Sean Peisert, “Security in High-Performance Computing Environments”, Communications of the ACM (CACM), 60(9):72-80, September 2017.

Sean Peisert, Von Welch, Andrew Adams, Michael Dopheide, Susan Sons, RuthAnne Bevier, Rich LeDuc, Pascal Meunier, Stephen Schwab, and Karen Stocks, Ilkay Altintas, James Cuff, Reagan Moore, and Warren Raquel, “Open Science Cyber Risk Profile,” February 2017.

Sean Whalen, Sean Peisert, Matt Bishop, “Multiclass Classification of Distributed Memory Parallel Computations,” Pattern Recognition Letters (PRL), 34(3):322-329, February 2013.

Projects

Trusted CI, the National Science Foundation Cybersecurity of Excellence

The mission of Trusted CI is to improve the cybersecurity of NSF computational science and engineering projects, while allowing those projects to focus on their science endeavors. Sean Peisert is the Director- and PI-Designate of Trusted CI.

Mitigation via Analytics for Inverter-Grid Cybersecurity (MAGIC)

Project MAGIC will develop artificial intelligence and machine learning algorithms to detect and mitigate cyber attacks on aggregations of Distributed Energy Resources (DER). The developed algorithms will be demonstrated in hardware-in-the-loop testing and integrated into an open source simulation tool. It is funded by DOE CESER’s RMT program and is led by Daniel Arnold.

Privacy-Preserving, Collective Cyberattack Defense of DERs

This project aims to develop, apply, and test a technique for enabling collective defense of distribution grids with significant penetration of distributed energy resources (DER) and responsive loads, by leveraging a privacy-preserving method of data sharing without exposing raw data that might contain personally identifiable information (PII) or that might otherwise be considered national security information that could be leveraged by adversaries to more effectively compromise and potentially destabilize portions of the electric grid. It is funded by DOE CESER’s RMT program and is led by Sean Peisert.

Using Fuzz Testing to Detect Software Tampering

This project aims verify that software operating on arms control monitoring equipment is within agreed parameters. It is funded by the National Nuclear Security Administration Office of Defense Nuclear Nonproliferation Research and Development and is led by Sean Peisert.

Data Enclaves for Scientific Computing

This project will develop secure computation architectures to ensure trustworthiness of scientific data while addressing the gaps left by existing solutions for scientific workflows to address the specific power, performance, and usability, and needs from the edge to the HPC center. It is led by Sean Peisert, Venkatesh Akella, and Jason Lowe-Power.

Privacy-Preserving Data Analysis for Scientific Discovery

This project aims to produce methods, processes, and architectures applicable to a variety of scientific computing domains that enables querying, machine learning, and analysis of data while protecting against releasing sensitive information beyond pre-defined bounds. It is supported by LBNL CSR funds and is led by Sean Peisert.

Securing Automated, Adaptive Learning-Driven Cyber-Physical System Processes

This project is developing secure machine learning methods that will enable safer operation of automated, adaptive, learning-driven cyber-physical system processes. It is supported by LBNL LDRD funds and is led by Sean Peisert.

Provable Anonymization of Grid Data for Cyberattack Detection

This project aims to develop techniques for enabling data analysis for the purposes of detecting and/or investigating cyberattacks against energy delivery systems while also preserving aspects of key confidentiality elements within the underlying raw data being analyzed. The result will be a complete solution for anonymization of data collected from OT and IT networks pertaining to energy grid cyberattack detection that has been tested for its ability to retain privacy properties and still enable attack detection. It is funded by DOE CESER’s CEDS program and is led by Sean Peisert.

Supervisory Parameter Adjustment for Distribution Energy Storage (SPADES)

This project is developing the methodology and tools allowing Electric Storage Systems (ESS) to automatically reconfigure themselves to counteract cyberattacks, both directly against the ESS control systems and indirectly through the electric grid. It is funded by DOE CESER’s CEDS program and is led by Daniel Arnold.

Securing Solar for the Grid (S2G)

This project aims to develop an understanding of security and performance requirements for the use of AI high solar / IBR / DER penetration scenarios, and also to develop an understanding of understanding power grid data confidentiality and privacy requirements. It is funded by DOE’s SETO office and is co-led by Sean Peisert and Daniel Arnold.

AOSCSWAP: Study of Academic, Open Source, and COTS Software Assurance Products

In this project, LBNL will help inform DHS S&T regarding the state of the art in software assurance tools and capabilities. It is funded by DHS S&T and is led by Sean Peisert.

Cybersecurity via Inverter-Grid Automatic Reconfiguration (CIGAR)

This project performed R&D to enable distribution grids to adapt to resist a cyber-attack by (1) developing adaptive control algorithms for DER, voltage regulation, and protection systems; (2) analyze new attack scenarios and develop associated defensive strategies. It was funded by DOE’s CEDS program and was co-led by Sean Peisert and Daniel Arnold.

Byzantine Security — Multi-layered Intrusion Tolerant Byzantine Architecture for Bulk Power System Protective Relays

This project aims to explore applications of a Byzantine Fault Tolerant (BFT) architecture in combination with ML/AI methods to ensure that the bulk power system, including protective relays in the transmission grid, and associated substation and control center systems, can perform intrusion tolerant operations. It is funded by the DOE Grid Modernization Initiative. The LBNL portion of this effort is led by Sean Peisert.

Synthetic Biology Automation

In this project, LBNL Computing Sciences Research supported the automation of synthetic biology research pipelines supporting the design-build-test-learn (DBTL) cycle, including ingest and analysis of liquid chromatography mass spectrometry and feedstocks-to-fuels pipelines.

Medical Science DMZ

We have defined a Medical Science DMZ as a method that allows data flows at scale while simultaneously addressing the HIPAA Security Rule and related regulations governing biomedical data and appropriately managing risk.

UC-Lab Center for Electricity Distribution Cybersecurity

This project will bring together a multi-disciplinary UC-Lab team of cybersecurity and electricity infrastructure experts to investigate, through both cyber and physical modeling and physics-aware cybersecurity analysis, the impact and significance of cyberattacks on electricity distribution infrastructure. It is funded by the UC-Lab Fees Research Program. The overall project is led by Hamed Mohsenian-Rad; the LBNL portion is led by Sean Peisert.

Integrated Multi Scale Machine Learning for the Power Grid

The goal of this project is to create advanced, distributed data analytics capability to provide visibility and controllability to distribution grid operators. It is funded by the DOE Grid Modernization Initiative. The LBNL portion of this effort is led by Sean Peisert.

Detecting Distributed Denial of Service Attacks on Wide-Area Networks

This project develops techniques for detecting DDoS attacks and disambiguating them from large-scale science flows. It is funded by the DOE iJC3 Cyber R&D program and is led by Sean Peisert.

Toward a Hardware/Software Co-Design Framework for Ensuring the Integrity of Exascale Scientific Data

This project takes a broad look at several aspects of security and scientific integrity issues in HPC systems. It is funded by DOE ASCR and is led by Sean Peisert.

Power Grid Threat Detection and Response with Data Analytics

The goal of this project is to develop technologies and methodologies to protect the nation’s power grid from advanced cyber and all-hazard threats. This will be done through the collection of disparate data and the use of advanced analytics to detect threats and response to them. It is funded by DOE OE’s CEDS program via the Grid Modernization Initiative and is co-led by Sean Peisert.

Inferring Computing Activity Using Physical Sensors

This project uses power data to monitor the use of computing systems, including supercomputers and large computing centers. It is led by Sean Peisert.

An Automated, Disruption Tolerant Key Management System for the Power Grid

This project is designing and developing a key management system to meet the unique requirements of electrical power distribution systems. It is funded by DOE OE’s CEDS program and is led by Sean Peisert.

Bedside to the Cloud and Back

This project is developing a system-based workflow to securely acquire wireless data from mechanical ventilators in critical care environments, and leverage scalable web-based analytic platforms to advance data analytics and visualization of issues surrounding patients with respiratory failure.

Host and Network Resilience

This project focused on mapping and analyzing the qualities of resilient networks by investigating components of redundancy, diversity, quality of service, etc… The project’s goal is to be able to quantify and compare the resilience of networks in a scientifically meaningful way. This project was led at LBNL by Sean Peisert.

Symbiosis in Byzantine Fault Tolerance and Intrusion Detection

This project was funded by NSF’s SaTC program, and was co-led by Sean Peisert. The theme of this effort was to integrate Byzantine fault tolerance (BFT) into intrusion detection systems (IDS), at both the fundamental and system levels, thereby improving both BFT and IDS. potential to improve BFT.

NetSage - an open privacy-aware network measurement, analysis, and visualization service

NetSage is a network measurement, analysis and visualization service funded by the National Science Foundation and is designed to address the needs of today’s international networks. This project is co-led by Sean Peisert at LBNL.

Cyber Security of Power Distribution Systems by Detecting Differences Between Real-time Micro-Synchrophasor Measurements and Cyber-Reported SCADA

This project used micro-PMU measurements and SCADA commands to develop a system to detect cyberattacks against the power distribution grid. It was funded by DOE OE’s CEDS program and was led by Sean Peisert.

LBNL Power Data

This distribution level phasor measurement data can be used to understand ways to enables advanced diagnostic, monitoring and control methodologies in distribution systems.

NNSA Cyber Sciences Lab (CSL)

Using seed funding from the NNSA CIO, this consortium of eight DOE laboratories worked to form an enduring, national computer security research laboratory to address cybersecurity threats. LBNL’s effort was led by Deb Agarwal and Sean Peisert.

The Hive Mind: Applying a Distributed Security Sensor Network to GENI.

This project sought to define and prototype a security layer using a method of intrusion detection based on mobile agents and swarm intelligence. The project was funded by NSF’s CISE Directorate, and was led by Sean Peisert.

Application of Cyber Security Techniques in the Protection of Efficient Cyber-Physical Energy Generation Systems

The goal of this project was to design and implement a measurement network, which can detect and report the resultant impact of cyber security attacks on the distribution system network. It was funded by DOE OE’s CEDS program and was co-led by Chuck McParland and Sean Peisert.