The Multi-Mission Data Processing System Study is being conducted to identify the best model for an open Mission Data Processing System (MDPS) to support upcoming NASA Earth science missions, particularly the coordinated missions comprising NASA's Earth System Observatory (ESO). The study will identify a data processing architecture that not only meets future mission science processing objectives and supports Earth system science, but also promotes open science principles and enables data system efficiencies.

The study comprises several phases, with each phase building on the results of the previous phase. In keeping with NASA open science guidelines and objectives, all phases in the study are open and encourage participation by the broader science and data processing communities. Phase 2: Design Review and Architecture Study is scheduled to conclude in fall 2024.

Phase 1: Identifying a Recommended Mission Data Processing System

The objective of Phase 1 was to evaluate and assess various system architectures and develop a recommendation for an architecture that would best serve future missions and data users. Two public workshops were conducted. Workshop #1 focused on collecting NASA stakeholder objectives and mission requirements. Workshop #2 studied practices across NASA and other agencies for developing science data processing systems. The information from these workshops was used to inform and identify potential architectures that could meet the study objectives. A technical trade study was performed along with a programmatic trade study of different architectures. The results from these studies were then combined to establish a final recommendation.

The recommendation of the study team is for upcoming missions to develop their own MDPS using a common architecture and services that are provided and managed by an overarching Multi-Mission Organization (MMO). The MMO will establish standards across missions and develop and deliver infrastructure, data catalog, analysis, and (potentially) processing services. This MDPS is designated Type 2 (Managed Services), Variant 4 (encompassing infrastructure, data, catalog, and analysis and processing services) and written as T2V4.

Workshop #1, October 19-20, 2021

We began collecting information with our first virtual two-day workshop on Tuesday, October 19, 2021 (programmatic presentations) and Wednesday, October 20, 2021 (flight project presentations).

The input provided, along with information collected in the second workshop (scheduled for February 2022) on the state-of-the-art in science processing and open science, will be used by the ESO Open Source Processing Study Architecture Working Group to develop and assess potential architectures.

Agenda

Open science is defined as a collaborative culture enabled by technology that empowers the open sharing of data, information and knowledge within the scientific community and the wider public to accelerate scientific research and understanding. Under the guidelines set forth in the 2017 Decadal survey, NASA's new set of Earth-focused missions that form the ESO will adhere to open data, software, algorithm, and publication policies where missions will develop algorithms (in Levels 1–4), software, and documentation in open systems from inception. In order to generate community input and support, this first workshop solicited input from NASA programmatic officials and flight project teams.

Recordings of the workshop are available: ESO Mission: Data Processing Study Workshop

The workshop report is also available: ESO Mission Data Processing Study - Summary of NASA Program Offices and ESO Missions Requirements, Constraints, Recommendations, and Opportunities.

Day 1: Tuesday October 19th, 2021 (1 - 5 PM EST)

Duration	EST	PST	Topic	Speaker
0:10	1:00 PM	10:00 AM	Welcome and Purpose	Andrew Mitchell
0:15	1:10 PM	10:10 AM	Earth Science Division (ESD) Opening Remarks	Kevin Murphy
0:15	1:25 PM	10:25 AM	SMD Scientific Information Policy	Steven Crawford
0:15	1:40 PM	10:40 AM	ESDS Perspective of Open Science	Katie Baynes
0:15	1:55 PM	10:55 AM	Overview of the Mission Processing Study	Andrew Bingham
0:15	2:10 PM	11:10 AM	Q&A with the SAWG
0:10	2:25 PM	11:25 AM	Break
0:15	2:35 PM	11:35 AM	Flight Projects Program Perspective	Charles Webb, Kathleen Boggs
0:15	2:50 PM	11:50 AM	Earth Science Technology Office (ESTO) Perspective	Pam Millar, Jacqueline Lemoigne-Stewart, Ben Smith
0:15	3:05 PM	12:05 PM	R&A Program Perspective	Jack Kaye
0:15	3:20 PM	12:20 PM	Q&A with the SAWG
0:10	3:35 PM	12:35 PM	Break
0:15	3:45 PM	12:45 PM	Applied Science Program Perspective	Nancy Searby
0:15	4:00 PM	1:00 PM	High Performance Computing (HPC) Perspective	Tsengdar Lee
0:10	4:15 PM	1:15 PM	Q&A with the SAWG
0:30	4:25 PM	1:25 PM	Open Discussion
	4:55 PM	1:55 PM	END

Day 2: Wednesday October 20th, 2021 (12:20 - 5 PM EST)

Duration	EST	PST	Topic	Speaker
0:10	12:20 PM	9:20 AM	Recap and Agenda
0:15	12:30 PM	9:30 AM	NISAR Programmatic Perspective	Gerald Bawden
0:15	12:45 PM	9:45 AM	NISAR Project Science	Paul Rosen
0:15	1:00 PM	10:00 AM	NISAR Project Data Processing Systems	Hook Hua
0:10	1:15 PM	10:15 AM	Q&A with the SAWG
0:05	1:25 PM	10:25 AM	Break
0:15	1:30 PM	10:30 AM	Atmosphere Observing System (AOS) Programmatic Perspective	Hal Maring
0:15	1:45 PM	10:45 AM	AOS Project Science	Scott Braun
0:15	2:00 PM	11:00 AM	AOS Project Data Processing Systems	Robert Wolfe
0:10	2:15 PM	11:15 AM	Q&A with the SAWG
0:05	2:25 PM	11:25 AM	Break
0:15	2:30 PM	11:30 AM	Mass Change (MC) Programmatic Perspective	Lucia Tsaoussi
0:15	2:45 PM	11:45 AM	MC Project Science	David Wiese
0:15	3:00 PM	12:00 PM	MC Project Data Processing Systems	David Wiese
0:10	3:15 PM	12:15 PM	Q&A with the SAWG
0:05	3:25 PM	12:25 PM	Break
0:20	3:30 PM	12:30 PM	Surface Biology and Geology (SBG) Overview and Project Science	Dave Schimel
0:20	3:50 PM	12:50 PM	SBG Project Data Processing Systems	Jeff Pon
0:05	4:10 PM	1:10 PM	SBG Programmatic Perspective	Ben Phillips
0:10	4:15 PM	1:15 PM	Q&A with the SAWG
0:30	4:25 PM	1:25 PM	Open Discussion
	4:55 PM	1:55 PM	END

Workshop #2, March 1-4, 2022

This workshop focused on understanding the state-of-the-art in Big Data processing systems. The workshop was open to the public. A Request For Information (RFI; submission period now closed) sought input from organizations with relevant expertise in Big Data processing and open science to help guide the study and workshop.

Recordings of the workshop are available: ESO Mission: Data Processing Study Workshop.

The workshop report is also available: ESO Mission Data Processing Study: Summary of State-of-the-Practice and State-of-the-Art Mission Data Processing System Architectures.

Please direct questions about the Open Source Science for the Earth System Observatory Mission Science Data Processing Study Workshops to: karen.yuen@jpl.nasa.gov or sara.h.lubkin@nasa.gov.

Agenda

Day 1: Tuesday, March 1, 2022 (12 - 5 pm EST)

Time ET	Time PST	Title	Speaker	Organization
12:00 PM	9:00 AM	Welcome and Study Overview	Andy Mitchell and Andy Bingham	GSFC/JPL
12:15 PM	9:15 AM	Logistics	Sara Lubkin and Karen Yuen	GSFC/JPL
12:20 PM	9:20 AM	Opening Remarks	Kevin Murphy	NASA
12:25 PM	9:25 AM	Open Source Science / SPD-41 doi:10.5281/zenodo.6320745	Katie Baynes	NASA
12:35 PM	9:35 AM	Session 1 introduction: Science collaboration approaches	Chelle Gentemann	Farallon Institute
12:40 PM	9:40 AM	SAR data exploration in environmental sciences: challenges and opportunities	Naiara Pinto	JPL
12:52 PM	9:52 AM	Imagining a new NASA computing platform with AI + big data-supported analysis doi:10.5281/zenodo.6320918	Willow Coleman	Harvey Mudd College
1:04 PM	10:04 AM	Big Data Community Algorithms: Deep Learning for Mapping doi:10.5281/zenodo.6320982	Morteza Karimzadeh	University of Colorado
1:16 PM	10:16 AM	Cloud Computing Platforms for Processing Geospatial Big Data: Current Status and Challenges. doi:10.5281/zenodo.6307732	Qiusheng Wu	University of Tennessee, Knoxville
1:28 PM	10:28 AM	Project Jupyter - Lessons and Principles from a Community-Driven Open Source Project doi:10.5281/zenodo.6320538	Fernando Perez	University of California, Berkeley
1:40 PM	10:40 AM	Fishbowl Discussion	Q&A led by the SAWG
2:00 PM	11:00 AM	Break
2:10 PM	11:10 AM	Session 2 introduction: NASA Earth Systematic Mission Processing Architectures	Andy Mitchell	GSFC
2:15 PM	11:15 AM	Terra MODIS doi:10.5281/zenodo.6320958	Robert Wolfe	GSFC
2:30 PM	11:30 AM	CLAREO doi:10.5281/zenodo.6345252	Chris Currey	LaRC
2:45 PM	11:45 AM	ICESat-2 doi:10.5281/zenodo.6320538	Tom Neumann	GSFC
3:00 PM	12:00 PM	PACE doi:10.5281/zenodo.6313234	Sean Bailey and Bryan Franz	GSFC
3:15 PM	12:15 PM	SWOT doi:10.5281/zenodo.6315458	Oh-Ig Kwoun	JPL
3:30 PM	12:30 PM	Fishbowl Discussion	Q&A led by the SAWG
3:45 PM	12:45 PM	Break
3:55 PM	12:55 PM	Break Room Topics & Logistics
4:10 PM	1:10 PM	Breakout Rooms 1. System development approaches & challenges 2. System operations approaches & challenges 3. Open-sourced science approaches & challenges 4. Data analysis needs for Mission Data Processing System (MDPS) 5. Open source software approaches & challenges 6. MDPS Architectures now and the future
4:40 PM	1:40 PM	Report out from Breakout rooms
5:10 PM	2:10 PM	END

Day 2: Wednesday, March 2, 2022 (12 - 4:30 pm EST)

Time ET	Time PST	Title	Speaker	Organization
12:00 PM	9:00 AM	Logistics	Sara Lubkin and Karen Yuen	GSFC/JPL
12:05 PM	9:05 AM	Session 3 introduction: NASA Earth System Science Pathfinder mission processing architectures	Andy Mitchell	GSFC
12:10 PM	9:10 AM	OCO-2/OCO-3 doi:10.5281/zenodo.6321343	Cecilia Cheng	JPL
12:25 PM	9:25 AM	EMIT	Phil Brodrick	JPL
12:40 PM	9:40 AM	MAIA doi:10.5281/zenodo.6323718	Jeff Walter	LaRC
12:55 PM	9:55 AM	Break
1:05 PM	10:05 AM	TROPICS doi:10.5281/zenodo.6323291	Jess Braun	University of Wisconsin, Madison
1:20 PM	10:20 AM	GEDI	Scott Luthke	GSFC
1:35 PM	10:35 AM	Fishbowl Discussion	Q&A led by the SAWG
1:50 PM	10:50 AM	Session 4 introduction: Non-NASA Earth science mission processing architectures	Andy Bingham	JPL
1:55 PM	10:55 AM	National Oceanic and Atmospheric Administration doi:10.5281/zenodo.6307369	Lihang Zhou and Joseph Mani	NOAA
2:10 PM	11:10 AM	U.S. Geological Survey	Chris Engebretson	USGS
2:25 PM	11:25 AM	Fishbowl Discussion	Q&A led by the SAWG
2:40 PM	11:40 AM	Break
2:50 PM	11:50 AM	Breakout Rooms 1. System development approaches & challenges 2. System operations approaches & challenges 3. Open-sourced science approaches & challenges 4. Data analysis needs for Mission Data Processing System (MDPS) 5. Open source software approaches & challenges 6. MDPS Architectures now and the future
3:20 PM	12:20 PM	Report out from Breakout rooms
3:50 PM	12:50 PM	Japan Aerospace Exploration Agency doi:10.5281/zenodo.6319381	Osamu Ochia	JAXA
4:05 PM	1:05 PM	END

Day 3: Thursday, March 3, 2022 (12 - 5:20 pm EST)

Time ET	Time PST	Title	Speaker	Organization
12:00 PM	9:00 AM	Logistics
12:05 PM	9:05 AM	National Remote Sensing Center, India Space Research Organization (ISRO) doi:10.5281/zenodo.6302928	Uzair Mujeeb	ISRO
12:20 PM	9:20 AM	Italian Space Agency (ASI)	Francesco Tataranni	ASI
12:35 PM	9:35 AM	German Aerospace Center (DLR) doi:10.5281/zenodo.6315122	Conrad Albrecht	DLR
12:50 PM	9:50 AM	European Space Agency (ESA)	Anca Anghelea, Klaus Scipal, and Clement Albinet	ESA
1:05 PM	10:05 AM	Fishbowl Discussion	Q&A led by the SAWG
1:20 PM	10:20 AM	Break
1:30 PM	10:30 AM	Session 5 introduction: Non-Earth science mission processing architectures	Luke Dahl
1:35 PM	10:35 AM	Vera C. Rubin Observatory's Legacy Survey of Space and Time (LSST) doi:10.5281/zenodo.6311597	KT Lim	Vera C. Rubin Observatory
1:50 PM	10:50 AM	Nancy Grace Roman Space Telescope doi:10.5281/zenodo.6326463	Megan Soey	Space Telescope Science Institute (STScI)
2:05 PM	11:05 AM	IPAC Missions	Gregory Dubois-Felsmann	Caltech
2:20 PM	11:20 AM	Fishbowl Discussion	Q&A led by the SAWG
2:35 PM	11:35 AM	Break
2:45 PM	11:45 AM	Session 6 introduction: System interfaces and standards	Andy Bingham
2:50 PM	11:50 AM	Earth Science Data and Information System (ESDIS) doi:10.5281/zenodo.6402353	Sara Lubkin	GSFC
3:05 PM	12:05 PM	NASA Transform to Open Science (TOPS) doi:10.5281/zenodo.6324320	Chelle Gentemann	Farallon Institute
3:20 PM	12:20 PM	NASA's High-End Computing Capability: Growing to Support Science Data Processing for the Earth System Observatory Missions doi:10.5281/zenodo.6330014	Jon Jenkins	Ames Research Center
3:35 PM	12:35 PM	Data System Standards	Ingo Simonis	OGC
3:50 PM	12:50 PM	Fishbowl Discussion	Q&A led by the SAWG
4:05 PM	1:05 PM	Break
4:05 PM	1:05 PM	Breakout Rooms 1. System development approaches & challenges 2. System operations approaches & challenges 3. Open-sourced science approaches & challenges 4. Data analysis needs for Mission Data Processing System (MDPS) 5. Open source software approaches & challenges 6. MDPS Architectures now and the future
4:35 PM	1:35 PM	Report out from Breakout rooms
5:05 PM	2:05 PM	END

Day 4: Friday, March 4, 2022 (12 - 4 pm EST)

Time ET	Time PST	Title	Speaker	Organization
12:00 PM	9:00 AM	Logistics
12:05 PM	9:05 AM	Session 7 introduction: Other Big Data processing system architectures	Luke Dahl
12:10 PM	9:10 AM	CyVerse	Tyson Swetnam, Edwin Skidmore	University of Arizona
12:25 PM	9:25 AM	Pangeo doi:10.5281/zenodo.6330215	Joe Hamman	Carbon Plan
12:40 PM	9:40 AM	Alsaka SAR Facility doi:10.5281/zenodo.6354724	Kirk Hogenson	Alaska Satellite Facility
12:55 PM	9:55 AM	Raytheon doi:10.5281/zenodo.6325581	Shawn Miller	Raytheon
1:10 PM	10:10 AM	Element 84 STAC doi:10.5281/zenodo.6326572	Dan Pilone	Element 84
1:25 PM	10:25 AM	Break
1:35 PM	10:35 AM	Fishbowl Discussion	Q&A led by the SAWG
1:50 PM	10:50 AM	Red Hat	Dan McGuan	Red Hat
2:05 PM	11:05 AM	Amazon Web Services (AWS)		AWS
2:20 PM	11:20 AM	Multi-Mission Algorithm and Analysis Platform (MAAP) doi:10.5281/zenodo.6345081	George Chang	JPL
2:35 PM	11:35 AM	OpenNEX doi:10.5281/zenodo.6345019	Jennifer Dungan and Ian Brosnan	ARC
2:50 PM	11:50 AM	Break
3:00 PM	12:00 PM	Unity (Science Data System as a Service) doi:2014/54386	Hook Hua and Laura Jewell	JPL
3:15 PM	12:15 PM	Science Data Analytics Platform (SDAP) doi:10.5281/zenodo.6310363	Nga Chung and Thomas Huang	JPL
3:30 PM	12:30 PM	NASA Earth Information System (EIS) doi.10.5281/zenodo.6353694	Alexey Shiklomanov	GSFC
3:45 PM	12:45 PM	Fishbowl Discussion	Q&A led by the SAWG
4:00 PM	1:00 PM	END of Workshop

Phase 1 Study Participants

Name	Focus Area	Organization	Required Expertise
Elias Sayfi	Co-Chair	JPL	Flight mission data processing system architect. Community builder
Natasha Stavros	Co-Chair	University of Colorado, Boulder	Science data processing system architect Community builder
Hook Hua	SBG and NISAR Science Processing Representative	JPL	Science data processing systems Level 3 (L3) requirements and mission objectives
Curt Tilmes	AOS Science Processing Representative	Goddard/Ames Research Center	Science data processing systems L3 requirements and mission objectives
Bernie Bienstock	MC Science Processing Representative	JPL	Science data processing systems L3 requirements and mission objectives
Wenying Su	Applied Science Algorithm Developer	Langley Research Center	Expertise developing L3+ algorithms for science applications Experience integrating algorithms into a pipeline. Experience working in a collaborative development environment.
Andy Michaelis	Data Processing Workflow Community-focused Developer	Ames	Expertise in modern open-source development practices, cloud-based orchestration, GitHub, Continuous Integration. Experience with big data applications (science preferred, but not necessary)
Lesley Ott	Geophysical Numerical Modeler	Goddard	Expertise with HPC Expertise with big data assimilation and operations
Qing Yue	Instrument Algorithm Developer	JPL	L2 algorithm development Science data processing systems
Evelyn Ho	Science Processing Architect from a comparable big-data science-based organization or project	Earth Science Data Information System (ESDIS) Science Investigator-led Processing Systems (SIPS)
Chris Engebretson	Science Processing Architect from a comparable big-data science-based organization or project	USGS
Adrian Parker	Science Processing Architect from a comparable big-data science-based organization or project	NOAA
Sean Harkin	Science Processing Architect from a comparable big-data science-based organization or project	Marshall Space Flight Center

Workshop Recrdings and Final Report

View the workshop recordings and read the final workshop report.