Open Source Science for the Earth System Observatory Mission Science Data Processing Study Workshops

Image based on a similar image created by Ramachandran, et al., and published in "From Open Data to Open Science." Earth and Space Science [doi:10.1029/2020EA001562].
NASA is formulating the Earth System Observatory (ESO), a set of Earth-focused missions to provide key information to guide efforts related to monitoring climate change, mitigating disasters, fighting forest fires, and improving real-time agricultural processes. With this integrated approach, Kevin Murphy (Chief Science Data Officer, Science Mission Directorate, and Program Manager, Earth Science Data Systems [ESDS] Program) has set forth a challenge to the mission processing community to:
Identify and assess potential architectures that can meet the ESO mission science processing objectives, enable data system efficiencies, promote open science principles, and seek opportunities that support Earth system science.
A mission data processing system is the set of algorithms, software, compute infrastructure, operational procedures, documentation, and teams that automatically process raw instrument data through to science-quality data products. This includes the software tools that support the development of the processing algorithms and the validation and analysis of the processed data.
Open science is a foundational objective of NASA’s Science Mission Directorate (SMD) and is defined as “a collaborative culture enabled by technology that empowers the open sharing of data, information, and knowledge within the scientific community and the wider public to accelerate scientific research and understanding” (Ramachandran, R., Bugbee, K. & Murphy, K.J. From Open Data to Open Science. Earth and Space Science, 8(5), doi:10.1029/2020EA001562). On October 14, SMD will report out on open science and activities at NASA's Open Source Science for Data Processing and Archives Workshop.
The Open Source Science for ESO Mission Science Data Processing Study has a Steering Committee and System Architecture Working Group (SAWG) to address the challenge laid out above. The Study will consist of three workshops and a trade study designed to solicit input, perform systematic trade, and make a recommendation.
NASA Open-Source Science Initiative will be holding open science listening sessions in 2022. For more information on these sessions, please subscribe to the open-source mailing list.
Workshop #1, October 19-20, 2021
We began collecting information with our first virtual two-day workshop on Tuesday, October 19, 2021 (programmatic presentations) and Wednesday, October 20, 2021 (flight project presentations).
The input provided, along with information collected in the second workshop (scheduled for February 2022) on the state-of-the-art in science processing and open science, will be used by the ESO Open Source Processing Study Architecture Working Group to develop and assess potential architectures.
Agenda
Open science is defined as a collaborative culture enabled by technology that empowers the open sharing of data, information and knowledge within the scientific community and the wider public to accelerate scientific research and understanding. Under the guidelines set forth in the 2017 Decadal survey, NASA's new set of Earth-focused missions that form the ESO will adhere to open data, software, algorithm, and publication policies where missions will develop algorithms (in Levels 1–4), software, and documentation in open systems from inception. In order to generate community input and support, this first workshop solicited input from NASA programmatic officials and flight project teams.
Recordings of the workshop are available: ESO Mission: Data Processing Study Workshop.
The workshop report is also available: ESO Mission Data Processing Study - Summary of NASA Program Offices and ESO Missions Requirements, Constraints, Recommendations, and Opportunities
Duration | EST | PST | Topic | Speaker |
---|---|---|---|---|
0:10 | 1:00 PM | 10:00 AM | Welcome and Purpose | Andrew Mitchell |
0:15 | 1:10 PM | 10:10 AM | Earth Science Division (ESD) Opening Remarks | Kevin Murphy |
0:15 | 1:25 PM | 10:25 AM | SMD Scientific Information Policy | Steven Crawford |
0:15 | 1:40 PM | 10:40 AM | ESDS Perspective of Open Science | Katie Baynes |
0:15 | 1:55 PM | 10:55 AM |
Overview of the Mission Processing Study |
Andrew Bingham |
0:15 | 2:10 PM | 11:10 AM | Q&A with the SAWG | |
0:10 | 2:25 PM | 11:25 AM | Break | |
0:15 | 2:35 PM | 11:35 AM | Flight Projects Program Perspective | Charles Webb, Kathleen Boggs |
0:15 | 2:50 PM | 11:50 AM | Earth Science Technology Office (ESTO) Perspective | Pam Millar, Jacqueline Lemoigne-Stewart, Ben Smith |
0:15 | 3:05 PM | 12:05 PM |
R&A Program Perspective |
Jack Kaye |
0:15 | 3:20 PM | 12:20 PM | Q&A with the SAWG | |
0:10 | 3:35 PM | 12:35 PM | Break | |
0:15 | 3:45 PM | 12:45 PM | Applied Science Program Perspective | Nancy Searby |
0:15 | 4:00 PM | 1:00 PM | Tsengdar Lee | |
0:10 | 4:15 PM | 1:15 PM | Q&A with the SAWG | |
0:30 | 4:25 PM | 1:25 PM | Open Discussion | |
4:55 PM | 1:55 PM | END |
Duration | EST | PST | Topic | Speaker |
---|---|---|---|---|
0:10 | 12:20 PM | 9:20 AM | Recap and Agenda | |
0:15 | 12:30 PM | 9:30 AM | NISAR Programmatic Perspective | Gerald Bawden |
0:15 | 12:45 PM | 9:45 AM | NISAR Project Science | Paul Rosen |
0:15 | 1:00 PM | 10:00 AM | NISAR Project Data Processing Systems | Hook Hua |
0:10 | 1:15 PM | 10:15 AM | Q&A with the SAWG | |
0:05 | 1:25 PM | 10:25 AM | Break | |
0:15 | 1:30 PM | 10:30 AM | Atmosphere Observing System (AOS) Programmatic Perspective | Hal Maring |
0:15 | 1:45 PM | 10:45 AM | AOS Project Science | Scott Braun |
0:15 | 2:00 PM | 11:00 AM | AOS Project Data Processing Systems | Robert Wolfe |
0:10 | 2:15 PM | 11:15 AM | Q&A with the SAWG | |
0:05 | 2:25 PM | 11:25 AM | Break | |
0:15 | 2:30 PM | 11:30 AM | Mass Change (MC) Programmatic Perspective | Lucia Tsaoussi |
0:15 | 2:45 PM | 11:45 AM | MC Project Science | David Wiese |
0:15 | 3:00 PM | 12:00 PM | MC Project Data Processing Systems | David Wiese |
0:10 | 3:15 PM | 12:15 PM | Q&A with the SAWG | |
0:05 | 3:25 PM | 12:25 PM | Break | |
0:20 | 3:30 PM | 12:30 PM | Surface Biology and Geology (SBG) Overview and Project Science | Dave Schimel |
0:20 | 3:50 PM | 12:50 PM | SBG Project Data Processing Systems | Jeff Pon |
0:05 | 4:10 PM | 1:10 PM | SBG Programmatic Perspective | Ben Phillips |
0:10 | 4:15 PM | 1:15 PM | Q&A with the SAWG | |
0:30 | 4:25 PM | 1:25 PM | Open Discussion | |
4:55 PM | 1:55 PM | END |
Workshop #2, March 1-4, 2022
This workshop focused on understanding the state-of-the-art in Big Data processing systems. The workshop was open to the public. A Request For Information (RFI; submission period now closed) sought input from organizations with relevant expertise in Big Data processing and open science to help guide the study and workshop.
Recordings of the workshop are available: ESO Mission: Data Processing Study Workshop.
The workshop report is also available: ESO Mission Data Processing Study: Summary of State-of-the-Practice and State-of-the-Art Mission Data Processing System Architectures
Please direct questions about the Open Source Science for the Earth System Observatory Mission Science Data Processing Study Workshops to: karen.yuen@jpl.nasa.gov or sara.h.lubkin@nasa.gov.
Agenda
Time ET |
Time PST |
Title | Speaker | Organization |
---|---|---|---|---|
12:00 PM | 9:00 AM | Welcome and Study Overview | Andy Mitchell and Andy Bingham | GSFC/JPL |
12:15 PM | 9:15 AM | Logistics | Sara Lubkin and Karen Yuen | GSFC/JPL |
12:20 PM | 9:20 AM | Opening Remarks | Kevin Murphy | NASA |
12:25 PM | 9:25 AM | Open Source Science / SPD-41 doi:10.5281/zenodo.6320745 |
Katie Baynes | NASA |
12:35 PM | 9:35 AM | Session 1 introduction: Science collaboration approaches | Chelle Gentemann | Farallon Institute |
12:40 PM | 9:40 AM | SAR data exploration in environmental sciences: challenges and opportunities | Naiara Pinto | JPL |
12:52 PM | 9:52 AM | Imagining a new NASA computing platform with AI + big data-supported analysis doi:10.5281/zenodo.6320918 |
Willow Coleman | Harvey Mudd College |
1:04 PM | 10:04 AM | Big Data Community Algorithms: Deep Learning for Mapping doi:10.5281/zenodo.6320982 |
Morteza Karimzadeh | University of Colorado |
1:16 PM | 10:16 AM | Cloud Computing Platforms for Processing Geospatial Big Data: Current Status and Challenges. doi:10.5281/zenodo.6307732 | Qiusheng Wu | University of Tennessee, Knoxville |
1:28 PM | 10:28 AM | Project Jupyter - Lessons and Principles from a Community-Driven Open Source Project doi:10.5281/zenodo.6320538 |
Fernando Perez | University of California, Berkeley |
1:40 PM | 10:40 AM | Fishbowl Discussion | Q&A led by the SAWG | |
2:00 PM | 11:00 AM | Break | ||
2:10 PM | 11:10 AM | Session 2 introduction: NASA Earth Systematic Mission Processing Architectures | Andy Mitchell | GSFC |
2:15 PM | 11:15 AM | Terra MODIS doi:10.5281/zenodo.6320958 |
Robert Wolfe | GSFC |
2:30 PM | 11:30 AM | CLAREO doi:10.5281/zenodo.6345252 |
Chris Currey | LaRC |
2:45 PM | 11:45 AM | ICESat-2 doi:10.5281/zenodo.6320538 |
Tom Neumann | GSFC |
3:00 PM | 12:00 PM | PACE doi:10.5281/zenodo.6313234 |
Sean Bailey and Bryan Franz | GSFC |
3:15 PM | 12:15 PM | SWOT doi:10.5281/zenodo.6315458 |
Oh-Ig Kwoun | JPL |
3:30 PM | 12:30 PM | Fishbowl Discussion | Q&A led by the SAWG | |
3:45 PM | 12:45 PM | Break | ||
3:55 PM | 12:55 PM | Break Room Topics & Logistics | ||
4:10 PM | 1:10 PM | Breakout Rooms 1. System development approaches & challenges 2. System operations approaches & challenges 3. Open-sourced science approaches & challenges 4. Data analysis needs for Mission Data Processing System (MDPS) 5. Open source software approaches & challenges 6. MDPS Architectures now and the future |
||
4:40 PM | 1:40 PM | Report out from Breakout rooms | ||
5:10 PM | 2:10 PM | END |
Time ET |
Time PST |
Title | Speaker | Organization |
---|---|---|---|---|
12:00 PM | 9:00 AM | Logistics | Sara Lubkin and Karen Yuen | GSFC/JPL |
12:05 PM | 9:05 AM | Session 3 introduction: NASA Earth System Science Pathfinder mission processing architectures | Andy Mitchell | GSFC |
12:10 PM | 9:10 AM | OCO-2/OCO-3 doi:10.5281/zenodo.6321343 |
Cecilia Cheng | JPL |
12:25 PM | 9:25 AM | EMIT | Phil Brodrick | JPL |
12:40 PM | 9:40 AM | MAIA doi:10.5281/zenodo.6323718 |
Jeff Walter | LaRC |
12:55 PM | 9:55 AM | Break | ||
1:05 PM | 10:05 AM | TROPICS doi:10.5281/zenodo.6323291 |
Jess Braun | University of Wisconsin, Madison |
1:20 PM | 10:20 AM | GEDI | Scott Luthke | GSFC |
1:35 PM | 10:35 AM | Fishbowl Discussion | Q&A led by the SAWG | |
1:50 PM | 10:50 AM | Session 4 introduction: Non-NASA Earth science mission processing architectures | Andy Bingham | JPL |
1:55 PM | 10:55 AM | National Oceanic and Atmospheric Administration doi:10.5281/zenodo.6307369 |
Lihang Zhou and Joseph Mani | NOAA |
2:10 PM | 11:10 AM | U.S. Geological Survey | Chris Engebretson | USGS |
2:25 PM | 11:25 AM | Fishbowl Discussion | Q&A led by the SAWG | |
2:40 PM | 11:40 AM | Break | ||
2:50 PM | 11:50 AM | Breakout Rooms 1. System development approaches & challenges 2. System operations approaches & challenges 3. Open-sourced science approaches & challenges 4. Data analysis needs for Mission Data Processing System (MDPS) 5. Open source software approaches & challenges 6. MDPS Architectures now and the future |
||
3:20 PM | 12:20 PM | Report out from Breakout rooms | ||
3:50 PM | 12:50 PM | Japan Aerospace Exploration Agency doi:10.5281/zenodo.6319381 |
Osamu Ochia | JAXA |
4:05 PM | 1:05 PM | END |
Time ET |
Time PST |
Title | Speaker | Organization |
---|---|---|---|---|
12:00 PM | 9:00 AM | Logistics | ||
12:05 PM | 9:05 AM | National Remote Sensing Center, India Space Research Organization (ISRO) doi:10.5281/zenodo.6302928 |
Uzair Mujeeb | ISRO |
12:20 PM | 9:20 AM | Italian Space Agency (ASI) | Francesco Tataranni | ASI |
12:35 PM | 9:35 AM | German Aerospace Center (DLR) doi:10.5281/zenodo.6315122 |
Conrad Albrecht | DLR |
12:50 PM | 9:50 AM | European Space Agency (ESA) | Anca Anghelea, Klaus Scipal, and Clement Albinet | ESA |
1:05 PM | 10:05 AM | Fishbowl Discussion | Q&A led by the SAWG | |
1:20 PM | 10:20 AM | Break | ||
1:30 PM | 10:30 AM | Session 5 introduction: Non-Earth science mission processing architectures | Luke Dahl | |
1:35 PM | 10:35 AM | Vera C. Rubin Observatory's Legacy Survey of Space and Time (LSST) doi:10.5281/zenodo.6311597 |
KT Lim | Vera C. Rubin Observatory |
1:50 PM | 10:50 AM | Nancy Grace Roman Space Telescope doi:10.5281/zenodo.6326463 |
Megan Soey | Space Telescope Science Institute (STScI) |
2:05 PM | 11:05 AM | IPAC Missions | Gregory Dubois-Felsmann | Caltech |
2:20 PM | 11:20 AM | Fishbowl Discussion | Q&A led by the SAWG | |
2:35 PM | 11:35 AM | Break | ||
2:45 PM | 11:45 AM | Session 6 introduction: System interfaces and standards | Andy Bingham | |
2:50 PM | 11:50 AM | Earth Science Data and Information System (ESDIS) | Sara Lubkin | GSFC |
3:05 PM | 12:05 PM | NASA Transform to Open Science (TOPS) doi:10.5281/zenodo.6324320 |
Chelle Gentemann | Farallon Institute |
3:20 PM | 12:20 PM | NASA's High-End Computing Capability: Growing to Support Science Data Processing for the Earth System Observatory Missions doi:10.5281/zenodo.6330014 |
Jon Jenkins | Ames Research Center |
3:35 PM | 12:35 PM | Data System Standards | Ingo Simonis | OGC |
3:50 PM | 12:50 PM | Fishbowl Discussion | Q&A led by the SAWG | |
4:05 PM | 1:05 PM | Break | ||
4:05 PM | 1:05 PM | Breakout Rooms 1. System development approaches & challenges 2. System operations approaches & challenges 3. Open-sourced science approaches & challenges 4. Data analysis needs for Mission Data Processing System (MDPS) 5. Open source software approaches & challenges 6. MDPS Architectures now and the future |
||
4:35 PM | 1:35 PM | Report out from Breakout rooms | ||
5:05 PM | 2:05 PM | END |
Time ET |
Time PST |
Title | Speaker | Organization |
---|---|---|---|---|
12:00 PM | 9:00 AM | Logistics | ||
12:05 PM | 9:05 AM | Session 7 introduction: Other Big Data processing system architectures | Luke Dahl | |
12:10 PM | 9:10 AM | CyVerse | Tyson Swetnam, Edwin Skidmore | University of Arizona |
12:25 PM | 9:25 AM | Pangeo doi:10.5281/zenodo.6330215 |
Joe Hamman | Carbon Plan |
12:40 PM | 9:40 AM | Alsaka SAR Facility | Kirk Hogenson | Alaska Satellite Facility |
12:55 PM | 9:55 AM | Raytheon doi:10.5281/zenodo.6325581 |
Shawn Miller | Raytheon |
1:10 PM | 10:10 AM | Element 84 STAC doi:10.5281/zenodo.6326572 |
Dan Pilone | Element 84 |
1:25 PM | 10:25 AM | Break | ||
1:35 PM | 10:35 AM | Fishbowl Discussion | Q&A led by the SAWG | |
1:50 PM | 10:50 AM | Red Hat | Dan McGuan | Red Hat |
2:05 PM | 11:05 AM | Amazon Web Services (AWS) | AWS | |
2:20 PM | 11:20 AM | Multi-Mission Algorithm and Analysis Platform (MAAP) doi:10.5281/zenodo.6345081 |
George Chang | JPL |
2:35 PM | 11:35 AM | OpenNEX doi:10.5281/zenodo.6345019 |
Jennifer Dungan and Ian Brosnan | ARC |
2:50 PM | 11:50 AM | Break | ||
3:00 PM | 12:00 PM | Unity (Science Data System as a Service) doi:2014/54386 |
Hook Hua and Laura Jewell | JPL |
3:15 PM | 12:15 PM | Science Data Analytics Platform (SDAP) doi:10.5281/zenodo.6310363 |
Nga Chung and Thomas Huang | JPL |
3:30 PM | 12:30 PM | NASA Earth Information System (EIS) | Alexey Shiklomanov | GSFC |
3:45 PM | 12:45 PM | Fishbowl Discussion | Q&A led by the SAWG | |
4:00 PM | 1:00 PM | END of Workshop |
Study Team
Dr. Andrew Bingham has more than 20 years of experience managing and leading teams responsible for delivering science processing, archive, distribution, and analysis systems for Earth and planetary missions. He is the manager for the Science Data and Instrument Operations System Section at NASA's Jet Propulsion Laboratory (JPL).
Luke Dahl has more than 20 years of experience as an architect and systems engineer supporting diverse activities ranging from enterprise infrastructure and applications, multi-mission instrument operations, and Earth and planetary science data systems.
Dr. Chelle Gentemann is a senior scientist at the Farallon Institute leading research on open science, cloud computing, remote sensing, and physical oceanography. For more than 20 years, she has worked on every aspect of passive microwave satellite missions, both domestically and internationally, from launch through decommission, including calibration, algorithm development, validation, data distribution, and science applications.
Dr. Sara Lubkin has a broad background in Earth science. She has worked with NASA since 2015 and is a Distributed Active Archive Center (DAAC) Operations Engineer with NASA’s Earth Science Data and Information System (ESDIS) Project at NASA's Goddard Space Flight Center.
Andrew Mitchell has multiple years of experience executing the technical and financial management of the science systems of NASA's Earth Observing System Data and Information System (EOSDIS). He is responsible for managing the processing, archiving, and distribution of Earth science data while ensuring scientists and the public have access to these data to enable the study of Earth. He is the project manager for ESDIS at Goddard.
Karen Yuen has 20 years of technical and management experience in science applications, project formulation, system engineering, and communications for multiple NASA missions and programs. She is the Science Data Applications Lead for the Orbiting Carbon Observatory-2 (OCO-2) and OCO-3 missions.
Study Architecture Working Group Members
Name | Focus Area | Organization | Required Expertise |
---|---|---|---|
Elias Sayfi | Co-Chair | JPL |
|
Natasha Stavros | Co-Chair | University of Colorado, Boulder |
|
Hook Hua | SBG and NISAR Science Processing Representative | JPL |
|
Curt Tilmes | AOS Science Processing Representative | Goddard/Ames Research Center |
|
Bernie Bienstock | MC Science Processing Representative | JPL |
|
Wenying Su | Applied Science Algorithm Developer | Langley Research Center |
|
Andy Michaelis | Data Processing Workflow Community-focused Developer | Ames |
|
Lesley Ott | Geophysical Numerical Modeler | Goddard |
|
Qing Yue | Instrument Algorithm Developer | JPL |
|
Evelyn Ho | Science Processing Architect from a comparable big-data science-based organization or project | Earth Science Data Information System (ESDIS) Science Investigator-led Processing Systems (SIPS) | |
Chris Engebretson | Science Processing Architect from a comparable big-data science-based organization or project | USGS | |
Adrian Parker | Science Processing Architect from a comparable big-data science-based organization or project | NOAA | |
Sean Harkin | Science Processing Architect from a comparable big-data science-based organization or project | Marshall Space Flight Center |
Related Resources
- Decadal Survey - Thriving on Our Changing Planet: A Decadal Strategy for Earth Observation from Space (2018)
- NASA Earth Science - Decadal Survey
- Surface Biology and Geology
- Aerosol, Cloud, Convection and Precipitation (ACCP)
- Mass Change
- Surface Deformation and Change
- NASA-Indian Space Research Organisation Synthetic Aperture Radar (NISAR) Mission
Page Last Updated: May 6, 2022 at 9:56 AM EDT