Skip to main content

With the launch of new, high-data-volume Earth observation missions, NASA’s ability to effectively ingest, process, and archive large amounts of data requires the most cost-effective, flexible, and scalable data-management architectures and technologies. To meet these demands, NASA's Earth Science Data Systems (ESDS) Program has implemented a strategic vision to develop and operate multiple components of NASA's Earth Observing System Data and Information System (EOSDIS) in a commercial cloud environment.

EOSDIS provides end-to-end capabilities for managing NASA Earth science data from satellites, aircraft, in situ measurements, and other sources. The migration of EOSDIS data into the Earthdata Cloud benefits users by giving them new ways to access NASA’s collection of Earth science datasets, improves the efficiency of data systems operations, increases user autonomy, maximizes flexibility, and offers shared services and controls.

The Earthdata Cloud is a key component of the ESDS Transform to Open Science (TOPS) mission, which provides the visibility, advocacy, and community resources to support and enable the shift to open science. TOPS, in turn, is part of NASA’s Open-Source Science Initiative, which promotes the open sharing of software, data, and knowledge (algorithms, papers, documents, ancillary information) as early as possible in the scientific process.

Data in the Cloud

The Earthdata Cloud architecture went operational in July 2019 and key EOSDIS services, such as NASA's Common Metadata Repository (CMR) and Earthdata Search, were deployed within it. Additionally, NASA’s Distributed Active Archive Centers (DAACs) are moving the data archives they manage into the cloud.

Benefits to Data Users and the Scientific Community

Moving EOSDIS data to the cloud has numerous benefits, including:

  • Easy access to data: Data users will be able to access data directly in the cloud, making the need to download volumes of data unnecessary (users will still have the ability to download data if they choose)
  • Rapid deployment: Users can bring their algorithms and processing software to the cloud and work directly with the data in the cloud, simplifying procurement and hardware support while expediting science discovery
  • Scalability: The size and use of the archive can expand easily and rapidly as needed
  • Flexibility: Mission needs can dictate options for selecting operating systems, programming languages, databases, and other criteria to enable the best use of mission data
  • Reduced redundancy: The use of a common infrastructure with cloud native services will reduce redundant tools and services, enable sharing, and enforce the use of community standards as well as uniform policies and processes
  • Cost effectiveness: Under NASA’s full and open data policy, all NASA data will continue to be free to access and download. This means that users will be able to employ cloud-based EOSDIS-provided services to discover, search, access, and download data at no cost. Users who wish to store data in their own Amazon Web Services (AWS) cloud instance or cloud storage, are responsible for covering these costs. For more information on this topic, see “Understanding and Managing Costs in the AWS Cloud,” one of several tutorials on how to get started in the AWS cloud

Earthdata Cloud also benefits the scientific community that uses NASA Earth science data for research. By making NASA data, algorithmic code, and metadata available in the cloud, the scientific processes of NASA researchers will become more transparent and their results more reproducible, which in turn lends clarity and validity to the scientific process. In addition, the use of standardized software and code makes it easier for new users to learn to interact with the data and become more involved in the scientific process.

Background

As the volume of data in the EOSDIS archive continues to grow, the EOSDIS archive’s data ingest rate is expected to increase dramatically along with it. By 2030, the volume of data in the EOSDIS archive is expected to surpass 320 PB.

For the Earthdata Cloud to meet users’ needs, NASA's Earth Science Data and Information System (ESDIS) Project is working to ensure it provides services in several key areas, including:

  • Data acquisition from data providers (such as NASA science teams)
  • Data ingest: The system must support multi-mission and multi-discipline data ingest
  • Data validation and processing
  • Data archive: The system must preserve and protect NASA Earth observation data
  • Data distribution, including disaster recovery: The system must support distribution of data, subsetting, and visualization, and must be adaptable to future technologies
  • Metadata: The harvest, creation, and publication of dataset metadata to the CMR
  • Data management: The system must meet the development and execution of information lifecycle needs of NASA mission-based Earth science datasets
  • Metrics: Publication of metrics to the ESDIS Metrics System (EMS), which collects and organizes various metrics from the DAACs and other data providers
  • NASA’s agreement with AWS has resulted in collaborations to improve the discovery, access, and use of NASA science datasets; the creation of data storage and staging areas to facilitate the community evaluation of data products; and workshops to expand the use of cloud-computing resources
  • NASA’s collaboration with Google has led to investigations into the transfer, storage, and value of making large volumes of NASA science datasets available on the Google Cloud and Google Earth Engine; making NASA Earth Science data accessible to users via the Google Cloud Public Dataset search engine and Earth Engine Catalog; and growing NASA’s artificial intelligence (AI) capabilities through joint efforts with NASA’s Frontier Development Lab (FDL) Challenges and SpaceML projects
  • NASA’s partnership with Microsoft has launched investigations into the value of making high-value NASA science datasets available on Azure; cost and performance evaluations of data storage methods and technologies and support analytics; the exploration of strategies to enable cloud-based analytics to promote science in the cloud; and analysis of the approaches to build and share training datasets for AI at a scale

For example, NASA implemented Cumulus, which provides a range of functionality in the cloud, including data acquisition from providers (such as NASA science teams); data ingest, including validation and processing; the harvest, creation, and publication of dataset metadata to the CMR; the storage and distribution of data, including disaster recovery; and publication of metrics to the EMS, which collects and organizes various metrics from the DAACs and other data providers.

Further, Cumulus is integrated with the NASA-Compliant General Application Platform (NGAP), a custom-built cloud optimized platform, which provides highly flexible cloud native infrastructure, NASA-compliant IT Security controls, networking services, and business cost control in Amazon Web Services (AWS).

Moving the collective data archive from the DAACs into the cloud puts NASA Earth observation data “close to compute,” giving users improved access to data, the ability to use large datasets more efficiently, and the ability to conduct a broader range of research. This move will not change existing methods of user interaction with EOSDIS data, but it does require new methods of accessing NASA data that differs from on-premises platforms. Further, as more datasets migrate to the cloud, the DAACs will continue to serve as the gateways to EOSDIS data holdings and provide a wide range of support services for users.

Earthdata Cloud Evolution

As the volume of data from NASA missions increases, so will the need for data management and archive technologies that are adaptable and scalable. Earthdata Cloud possesses these attributes, which will serve NASA’s forthcoming, high-data-volume missions, such the Surface Water Ocean Topography (SWOT) mission and the upcoming NASA-Indian Space Research Organisation Synthetic Aperture Radar (NISAR) mission. To accommodate these data volumes, NASA's Alaska Satellite Facility DAAC (ASF DAAC) worked collaboratively with NASA’s Jet Propulsion Laboratory to test and prototype ways of archiving and distributing NISAR data using the commercial cloud. This project, known as Getting Ready for NISAR (GRFN), successfully demonstrated key components for efficiently handling very large data volumes in a commercial cloud. ASF DAAC is also archiving and distributing Sentinel-1 data from the European Commission’s Copernicus Program into NASA-managed cloud accounts.

These missions are presenting NASA with unrivaled opportunities to further develop and test systems and architectures for providing improved data management and user access to the unprecedented volumes of data future Earth science missions are expected to generate.

Achieving the Potential of Cloud Computing through Partnerships

To capitalize on the benefits that cloud computing offers to the scientific community, ESDS’s Interagency Implementation and Advanced Concepts Team (IMPACT) program has executed Space Act Agreements (SAAs)— legal agreements between NASA and another party to work collaboratively on a project or technology—with private companies, including AWS, Google, and Microsoft. In the process, NASA is building a network of technical experts whose knowledge it can leverage for the benefit of Earth Science data users. For example:

In addition, NASA is considering additional agreements with some additional companies—Esri, IBM, and Nvidia—to accelerate the development, delivery, and adoption of AI to further NASA’s science research and applications, explore new opportunities in cloud technologies that enable and accelerate open science, and collaboration on effective joint solutions promoting open science and a better experience for those who user NASA data.

Continuing NASA’s Tradition of Free and Open Data

NASA Earth science data have been freely openly and available to all users since EOSDIS became operational in 1994. Under NASA's full and open data policy, all NASA mission data (along with the algorithms, metadata, and documentation associated with these data) must be freely available to the public. This means that anyone, anywhere in the world, can access the more than 59 PB of NASA Earth science data without restriction. Further, since 2015, the data systems software developed through NASA awards and research and technology grants are available as open-source software, which means the software’s source code for is freely available for inspection, modification, and enhancement. This allows enabled software and code to be available more broadly and shared collaboratively with diverse groups to accelerate software development.

The concept of open science builds on the philosophy and spirit of the open-source software movement, and endeavors to create a collaborative culture enabled by technology that empowers the open sharing of data, information, and knowledge within the scientific community and the public to accelerate scientific research and understanding. As the term implies, open science aims to make scientific findings as transparent as possible by making all elements of a claimed discovery readily accessible, which enables results to be repeated and validated. Now, a new scientific paradigm—open-source science—is emerging from the open science concept.

Open-source science takes these notions of openness and transparency even further by applying them to the entire scientific process. Its goal is to accelerate discovery by conducting science openly, from project initiation through implementation. The result is the inclusion of a wider, more diverse community in the scientific process as close to the start of research activities as possible, which engenders trust in the scientific process. It also represents a cultural shift that encourages collaboration and participation among practitioners of diverse backgrounds, including scientific discipline, gender, ethnicity, and expertise. Open-source science is more equitable science.

Open-source science is a foundational objective of NASA’s Science Mission Directorate (SMD) and SMD's ESDS Program. Along with the wide dissemination and use of openly available Earth-observing data, the SMD promotes and facilitates the full and open sharing of all metadata, documentation, models, images, and research results achieved using these data and makes available the source code used to generate, manipulate, and analyze them. Open-source science will also be a key attribute of NASA’s Earth System Observatory (ESO), a new set of Earth-focused satellite missions that will work in tandem to provide a holistic view of Earth and collect key information to guide efforts related to climate change, natural hazard mitigation, fighting forest fires, and improving real-time agricultural processes.

As part of its commitment to open-source science, NASA will make all ESO mission data, code, and supporting documents available as early in the mission life cycle as feasible. Given the expected high volume of ESO data, these data will be stored in Earthdata Cloud and tools will be provided for working with these data directly in the cloud environment. This strategy will expand the ability of global research teams to collaboratively work with and conduct research using more NASA Earth science data than ever before, and the result will be the availability of these data to a broader, more diverse global community of users with the attendant increase in opportunities for scientific discovery.