Community Data System Capabilities Overview
Revision 1.0, 2 November 2017
The purpose of this Web page is to provide supplementary material in support of NASA’s Research Opportunities in Earth and Space Science 2017 (ROSES 17) solicitation for A.42 Advancing Collaborative Connections for Earth System Science (ACCESS). Specifically, this Web page is intended to provide ACCESS 2017 proposers with pertinent information about the Earth Observing System Data and Information System (EOSDIS). EOSDIS is a distributed data system that supports Earth science research and applied science users with a wide range of science data from a large number of satellite, airborne, and field campaign missions as part of the NASA’s Earth Science Program. The data cover a wide range of Earth science disciplines. EOSDIS consists of a constellation of discipline-specific Distributed Active Archive Centers (DAACs), each with expertise in particular disciplines and tasked with supporting the data in those disciplines and their user communities.
All comments and questions regarding this Web page or the ACCESS 2017 solicitation should be directed to the Program Manager, Kevin.J.Murphy@nasa.gov.
This Web page covers the high-level organization of EOSDIS into constituent Distributed Active Archive Centers (DAACs) and EOSDIS common elements that apply across the system. The material covered focuses primarily on the functional aspects of EOSDIS that are applicable to ACCESS projects planning to develop and deploy tools and capabilities at one or more DAACs.
1.2 Configuration Management of This Document
This Web page maintains tight configuration control and will not be altered, except as directed by the ACCESS Program Manager. Note, however, that information on Web pages linked to from this Web page is subject to change without notification.
2. EOSDIS Common Services
A number of shared common services and tools are offered by EOSDIS that may be pertinent to the ACCESS 2017 solicitation. Awarded projects are expected to use these services where applicable. The table below summarizes these services, and provides links to other related resources, application programming interface (API) documentation, and software source code.
Note that some services and tools listed below (where there is no link) can only be accessed via a NASA approval process that can only be considered for funded projects upon execution. Note also that some information resources below require an Earthdata Login (EL) account to access. Proposers can obtain the requisite account here.
* Availability of open source software pending
3. Description of Capabilities
3.1.1 Earthdata Developer Portal
The NASA Earthdata Developer Portal is the central location for all publicly accessible developer APIs and other documentation related to EOSDIS common services and applications.
3.1.2 Earthdata Website
The NASA Earthdata website is a collaborative environment integrating information from across EOSDIS. Earthdata is the entry point for EOSDIS data, articles, documentation, and collaboration. It also leverages NASA’s Common Metadata Repository (CMR) to provide comprehensive search capabilities. Earthdata offers new and experienced users an organized view of EOSDIS resources and latest events. EOSDIS DAACs are key contributors to Earthdata, providing the latest information on Atmosphere, Solar Radiance, Cryosphere, Human Dimensions, Land, and Ocean Science.
The Earthdata Website is supported by a number of components that support and encourage a common User Interface/User Experience (UI/UX) for all tools across EOSDIS for the benefit of end users. The Earthdata User Interface (EUI) is a toolbox of guidelines, examples, and best practices for implementing a common UI/UX across EOSDIS.
3.1.3 Conduit Content Management System
Conduit is a lightweight Content Management System that supports the Earthdata Website.
3.1.4 Status Application
3.1.5 User Feedback Module
3.1.6 Earthdata Tophat
3.2 Metadata Management
3.2.1 Common Metadata Repository (CMR)
The Common Metadata Repository (CMR) is a high-performance, high-quality metadata system providing Ingest and Discovery capabilities for EOSDIS metadata. The CMR contains metadata for nearly 6400 EOSDIS collections and more than 270 million science granules as of March 2016. The CMR provides a single, authoritative metadata repository though the Unified Metadata Model (UMM). The UMM is an extensible metadata model that provides mappings between supported metadata standards and identifies EOSDIS- compliant profiles.
Detailed information about the CMR architecture and the CMR API is available on the Earthdata Developer Portal.
CMR is open source and available on the NASA Github portal here.
3.2.2 Metadata Management Tool (MMT)
The Metadata Management Tool (MMT) is a Web-based user interface which allows metadata authors to create and manage their metadata records using the CMR’s Unified Metadata Model (UMM). The MMT allows metadata authors to create and update CMR metadata records by importing XML or by using a data entry form based on the metadata fields in the CMR Unified Metadata Model (UMM). Metadata authors may also publish, view, delete, and manage CMR metadata records and and access using the MMT. Additional information is available here. MMT source code is open source and available on the NASA Github portal here.
3.3.1 Earthdata Search
Earthdata Search is an open source EOSDIS web application that provides easy-to-use access to EOSDIS services for Earth Science data discovery, filtering, visualization, and access. It serves as a platform to feature planned EOSDIS services as they become available and integrates several core EOSDIS services into a unified user experience. Earthdata Search is able to take advantage of data services exposed by the EOSDIS community, in many cases simplifying data access and providing more targeted information to the user.
Earthdata Search allows users search for Earth observation data, preview Earth observation data, and download data using data visualization from GIBS. Users also have access to customized data services through Earthdata Search including capabilities from OPeNDAP servers and advanced services like the HDF-EOS to GeoTIFF Converter (HEG), the MODIS Reprojection Tool (MRT), and Geospatial Data Abstraction Library (GDAL) Translate.
Developers can leverage Earthdata Search to design a customized portal that targets a specific user community. Instructions for creating an Earthdata Search portal are available on the Earthdata Developer Portal Website here.
Earthdata Search software is open source and available on the NASA Github portal here.
3.4 Browse Imagery
3.4.1 Global Browse Imagery Services (GIBS)
The Global Imagery Browse Services (GIBS) system is a core EOSDIS component which provides a scalable, responsive, highly available, and community standards based set of imagery services. These services are designed with the goal of advancing user interactions with EOSDIS’ inter-disciplinary data through enhanced visual representation and discovery. For a full overview and access to the source code visit the GIBS home page on Github. Documentation on using GIBS is available on the Earthdata Developer Portal here.
Worldview is an open source EOSDIS web application allowing users to interactively browse global satellite imagery within hours of it being acquired then download the underlying data. Most of the 100+ available products are updated within three hours of observation, essentially showing the entire Earth as it looks "right now". This supports time-critical application areas such as wildfire management, air quality measurements, and flood monitoring. Arctic and Antarctic views of several products are also available for a "full globe" perspective. Browsing on tablet and smartphone devices is generally supported for mobile access to the imagery. Worldview uses the Global Imagery Browse Services (GIBS) to rapidly retrieve its imagery for an interactive browsing experience. Additional information is available here. Also, see the Worldview/GIBS Blog.
3.5 User Profile Management
EOSDIS includes a component named Earthdata Login to help track data usage by users, enabling both accurate metrics on data usage and the ability to send critical data update messages. Earthdata Login includes a User Registration site and protocols that allow both DAAC-developed and ESDIS-developed services to look up the userid for anyone accessing data. In general, the intent is to require registration for data access, but to leave it optional for search and browse functions. Users who register for an Earthdata Login account may potentially use the same account for all EOSDIS web services. Information on how to integrate applications with Earthdata Login is available on the Earthdata Developer Portal here. To create a new user profile in Earthdata Login, go here.
3.6 Data Services
OPeNDAP is the developer of client/server software, of the same name, that enables scientists to share data more easily over the internet. The OPeNDAP group is also the original developer of the Data Access Protocol (DAP) (DAP) that the software uses. Many other groups have adopted DAP and provide compatible clients, servers, and SDKs. OPeNDAP’s DAP is also a NASA community standard. An OPeNDAP User Guide and examples of using OPeNDAP with NASA data are available on the Earthdata Developer Portal here.
3.6.2 Programmatic Access
The Programmatic Access interface is used to locate and access DAAC hosted data, optionally performing services on the data. The resulting output data is synchronously streamed back to the user, either as a single file or a multi-file zip file. It combines into one interface functions that formerly required multiple interfaces: searching CMR for science granules of interest, and then obtaining the science data files with optional services applied. Details are available on the Earthdata Developer Portal here.
3.7 Development Framework
3.7.1 Earthdata Code Collaborative (ECC)
The Earthdata Code Collaborative (ECC) provides a full suite of development tools to EOSDIS projects that embody current best practices for software development. To request access to ECC, you must first have an Earthdata Login ID. Access to ECC can then be requested here.
The ECC provides access to the following capabilities for supporting development projects:
184.108.40.206 USER STORY / WORK TRACKING
The ECC offers full Agile Development process support for implementation through Atlassian JIRA. User Stories can be automatically synchronized with requirements to minimize process overhead while providing a consistent view of project progress and features. The ECC offers a standard workflow for projects with support for developers, internal testing, external testing, and deployments as well as custom workflows if needed.
220.127.116.11 SOURCE CODE CONTROL
The ECC offers full lifecycle source code control through Atlassian’s BitBucket. This git-based source code control has support for branching and merging, code reviews, and linkage to JIRA tickets. A BitBucket user’s guide is available here.
18.104.22.168 AUTOMATED BUILDS AND CONTINUOUS INTEGRATION
The ECC offers sophisticated support for automated builds and continuous integration through Bamboo. The ECC is capable of monitoring a project’s source code repository for changes and then invoking custom builds and automated tests when detected. The ECC can monitor for branch creation as well, providing full support for feature-branch style development coupled with code reviews and subsequent merging. For approved, hosted EOSDIS projects, the ECC supports Continuous Deployment, automatically deploying the latest build of a project after a successful test execution and properly monitor versions by environment.
22.214.171.124 TEAM AND USER COLLABORATION
Each registered ECC project gets a dedicated space in the Earthdata Wiki for team and user support. Team members can control access to their project wiki allowing for internally focused communication and collaboration as well as public-facing support and communication. The ECC Wiki is aware of the other ECC components and can easily surface common agile graphs including burndowns, backlogs, build and test results, and latest deployment information. A Wiki user’s guide is provided here (on the Wiki). Additional information on the Confluence Wiki product is also available from Atlassian here.
4. Getting Started
4.1 Earthdata Login
All members of the project team should have Earthata Login accounts. This provides access to multiple tools and services. First time users should go to https://urs.earthdata.nasa.gov/ and click on the Register button to create an account. Note that Earthdata Login accounts require an activation step via an email that new users should receive. The account is not active until this step is taken.
4.2 Earthdata Code Collaborative (ECC)
A prerequisite for ECC access is an activated Earthdata Login account (Section 4.1). Next, request access to the ECC at https://ecc.earthdata.nasa.gov/. Project teams should designate the project lead (typically, the Project PI) to manage the permissions of other members in the project. To create a new project in ECC, see instructions here after you get an Earthdata Login account. Once created, the project will be automatically set up in the Confluence Wiki, JIRA bug tracking, and in BitBucket Git.
4.3 Development Environments
All software developed is submitted to a version control repository, BitBucket (Git-based), and tested prior moving to production systems. Integration begins behind the firewall in a development environment, where code developed by projects is tested alongside functionality developed by other projects. After testing is confirmed, features are moved from the DIT environment to a System Integration and Test (SIT) environment, where regression testing is performed. Features are then moved from the SIT environment to a User Acceptance Testing (UAT) environment, allowing for testing by selected end users. Once the release candidate is successfully tested in UAT, it is then moved into the Production (PROD) environment.
The BitBucket is intended for projects that are not open source or not yet open source. Open source projects should make software available in a public repository, such as the NASA Github portal.
4.4 Getting Help
To request help, report a problem, or to submit feedback regarding any data, Web tool, application, or service across EOSDIS (i.e at any DAAC), send an email to firstname.lastname@example.org. This will automatically result in the creation of a ticket in the Kayako ticketing system that will be routed to the DAAC or organization most able to respond. Alternatively, on the Earthdata Website and on many DAAC Websites, you can submit feedback (including questions) by clicking on the feedback icon in the top hat of the Web page (upper right). This also results in a Kayako ticket being created. In addition, a number of the DAACs provide other means to provide feedback or submit questions through their own interfaces.
Last Updated: Nov 15, 2017 at 2:49 PM EST