ESDS-RFC-016

Title

Lessons Learned Regarding WCS Server Design and Implementation 

Abstract

This document provides lessons learned regarding a WCS server implementation for the Coordinated Energy and Water Cycle Observations Project (CEOP) Satellite Data Server, a NASA ACCESS project that provides a gateway between the Open-source Project for a Network Data Access Protocol (OPeNDAP) and WCS protocols.  The gateway allows a user to use an OPeNDAP-enabled client to access satellite data held at a WCS server with services such as subsetting, reprojection, mosaicking and time series support.  This project was done as part of the Committee on Earth Observation Satellites (CEOS) Working Group and Systems and Services (WGISS) project activity, the WGISS Test Facility for CEOP.  This Technical Note presents several challenges encountered in the design and implementation of the WCS server, mostly arising from user access patterns, as well as the approaches taken to address those challenges.

Synopsis

Final Recommendation

The ESDS-RFC-016 Technical Working Group (TWG) has conducted a review of ESDS-RFC-016 - Lessons Learned Regarding WCS Server Design and Implementation with the following conclusion:

That the Standards Process Group should endorse ESDS-RFC-016 as a Technical Note.

Recommendation

The TWG bases its recommendation on an analysis of the following factors in a NASA context.

Strengths: Interoperability prototypes such as those described in this technical note are of fundamental importance in helping developers and system architects understand the issues involved in bridging related science domains both in terms of protocols as well as information schemata. Users cite the following uses for this technical note:

“…they each provide great information to help investigation of these described services.”
“These tech notes were informative to me and boosted my confidence in discoveries I had made regarding these services.”
“This document describes a useful scenario to bridge Earth science community standards with OGC standards and combine CS/W and WCS together”
“RFC 16 not only gives developers guidance on some issues to be faced when implementing WCS servers, but also raises some thoughts on [the] OGC WCS specification.”

Weaknesses: Ideally, developers would have liked to have source code and schema mapping files to work with, however as the prototypes described in this technical note were not billed as open-source, the issue of code sharing was not applicable.

Applicability: As already mentioned, this technical note not only aids developers who may be either in the process of developing similar gateways or contemplating developing them, understand the issues involved, but also help information architects and specification implementers “measure the maturity and shortcomings of standards”.

NASA centers that have data to share are looking at alternate ways to serve up their data holdings and one way to do this is via application gateways and protocol translators such as those described in these technical notes, to make them available seamlessly, with a wide variety of clients.

Limitations: The TWG realized that reviewers would request that some of the shortcomings in the underlying standards on which the prototypes were built, be fed back to the standards body, i.e., OGC. While the RFC authors did indeed itemize these issues and suggest alternate implementation approaches, recommending changes to an existing standard, published by a standards body is currently limited to information exchange only. Ensuring that the changes are incorporated into these standards is currently outside the scope of the SPG - this is a limitation of the SPG process and should not be construed as a limitation of the submitted RFC.

History

ESDS-RFC-016 was submitted by Chris Lynnes from NASA GSFC as a NASA Standards Process Group (SPG) Technical Note. A technical note, in SPG parlance is a document that contains useful information but is not a "standard". Technical notes must be relevant to the domain of NASA Earth Science data systems, serve a useful purpose, be technically of high quality, be well written and undergo a process of review from relevant stakeholder communities to prove their relevance.

The ESDS-RFC-016 TWG conducted a Technical Review designed to determine the technical validity of this technical note within a NASA context. The review of the technical note was completed over a period of approximately 8 months. An initial review by the members of the pre-TWG resulted in comments that addressed minor issues with the document along with suggestions to make some concepts clearer. This feedback was provided to the RFC author, and the documents were updated to incorporate this feedback. The revised documents were then sent out to the wider SPG community, via the spg-announce e-mail list, along with a review questionnaire, to elicit feedback in evaluating their relevance and usefulness.

The reviews received were positive, and stressed the need for the SPG to publish more of these ‘lessons learned’ type technical notes.

RFC Overview

The functionality of ESDS-RFC-016 - Lessons Learned Regarding WCS Server Design and Implementation is described in the excerpt below:

This Technical Note describes lessons regarding a Web Coverage Service (WCS) server implementation from a NASA ACCESS project to provide a gateway between the Open-source Project for a Network Data Access Protocol (OPeNDAP) and WCS protocols. The project developed a data handler for the OPeNDAP server that enabled serving of data obtained from a WCS server. The aim was to allow access to the data by analysis clients that have OPeNDAP support but not WCS support. For its part, the WCS server enables the serving of high-resolution satellite swath data (suitably reprojected to the WCS supported Earth spatial reference system) in a gridded form intelligible to the OPeNDAP client. A useful byproduct of the gateway architecture also allows third parties to provide OPeNDAP interfaces to data they do not actually hold but simply access remotely through WCS.

The project was user-driven, by the Coordinated Enhanced Observing Period (CEOP) community, which afforded us the opportunity to see how the WCS server implementation served (or did not serve) the analysis clients employed by those users. This project was done as part of the Committee on Earth Observation Satellites (CEOS) Working Group on Information Systems and Services (WGISS) project activity, the WGISS Test Facility for CEOP. The CEOP Satellite Data Server was initially designed to include a gateway (middleware) between a WCS server for NASA satellite data and the OPeNDAP server. However, the re-architecture of the new version of the OPeNDAP server, Hyrax, allowed the key gateway functionality to be implemented instead as a “format handler” in the Hyrax’s Back-end Server (BES).  In addition, significant customization of the WCS server implementation and configuration was necessary as well and forms the main basis for the lessons documented in this Technical Note.

The basic process of fulfilling a request is:

1. The client submits an OPeNDAP request to the CEOP Satellite Data Server, which is handled by its front end, the OPeNDAP Light Front End Service (OLFS).
2. the OLFS interacts with local catalog to identify the data source as WCS;
3. the OLFS instructs its BES to set container type to WCS and passes identifying information about the data to be retrieved;
4. BES formulates a WCS request to the WCS server;
5. BES stores the WCS response to local cache;
6. BES uses the NetCDF format handler to process cached file to satisfy the Data Access Protocol (DAP) request; and
7. Subsequent DAP requests operate against local cache.

The key lessons learned from attempting to serve this time-oriented community with WCS technologies are relevant to archives contemplating the use of WCS to serve data covering long time periods (i.e., years) to the science research community. Although some of these lessons may have implications as to how the OGC WCS protocol might be extended to better serve the temporal dimension, our primary goal is document our experiences for the benefit of WCS server implementers.