ESDS-RFC-018

Title

Creating File Format Guidelines: The Aura Experience

Abstract

This technical note describes the process that the Aura teams used to create file format guidelines for their data products.

Synopsis

Final Recommendation

The ESDS-RFC-018 Technical Working Group (TWG) has conducted a review of ESDS-RFC-018 "Creating File Format Guidelines:  The Aura Experience" with the following conclusion:

That the NASA Earth Science Division should endorse ESDS-RFC-018 as a Technical Note.

Recommendation

The TWG bases its recommendation on an analysis of the following factors in a NASA context:

Strengths: ESDS-RFC-018documents the methodology and the results of one of the more successful efforts to improve data access to a broad community.  Reviewers felt that the process used by the Aura Data Systems Working Group was a good “grass-roots” model for coming to an agreement among several data producers for the benefit of each other as well the broader user community.  As such, the process is well worth documenting and sharing with future teams.

Weaknesses:  One reviewer would have liked more of the perceived weaknesses of the Aura File Format Guidelines, which are documented in ESDS-RFC-009, to be documented as lessons learned in ESDS-RFC-018.  For example:

• “Perhaps the technical note could mention, as one of the lessons learned, that more work on metadata guidance is needed in the future.”
• “The Technical Note did not address the issue of multiple teams that use the same field name, but assign slightly different meanings to the data contained in the fields.”

Applicability:  We expect that ESDIS would encourage use of a common data file format for data products generated by different sources, such as instrument teams on future missions or other NASA-funded data generation products (e.g., Decadal Survey missions).  At least one of the reviewers plans to look to the Aura Guide as a starting point in developing data products for a mission currently being planned.

Limitations:  ESDS-RFC-018 documents a grass roots, consensus based approach to file format guidelines.  This is certainly not the only approach for standardization.  Some of the perceived weaknesses mentioned might have been avoided with a more top-down approach.  However, getting community buy-in might have been more difficult.

Overall, reviewers were pleased with the Aura Guidelines (ESDS-RFC-009), and agreed that this “Aura Experience” document (ESDS-RFC-018) was a valuable record of the approach taken and lessons learned by the Aura team.

History

ESDS-RFC-009, “A File Format for Satellite Atmospheric Chemistry Data Based on Aura File Format Guidelines,” describes the HDF-EOS5 profile developed for data products from the instruments on the Aura satellite.  It was endorsed by the Standards Process Group in 2008.  During the review process for this earlier RFC, one reviewer commented that the process by which the Aura Guidelines were created was in itself valuable, and should be made available for re-use by other missions: “It's no
small accomplishment that this was actually agreed-to by such a large team."

The Aura Guidelines team decided that it was also important to document the process that they went through to develop the Aura Guidelines, and with encouragement from the ESDIS project they developed ESDS-RFC-018, “Creating File Format Guidelines:  The Aura Experience.”

A technical note, in SPG parlance is a document that contains useful information but is not a "standard". The process of review is less rigorous that that of RFC’s which are submitted to the SPG Standards track.  However, proposed technical notes must nonetheless, be relevant to the domain of NASA Earth Science data systems, serve a useful purpose, be technically of high quality, be well written and undergo a process of review from relevant stakeholder communities to prove their relevance.

The ESDS-RFC-018 TWG conducted a Technical Review within the SPG, designed to determine the technical validity of this technical note within a NASA context. The review of the technical note was completed over a period of approximately one year. An initial review by the members of the pre-TWG resulted in a request for more details on the process used by the Aura team, as well as more explicit lessons learned. This feedback was provided to the RFC authors, who returned a much improved second draft. The revised documents were then sent out to the wider SPG community, via the “spg-announce” e-mail list, along with a review questionnaire, to elicit feedback in evaluating their relevance and usefulness.

The reviews received were all positive, and stressed the need for the SPG to publish more of these ‘lessons learned’ type technical notes.

RFC Overview

ESDS-RFC-018 covers the Aura team’s entire process of developing guidelines for a common file format.  It covers how the team came together; the processes used for developing, refining, documenting and verifying the guidelines; summarizes key considerations for developing similar guidelines; and documents lessons learned.  The motivation behind this technical note is described in the excerpt below.

This technical note describes the creation of a common file format developed and used by the individual teams working on the four instruments on NASA’s Aura satellite (HIRDLS, MLS, OMI and TES)velop their standard.  ill be described, but a t.  Each of these teams was independent and there was no mandate that they all use a common file format.  The decision to do so and the implementation of it was a grassroots effort that was accepted by all of the PIs and instrument teams’ leading scientists.  This document describes the process used in developing the guidelines and the keys to its success...

Early on in the Aura program, the Aura teams realized that common data and file formats would greatly facilitate the sharing of data.  The teams agreed to use the HDF5/HDF-EOS5 data format.  It was also necessary to concur upon specific details within the file itself.  The teams agreed upon the names, data types and dimension order of fields.  There was also agreement regarding the file-, group- and field-level attributes to include in each product file.  A file-naming convention was also adopted.

Future NASA missions can use this technical note to learn from the process that the Aura instrument teams went through to develop their own set of guidelines.  The process can be adopted and modified as needed.