Provenance for AMSR-E Data Production System

Michael Goodman, NASA/MSFC, reports the successful integration of the provenance capture tools into the AMSR-E (Advanced Microwave Scanning Radiometer – Earth Observing System) data production system.  The AMSR-E provenance tools are the result of Goodman’s ACCESS (Advancing Collaborative Connections for Earth System Science) project “Instant Karma: Applying a Proven Provenance Tool to NASA’s AMSR-E Data Production Stream.” The objective of the project was to improve the collection, preservation, utility and dissemination of provenance information within the NASA Earth Science community. NASA has been collecting, storing, archiving and distributing vast amounts of data collected from satellite sensors for several decades now. The raw data collected from the different sensors undergoes many different transformations before it is distributed to the science community as climate research quality data products. These transformations are based on scientific algorithms and may also utilize ancillary data. There may be changes made to the algorithms, different ancillary datasets may be used by these algorithms, underlying hardware and software may get upgraded, etc. These variabilities impact the data product and thereby influence the scientific analyses. The history of these changes--the provenance of the data product--needs to be captured, documented, and made accessible to the scientific community so they can be properly accounted for in analyses.

The project built on research at Indiana University led by co-I Dr. Beth Plale, whose “Karma” tool suite for provenance collection and representation is designed and developed for data driven workflows such as those used to generate NASA standard products.  Co-I Helen Conover of the University of Alabama in Huntsville, who manages the AMSR-E SIPS with Mr. Goodman, oversaw the development and integration of streamlined provenance tools for the AMSR-E system.  Co-I Dr. Rahul Ramachandran, also of UAH, led development of a Provenance Browser to allow science users to explore lineage and context of an individual data file, or general context information for a data collection.  Co-I Dr. Thorsten Markus and others on the AMSR-E Sea Ice algorithm team provided a science focus for the information collected and displayed.

The AMSR-E Provenance Browser, available for test at the project web site, http://provenance.itsc.uah.edu/, provides access lineage and context information for a test set of AMSR-E products from September and October 2011.  Results from this project will be integrated into the AMSR-E SIPS production environment, to provide for provenance collection, storage and dissemination.  Provenance metadata and tools will be made available to the AMSR-E data user community after reprocessing to begin in the fall of 2012.

For more information, see the poster from the ESDSWG meeting.