EOSDIS SIPS and their Role Ensuring Consistent Long-Term Global Observations of Atmospheric Ozone

NASA’s Earth Observing System Data and Information System (EOSDIS) provides end-to-end capabilities for managing NASA’s Earth science data. These data are archived and disseminated through discipline-specific Distributed Active Archive Centers (DAACs), which are managed by NASA’s Earth Science Data and Information System (ESDIS) Project. 

Processing instrument data and providing a wide range of processed data products to the DAACs is the responsibility of EOSDIS Science Investigator-led Processing Systems (SIPS). Thanks to the SIPS, data users have access to more than 10,000 unique data products that are constantly reprocessed and improved to provide the best research-caliber data possible.

EOSDIS SIPS are under the direction of instrument or mission Principal Investigators or Team Leads, and are generally (but not necessarily) located at or near the Scientific Computing Facility (SCF) responsible for processing specific instrument or mission data. The SIPS incorporate science algorithms and process complex satellite data into well-calibrated observational products, which are sent to the appropriate DAAC for archiving and distribution.

The work of the SIPS is challenging for several reasons. For one, raw satellite telemetry data often arrive at different times and in separate data streams and formats, and must be aligned to meet algorithm dependencies. This means that the SIPS must perform data accounting on each interface and track down missing data to ensure that processing keeps up with the continuous, operational instrument data flows. Additionally, they must regularly and quickly incorporate changes to data calibration parameters and algorithms as well as reprocess existing data to ensure long-term data accuracy over the life of a mission. 

Along with these responsibilities, SIPS simultaneously maintain and upgrade infrastructure to support evolving requirements, such as updated system security enhancements. Ozone data and products in the EOSDIS archive are a good example of the vital role played by the SIPS and the challenges they face.

Stratospheric ozone (O₃) absorbs incoming solar ultraviolet (UV) radiation. While atmospheric ozone is beneficial, ozone at ground level is considered a pollutant, and can damage vegetation and harm health. Monitoring atmospheric ozone is not only crucial to safeguarding life on Earth, but also a job perfectly suited for instruments aboard Earth observing satellites.

NASA efforts to establish a consistent, long-term ozone data collection began with NASA's Total Ozone Mapping Spectrometer (TOMS) program. Starting in 1978 with the launch of the first TOMS instrument aboard the joint NASA/National Oceanic and Atmospheric Administration (NOAA) Nimbus-7 spacecraft and concluding in 2007 with the decommissioning of the last of the three TOMS instruments aboard the NASA TOMS-Earth Probe spacecraft, the daily TOMS data gave scientists their first opportunity to study ongoing changes in the ozone layer.

Two principal current satellite-based ozone-observing instruments are the Ozone Monitoring Instrument (OMI), which is aboard NASA’s Aura satellite) and the Ozone Mapping and Profiler Suite (OMPS),which is aboard the joint NASA/NOAA Suomi National Polar-orbiting Partnership (Suomi-NPP) satellite. The OMI and OMPS SIPS are responsible for processing data from these instruments into numerous data products, and are both located at NASA’s Goddard Space Flight Center in Greenbelt, Maryland. 

A second OMPS instrument is aboard the recently-launched NOAA-20 satellite. Data from this second OMPS instrument that are consistent with the long-term NASA ozone record are expected to be publicly available later this year.

Standard ozone data products from TOMS, OMI, and OMPS that are intended for use in scientific research are archived and disseminated through NASA’s Goddard Earth Sciences Data and Information Services Center (GES DISC), which is the EOSDIS DAAC responsible for NASA atmospheric composition and dynamics data. These products can be accessed through the GES DISC as well as through the EOSDIS Earthdata Search application by searching with the keyword “ozone.”

In addition, near real-time (NRT) OMI and NRT OMPS data products are available through the EOSDIS Land, Atmosphere Near real-time Capability for EOS (LANCE) system. While NRT products do not have the extensive processing, quality assurance, or validation required for use in scientific research, they are available soon after a sensor overpass and are an excellent resource for tracking on-going events or monitoring daily ozone concentrations.

OMI is one of four instruments aboard the Aura spacecraft, and is a contribution of the Netherland’s Agency for Aerospace Programs (NIVR, which was merged into the Netherlands Space Office in 2009) in collaboration with the Finnish Meteorological Institute (FMI). OMI uses hyperspectral imaging in a push-broom mode to observe solar backscatter radiation in the visible and ultraviolet electromagnetic spectrums. Earth is viewed in 740 wavelength bands along the satellite track with a swath large enough to provide global coverage in a day with a nominal 13x24 km spatial resolution.

The OMI SIPS began operations in 2004 after the launch of Aura, and currently generates data products including total ozone and other atmospheric parameters related to ozone chemistry and climate. The Royal Netherlands Meteorological Institute (KNMI) provides the instrument calibration software and parameters required to generate the OMI calibrated and geo-located Level 1b products. Algorithms for higher-level OMI products, such as nitrogen dioxide (NO₂), sulfur dioxide (SO₂), and total column ozone, are provided by OMI science team Principal Investigators.

The OMI SIPS facility consists of the OMI Data Processing System (OMIDAPS), which was originally adapted as a scaled-down version of the processing system that was developed for processing Moderate Resolution Imaging Spectroradiometer (MODIS) data from the Terra and Aqua missions. Rather than use the large, high-performance servers that were required for MODIS processing, OMIDAPS was moved to a cluster of Linux-based commodity computers. A second instance of the OMIDAPS was established for integration and testing of OMI science processes as algorithms are delivered from the science team.

OMIDAPS software originally was developed based on technology prevalent in 2001, using 32 bit Fortran and C computing systems with Hierarchical Data Format-Earth Observing System (HDF-EOS) libraries. The OMI SIPS has continually upgraded versions of the compiler and libraries every few years, making incremental changes to the software. 

One of the more significant recent processing challenges faced by the OMI SIPS has been converting all processing code from 32 bit to 64 bit processor systems. This necessitated verifying that existing 32 bit science software would run the 64 bit systems and prioritizing the order in which the science software would be converted. This upgrade also required the conversion of all libraries along with the development of guides for the conversion and documentation of common problems and their solutions. The conversion to 64 bit not only improves the processing capability of the SIPS, it also provides the ability for science algorithms to use larger amounts of memory, access larger files, and use newer hardware instructions.

Most of the OMI processing algorithms also have benefitted from the move to new hardware. In the case of central processing unit (cpu)-bound algorithms, this has resulted in a 50% increase in performance. In addition, the OMI SIPS team has designed the system for rapid algorithm development. This allows a scientist to make a change to their data processing algorithm, test this change, and deliver the updated algorithm to the OMI testing team with one command.

As the OMI science algorithms and knowledge of instrument calibration improve, OMI data products need to be reprocessed. Most recently, the OMI SIPS reprocessed the UV Aerosol data product (OMAERUV) doi:10.5067/Aura/OMI/DATA2004 , OMI/Aura and MODIS/Aqua Merged Aerosol Geo-colocation Product (OMMYDAGEO), and the OMI/Aura Global Ground Pixel Corners product (OMPIXCOR) doi:10.5067/Aura/OMI/DATA2020

OMI SIPS also produces special validation products that are used by the OMI science team and stored at the Aura Validation Data Center (AVDC). Reprocessing of these OMI validation products is required when algorithms or calibration are updated. OMI SIPS recently reprocessed the OMI Above-Clouds Aerosol Data research product that is used for validating above-cloud aerosol data products derived from OMI near-UV observations.

In addition to OMI standard data products, five OMI near real-time (NRT) data products are available through LANCE. The OMI NRT capability is a joint development of NASA and the KNMI.

Since the Aura launch in 2004, over 100 TB of OMI observations have been generated by OMI SIPS and archived in more than 50 standard data products at GES DISC. Each year, approximately 14 TBs of new OMI observations are generated by the SIPS and added to the archive. More than 1 PB of OMI data has been distributed from GES DISC and LANCE over the life of the mission.

Data from the OMPS instrument aboard the joint NASA/NOAA Suomi-NPP satellite (operational 2011 to present) compliment OMI data and enhance the ongoing NASA ozone data record. OMPS measures the global distribution of the total atmospheric ozone column on a daily basis. It also measures the vertical distribution of ozone from about 15 to 60 km/about 9 to 37 miles in the atmosphere, although this measurement is conducted less frequently. 

OMPS is a three-part instrument build by Ball Aerospace in Boulder, CO: a nadir mapper that maps global ozone, a nadir profiler that measures the vertical distribution of stratospheric ozone, and a limb profiler that measures ozone in the lower stratosphere and troposphere. The nadir mapper and profiler look directly below the instrument through the atmosphere, while the limb profiler collects data looking through the atmosphere at an angle.

OMPS SIPS began operation with the launch of the Suomi-NPP satellite in 2011, and is responsible for processing and reprocessing Suomi-NPP Nadir Products, including OMPS total column, total column Earth view, total column calibration, nadir profile Earth view, and nadir profile calibration. OMPS SIPS recently established a new processing string to process NOAA-20 OMPS data.

Currently, 12 Suomi-NPP OMPS datasets are available for download through GES DISC. In addition, Total Column Ozone and Aerosol Index, SO₂, and Ozone Profile NRT products are available through LANCE. Since the Suomi-NPP launch, over 3 TB of OMPS data have been generated by OMPS SIPS and archived at GES DISC; over 6 TB of OMPS data have been distributed by GES DISC.

As with OMI, an important recent OMPS SIPS accomplishment is the conversion of processing code from 32 bit to 64 bit, resulting in increased performance for processing OMPS limb data. The combination of converting the code to 64 bit along with running the code on the newest 64 bit machines significantly decreases the time needed to process one orbit’s worth of limb data—from 2.5 hours to 0.5 hours.

OMPS data products, like OMI products, occasionally need to be reprocessed to maintain a consistent record of observations over the entire mission. OMPS SIPS recently reprocessed Version 2 of the OMPS nadir sensor datasets (total column ozone, nadir ozone profile, SO₂, and NO₂) and Version 2 of the OMPS limb ozone profile product (OMPS_NPP_LP_L2_O3_DAILY). doi:10.5067/X1Q9VA07QDS7 

OMPS SIPS also generated a new product: the OMPS limb aerosol profile (OMPS_NPP_LP_L2_AER675_DAILY) doi:10.5067/2CB3QR9SMA3F In addition, OMPS SIPS packaged the OMPS nadir sensor science processing algorithm software for distribution by the Direct Readout Laboratory at Goddard.

New instruments constantly are joining the constellation of Earth observing satellites, such as the second OMPS instrument aboard the NOAA-20 satellite. The work of EOSDIS SIPS ensures that accurate, valid data products derived from data collected by these instruments are fully and openly available to data users around the world.

Published January 24, 2018