The Level-1 and Atmosphere Archive & Distribution System Distributed Active Archive Center's (LAADS DAAC) Search and Order Interface is a web-based portal that allows researchers to search for, access, and download data from the LAADS DAAC archive of Level-1 atmospheric science data from several sources, including two critical Earth-observing instruments: the Moderate Resolution Imaging Spectroradiometer (MODIS) aboard NASA's Terra and Aqua satellites and the Visible Infrared Imaging Radiometer Suite (VIIRS) aboard the joint NASA-NOAA Suomi National Polar-orbiting Partnership Satellite (Suomi NPP), NOAA-20, and JPSS-2 satellites of the Joint Polar Satellite System (JPSS).
Yet, beyond finding and downloading data, the interface also allows users to transform the data they want by applying several post-processing options. Traditionally, these options have included:
- Parameter subsetting, which allows users to select only the science datasets (SDS) they need from a file. Most LAADS products contain many datasets. If you will not need most of them, choose only the ones you will use. This can reduce the size and number of files you will need to download.
- Geographic subsetting, which allows users to select only the data from a specific area of interest and also remove any extraneous data that may have been inadvertently included. This can further reduce the size of files you will need to download and will also reduce the amount of data you need to process.
- Mosaic creation, which allows users to combine data from multiple files into one file, with all the data correctly geolocated.
- Reprojecting, which allows users to change the map projection or sample size.
- Reformatting, which allows users to change the format (or type) of the files they want.
In the past, the LAADS DAAC Post-Processing System has only allowed users to reformat their data files to the GeoTIFF file format, which is commonly used in Geographic Information System (GIS) tools. However, as of December 2022, LAADS DAAC's updated Post-Processing System gives users the option of selecting a wider variety of file formats.
Although the ability to change file formats is nothing new, the expanded types of formats now offered by the DAAC's updated Post-Processing System will help users work with MODIS and VIIRS products more efficiently, said Greg Ederer, Software Engineer with the LAADS DAAC.
“Currently users need to use different tool sets, one for MODIS data (which is in HDF4 format) and one for VIIRS data (which is in HDF5 or NetCDF format),” he said. “Now, they can convert all the files to the same format, whichever one they prefer.”
Researchers have relied on MODIS data for the past 20 years because of their high quality and consistency. Now that MODIS instruments will soon be retiring, researchers might want to compare MODIS data with corresponding VIIRS data to see how compatible they are for maintaining data continuity into the future. The LAADS DAAC’s updated Post-Processing System will make these comparisons less cumbersome.
“The data will be easier to compare if they are in the same file format because users will be able to use the same tools to process both MODIS and VIIRS data,” said Ederer. “Otherwise, with two separate formats, users will need two separate tools. If the comparisons are then troublesome, the question becomes is it the tool sets or the data?”
So which file format—GeoTIFF, HDF4, HDF5, or NetCDF—should a user choose when using the Post-Processing System? That depends on the data application the user has at his or her disposal.
“There is a tool used by modelers—the Grid Analysis and Display System (GrADS)—that only understands NetCDF. In the past, those modelers could not use our data. Now they can. So for them, this update it is valuable,” said Ederer. “For other users, it doesn't really matter which format you use—HDF4 vs HDF5 vs NetCDF—but it’s helpful if all products are in one format so the user’s application only has to handle one.”
The Post-Processing System's expanded offering also give users options for circumventing the limitations associated with certain file formats.
“There are several known issues with HDF4 formatting, such as its two gigabyte size limitations,” said LAADS DAAC Software Engineer Christine Brown-Stevenson. “With this new Post-Processing System release, users will be able to reformat their LAADS product orders from HDF4 to GeoTIFF, HDF5, or NetCDF to better meet their needs."
There is, however, one data format the updated Post-Processing System cannot work with: Vector data.
“The data in image arrays are usually at least 2D, with one dimension the x and the other on the y of the image pixels. These can be mapped to geographic locations by applying various calculations on the x and y coordinates,” said Ederer. “Vector data are arranged differently, like a list of arrows pointing in a direction with a value associated with them, such as for wind speed and direction.”
A user’s choice of file format may also depend on whether he or she is working in the cloud.
The updated Post-Processing System gives users the option to use a variety of file formats, including NetCDF, which seems to be the preferred format for working with Python/Jupyter Notebooks. GeoTIFFs, or Cloud-Optimized GeoTIFFS (COGs), are commonly used in cloud data systems as well.
Nevertheless, Ederer cautions that this could change, as there is no community-wide consensus about which file format is best for working with data in the cloud and it's possible that an unknown file format could emerge as the favorite.
“One possibility is that all the files in the cloud will be transformed into a new format, although we do not have much information on how those formats will be exposed to users,” he said. “Another is that the files will remain in the cloud in their current formats, and users will have an option to select a file transformation when they access the data.”
In addition to offering users the ability to work with a wider range of file formats, the updated Post-Processing System will allow users to post-process several VIIRS Suomi NPP products that could not undergo post-processing in the past. These include:
- Daily Gridded Day/Night Band 15 arc-second Linear Latitude and Longitude (VNP46A1)
- Gap-Filled Lunar BRDF-Adjusted Nighttime Lights Daily Level 3 Global 500-meter Linear Latitude and Longitude Grid (VNP46A2)
- Moderate Resolution Terrain-Corrected Geolocation 6-Minute Level 1 750-meter Swath (VNP03MOD)
- Day/Night Band 6-Minute Level 1 750-meter Swath (VNP02DNB)
- Imagery Resolution 6-Minute Level1B 375-meter Swath (VNP02IMG)
- Moderate Resolution 6-Minute Level1B 750-meter Swath (VNP02MOD)
- Imagery Resolution Terrain-Corrected Geolocation 6-Minute 375-meter Swath (VNP03IMG)
- Atmospherically Corrected Surface Reflectance 6-Minute Level2 375-meter, 750-meter Swath IP (VNP09)
- Land Surface Temperature and Emissivity 6-Minute Level 2 750-meter Swath (VNP21)
- Cloud Mask 6-Minute 750-meter Swath (CLDMSK_L2_VIIRS_SNPP)
- Deep Blue Aerosol L2 6-Minute 6-kilometer Swath (AERDB_L2_VIIRS_SNPP)
With these VIIRS products, users will be able to subset by SDS and region, create mosaics, reproject to a geographic transformation of their choosing (e.g., World Geodetic System coordinates) if applicable, and reformat to GeoTIFF, HDF4, HDF5, or NetCDF.
The updated Post-Processing System is now available on the LAADS DAAC’s website during “Step 5” of the Search and Order interface process.
For more information, view the How to Order Post-Processed Data from NASA's LAADS DAAC tutorial.