Data format refers to standardized ways that Earth science information is encoded for storage in a computer file. Different data formats describe the structure and compression of data, metadata requirements, and the scope of the variables included to meet the varying needs of the data users.
Commonly-used formats are listed below. More information about data formats is available on the Earth Standards Coordination Office (ESCO) Document List page.
ASCII a character encoding standard. It is binary code used by electronic equipment to handle text using the English alphabet, numbers, and other common symbols. The code includes definitions for 128 characters, which are assigned numbers from 0 to 127. Numbers 32 to 126 stand for letters, numbers and symbols such as abc, ABC, 123, and ?&!. Numbers 0 to 31 and the number 127 stand for characters that control how text is processed and are not printed.
The ASCII File Format Guidelines for Earth science data was recommended for use in NASA Earth science data systems in 2016. More information about ASCII NASA Earth Science Community Recommendations for Use is available through the ESDIS Standards Coordination Office (ESCO).
The CEOS data format is used for radar data. The format does not specify a naming convention, so naming is facility or agency-specific.
| File Type | Description | ERS-1, ERS-2, JERS-1 and RSAT-1 File Extension (L1) | ALOS PALSAR File Prefix (L1.5) |
|---|---|---|---|
| SAR VOLUME file | Stores the volume-management and file-management information. | N/A | VOL- |
| SAR LEADER file | Contains detailed metadata. Useful records include the Dataset Summary, Platform Position, and the Facility-Related records. Contents can be read in plain text if the .L.txt file is available. | .L | LED- |
| SAR DATA file | Contains image data | .D | IMG- |
| SAR TRAILER file | .P contains basic metadata including orbit, beam mode, start/stop times and lat/long corners and center. Can be read with a text editor. TRL contains a file descriptor, and for L1.1 and L1.5 data, low resolution image data. | .P | TRL- |
The International Global Navigation Satellite System (GNSS) Service (IGS) analysis centers provide satellite orbit solutions to NASA's Crustal Dynamics Data Information System (CDDIS) using pre-determined schedules, e.g., sub-daily, daily, or weekly, depending upon the data product. All orbit solution files utilize the Extended Standard Product- 3 (SP3c) format.
GeoTIFF, an Open Geospatial Consortium (OGC) Implementation Standard, is based on the TIFF format and is used as an interchange format for georeferenced raster imagery. While GeoTIFF has widespread use, it is not suitable for storing complex multi-dimensional data structures nor for storing vector data with many attributes or topology information.
TIFF is an useful format for handling images and data within a single raster file by including header tags such as size, definition, image-data arrangement, and applied image compression. The GIS-friendly GeoTIFF format is an extension of TIFF that includes georeferencing or geocoding information embedded within a TIFF file (such as latitude, longitude, map projection, coordinate systems, ellipsoids, and datums) so an image can be positioned correctly on maps of Earth. It is a public domain metadata standard.
Credit: JAXA (Japan Aerospace Exploration Agency) ALOS/PALSAR Level 1.1/1.5 Product Format Description document.
A Cloud Optimized GeoTIFF (COG) is a GeoTIFF that contains internal organization, which enables it to be efficiently delivered and processed on the cloud. Specific NASA Earth science datasets are available in COG format.
Tutorials
Tools and Applications
View Radiometrically Terrain-Corrected (RTC) Images in ArcGIS
View Radiometrically Terrain-Corrected (RTC) Images in QGIS
Radiometrically Terrain-Correct (RTC) Sentinel-1 Data Using the ESA Toolbox
Radiometrically Terrain-Correct (RTC) Sentinel-1 Data Using Sentinel-1 Toolbox Script
Radiometrically Terrain-Correct (RTC) Sentinel-1 Data Using GAMMA Software
OGC GeoTIFF 1.1 is an approved NASA Earth Science Data Systems standard.
NASA's Earth Science Data and Information System (ESDIS) Project recommends the use of Cloud Optimized GeoTIFF (COG) for any data distribution that would be appropriate for a traditional GeoTIFF. COG files are fully backward compatible with all tools that are used in the reading of GeoTIFFs. Using COG files allows those with access to cloud-based computing environments to have all the benefits of the expanded COG format without limiting those still using traditional tools.
View NASA ESDIS recommendations for the use of GeoTIFF.
HDF is a set of file formats designed to store and organize large amounts of data, including scientific, engineering, and remote sensing data. The latest version of HDF—HDF5—allows users to read only the data that they need, not the whole file. Data producers can put images, tables, multidimensional arrays, and more into the same file.
HDF5 simplifies the file structure to include only two major types of object:
HDF5 is a general purpose file format and programming library for storing scientific data. Use of the HDF library enables users to read HDF files on multiple platforms regardless of the architecture the platforms use to represent integer and floating point numbers.
View Seasat HDF5 Files in ASF MapReady
The HDF Group offers tools and technologies to support HDF on GitHub.
HDF-EOS is a software library designed to support NASA Earth Observing System (EOS) Earth science data. A key feature of HDF-EOS is a standard prescription for associating geolocation data with science data through internal structural metadata. The relationship between geolocation and science data is transparent to the end-user. Instrument and data type independent services, such as subsetting by geolocation, can be applied to files across a wide variety of data products through the same library interface. The library is extensible and new data structures can be added.
The HDF-EOS5 Data Model, File Format and Library is an approved standard recommended for use in NASA Earth Science Data Systems in January 2007. Minor updates to correct typographic errors and URLs that had become unreachable were made in May 2016. More information about HDF-EOS5 Data Model, File Format, and Library is available through the ESDIS Standards Coordination Office (ESCO).
JPEG 2000 is an image compression standard and coding system. JPEG 2000 provides a superior image resolution compared to JPEGs since it can compress files without compromising as much visual data through lossless compression, while also leaving the option open for high-quality lossy compression if needed.
JSON is an open standard file format and data interchange format that uses human-readable text to store and transmit data objects consisting of attribute–value pairs and arrays. It is completely language independent but uses conventions that are familiar to programmers of the C-family of languages, including C, C++, C#, Java, JavaScript, Perl, Python, and many others.
The netCDF file format provides a data model, software libraries, and machine-independent data format for geoscience data. Together, the netCDF interfaces, libraries, and format support the creation, access, and sharing of scientific data.
Data in netCDF format are:
RINEX is a data interchange format for raw satellite navigation system data. RINEX Version 2 is the format used by the International Global Navigation Satellite System (GNSS) Service (IGS) for operational data holdings. However, data support for additional GNSS, such as Galileo, QZS, and BeiDou, uses an enhanced version of RINEX available in Version 3.
IGS analysis centers currently provide station position, velocities, daily length-of-day (LOD) and X, Y pole positions and rates, and geocenter position solutions. The IGS reference frame coordinator retrieves these solutions and produces a combined product, which is then in turn archived in Solution Independent Exchange (SINEX) format. These combination solutions are considered the official IGS products, which contribute to the International Terrestrial Reference Frame (ITRF).
Some data products generated in partnership with international collaborators, such as ESA (European Space Agency), are delivered in the SAFE file format. Sentinel-1 is an example.
SAFE product folder structure
Sentinel data products use a Sentinel-specific variation of the SAFE format, an ESA folder structure containing data and information as follows:
The "manifest.safe" file is an XML file containing the mandatory product metadata common to all Sentinel-1 products.
"Annotation" datasets contain metadata describing the properties and characteristics of the measurement data or how they were generated. For each band of data there is a product annotation data set that contains metadata describing the main characteristics corresponding to that band such as the state of the platform during acquisition, image properties, polarization, Doppler information, swath merging and geographic location. Calibration annotations contain calibration information and the beta naught, sigma naught, gamma and digital number look-up tables that can be used for absolute product calibration. Noise data annotations contain the estimated thermal noise look-up tables. Annotated data sets are provided in XML format.
"Measurement" datasets contain the binary information of the actual acquired or processed data. For Level 0 this is the instrument data, for Level 1 it is processed data. Measurement datasets are provided in GeoTIFF format (georeferenced) for Level 1 products. There is one measurement data set per polarization and per swath. TOPSAR SLC products contain one complex measurement data set in GeoTIFF format (georeferenced) per swath per polarization. Level 1 GRD products contain one detected measurement data set in GeoTIFF format (georeferenced) per polarization.
In the "Preview" folder, quick-look datasets are power detected, averaged and decimated to produce a lower resolution version of the image. Single polarization products are represented with a grey scale image. Dual polarization products are represented by a single composite color image in RGB with the red channel (R) representing the first polarization, the green channel (G) represents the second polarization and the blue channel (B) represents an average of the absolute values of the two polarizations.
Representation datasets found in the "Support" folder contain information about the format or syntax of the measurement and annotated data sets and can be used to validate and exploit these data. Representation data sets are provided as XML schemas.
Zarr is a specification for storage of and access to multi-dimensional array data. Its development targeted data in cloud environments (i.e., in object store), and the specification is optimized for cloud access via capabilities such as metadata consolidation and ability to chunk across any dimension.
Zarr Storage Specification V2 is an approved data format convention for use in NASA Earth Science Data Systems (ESDS). More information about Zarr Storage Specification Version 2 is available through the ESDIS Standards Coordination Office (ESCO).