What is Data Latency?
Several factors affect the speed at which data are processed and made available to users. Here's how NASA's Earth Observing System Data and Information System (EOSDIS) defines data latency.
When a major fire starts in a forest, emergency responders and resource managers need data fast. Data providers use terms such as near real-time and low latency to refer to data provided to users quickly, but what do these terms mean?
NASA’s Earth Observing System Data and Information System (EOSDIS) distributed about 36 petabytes of Earth science data products in 2019 for interdisciplinary studies. These data products can be satellite observations, imagery, airborne observations, field data, and data published by scientists. Some of the data are available to users within hours. Other data require extensive processing before they are useful.
Data latency is defined as the total time elapsed between when data are collected, acquired by a satellite, airborne, or in situ sensor, and when they are made available for public access via the internet.
For satellite data, data latency refers to the time between satellite observation and the time data are available to users. For example, NASA's Fire Information for Resource Management System (FIRMS) distributes near real-time active fire data within three hours of satellite observation from NASA's Moderate Resolution Imaging Spectroradiometer (MODIS) onboard the Terra and Aqua satellites, and the joint NASA/NOAA Visible Infrared Imaging Radiometer Suite (VIIRS) onboard the Suomi National Polar-orbiting Partnership (Suomi NPP) and NOAA-20 satellites. While it is important for emergency responders to have data as quickly as possible, several factors affect the speed at which data are processed and made available to users.
Data Latency Terminology
NASA and other Earth science data providers use different terminology to describe data latency. The terms near real-time (NRT), low latency, and expedited are often used interchangeably to refer to data that are made available more quickly than routine processing allows.
In 2016, a workshop titled “Time Sensitive Applications of NASA Data” was held at NASA's Langley Research Center, located in Hampton, Virginia, to identify, coordinate, and focus attention on low latency satellite data. At this workshop the terms for latency were defined and agreed to by participants across NASA for all data managed by EOSDIS on behalf of NASA’s Earth Science Division. The table below summarizes these terms.
|Real-time||Less than 1 hour||These terms are often used to refer to data that are made available quicker than routine processing allows. They are used for a range of applied sciences, decision and tactical support, monitoring and early warning of events.|
|Near real-time (NRT)||1-3 hours|
|Low latency||3-24 hours|
|Standard routine processing||Generally, 8 – 40 hours but up to 2 months for some higher-level products||Standard products provide an internally consistent, well-calibrated record of the Earth’s geophysical properties to support science.|
Low latency data products are made available more quickly than standard processing to support users interested in monitoring a wide variety of natural and man-made phenomena. In the context of NASA data, low latency products are distinct from standard data products in that the algorithms for low latency products are modified to expedite data availability.
One key difference between some NRT and standard products is that NRT data use predictive orbit information for geolocation instead of definitive orbit information, which may not be available until after the NRT products are produced. A second key difference applies to NRT products that make use of ancillary data as part of the processing algorithm from sources other than the satellite. These ancillary data are combined with the satellite data to create higher-level products. Some algorithms for NRT data products use different or less accurate ancillary data than standard products. Standard products are processed using definitive geolocation and instrument calibration, and provide an internally consistent, well-calibrated record of Earth’s geophysical properties to support scientific research.
These science-quality standard data products require routine processing, which takes time. They are typically made available between eight hours to two months from data acquisition. If latency is not a primary concern, users are encouraged to use the standard science products.
EOSDIS data products are processed at various levels ranging from level 0 to level 4. Level 0 products are raw data at full instrument resolution. At higher levels, the data are converted into more useful parameters and formats. NRT data are often provided at level 0, 1, and 2, and some level 3 products.
|Level Name||Processing Level|
|Level 0||Level 0 data products are reconstructed, unprocessed instrument/payload data at full resolution; any and all communications artifacts, e.g., synchronization frames, communications headers, duplicate data removed.|
|Level 1A||Level 1A data products are reconstructed, unprocessed instrument data at full resolution, time-referenced, and annotated with ancillary information, including radiometric and geometric calibration coefficients and georeferencing parameters, e.g., platform ephemeris, computed and appended but not applied to the Level 0 data.|
|Level 1B||Level 1A data that have been processed to sensor units (not all instruments will have a Level 1B equivalent).|
||Level 2 data products are derived geophysical variables at the same resolution and location as the Level 1 source data.|
|Level 3||Level 3 data products are variables mapped on uniform space-time grid scales, usually with some completeness and consistency.|
|Level 4||Level 4 data products are model output or results from analyses of lower level data, e.g., variables derived from multiple measurements.|
Explore NRT Imagery and Data
NASA's Land, Atmosphere Near real-time Capability for EOS (LANCE) supports users interested in monitoring a wide variety of natural and man-made phenomena. Explore NRT satellite data products provided through LANCE.
NASA provides satellite imagery in real-time and NRT on the Worldview interactive data visualization application. For example, NASA’s Global Imagery Browse Services (GIBS) and Worldview provide imagery from geostationary satellites available in 10-minute increments approximately 40 minutes after it is captured by the satellite. Many products are available within three to five hours of being observed.
“Summary of the Workshop on Time-Sensitive Applications of NASA Data.” The Earth Observer, March - April 2017.
Published August 28, 2020
Page Last Updated: Sep 17, 2020 at 1:24 PM EDT