NASA Earthdata Wiki
ISO and HDF5
Implementing ISO Information Models in HDF5
Today's information modeler can rely on well-established methodologies for information model development and expressive languages for their formalization. When it comes to implementing ISO information models the picture appears much less settled. Implementation methods such as databases and XML assist the implementer with sopisticated tool chains and machinery, adding to their appeal. Unfortunately, the information model is only one of two key ingredients on the road to a valid implementation. The second, equally important, ingredient, and not part of the information model, is usage. A particular implementation method may deliver a verified implementation, but it may turn out to be not valid (i.e., not fit for its intended purpose). It may turn out to be valid, but the community may not accept it or a hostile ecosystem may prevent it from gaining a foothold.
Since its initial release in 1998, HDF5 has been in the sights of implementers in search of suitable targets for their models. HDF5 distinguishes itself from competitors through a unique combination of features related to its data model, its software stack, and its file format:
- Facilities for item association, hierarchies, and annotation
- Files are self-contained and self-describing
- Unlimited file size support
- Free and open source (BSD license)
- Portable across platforms and architectures
- Flexible user-defined types
- High I/O performance, parallel I/O
- Out-of-core data access (partial I/O)
- Support for compression and other custom filters
- Suitable for long-term data archiving
- High-quality support, training, and documentation
These and other capabilities have led to the widespread adoption of HDF5 in many fields, ranging from earth science to product models.
The HDF5 ecosystem includes a wide variety of APIs, tools, and applications. An HDF5 implementation of any ISO information model will be constrained by community traditions and preferences, and the pervasiveness of the support of certain HDF5 features.
For example, if the support of user-defined data types (UDT) is generally limited or diverging throughout the ecosystem, a solution solely based on technical excellence may not find wide adoption. Even with the best support for UDTs this might be a poor choice if, not informed by its intended use, it was made as a result of taking a formal specification too literal.
What looks like a perfect match on paper may incur a hefty performance penalty when the implementation is brought into its intended environment. To make matters worse, but to meet its performance targets, a successful implementation may have to opt for different HDF5 represenations of formally similar concepts in the same information model. For example, a sparsely populated record type might be more efficiently represented by an HDF5 group decorated with HDF5 attributes or, depending on use, a collection of HDF5 datasets.
As there is no "royal road to geometry" (Euclid's alleged reply to King Ptolemy's inquiry about a shortcut to learning the subject), there is no "easy way" to implementing rich ISO information models in HDF5. On the upside, you can always count on the advise and a helping hand from the dedicated individuals at The HDF Group and the wider HDF community. Talk to them at mailto:help@hdfgroup.org or the HDF-Forum !
- Wikipedia, Information model, http://en.wikipedia.org/wiki/Information_model
- Y. Tina Lee, Information Modeling: From Design To Implementation, NIST 1999
- Vailin Choi and Mike Folk, Investigations into using HDF5 as an alternative to STEP for Finite Element Modeling data, April 2007
- (GEO-IDE) Guidelines and Best Practices Wiki, NetCDF, HDF and ISO Metadata, https://geo-ide.noaa.gov/wiki/index.php?title=NetCDF,_HDF_and_ISO_Metadata
