Operations Concepts DocumentsThe ECHO Team develops Operational Concepts to propose changes to ECHO, PUMP, or WIST functionality as a means of reviewing impacts to current functionality and all ECHO Partners. Each Operations Concept document is reviewed during an ECHO Technical Committee Meeting. ECHO Partners are encouraged to attend this meeting or to provide feedback via email. After the review, and after all outstanding issues have been resolved to the approval of the ECHO Partners, the Operations Concept will then be scheduled for implementation in an ECHO development sprint. The Operational Concept Documents covered on this page include the following items: Note: Not all Ops Concepts have been externally reviewed at an ETC, and therefore have not been scheduled for implementation.
DocumentsECHO User Address Updates (ECHO_OpsCon_025) This document describes work to be performed as part of EED Task 2 Revision 4. With the Introduction of the EOSDIS User Registration System (URS) disparate EOSDIS applications now have access to a common set of authentication and core user information fields. As part of URS Phase I, ECHO moved to be compatible with URS and now all authentication, user creation, and user updates are proxied to the URS. In general the ECHO User concept and the URS User concept fit well together, however, the location of the URS mandatory Country field in the optional ECHO User Address fields has become problematic. Specific enhancements have been proposed to make several fields in the legacy Address information optional and require exactly one Address for a user.
Visualization Tools (ECHO_OpsCon_024) This document describes work to be performed as part of EED Task 2 Revision 4. Specific enhancements have been requested to add capabilities to Reverb so that it will support image browsing for granules with associated browse imagery. Users will be able to select granules through their associated imagery as an additional mechanism for moving items into and out of the Reverb shopping cart. These enhancements are categorized under the heading “Image View.” Additionally, users will be able to select groups of granules temporally by supplying a set date/time range. Users can adjust the range through UI controls and view granule imagery across datasets. These enhancements are categorized under the heading “Timeline View.”
Browse Reclassification (ECHO_OpsCon_023) This document describes proposed changes to ECHO data model to reclassify the browse record. It is proposed that the ECHO data model will no longer have a formal browse record. If approved, all changes proposed in this document will be implemented as a part of the ECHO Task 2 Rev 2 Multi-Format Ingest work, currently scheduled for Operational deployment in late-Summer 2011.
Browse MimeType Updates (ECHO_OpsCon_022) This document describes the proposed changes to the existing FTP Ingest service and Reverb. There will be no visible changes to the ECHO kernel and no changes to WIST. If approved, all changes proposed in this document will be implemented as a part of the ECHO Task 2 Rev 2 Multi-Format Ingest work, currently scheduled for Operational deployment in late-Summer 2011.
Task 2 Rev 2 - Multi-Format Metadata Support (ECHO_OpsCon_020) This document describes the work to be performed as a part of EED Task 2 Revision 2. Specific enhancements have been requested to add capabilities to ECHO so that it will support Ingest and metadata presentation in multiple formats, specifically ISO 19115. Updates will also be implemented in Reverb to leverage the new capabilities and to display the ISO 19115 metadata fields.
ECHO Services (ECHO_OpsCon_019) This document describes proposed changes to the ECHO ExtendedServices service. The proposed changes are in response to the EED Task 3 statement of work. Specifically, the following items have been requested:
ACL Reporting (ECHO_OpsCon_018) This document describes a proposed Operational solution for providing enhanced status notifications to ECHO Client Partner applications. Implementation of the proposed capability will be performed by the ECHO Operations team and will not be a part of the core ECHO baseline.
ACL Reporting (ECHO_OpsCon_017) This document describes the proposed changes to PUMP and the ECHO API to provide reporting on Catalog Item and Provider Object ACLs.
Bulk Option Assigment Deletion (ECHO_OpsCon_016) As documented in NCR 1100991, PUMP does not allow users to delete option assignments in bulk. To delete option assignments in PUMP, users have to perform an option assignment search for the target collections and then delete the option assignments one-by-one from the option assignment search results page. Depending on the number of collections and options assignments this delete process can be cumbersome.
Ingest File Error Handling (ECHO_OpsCon_015) This document describes proposed changes to Ingest reporting to add functionality which will limit the number of reported file errors for each metadata file processed in each job.
ECHO Group Management Ops Concept (ECHO_OpsCon_014) This document describes proposed changes to the group management API to add functionality needed to support the proposed access control level (ACL) changes presented in a separate operations concept.
ECHO ACLs and Roles Ops Concept (ECHO_OpsCon_013) This document describes proposed changes to the access control level (ACL) system to reduce complexity and implement requested functionality which includes expanding the system beyond the current metadata catalog scope. The current ACL system feature-set is a carryover from earlier, versions of ECHO. Now that that API is used regularly by providers and client applications, a number of limitations, complexities, and unnecessary features have been identified. In order to addresses these issues, the ACL API needs to be modified based on past experience and future needs. At the same time, the need for a more encompassing ACL system has been identified. To support more complex use cases proposed by the DAACs, ECHO needs to support ACLs for more domain objects including but not limited to provider orders and provider policies.
Additional Attribute Order (ECHO_OpsCon_012) ECHO allows for multiple additional attributes per collection or granule and does not associate any order to those additional attributes. However, providers and users of ECHO metadata may consider additional attributes to be ordered by the order in which they appear the metadata. Ingest does not preserve the original order of additional attributes sent by providers. As a result, additional attributes will be presented in any order when collection or granule metadata is retrieved from ECHO. As part of ECHO 10.12, Ingest was modified to preserve the order of additional attribute values for a single granule additional attribute.
Order Dispatch Fault Tolerance (ECHO_OpsCon_011) ECHO dispatches provider orders to multiple providers using up to 5 concurrent threads. Each thread delivers one order using a SOAP/HTTP transfer to the provider's endpoint. Once the transfer is complete, the thread becomes available again to dispatch the next queued order. In most cases the preparation and dispatch of an order takes only a few minutes and the order queue remains relatively short or empty. In the event that a provider's endpoint is unreachable or there is an error processing the order for the provider (for example, a SOAP fault), Echo delays the order transmission and will automatically requeue the order at a later date based on the provider's policies.
ECHO Reconciliation (ECHO_OpsCon_010) All reconciliation activities performed by ECHO data providers are currently facilitated through the GetDatasetInformation method within the ECHO API's DataManagementService. This method generates an XML file containing the following metadata items and transmits that file via ftp to a configured destination:
Providers can specify one collection at a time for reconciliation and have ECHO filter on the following data characteristics to limit the data which is generated by the method.
Spatial Keywords Restructuring (ECHO_OpsCon_009) The ECHO data model was historically based upon existing metadata models at the time of its creation. Since that time there have been modifications to certain externally managed metadata fields which result in an inconsistency when compared with the ECHO data model's representation. Previously, ECHO updated its 'ScienceKeyword' structure to correctly match the updated GCMD structure for its science keywords. Similarly, the GCMD manages a list of 'location' keywords which have a historical association with ECHO's 'SpatialKeyword' structure. The new GCMD structure is as follows: Category > Type > Sub-Region1 > Sub-Region2 > Sub-Region3 > Detailed Location. ECHO currently has a single tiered structure only allowing for a single value in its SpatialKeywords.
Data Quality Summary Management (ECHO_OpsCon_008) The WIST wwwvalids.conf contains data quality summaries which are assigned to datasets based upon a small number of characteristics (e.g. short name, instrument, DAAC, etc). These summaries are maintained by the ECHO Operations group and any changes must be coordinated with ECHO data partners to ensure the proper information is displayed. Each data quality summary which is associated with a granule that is added to a WIST user's shopping cart will be displayed once during the process of adding granules to the user's cart. Users must choose to accept or reject the data quality summary which is displayed to them. This functionality is a carryover from EDG functionality, as are many of the summaries which are displayed to end users. This document outlines changes to ECHO and PUMP which will allow ECHO Data Partners to create, update, delete, and assign Data Quality Summaries to their data without requiring ECHO Operations coordination. The proposed functionality will also allow all ECHO clients to display the confnigured Data Quality Summaries, providing consistent user experiences within ECHO clients.
Suspend and resume provider (ECHO_OpsCon_007) This document describes the proposed implementation of a mechanism to suspend and resume search and order capabilities on a per provider basis. This operations concept was prepared in response to the trouble ticket 11004665 'Allow disabling a provider for search and order by Admin via PUMP'. The trouble ticket states the following, In an effort to reduce scheduled downtime of ECHO it is desirable to disable a single DAAC from the query and ordering for a short period of time. This time can be used to rebuild/modify indexes, update metadata, migrate table structures, etc. without needing to completely shutdown ECHO. We can approximate this functionality by disabling the provider at the DB level for specific situations. However, this scenario should be fully planned and made accessible via the ECHO API and surfaced as Admin functionality in pump.
ECHO Online Access & Resource URL Uniqueness (ECHO_OpsCon_006) The ECHO data model currently includes OnlineAccess and OnlineResource URLs associated with collections and/or granules. The intention of an OnlineAccess URL is to provider end users with a link that will directly take them to an http or ftp site where the collection or granule can be accessed (e.g. downloaded). The intention of an OnlineResource URL is to provider end users with links providing additional information regarding the collection or granule (e.g. instrument info, dataset descriptions). The metadata for OnlineAccess URLs also has URLDescription and MimeType fields, while OnlineResource URLs have Description, Type, and MimeType fields. Both OnlineAccess and OnlineResource URLs are currently uniquely identified by the actual URL itself, not using either the Type or MimeType fields in identifying URLs. Currently within ECHO all OnlineResource URLs can be uniquely identified by a combination of the URL and Type (not MimeType).
ECHO Resource Reconcilliation (ECHO_OpsCon_005) This document describes the proposed implementation of a URL resource reconciliation service that will validate persisted URLs in ECHO. The need to validate URLs persisted in ECHO has been identified in the BMGT Reconciliation White Paper as follows, 'ECHO will also perform periodic checks of datapool URLs that have been ingested into ECHO. These checks will make sure that the URLs are well formed and that they point to an existing endpoint. Any errors found will be reported to the DAAC in a manner that is TBD. While ECHO doesn't modify online access or online resource URLs as part of Ingest, there is a desire to ensure that the URLs stored have not expired or that there was not a data entry problem or mis-configuration of the exported data at the DAAC.
Collection Level TwoDCoordinates (ECHO_OpsCon_004) Granule metadata includes an element named TwoDCoordinate which allows for a granule to be spatially located via a two-dimensional coordinate system. This information is only available in the granule metadata. ECHO currently receives metadata from USGS_EROS for the WRS-1 and WRS-2 projections within these elements. ECHO Ingest also adapts specific PSAs from ECS metadata to create 2D coordinate system information for MISR and MODIS Tile grids. In instances such as Platforms/Instruments/Sensors and Additional Attributes, the collection level metadata contains defining information for these metadata constructs. The granule metadata then references a collection-level defined item and supplies values for the necessary fields. There is no mechanism to do this with TwoDCoordinates.
ECHO Calendar & Provider Information (ECHO_OpsCon_003) This document describes the proposed replacement of the Uptime Calendar, extensions to the provider profile's additional information, and associated enhancements to ECHO Services and PUMP. The Uptime Calendar is external to ECHO and is mainly used to pass information from providers to WIST users. Providers use the Uptime Calendar web client to publish alert and events that are then persisted by the Uptime Calendar service. WIST uses the Uptime Calendar service to retrieve any alerts or events that have been published by providers for display in WIST. Because the Uptime Calendar is separate from ECHO it requires separate user accounts and account management. Although WIST is currently the only client using the Uptime Calendar service, the information published by providers could be useful to any ECHO client as it commonly pertains to data-access outages or the status of data products. To address these issues with the Uptime Calendar, a new ECHO Calendar Service will be developed to replace the current Uptime Calendar service and PUMP will be augmented to allow for managing ECHO Calendar events using the ECHO Calendar Service. Also, to support an enhancement request from providers to allow configurable provider information such as an overview description and general information links, extensions will be made to the provider profile's additional information field.
Additional Attribute Type Handling (ECHO_OpsCon_002) Additional attributes are simply key/value pairs associated with collections and granules. A collection defines the name, type, and description of all supported additional attributes. A collection may also optionally define a value for the attribute. A granule additional attribute must reference an attribute defined in the parent collection and specify a specific value. A granule specified value for an additional attribute will override a value specified for the same additional attribute at the collection level. Additional attributes contain different types of information such as numbers (integer and floating point), dates, times, and text strings. Currently all of these values are stored as text strings in the ECHO database. Because of this storage design, clients are now seeing errors when searching for numeric values because the database cannot convert the string and date types to numbers for comparison.
ECHO & WIST Science Keywords Modifications (ECHO_OpsCon_001) This document describes proposed changes to the ECHO Ingest Schema, AQL DTD , Collection Results DTD, and Taxonomy Service and WIST User Interface in order to align the usage of GCMD managed science keywords. The current Science Keywords data model does not match its analogous structure managed by the GCMD. Data centers are choosing science keywords based upon this externally managed list, however they are unable at this time to submit the full and correct keywords to ECHO in their collection metadata. There is provider interest in adhering to this list, and thus ECHO too should make it possible for data centers to have their data properly represented within ECHO's holdings.
|
