Table of Contents
1. Acknowledgements 4
2 Introduction 5
2.1 Background 5
2.2 How to use this FCD 5
3 Component Alignment and Change History 8
3.1 Flow Component Version History 8
3.2 Flow Component Versions Currently Supported 8
3.3 Business Rule Change History 9
4 Flow Summary Information 10
4.1 Flow Identification 10
4.2 Data Flow Overview 10
4.3 Flow Access and Security 19
4.4 Flow-level Business Rules 19
4.4.1 General Business Rules Relating to Local Updating of the NHD 19
4.4.2 Business Rules for Submitting Local Updates to USGS via this Data Flow 20
4.4.3 Business Rules Relating to USGS Distribution of NHD Updates 21
4.5 Additional Flow Tools and Resources 21
5 Schema Information 23
5.1 Schema Structure 23
5.2 Schema Components 23
6 Data Service Information 24
6.1 Data Services 24
6.2 SubmitNHDUpdate 25
6.3 GetStatusNHDUpdate 26
7 Exchange Network Header Information 27
7.1 Header Elements, Description and Examples 27
8 Supplemental Information 30
8.1 Processing Overview 30
8.1.1 Local to USGS Data Flow (To_USGS): 30
8.1.2 USGS to Local Data Flow (From_USGS): 31
8.2 Operational Data Store Interface - USGS 31
8.3 Submission Processing and Feedback 32
8.3.1 Local to USGS (To_USGS): 32
8.3.2 USGS to Local (From_USGS): 33
Attachment A – NHDUpdate Submit Data Service Payload Information 35
Information Carried by the Data Flow 35
Data Flow Payloads 37
Attachment B – About the Minnesota Stewardship Pilot Project 39
Attachment C – Using Windsor Solutions’ NodeClientLite 41
1. Acknowledgements
This Flow Configuration Document was produced by the Land Management Information Center (LMIC) for the Environmental Protection Agency’s Office of Environmental Information under and Exchange Network FY 05 Challenge Grant - Assistance ID # OS-83260201. This document was prepared in cooperation with: United States Geological Survey (USGS) – NHD Team; EPA Office of Water, Wetlands, Oceans, and Watersheds (EPA-OWOW); Minnesota Pollution Control Agency (MPCA); and the Exchange Network CDX Team.
This document was prepared with input and support from the following individuals:
Susanne Maeder / LMICJerry Ornelas / USGS
Colleen Keeling / USGS
Jessica Janssen / USGS
Paul Kimsey / USGS
Steve Andrews / Indus Corp.
Scott Kocher / Indus Corp.
Mark Olsen / MPCA
Tommy Dewald / EPA - OWOW
Cindy McKay / Horizon Systems
2 Introduction
2.1 Background
The purpose of this project is to develop a data flow to transmit geospatial information updates to the National Hydrography Dataset (NHD). The National Hydrography Dataset (NHD) is a nationwide framework GIS database supporting surface water hydrologic applications. The data flow is designated as ‘NHDUpdate’ – or the ‘NHD Update Flow’. The NHD Update Flow enables the transmittal of updates to the National Hydrography Dataset created by local updaters or state data stewards of the NHD to the Master NHD data repository administered by the United States Geological Survey (USGS). The NHD Update Flow also enables the transmittal of updates from the master NHD database at USGS to satellite NHD databases administered by other agencies, including state stewards of the NHD.
This data flow is being developed by the United States Geological Survey – NHD Team and the Land Management Information Center, Office of Geographic and Demographic Analysis, Minnesota Department of Administration. Supporting agencies include the Minnesota Pollution Control Agency and the Environmental Protection Agency – Office of Water.
This project will develop the procedures necessary to support the transmittal of NHD update data. USGS NHD update procedures and tools are already designed to create a standardized XML as the update medium. As such, table definition structures have already been defined, if not described in the standard schema format required to support a data flow. This project will complete the process and develop the documentation and tools necessary to formally transmit this data set as an Exchange Network Data Flow. At the state level the data will be transmitted through Minnesota’s state node maintained by the Minnesota Pollution Control Agency. To accommodate this project the USGS is constructing its own node to distribute and accept the NHD Update Flow.
Note that this data flow defines only updates passed between NHD data stewards and updaters and the USGS NHD Database. Users of NHD data who are not updaters of the data are able to use the data directly via an online internet map service, or download NHD data for an area of interest in multiple formats through an already established web-based data distribution mechanism (See http://nhd.usgs.gov/data.html). This data flow will not be used to support routine data distribution to NHD users.
2.2 How to use this FCD
The purpose of this FCD is to describe the operation of the NHD Update Flow using XML-based data submissions through Node-to-Node or Client-to-Node transfers. The focus of the document is the spatial data flow of NHD updates between the state, local, and tribal agencies and USGS that meet the NHD requirements.
This Flow Configuration Document (FCD) is intended to define the supported data services, the approaches and processes that are used to exchange information over the Exchange Network via Web services technology. In addition, the FCD serves as a guide for trading partners to the details and challenges associated with a specific flow.
The scope of this FCD has been limited to the following primary functions:
· Submission of the data; and
· Reporting of the submission status.
This document does not include any query or data retrieval services. The data is to be submitted from local NHD updaters through state nodes to a node at the USGS, from which the data can be incorporated into the master NHD data repository using standard update processes already defined by the USGS. Updates from USGS back to the stewards can be transmitted in reverse using the same services.
This document includes the following five main sections and three attachments:
Flow Summary Information (Section 4)
This section describes the process of submitting NHD update data from the submitter environment via the USGS node, through the loading of the data into the NHD data repository at USGS. It also describes the reverse process by which USGS can send updates to satellite NHD databases hosted by states, tribes, or other federal entities.
Schema Information (Section 5)
This section provides basic overview information about the schema and references to the full schema.
Data Service Information (Section 6)
This section describes the ‘Submit’ and ‘GetStatus’ data services used to transact the NHD Update Flow.
Exchange Network Header Information (Section 7)
This section describes the Exchange Network Header information used in the NHD Update Flow. This submission structure will be used for both Node-to-Node and Client-to-Node submissions.
Supplemental Information (Section 8)
This section includes additional information considered to be useful to the understanding of the data flow.
Attachments
Attachment A describes in more detail the payloads carried by the NHD Update Data Flow.
Attachment B describes the Minnesota Stewardship Pilot Project.
Attachment C describes data submission using Windsor Solutions’ NodeClientLite software.
3 Component Alignment and Change History
Note: The major and minor version number of the FCD should be identical to the major and minor version number of the current schema supported by this exchange. If the FCD is changed without changes to the schema, then a revision number should be used on the FCD’s version number.
3.1 Flow Component Version History
Component / Version / Date / Changed By / Description of ChangeFCD / 0.3 / 2/20/2008 / S. Maeder / Internal Project Distribution draft; also submitted to NRB for pre-review, informational purposes.
FCD / 1.0 / 6/27/2008 / S. Maeder / Submission Draft Version 1.0
Schema / 1.0 / 6/27/2008 / J. Ornelas/ C. Keeling / Submission Schema Version 1.0
DET / 1.0 / 6/27/2008 / J. Ornelas/ C. Keeling / Submission DET Version 1.0
FCD / 1.1 / 8/20/2008 / S. Maeder / Revision in response to NTG Review
Schema / 1.1 / 8/20/2008 / J. Ornelas/ C. Keeling / Revision in response to NTG Review
DET / 1.1 / 8/20/2008 / J. Ornelas/ C. Keeling / Revision in response to NTG Review
3.2 Flow Component Versions Currently Supported
Component / Version(s) Supported / Explanation (optional)FCD / 1.1
Schema / 1.1
DET / 1.1
Submit / 1.1
GetStatus / 1.1
3.3 Business Rule Change History
Current business rules are listed in Section 4.4.
Business Rule Change / Date of Change / Explanation (optional)4 Flow Summary Information
4.1 Flow Identification
Flow Name: NHDUpdate
Flow Description:
This data flow is being created to support the transmission of updates to the National Hydrography Dataset. The data flow supports updates submitted to the United States Geological Survey master NHD data repository from local updaters and data stewards. The flow also supports updates extracted from the master NHD data repository by USGS and distributed to state stewardship databases and other satellite databases of the NHD.
Flow Steward:
This data flow is being developed by the United States Geological Survey – NHD Team and the Land Management Information Center, Office of Geographic and Demographic Analysis, Minnesota Department of Administration. Supporting agencies include the Minnesota Pollution Control Agency and the U. S. Environmental Protection Agency – Office of Water.
Flow Steward Contact Information:
Questions about this data flow can be directed to Jerry Ornelas, USGS-NHD Team (303-202-4143, ) or Susanne Maeder, Land Management Information Center, Mn Dept. of Administration (651-201-2488, ).
4.2 Data Flow Overview
Background
The National Hydrography Dataset (NHD) is a nationwide framework GIS database supporting surface water hydrologic applications. NHD is the national geospatial framework theme for hydrography. The United States Geological Survey serves as the theme lead for the National Hydrography Dataset. Additional support at the federal level comes from the U.S. Environmental Protection Agency Office of Water and the United States Forest Service. For more information on the content, theory, schemas, and activities supporting the NHD see http://nhd.usgs.gov.
NHD data nationwide is available at two scales: medium-resolution (1:100,000 scale) and high-resolution (1:24,000 scale). Users in some parts of the U.S. have created ‘local resolution’ data (e.g. 1:4800), which will be incorporated into the high-resolution database. The United States Geological Survey (USGS) and the United States Environmental Protection Agency (EPA) created the medium-resolution data by combining key elements of their native hydrography data sets. Creation of the high-resolution NHD was a cooperative effort which involved the USGS and other federal agencies, as well as many state, local, and tribal governments. High-resolution NHD was completed nationwide in August, 2007.
USGS maintains the national NHD database in a dual storage framework:
· A Production database houses the main database and is used to process updates provided by USGS and contributors. This is the ‘Master’ NHD database for the entire U.S. This database is versioned and processes updates reported as XML documents defined in a USGS native schema and format - Feature Communication Protocol (FCP).
· A Distribution database is a mirror image of the most current version of the database that can be accessed by users as a map service or through a data extraction process. Users can choose to extract the data in several formats.
As creating the NHD was a cooperative effort, updating and maintaining the NHD is also intended to be an effort involving many partners. The NHD was designed with the intention that the data can and will be updated and that all updates must be carefully tracked, coordinated, and validated. USGS is currently performing prescribed maintenance data steps (designated as ‘maintenance lite’) against the entire database. USGS is also working with numerous partners under ‘stewardship’ arrangements. Through the stewardship process USGS recognizes that local units of government are more familiar with their own hydrography, and supplies tools, training, and support to partners who wish to function as technical or administrative stewards. USGS tools implemented at both ends of the process assure that updates follow specified rules and are submitted in a specified format. (USGS stewardship information can be found at http://webhosts.cr.usgs.gov/steward/).
The NHDUpdate data flow does not pass through the Central Data Exchange, but is transmitted directly between the United States Geological Survey node and state nodes.
Many organizations (state, tribal, county, regional, local) may perform updates to the NHD and submit them to USGS: the entire stewardship environment was designed by the USGS to accommodate updaters from many types of jurisdictions. Entities exchanging updates with USGS through the NHD Update Data Flow will work through an established Exchange Network node to do so. Although this document generally describes the non-USGS node as a ‘state node’, it is possible that the Exchange Network node used by updaters could be a tribal or other node.
The USGS – NHD update process already operates in a controlled environment. Since the NHD Update programs already write out XML according to the NHDUpdate_v1.0 schema, there is no reason for an additional validation of the XML during the data flow transmission. Further validation at both ends of the process checks the update for validity within the logic of the NHD database.
The update processes described in this document are based upon current USGS practice for the NHD. This practice was developed using ArcGIS 9.1 capabilities and tools. There is the potential that update tools will be enhanced or changed in the future with newer database replication tools. At that point this document may have to be modified.
NHD Stewardship Roles and Responsibilities
The NHD Stewardship overview defined by the USGS is as follows:
· The community of users becomes the stewards of the data.
· The USGS facilitates the stewardship process.
· The users evolve the data to best meet their needs.
· The USGS guides the evolution for national continuity.
For the NHD Updating process, USGS defines three layers of roles:
· NHD National Steward (USGS): provide for continuity of the NHD; provide tools, training, quality control, database definition, ‘master’ NHD national database; (U=USGS)
· NHD State Steward (Administrative): provide area coordination; possibly also provide updating and local quality control; (S=State Steward)
· NHD technical stewards (local updaters/maintainers): provide enhancements to the data using the NHD tools, submitting updates either to the NHD State Steward or directly to the USGS. (L=Local Updater)