GISdevelopment.net ---> GITA 2001 ---> Direction for Data

3-Dimensional utility data conversion and utilization

Adam Chadwick
GIS Manager City of Kamloops 105 Seymour St
Kamloops, BC, Canada, V2C 2C6
email:achadwick@city.kamloops.bc.ca


Introduction
Hardcopy utility records, constructed and maintained over time to reflect simply the planimetric location of underground facilities, are quite often augmented with 'profile' information in order to show the distance between the ground (typically a road surface) and the underground facilities - depth of utility. Utility facility profiles are added to the same hardcopy medium as the planimetric drawing to create a plan & profile drawing. This profile information, initially captured to increase the accuracy of the underground location information in the case where digging has to occur in the vicinity of the facilities, can also be used to help build a network model of the facilities for analysis purposes. This is especially apparent for pipe-based, fluidcarrying types of utility facilities where the slope of pipes (necessary to properly define the model) are indicated in the profile view.

Once a complete set of plan, profile, and model records are in place for a utility network, a quite accurate "definition" of the utility network is available to support the majority of activities - planning, design, construction, maintenance and repair. However, these three types of records, having been built up over time using the most appropriate technology available at that time, typically are not all in a digital format and, almost certainly, are not housed within the same "database". This diversity results in utility facilities being stored and maintained in potentially three recording systems: one to represent a facility's plan location, one to represent a facility's profile location, and one to represent a facility's network model representation. Storing and maintaining the various components of utility facilities in different recording systems presents two main problems:
  1. Invariably, maintenance of utility facility records is not performed equally in each system. For example, a pipe may be added to the plan and profile records but is not added to the model either due to time constraints, lack of immediate need, operator skills, or process flaws. In addition, given the duplicity of various data items among the recording systems, and in the absence of technology or methodology to ensure a tight link between representations of facilities in the various recording systems, a pipe's diameter recorded in the plan system, for example, might not be the same as the same pipe's diameter in the model system. This is the standard problem with duplicate data storage.

  2. The components of facility records from the various recording systems: plan (xy), profile (z), and model (connectivity) cannot be easily integrated and analyzed in concert in a single system.
These problems ultimately lead to a desire to house and maintain utility records, and each of their three components - xy, z, and connectivity, in the same recording system in order to eliminate the problems associated with disparate systems as described above. In order to do this, typically an intensive data cleaning process is undertaken (known as 'scrubbing'), as well as a data conversion task that sees current utility records 'translated' to a system capable of housing and maintaining each of the three feature components previously mentioned.

Information analysis and utilization - requirements first
Prior to converting utility facility data to a single integrated system, the possibilities available in an integrated digital environment for plan/profile/model data should be considered*. By doing this it should be possible to completely replace the original source records with the newly converted data. It is critical at this point however to understand the issues surrounding conversion and integration of these records as there is a big difference between simply making digital the existing records and then replicating the original source products from the new system, as opposed to creating an entirely new data repository and output products. The former approach, which initially might seem simpler as it introduces less change into the organization, can lead to additional complications to produce the original products (as originally they were tailored for a non-digital environment), and limitations in the capabilities of the new system. Conversely, the latter approach, while initially more difficult due to the degree of change that is introduced, ultimately provides for a more complete and capable system in the longer term. The approach described in this paper is one that not only considers the integration of the three main data sources discussed so far into one data source, but also looks at redefining both in content and structure the existing output products, and also how, when, and where those products are used. Of primary significance here is that the three original data sources are migrated into one data source and the three previous data sources simply become output products rather than sources.

* Since it is believed that the majority of utility facility data sources, and this author's experience, is with hardcopy plan/profile maps and digital network model data, the data conversion issues discussed in this paper assume these types of data sources.

Planimetric Map
Planimetric maps in the pipe utility industry typically have to serve two main purposes: 1) to record the accurate location of utility facilities and, 2) to represent the operational network*, **. In the hardcopy environment this is typically satisfied by two separate map sets at different map scales. In the digital environment these two scales of maps can be produced from the single set of planimetric data using generalization and scale-based symbology techniques. Since the operational network map is, for the most part, simply 'less' information than the infrastructure location record map (as built), this product can be generated by dropping detail from the as built map and enhancing those devices of interest to an operations staff - 'Normally Open' valves for example.

Profile
In the hardcopy records environment, profile drawings are typically located either 'above' or 'below' the plan drawing on the same map sheet (hence the term, 'Plan & Profile' drawing). This enables the user to make the relationship between a valve or manhole on the profile drawing with the same valve or manhole on the planimetric drawing by simply looking vertically 'up' or 'down' the sheet between the separate plan and the profile drawings. To replicate this scenario in the digital environment would mean that fixed map sheets would have to be maintained with some method of associating pre-generated profiles with a certain location on a map page. Since this would not allow for the production of plan drawings with associated profiles for any area of the digital utility database this approach would be overly restrictive.

To avoid this problem a method has to be developed to allow for the generation of profiles for any area and for the profile to be placed anywhere on a plan map (or even separately) yet still allow a user to visually associate facilities in the plan drawing with the same facilities in the profile drawing. One method of accomplishing this is to ensure, during conversion, that unique identifiers for valves or manholes (or whatever facilities are to be shown on the profile) are captured and displayed on the plan drawing. When the profile drawing is generated from the plan information, these unique identifiers are displayed so that facilities on two physically separate outputs - one plan and one profile - can be visually associated using the unique identifiers. An additional consideration for profile drawings is the potential for them to be available in the field via handheld devices. Currently, for this to be realistically possible it is necessary to pregenerate profiles and then make them available either simplistically through some sort of index list with a descriptor identifying the location of the profile. Another indexing option would be through the use of "profile polygons" as a separate layer in the utility database that when a particular profile polygon was selected, the associated pre-generated profile would be displayed (or plotted, dependant on the environment). While this does not necessarily have an impact on

* "Wire", or non-pipe utilities usually have facilities so closely spatially located to each other that the planimetric map showing utility facilities at accurate locations is unable to show the operational network at a useful scale. Typically, schematic maps are used at quasi-geographically accurate locations for this purpose.

** While the model of a utility network, using a system with sufficient graphic interface and output capabilities, could represent the operational network, this is typically used instead for planning modifications or additions to the existing network.

data conversion activities, it is yet another way that profile drawings can be made available to users.

Another solution, and one that simply reduces, but does not totally eliminate the need for a profile drawing yet still relays the necessary elevation information, is to display on the planimetric map the depth of pipe endpoints where devices such as valves, manholes, catch basins etc., are located. Since inverts and rim elevations (road surface) can be recorded in the GIS with the pipes and devices respectively, by utilizing the connectivity of pipes to devices the pipe depths at the devices can be determined. The difference between the pipe inverts and rim elevations can be displayed as annotation on the map at a predefined distance from the pipe endpoints (to reduce clutter given that at pipe endpoints typically utility devices are shown) showing the depth of pipe at that location. For in-field use this information can be used as an approximate guide for field users when digging in the vicinity of underground utilities.

Network Modeling
For the connectivity model of the network to be derivable from the infrastructure facility records, at a minimum, it must be possible to export the data to the network modelling system of choice. Ideally, and given that data model standards are now emerging in the GIS industry, network modelling systems would be able to 'read' the GIS data directly. However, assuming that that type of higher level of systems integration is not available, output routines must be able to format the utility infrastructure data to suit the network modelling system. Of importance, from a data conversion perspective however, is that the information required to adequately create the network model is available in the GIS after loading it with the converted utility data. This implies therefore that sufficient infrastructure information (other than the traditional pipe diameter and material), from a 3-dimensional perspective, needs to be available. By converting pipe elevations into the GIS and including this information in the infrastructure utility facility data, pipe slopes and true utility facility connectivity can be derived.

Of importance here is that the network model needs to be able to be created easily and quickly so that:
  1. network modifications are done on the source infrastructure facility records data in the GIS and then exported to the network modelling system as opposed to modifications being made directly in the network modelling system, and,

  2. The data in the network modelling systems is updated frequently (as the infrastructure facility data changes) so that network operations engineers are always working with the most current information. For this to happen, the transfer of the source infrastructure facility information to the network modelling system (if that is even necessary), must be simple and fast.
All too often, after struggling to get infrastructure facility information into a network modelling system and then struggling to correctly model the network, operators are reluctant to refresh the information due to the amount of work involved in re-creating the network representation. The utility network, modeled in the network modelling system, becomes a new 'source' of information as changes to the network start to happen there as opposed to the 'true' source of the information; that being the infrastructure facility records data in the GIS.

Data Conversion
The adage of "Garbage in, Garbage out" couldn't be more true when it comes to converting any kind of data, however it is even more apparent when the data being converted comes from; 1) many sources, and, 2) contains complexities not typically encountered. With data conversion involving 3-dimensional information both of these are the case. It is for this reason that data conversion must be approached in a systematic and controlled manner.

Data Scrubbing
Whatever the decision is on the most appropriate source(s) for conversion it is important that there be as few different sets of sources as possible, and that to the degree possible, there be minimal anomalies or differences between the various files, maps, or records. In cases where automation may have occurred to some degree and for some source types - possibly the network model and the operational map, complete digital information is likely not available for all sources. In this case it might be tempting to convert the "new and accurate" digital information and the older hardcopy information. However, integrating hardcopy and digital information during conversion can be problematic due to the following reasons:
  1. The difficulty of automatically checking the various aspects of a "record" during scrubbing
  2. Difficulty of combining information from the different media during conversion
  3. Different levels of currency and accuracy of the differently managed sources
  4. Handling of different sources during conversion distracts operators from their real data conversion tasks
  5. confirmation of data integrity
For these reasons, it is prudent to reduce the number and types of sources. Typically this would be hardcopy (as opposed to digital) and most likely the infrastructure facility records map. The infrastructure facility records map usually infers the majority of infrastructure connectivity* however the profile drawings would likely have to be included as a conversion source as well as the amount of potential elevation information available for conversion could be too much information to realistically scrub onto the infrastructure facility records maps. Since it is likely that older records are the source information it is imperative that experienced people are used for the scrubbing process. The understanding that long-time employees have of

* The small scale operations map, if one exists, might have to be consulted to verify connectivity and operating device status.

the records and why they are the way the are is something that is very difficult to effectively teach to a newcomer. While these types of experienced people are usually the most difficult to get on a data scrubbing project, they are essential to producing good quality data conversion sources. Quite often these people are knowledgeable of the areas of the system that are not built to standards and why. And it's always the 10 percent of the data anomalies that cause the majority of the problems. Yes, when the sources are clear and understandable novices can be trained to scrub the sources appropriately. However, it's when anomalies arise that either novices are not able to perform the scrubbing accurately or are even unaware of the anomalies and pass them through the scrub process and onto data conversion where it will either hold-up conversion or even worse, will be converted without knowledge of the error. For these reasons, the most important aspect of this type of data conversion, given the difficulties introduced with 3- dimensional data, is the experience of the data scrubbing personnel.

In addition, while the experience of data scrub personnel is important, no one can know everything. Field checking is a must for those cases where sources conflict or are missing information and there is no obvious way of determining the correct answer. If the utility company already has staff in the field performing other tasks (and since they usually have at least radio communications), it is very useful to make these staff available to the data scrubbing staff for in-field verifications. This not only saves the time of the scrub staff, but also involves field personnel in the conversion project. Once the data is converted and put into production, the field personnel will be expected to depend on the new information so using them to perform in-field verifications helps with this as people tend to be more confident of something they've had a part in creating.

It is also useful to test a sample of scrubbed data by performing, to the extent feasibly possible, your own conversion from your own sources. By doing this (without writing custom data conversion efficiency programs) it should be possible to detect problems with your data model and your scrub sources: problems such as attributes or elevations required in the data model not being scrubbed onto the source documents.

Quality Assurance
Quality assurance, while always part of a data conversion project, gets more complicated when 3-dimensional issues are involved. Not only do layering, connectivity, attribution, annotation, and location aspects of the newly converted data have to be checked, but issues related to 3- dimensionality also have to be checked. For example:
  1. Facility features that are supposed to have elevations, do actually have meaningful elevations
  2. Flow directions tend to match the slope of pipes - (with some specific exceptions) fluid does flow uphill in gravity networks*

    * It could be that "flow direction" is not something that is captured but rather is something that is determined after conversion from facility elevation information. No double-checking of source flow directions against elevations is possible however unless flow direction is captured. It is also possible that flow direction be captured for checking purposes only and then discarded afterwards (as it can be determined at any time from the elevation information).

    Elevations of facility features that are supposed to be connected at the same location, have the same elevations

  3. Elevations of facilities that are supposed to be connected but at different elevations (at manholes for example), have different elevations.
While views differ as to whether outside data conversion contractors should be given the same data checking routines as the customer uses, the finer aspects of network connectivity and 3- dimensionality may not be apparent to most data conversion contractors and therefore may not get checked. The more information and understanding the conversion contractor has about 3- dimensional network connectivity the better the converted data will be.

Data Maintenance and Integration
Primarily for data volume and source content reasons, data is converted a "piece" at a time - that piece may be area based, network zone based, theme based, or some other type of data segmentation. Regardless of the approach taken, data already received, checked, and accepted from conversion, must be matched-up with neighbouring data as it is received, checked, and accepted. Of relevance here with respect to 3-dimensionality, is that facility features that must be connected must have the appropriate elevations as describe above (either feature elevations that are supposed to be the same are the same, or feature elevations that are supposed to be different are different). These "dangles" typically would be at pipe endpoints and could either be valid dangles or invalid dangles. Valid dangles are those pipes or devices that are endpoints to the utility system and may act as inlets or outlets of system fluids. Invalid dangles are those that exist as simply a place to stop conversion in order to constrain the size of the conversion area. A method of automatically differentiating between the two types can prevent having to repeatedly check the valid dangles in cases where data shipments have to be checked more than once. Once integrated with existing converted data, data maintenance must be performed as changes to the system occur. In addition to traditional data capture automation tools, 3-dimensionality needs to be considered. For example, capturing coordinates for pipes on curves (in xy) that are not perfectly flat, needs to be automated so that elevations for each xy coordinate does not have to be calculated manually - a very painstaking task. When pipes or devices attach at the same elevation, data capture tools can be made to assume the "connected to" feature's elevation for the feature currently being captured.

Opportunities in the digital world
Once data has been converted to a single 3-dimensional data source, new possibilities become available.
  1. Utility Operations

    2. Profile drawings, once a source of information on their own, can be now generated either for general use throughout the organization, or for use in the field. Additionally, profiles can be created for specific areas "on-the-fly" for example, in the case where a non-traditional area needs to be profiled, possibly many city blocks for the purpose of design drawings for street reconstruction and utility replacement.

    Custom content profiles can also be created whereby only some system types (possibly only drainage and sanitary utilities) are profiled; or possibly additional pipe attributes are shown on the profile.

    Operational maps can now be created at varying scales and content possibly showing a highly generalized view of the utility system with only major devices shown, or a more neighbourhoodtype view of the operational system with all pipes, devices, and their normal operating status shown.

    System modelling software can now be used on a more ad hoc basis. For example, for checking current system capacity in a specific area of the system for the purpose of verifying capacity for adding new utility customers. Ad hoc modeling can now be done on specific areas of the system in order to track-down chronic system problems. These activities should now be able to be done on the 3-dimensional source data through either directly reading the 3-dimensional data that is stored within the GIS either from within the network modelling system or through a simple transfer process. No longer should it be necessary to permanently store network information in the network modelling system as source data should be available through the newly converted and integrated data.

  2. Foreign Utilities

    3. While less dependant on the 3-dimensionality of the converted data, once digital, utility infrastructure information can now be provided to other utility service providers for use in their operations, and vice-versa. Of particular benefit, the ability to view other company's utility records in conjunction with your own can assist with preliminary designs for system modification or additions, or for planning road openings for utility repair or servicing. An essential component of utility infrastructure data exchange is for all parties to use either the same land base (from which utility lines are offset) or ones that are spatially within similar accuracies. Without this when other utility companies view each others utility infrastructure data, placement of the each others lines will not be correct enough to be of any value. Municipal utility providers can offer to exchange their (usually) up to date and spatially accurate land bases with the foreign utility companies in exchange for access to the foreign utility company infrastructure data. This allows utility companies to avoid having to capture land bases and for municipalities to gain access to up to date digital foreign utility locations for use in their own operations*.

    * The Province of British Columbia, utility companies, and municipalities and regional district have just completed a "proof of concept" pilot project which has all three groups integrating their various data sets in a central data warehouse. For more information on this initiative, please visit the web site at: http://www.elp.gov.bc.ca/clrs/business_solutions/ici/index.html
Summary
Given the recent possibilities brought about by advances in GIS capabilities for the storage and manipulation of 3-dimensional geographic data, new ways of managing utility information are now available. By carefully considering how these capabilities can be used, conversion of utility information may have to be done differently, and more extensively, to ensure that more usable and complete utility information is available in the GIS. The biggest change that this brings about is the migration from multiple hardcopy and/or digital "sources" to a single digital source from which traditional map and data products are derived. Once this concept is adopted, plan, profile, and network model representations of the utility facilities become outputs derived at will from the single-source GIS utility data set.
© GISdevelopment.net. All rights reserved.