| GISdevelopment.net ---> GITA 1999 ---> User Perspectives |
Networked Electronic and Manual QA of Conversion Data
MaryAnn Stewart
AM/FM Conversion Manager
UtiliCorp United
20 W. 9’hStreet
Kansas City, MO 64105
MaryAnn Stewart
AM/FM Conversion Manager
UtiliCorp United
20 W. 9’hStreet
Kansas City, MO 64105
Project Overview
UtiliCorp United is a Kansas City, MO-based company providing gas and electric utility services to 1.1 million customers in eight states (Missouri, Kansas, Colorado, Nebraska, Iowa, Minnesota, Michigan, and West Virginia), British Columbia, New Zealand and Australia. UtiliCorp is currently engaged in an aggressive conversion project, taking paper and electronic maps of holdings in the eight states to a common electronic format. This conversion is part of a larger reengineering effort, which moves the primary mapping fimctions from centralized drafting personnel to decentralized construction coordinators in field offices. Applications development for input of designed work is proceeding in parallel with data conversion. The resulting AM/FM/GIS of UtiliCorp is evolving as FAME (Facilities and Mapping Enabler).
Utilities under conversion are: Kansas Public Service, Michigan Gas Utilities, Missouri Public Service, Northern Minnesota Utilities, Peoples Natural Gas, West-Plains Energy and West Virginia Power. Map formats are diverse, from manual sources on paper, Mylar and linen to various electronic implementations of AutoCAD, CableCAD, Gentry, and ProScan. Age of maps ranges from Kansas Public Service linen maps dating back to 1906 to West-Plains Gentry maps being created from multiple sources just in time for the conversion. The historic mapping standards of the seven companies were predictably diverse, with an especially troublesome variety of information mapped and considered essential.
End users participated in data modeling sessions and other decision making meetings through the summer of 1997, during which common gas and electric data models were forged. The contract with a conversion vendor was signed in September, 1997, and we proceeded with gas and electric proof of concept conversion throughout the following winter. The production conversion schedule is aggressive, with all maps to be converted in 1½ years, resulting in a complete hardware, software, network and data rollout to 114 initial FAME end users by the end of 1999.
Electronic QA and Conversion Issues
The combination of an aggressive conversion schedule and the accuracy requirements of the conversion contract (100% connectivity and 98% attributes) created a challenge in QA of converted data. UtiliCorp’s intent is to be able to receive converted data as rapidly as possible and then rigorously examine the quality of this data. It became necessary to develop efficient and flexible QA procedures to detect the types of errors encountered in each phase of the conversion.
The electronic QA tool became the first line of defense in the examination of 63 conversion delivery packets. The electronic QA tool was originally developed by UtiliCorp’s application vendor and was configured by the vendor and UtiliCorp for use on this conversion project. The primary capabilities of the tool are to verify connectivity, object placement, and object attribute values. Toggle settings allow the user to query on all objects or on a partial subset of objects. The size and composition of the sample data set is also user defined. The sample can consist of the data representing the facilities in a defined geographical area, in an entire network, or in the total GIS. The tool also can generate a random sampling to check attributes for a subset of a large number of facilities.
The electronic tool can be configured to make connectivity checks based on UtiliCorp conversion rules. It is also possible to configure the tool to make attribute checks based on UtiliCorp conversion rules. The configuration can be readily altered based on changes in rules from company to company or on evolving standards. The tool provides interactive capabilities— it provides GoTo functionality on flagged errors as well as the opportunity to examine the attributes of a flagged object while within the tool.
The electronic tool is very good at checking facilities information-phasing compatibility, pipe termination, correct number of connections, annotation placement with respect to its object. However, it can’t be expected to do much for certain proofreading needs, especially with regard to landbase features, such as correct road names, presence or absence of features, or non-rulebased attribute checks. Thus it became necessary to develop a scheme for conducting manual QA.
UtiliCorp was fortunate in acquiring the services of a Kansas City minority small business engaged in AM/FM work for utilities. A methodology was developed for taking a random sample of the 1:100 scale map grids, plotting the grids, proofreading paper copies of converted data against paper originals, and generating statistics on each map grid. The statistical methodology is based on UtiliCorp’s U.S.-wide grid system and FAME’s capability to count total objects by type in each grid. Sample sizes are determined using a single sampling method based on Dodge and Romig’s Sanmling Irumection Tables.
This manual proofreading effort was initially painful in that it required tons of paper to move through our office space. The solution was to provide our manual QA vendor with network connectivity and a FAME workstation to facilitate plotting in-house. Due primarily to this connectivity, the manual QA vendor and UtiliCorp are able to develop innovative techniques to streamline and automate the manual QA process.
UtiliCorp QA Plan
UtiliCorp’s QA plan for the conversion requires the participation of FAME end users in final acceptance or rejection of the data. This participation is critical to the success of the project, as the end users know the most about their data and will be responsible for upkeep of the maps once conversion is complete. However, participation by end users at the beginning of the FAME rollout was challenging on all fronts – from training to network connectivity to system administration.
The FAME project initial rollout has 114 end users at sites in 48 towns in eight states. The first site requiring network connectivity for viewing converted data was in northern Minnesota and suffered delays because the telecom service provider couldn’t place lines until ground thawed in the spring. Once lines were laid, it was full-tilt construction season and the Construction Coordinator had no time for map proofreading. It is anticipated that end users at the middle or end of the rollout will not be exposed to so much uncertainty in acquisition of their hardware or network lines. However, there will probably continue to be the scheduling problem that all Construction Coordinators want their data to be delivered in the dead of winter.
Prior to the end user rollout, FAME had established network connectivity between FAME offices in Kansas City, the application development vendor in Austin, the data server in Omaha and the conversion vendor in Indianapolis. An early connection was made to the application developer, to provide remote development capability. As a result, UtiliCorp staff developed technical expertise in the desired network configuration. The process of the subsequent connections benefited from the early knowledge and UtiliCorp was able to provide full technical support for the connections. We found it substantially less confksing to take charge of all network issues, rather than sharing these issues with the connecting side staff.
The UtiliCorp QA plan outlines the acceptance process for converted data. Data must pass electronic QA, manual QA, end user spot checks, FAME staff spot checks and move to acceptance by the end users and the UtiliCorp Conversion Coordinator. The overall goals of the QA plan are: detect systematic conversion errors early, identify all possible problems with the conversion and expedite acceptance of conversion packets. The general QA flow of a delivered work packet is from electronic check to manual check, with spot checks occurring throughout the work flow.
A crucial first step in working with the electronic QA tool was for UtiliCorp and the conversion vendor to agree on use of the error codes flagged at conversion startup for each operating company. This was not a trivial process, as the original version of the Missouri Public Service checker for electric objects contained 228 error codes for 18 objects. The gas object checker is somewhat simpler, with 41 error codes for ten objects. The electronic tool is fully configurable, so checks can be added, deleted, and/or modified as data from the different companies is examined or as new conversion problem areas are identified.
In addition to the resolution of error codes for a company, there is the issue of incomplete or incorrect data from existing sources. To deal with source anomalies, we devised a discrepancy object. The conversion vendor can place this object near the object with problematic information, link the discrepancy object to the problematic object, and make detailed notes on the data issues with this object. One function of the discrepancy object is to cause the object to which it is linked to be removed from the statistical calculations of the QA tool. A typical scenario would be for discrepancy objects to be non-issues for delivery of converted data and for the discrepancies to be resolved by UtiliCorp staff once the data is in production.
Another addition to the basic QA process is the mutual use of the QA tool by UtiliCorp and the conversion vendor to clear data prior to delivery. This mutual use and discussion of electronic QA results allows the conversion vendor to deliver data that has already passed electronic QA (that is, errors are fixed until the tool shows 100% connectivity). This prescreening of data prior to delivery helps speed the acceptance process and reduces the QA time spent by UtiliCorp staff and the manual QA vendor.
It has been mentioned previously that the manual QA vendor has network access to FAME. Although the original and primary purpose for this connection is to permit in-house plotting for use by proofreaders, we anticipate innovation from this vendor in developing a more full utilization of their connectivity. The starting point is for the manual QA staff to have immediate access to the common data store, with one obvious extension being the capability to do a basic sanity check on-line prior to plotting.
Statistics and Reports
One of the more useful capabilities of the electronic QA tool is its ability to work with several statistical models and to generate reports. The ANSI Standard and Inferential Statistics methods are used to determine the sample size to select objects for object attribute check. For ANSI Standard, the sample size is determined by either lot or batch size (number of objects to be selected), inspection level and inspection plan through ANSI Standard sampling tables.
An inferential statistics formula is used to calculate the sample size by presetting the confidence factor z, pass factor p, and error factor e so the sample size = (z ** 2) * p * (1 -p) / (e ** 2).
The tool’s analysis process generates data that can be viewed interactively on-screen or printed as a set of reports. Because the conversion vendor and UtiliCorp can view the same Smallworld data store, a QA run made by either party can be saved as a QA case in a suitable alternative and viewed by both parties. This reduces run time for large data sets and facilitates communication concerning errors in converted data.
Report data is compiled in ASCII files, which can be exported to Microsoft Excel files for firther manipulation. The QA tool generates four types of reports for each test case: summary, attribute errors, connectivity errors and overlapping errors. The summary report provides number of objects sampled, number of objects passing and failing for connectivity and attributes, total number of objects in data set, accuracy percentages, mean, standard deviation, variance and ranges. All other reports show object name, system ID and status for each object examined. System ID can be used to locate flagged objects in cases where the QA tool’s interactive screen is not used.
The manual QA vendor participates in statistical reporting by populating a spreadsheet with error results by object for each grid proofread. Total percent error as well as percent error by object can then be calculated for each sampling set.
Problems and Solutions
The most glaring problem with a shared network environment is that problems will be pushed out to all users. If the production database is down, everyone is down. No electronic QA can take place. End users can’t edit (or even view). The manual QA vendor can’t plot. Central problems can bring all activity to a screeching halt. This makes planning for work flow and scheduled deliverables extremely difficult. The scheduling advantages to be gained from parallel work activities can be severely adjusted by system outages.
The difference in running the QA tool in a development environment and a production environment has been a source of fi-ustration for the conversion vendor. Each new type of converted data brings data quality challenges and prompts UtiliCorp to work with configuration of the tool to solve problems associated with that data set. The tool is readily configured and tested on the UtiliCorp side, but the necessary time lag from development to test to production code results in the conversion vendor not being able to run a recently revised QA tool at their site. One obvious work-around is to have the conversion vendor use UtiliCorp runs of the tool, which is expedient but not totally satisfactory.
The electronic QA tool can be easily configured to check attributes for objects based on if/then conditions. However, source maps may not provide the attributes expected by the data model. In many cases it is possible to provide rules for populating fields with default values, but there can be problems with defaults. Gas staff tends to believe that default values, such as pressure, should not be inserted into the database due to liability and safety concerns. Thus, the tool has to date been used more effectively for connectivity issues than for attribute checking.
The manual QA vendor is easily capable of performing basic proofreading finctions for utility maps and providing statistics on the results. There is desire on the part of UtiliCorp and the vendor to more filly utilize the skills of this small business which specializes in CAD and AM/FM work for utilities. Providing network connectivity and a FAME workstation for their site has been an important first step in optimizing their QA efforts. Their staff is drawn from local vocational technology schools and is highly motivated to develop new AM/FM skills. It is anticipated that continued collaboration on the manual/electronic QA process will provide innovative solutions.
The QA process has been a necessary but somewhat peculiar introduction to FAME for the end users. Initially, extensive (four-day) training was provided to end users prior to the delivery of their first converted work packets. The end users were then asked to assist in spot checking the data and, in particular, to provide feedback about the quality of the converted data for their use. Early end users have worked extensively with FAME staff to develop standards for spot checking and to provide feedback concerning the process and the data. Spot checking has been a source of great interest and confusion for the end users. They are very aware that they are the ultimate owners of the data and want to make sure that everything is correct. However, their time is limited (the first deliveries arrived during the height of the short construction season in northern Minnesota) and so is their experience with FAME functionality.
In some early cases, providing the network and hardware rollout to the end users has been challenging, resulting in such circumstances as northern Minnesota staff with no network connection driving three hours to southern Minnesota to view data. Furthermore, the QA process does not much resemble the work flow of an end user using the developed applications to design a job. End user spot checkers have commented that the fill FAME training program is premature at the QA stage and that they forget how to do design work by the time production data is available to them.
Conclusions and Recommendations
UtiliCorp United is a Kansas City, MO-based company providing gas and electric utility services to 1.1 million customers in eight states (Missouri, Kansas, Colorado, Nebraska, Iowa, Minnesota, Michigan, and West Virginia), British Columbia, New Zealand and Australia. UtiliCorp is currently engaged in an aggressive conversion project, taking paper and electronic maps of holdings in the eight states to a common electronic format. This conversion is part of a larger reengineering effort, which moves the primary mapping fimctions from centralized drafting personnel to decentralized construction coordinators in field offices. Applications development for input of designed work is proceeding in parallel with data conversion. The resulting AM/FM/GIS of UtiliCorp is evolving as FAME (Facilities and Mapping Enabler).
Utilities under conversion are: Kansas Public Service, Michigan Gas Utilities, Missouri Public Service, Northern Minnesota Utilities, Peoples Natural Gas, West-Plains Energy and West Virginia Power. Map formats are diverse, from manual sources on paper, Mylar and linen to various electronic implementations of AutoCAD, CableCAD, Gentry, and ProScan. Age of maps ranges from Kansas Public Service linen maps dating back to 1906 to West-Plains Gentry maps being created from multiple sources just in time for the conversion. The historic mapping standards of the seven companies were predictably diverse, with an especially troublesome variety of information mapped and considered essential.
End users participated in data modeling sessions and other decision making meetings through the summer of 1997, during which common gas and electric data models were forged. The contract with a conversion vendor was signed in September, 1997, and we proceeded with gas and electric proof of concept conversion throughout the following winter. The production conversion schedule is aggressive, with all maps to be converted in 1½ years, resulting in a complete hardware, software, network and data rollout to 114 initial FAME end users by the end of 1999.
Electronic QA and Conversion Issues
The combination of an aggressive conversion schedule and the accuracy requirements of the conversion contract (100% connectivity and 98% attributes) created a challenge in QA of converted data. UtiliCorp’s intent is to be able to receive converted data as rapidly as possible and then rigorously examine the quality of this data. It became necessary to develop efficient and flexible QA procedures to detect the types of errors encountered in each phase of the conversion.
The electronic QA tool became the first line of defense in the examination of 63 conversion delivery packets. The electronic QA tool was originally developed by UtiliCorp’s application vendor and was configured by the vendor and UtiliCorp for use on this conversion project. The primary capabilities of the tool are to verify connectivity, object placement, and object attribute values. Toggle settings allow the user to query on all objects or on a partial subset of objects. The size and composition of the sample data set is also user defined. The sample can consist of the data representing the facilities in a defined geographical area, in an entire network, or in the total GIS. The tool also can generate a random sampling to check attributes for a subset of a large number of facilities.
The electronic tool can be configured to make connectivity checks based on UtiliCorp conversion rules. It is also possible to configure the tool to make attribute checks based on UtiliCorp conversion rules. The configuration can be readily altered based on changes in rules from company to company or on evolving standards. The tool provides interactive capabilities— it provides GoTo functionality on flagged errors as well as the opportunity to examine the attributes of a flagged object while within the tool.
The electronic tool is very good at checking facilities information-phasing compatibility, pipe termination, correct number of connections, annotation placement with respect to its object. However, it can’t be expected to do much for certain proofreading needs, especially with regard to landbase features, such as correct road names, presence or absence of features, or non-rulebased attribute checks. Thus it became necessary to develop a scheme for conducting manual QA.
UtiliCorp was fortunate in acquiring the services of a Kansas City minority small business engaged in AM/FM work for utilities. A methodology was developed for taking a random sample of the 1:100 scale map grids, plotting the grids, proofreading paper copies of converted data against paper originals, and generating statistics on each map grid. The statistical methodology is based on UtiliCorp’s U.S.-wide grid system and FAME’s capability to count total objects by type in each grid. Sample sizes are determined using a single sampling method based on Dodge and Romig’s Sanmling Irumection Tables.
This manual proofreading effort was initially painful in that it required tons of paper to move through our office space. The solution was to provide our manual QA vendor with network connectivity and a FAME workstation to facilitate plotting in-house. Due primarily to this connectivity, the manual QA vendor and UtiliCorp are able to develop innovative techniques to streamline and automate the manual QA process.
UtiliCorp QA Plan
UtiliCorp’s QA plan for the conversion requires the participation of FAME end users in final acceptance or rejection of the data. This participation is critical to the success of the project, as the end users know the most about their data and will be responsible for upkeep of the maps once conversion is complete. However, participation by end users at the beginning of the FAME rollout was challenging on all fronts – from training to network connectivity to system administration.
The FAME project initial rollout has 114 end users at sites in 48 towns in eight states. The first site requiring network connectivity for viewing converted data was in northern Minnesota and suffered delays because the telecom service provider couldn’t place lines until ground thawed in the spring. Once lines were laid, it was full-tilt construction season and the Construction Coordinator had no time for map proofreading. It is anticipated that end users at the middle or end of the rollout will not be exposed to so much uncertainty in acquisition of their hardware or network lines. However, there will probably continue to be the scheduling problem that all Construction Coordinators want their data to be delivered in the dead of winter.
Prior to the end user rollout, FAME had established network connectivity between FAME offices in Kansas City, the application development vendor in Austin, the data server in Omaha and the conversion vendor in Indianapolis. An early connection was made to the application developer, to provide remote development capability. As a result, UtiliCorp staff developed technical expertise in the desired network configuration. The process of the subsequent connections benefited from the early knowledge and UtiliCorp was able to provide full technical support for the connections. We found it substantially less confksing to take charge of all network issues, rather than sharing these issues with the connecting side staff.
The UtiliCorp QA plan outlines the acceptance process for converted data. Data must pass electronic QA, manual QA, end user spot checks, FAME staff spot checks and move to acceptance by the end users and the UtiliCorp Conversion Coordinator. The overall goals of the QA plan are: detect systematic conversion errors early, identify all possible problems with the conversion and expedite acceptance of conversion packets. The general QA flow of a delivered work packet is from electronic check to manual check, with spot checks occurring throughout the work flow.
A crucial first step in working with the electronic QA tool was for UtiliCorp and the conversion vendor to agree on use of the error codes flagged at conversion startup for each operating company. This was not a trivial process, as the original version of the Missouri Public Service checker for electric objects contained 228 error codes for 18 objects. The gas object checker is somewhat simpler, with 41 error codes for ten objects. The electronic tool is fully configurable, so checks can be added, deleted, and/or modified as data from the different companies is examined or as new conversion problem areas are identified.
In addition to the resolution of error codes for a company, there is the issue of incomplete or incorrect data from existing sources. To deal with source anomalies, we devised a discrepancy object. The conversion vendor can place this object near the object with problematic information, link the discrepancy object to the problematic object, and make detailed notes on the data issues with this object. One function of the discrepancy object is to cause the object to which it is linked to be removed from the statistical calculations of the QA tool. A typical scenario would be for discrepancy objects to be non-issues for delivery of converted data and for the discrepancies to be resolved by UtiliCorp staff once the data is in production.
Another addition to the basic QA process is the mutual use of the QA tool by UtiliCorp and the conversion vendor to clear data prior to delivery. This mutual use and discussion of electronic QA results allows the conversion vendor to deliver data that has already passed electronic QA (that is, errors are fixed until the tool shows 100% connectivity). This prescreening of data prior to delivery helps speed the acceptance process and reduces the QA time spent by UtiliCorp staff and the manual QA vendor.
It has been mentioned previously that the manual QA vendor has network access to FAME. Although the original and primary purpose for this connection is to permit in-house plotting for use by proofreaders, we anticipate innovation from this vendor in developing a more full utilization of their connectivity. The starting point is for the manual QA staff to have immediate access to the common data store, with one obvious extension being the capability to do a basic sanity check on-line prior to plotting.
Statistics and Reports
One of the more useful capabilities of the electronic QA tool is its ability to work with several statistical models and to generate reports. The ANSI Standard and Inferential Statistics methods are used to determine the sample size to select objects for object attribute check. For ANSI Standard, the sample size is determined by either lot or batch size (number of objects to be selected), inspection level and inspection plan through ANSI Standard sampling tables.
An inferential statistics formula is used to calculate the sample size by presetting the confidence factor z, pass factor p, and error factor e so the sample size = (z ** 2) * p * (1 -p) / (e ** 2).
The tool’s analysis process generates data that can be viewed interactively on-screen or printed as a set of reports. Because the conversion vendor and UtiliCorp can view the same Smallworld data store, a QA run made by either party can be saved as a QA case in a suitable alternative and viewed by both parties. This reduces run time for large data sets and facilitates communication concerning errors in converted data.
Report data is compiled in ASCII files, which can be exported to Microsoft Excel files for firther manipulation. The QA tool generates four types of reports for each test case: summary, attribute errors, connectivity errors and overlapping errors. The summary report provides number of objects sampled, number of objects passing and failing for connectivity and attributes, total number of objects in data set, accuracy percentages, mean, standard deviation, variance and ranges. All other reports show object name, system ID and status for each object examined. System ID can be used to locate flagged objects in cases where the QA tool’s interactive screen is not used.
The manual QA vendor participates in statistical reporting by populating a spreadsheet with error results by object for each grid proofread. Total percent error as well as percent error by object can then be calculated for each sampling set.
Problems and Solutions
The most glaring problem with a shared network environment is that problems will be pushed out to all users. If the production database is down, everyone is down. No electronic QA can take place. End users can’t edit (or even view). The manual QA vendor can’t plot. Central problems can bring all activity to a screeching halt. This makes planning for work flow and scheduled deliverables extremely difficult. The scheduling advantages to be gained from parallel work activities can be severely adjusted by system outages.
The difference in running the QA tool in a development environment and a production environment has been a source of fi-ustration for the conversion vendor. Each new type of converted data brings data quality challenges and prompts UtiliCorp to work with configuration of the tool to solve problems associated with that data set. The tool is readily configured and tested on the UtiliCorp side, but the necessary time lag from development to test to production code results in the conversion vendor not being able to run a recently revised QA tool at their site. One obvious work-around is to have the conversion vendor use UtiliCorp runs of the tool, which is expedient but not totally satisfactory.
The electronic QA tool can be easily configured to check attributes for objects based on if/then conditions. However, source maps may not provide the attributes expected by the data model. In many cases it is possible to provide rules for populating fields with default values, but there can be problems with defaults. Gas staff tends to believe that default values, such as pressure, should not be inserted into the database due to liability and safety concerns. Thus, the tool has to date been used more effectively for connectivity issues than for attribute checking.
The manual QA vendor is easily capable of performing basic proofreading finctions for utility maps and providing statistics on the results. There is desire on the part of UtiliCorp and the vendor to more filly utilize the skills of this small business which specializes in CAD and AM/FM work for utilities. Providing network connectivity and a FAME workstation for their site has been an important first step in optimizing their QA efforts. Their staff is drawn from local vocational technology schools and is highly motivated to develop new AM/FM skills. It is anticipated that continued collaboration on the manual/electronic QA process will provide innovative solutions.
The QA process has been a necessary but somewhat peculiar introduction to FAME for the end users. Initially, extensive (four-day) training was provided to end users prior to the delivery of their first converted work packets. The end users were then asked to assist in spot checking the data and, in particular, to provide feedback about the quality of the converted data for their use. Early end users have worked extensively with FAME staff to develop standards for spot checking and to provide feedback concerning the process and the data. Spot checking has been a source of great interest and confusion for the end users. They are very aware that they are the ultimate owners of the data and want to make sure that everything is correct. However, their time is limited (the first deliveries arrived during the height of the short construction season in northern Minnesota) and so is their experience with FAME functionality.
In some early cases, providing the network and hardware rollout to the end users has been challenging, resulting in such circumstances as northern Minnesota staff with no network connection driving three hours to southern Minnesota to view data. Furthermore, the QA process does not much resemble the work flow of an end user using the developed applications to design a job. End user spot checkers have commented that the fill FAME training program is premature at the QA stage and that they forget how to do design work by the time production data is available to them.
Conclusions and Recommendations
- An electronic QA tool provides the most direct path to determining 100% connectivity and 98% correct attributes.
- Manual QA is still necessary; how else to determine completeness or non-rulebased attributes?
- Simultaneous viewing in FAME implemented via fast network connections expedites the conversion process and adds clarity to discussions between client and vendor.
- End user input during QA is critical; they’re the best resource for a sanity check.
- Keep paper maps out of your office whenever possible.