| GISdevelopment.net ---> GITA 2000 ---> Exploiting Field and Mobile Technologies |
Data Collection - Merging old methods with new technology
Alan Johnson, P.E.
Project Manager, Associate
Wade & Associates, Inc.
Alan Johnson, P.E.
Project Manager, Associate
Wade & Associates, Inc.
Introduction
Underground utilities infrastructure is deteriorating. Utilities that are underground and out of sight face increasing maintenance and repair problems. Waterline and gas line inspections, storm sewer inspections, and Sanitary Sewer Evaluation Studies (SSES) are necessary and important parts of maintaining and improving underground utilities. The nation's population is becoming more environmentally conscious and environmental regulators have seen increased pressure to implement tougher regulations and enforcement policies on utilities to prevent leakage and contamination. Most regulatory agencies encourage and in some cases require cities to implement SSES as a first step in reducing sanitary sewer overflows and bypasses. Potable water systems are facing stringent regulations to maintain water purity. Utilities that deliver gas and other petroleum products are interested in and required to protect their infrastructure to avoid dangerous and costly breaks in pipelines. Communications providers are concerned about the condition and protection of underground cables. In order to properly diagnose utility problems, data must be collected on structure locations, material types, and the conditions of all components.
Management of the large amount of data collected while inspecting the utility infrastructure during field activities can become overwhelming and generally requires automated tracking methods. Data collected may include review of manned entry structures, smoke and dyed-water testing, CCTV (Closed Circuit Television) inspections, flow monitoring, quality testing, and building inspections. A review and audit of maintenance and repair records is also important for evaluating overall system condition and for use as a basis for further field investigations.
Our goal is to identify the methods of data collection and storage that will give us the quickest and most accurate access to the data and results of analyzing the data while protecting the original source documents and preserving a coherent trail for review and audit.
Traditional Data Collection Methods
There are traditional, tried and proven, methods of data collection. There are cutting edge technology data collection methods. In general, the same information is collected and each method has advantages and disadvantages.
Traditional inspection methods preceded our current age of technology. Physical inspections of underground utilities have been conducted with manned entry being a mainstay of the operations. Inspectors physically enter utility structures, when possible, to visually inspect conditions.
Data, comments, and sketches have been recorded on paper forms. Field technicians collect instant photos for location and defect identification, and record time, date, location, defect and structure number on each photo. Field forms and photos must be filed in folders or notebooks for storage and reference. Utility atlas maps must be created or updated for facility locations. All this information must be entered into computerized databases for review and evaluation.
Monitoring flow in conduits is another area where accepted traditional methods may be supplanted with newer technologies. Open channel flows, such as gravity sanitary sewers or storm sewers, and pressure system flows, such as potable water systems, gas mains and sanitary sewer force mains, have often been recorded using continuous paper strip charts, drum charts or circular charts. Analysis of the flows recorded on paper charts requires that operators manually pick data from the charts and tabulate the information. There are many problems associated with using paper chart recorders including loss of data due to ink drying out, data being overwritten if charts aren't replaced at proper intervals, and fading of ink over time.
Data collection and utility system evaluation must include review and confirmation of existing data including utility atlas maps, as-built and record drawings, and field surveys. Surveys are time consuming and expensive, so surveyor activities are normally conducted to rectify datum differences and to verify key data. In order to utilize modern mapping tools, coordinates must be determined for utility structures, labels or structure identifiers must be linked, and spatial relationships must be determined for all entities.
Leading Edge Data Collection Methods
A concern for inspector's safety, increased productivity, inspections where people simply can not go, and reducing the overall cost of information collection has driven the industry to develop more modern inspection methods. Technological advances in microchips and miniaturization, improvements in computers, digital electronic devices and telecommunications, and cost- reductions for electronic devices has changed data collection. Modem inspection and data collection methods includes a wide range of devices and systems.
Digital or VHS TV cameras and SONAR devices are increasingly being used for inspecting structures and conduits. A video tape of a sewer pipe or water line can give invaluable information concerning pipeline alignment and condition.
Digital cameras can be used for capturing still photos of surface conditions and topography, internal conditions and various defects. Digital cameras have improved dramatically recently, and are easy to use and provide high quality computerized images.
Advances are being made for automating or at least streamlining data entry by using specially developed forms and scanning or faxing each form that is subsequently read by and evaluated by companion software. Data taken from the forms can be automatically entered into the databases.
Laptop computers or other handheld devices can be used to enter data directly into the analysis software. These devices can be used in the field which eliminates data entry from paper forms. Laptops and handheld dataloggers can provide direct links from the electronic field forms to evaluation software.
Surveys can be conducted using GPS (Global Positioning System) to obtain location coordinates for each structure. GPS has improved immensely and can be accurate enough for valuable location information and even for design surveys. It is easier to collect more data using electronic tools; more data collected coupled with computerized analytic tools will yield better, more accurate and comprehensive results.
Flow monitoring can be automated by replacing mechanical recorders with electronic devices. Results will be more easily added to databases. Various sensors have been developed to detect mass flow, some are intrusive, being mounted in the flow, others are mounted outside of the stream of flow. Pressure transducers that measure velocity and depth are generally installed at the bottom of a pipe to record gravity flows, while ultra-sonic sensors are mounted above the stream and measure depth. Dopier sensors measure flow in pressurized systems and provide data on mass flow. All of these sensors are attached to data loggers that electronically record measurements at predetermined intervals. The data loggers are interrogated periodically using laptop computers to get the data into evaluation software. Data loggers and office based computers can also be linked by telemetry for real-time, automatic data transfer.
Advantages and Disadvantages
There are several compelling advantages to using paper inspection forms. Retaining field forms assures having the original source document for future reference. This will be helpful in verifying data and assuring reproducibility of analysis results. Inspectors and field technicians are more apt to record comments and draw sketches of field conditions and comment on problems. Sketches may add very important detail for evaluating system conditions. For instance, a field form may identify a crack in a manhole structure as a defect that might not make a rehabilitation schedule for repair. A well drawn sketch of the crack may lead the engineer to understand that the crack is diagonal and is separating and, therefore, a defect that must be addressed.
On the other hand, there are significant disadvantages to using paper inspection forms. First of all, paper forms are bulky and may take up dozens, if not hundreds, of large notebooks or files to hold and organize the documents. Paper forms are subject to being lost, misplaced or inappropriately filed and can be rendered unreadable if handled with dirty hands. In order to report on inspection results, the data from the forms must be summarized by hand or entered into computerized databases. This effort adds a significant labor cost to the project and may increase the chance of mistyping information or overlooking important data. Retrieval of information from paper forms can be difficult and time consuming, and summarizing results or plotting trends is cumbersome at best.
Photos are important for documenting structure locations and important conditions. Using instant film pictures allows the inspector to annotate on the border of the picture as it develops. The pictures can be filed in notebooks with field forms for visual review and confirmation. Instant film photos are expensive, however, costing about $1.00 per picture. On a large project, tens of thousands of dollars may be spent on instant film. Instant photos may fade over time if not protected from sunlight. Digital images won't fade and will last forever, essentially. The only problem is in annotating the pictures; the digital cameras we have tried provide no means of identifying the location or structure so a photo log must be maintained. Pictures can and will be logged incorrectly and images misidentified. After the initial investment for camera and media, pictures cost almost nothing to collect, develop and retain. Printing digital images requires a color inkjet or laser printer.
Collecting data electronically dramatically reduces the time and effort of getting field data into the computer for evaluation and using cameras to inspect structures eliminates the need for inspectors to enter the hazardous environment. Reporting and data summarizing is streamlined and easily customized using common computer applications and custom applications developed specifically for utility problem solving. On the other hand, inspectors are less inclined to write important observations or draw sketches of problems on electronic devices.
Mapping
An important aspect of utility inspection and evaluation is developing and updating location maps. Most utilities have atlas maps of their underground facilities. Many continue to use hand or CAD drafted maps that have served over the years. These atlases are used to locate and identify facilities and structures. Older maps, though technologically outdated, represent an important historic record of utility change over time. These maps are still used to identify unmarked pipes and lines and match utility owners with their facilities.
With the development of powerful yet relatively inexpensive GIS (Geographical Information Systems), more and more utilities and municipalities are moving utility maps to electronic GIS maps. GIS that is available to everyone is a very important development because the crucial information contained on maps is more available for use by locating services, underground utility contractors, engineers, and surveyors. GIS maps are easier to maintain and update for system changes allowing the facility owner to improve map accuracy over time.
What works for us
Our firm specializes in sanitary sewer studies. We provide extensive field inspection services. We have tried most every method available to collect, record, store and evaluate data. Our experience leads us to believe that there are appropriate technologies for inspecting underground utilities and there are inappropriate ones. I will show how we have merged the traditional data collection methods with appropriate new technologies to arrive at an expedient, accurate and safeguarded method of gathering field data. Our industry is not static; we will continue to evaluate technology to assure data we collect and use is the best it can be.
First of all, we enter manhole structures. In spite of the inherent danger associated with confined space entry, we have found that the best information is collected most reliably by having trained inspectors enter sewer facilities. We use only the best confined space entry equipment, keep current and accurate entry records, follow OSHA regulations and guidelines, and have a comprehensive and ongoing safety training program for all our field people. We strongly believe that a well trained inspector can gather the best information when seeing the structure live, in person. We have tried various methods of inspecting manholes that eliminate manned entry, including using video cameras attached to telescopic poles, digital cameras on poles, and top-side visual inspection. For structure evaluation, all of these remote viewing methods have problems, most of which are negated when an inspector enters and physically inspects the structure.
We use paper field inspection forms. Several years ago we developed a handheld data logger application. The application was easy to use and closely followed our field forms to minimize training requirements. The devices we used could download data directly to desktop computers to merge data with existing databases. Laptop computers have also been used to log data in the field. We abandoned the electronic field forms for several reasons. Some field technicians are intimidated by new devices; a problem easily overcome with adequate training. The biggest problem, however, was that it is difficult to sketch on a handheld device or laptop, so diagrams of defects or locations were not drawn. Special comments or observations weren't written down because of space limitations or time constraints.
We use instant film for surface condition photos and digital images for logging many internal defects. The ability to note the location and structure ID on each instant film picture is too important to ignore. Digital cameras work well in locations and environments where it is impossible for men to enter.
We use digital CCTV or digitize VHS tapes of pipes. The analytic software we developed for infrastructure evaluation has the capability of displaying streaming digital images or still images. Images are burned onto CD-ROM and are filed with other project documentation, available to users needing to review images.
We enter data in computers from field forms. We do not use fax or scanned image interpretation software. The accuracy of form interrogation software is too poor to be reliable for data conversion. We have found that these systems will have as much as 20% errors per page, depending, of course, on the care with which the original form was filled out. Trained and experienced data entry personnel can enter as many as 20 forms per hour with a very high rate of accuracy. We have staff engineers and senior technicians review every form for completeness and accuracy on a daily basis. Data entry is checked regularly.
We store field forms in notebooks. Since we use paper forms, project notebooks have become the source document storage means of choice. Notebooks are easy to label and identify. It is convenient to dedicate shelving for easy access. Documents for older projects that have been completed can be kept in the notebooks or scanned and the images archived on CD-ROM or micro-fiche.
We use electronic flow monitors and interrogate the devices with laptop computers. Paper data recording charts are too unreliable and too hard to extract data from. Data collected from electronic data loggers is easily downloaded to software such as spreadsheets or other analytic programs. Using telemetry is a viable option for permanent recorders, but not for temporary installations.
We use ArcView GIS for displaying data. Maintaining paper, hand drafted maps and atlases is too time consuming and expensive. We convert all maps to CAD for drafting and maintenance, and use ArcView for data display and results interpretation. ArcView is inexpensive, relatively powerful, compatible with many database management file formats, and is commonly used by municipalities around the country.
Conclusion
Our goal has been to identify methods of data collection and storage that will provide the quick, accurate access to data and results of analyzing the data while protecting the original source documents and preserving a coherent trail for review and audit. Our approach has been to combine accepted traditional inspection methods with leading edge technological solutions.
Underground utilities infrastructure is deteriorating. Utilities that are underground and out of sight face increasing maintenance and repair problems. Waterline and gas line inspections, storm sewer inspections, and Sanitary Sewer Evaluation Studies (SSES) are necessary and important parts of maintaining and improving underground utilities. The nation's population is becoming more environmentally conscious and environmental regulators have seen increased pressure to implement tougher regulations and enforcement policies on utilities to prevent leakage and contamination. Most regulatory agencies encourage and in some cases require cities to implement SSES as a first step in reducing sanitary sewer overflows and bypasses. Potable water systems are facing stringent regulations to maintain water purity. Utilities that deliver gas and other petroleum products are interested in and required to protect their infrastructure to avoid dangerous and costly breaks in pipelines. Communications providers are concerned about the condition and protection of underground cables. In order to properly diagnose utility problems, data must be collected on structure locations, material types, and the conditions of all components.
Management of the large amount of data collected while inspecting the utility infrastructure during field activities can become overwhelming and generally requires automated tracking methods. Data collected may include review of manned entry structures, smoke and dyed-water testing, CCTV (Closed Circuit Television) inspections, flow monitoring, quality testing, and building inspections. A review and audit of maintenance and repair records is also important for evaluating overall system condition and for use as a basis for further field investigations.
Our goal is to identify the methods of data collection and storage that will give us the quickest and most accurate access to the data and results of analyzing the data while protecting the original source documents and preserving a coherent trail for review and audit.
Traditional Data Collection Methods
There are traditional, tried and proven, methods of data collection. There are cutting edge technology data collection methods. In general, the same information is collected and each method has advantages and disadvantages.
Traditional inspection methods preceded our current age of technology. Physical inspections of underground utilities have been conducted with manned entry being a mainstay of the operations. Inspectors physically enter utility structures, when possible, to visually inspect conditions.
Data, comments, and sketches have been recorded on paper forms. Field technicians collect instant photos for location and defect identification, and record time, date, location, defect and structure number on each photo. Field forms and photos must be filed in folders or notebooks for storage and reference. Utility atlas maps must be created or updated for facility locations. All this information must be entered into computerized databases for review and evaluation.
Monitoring flow in conduits is another area where accepted traditional methods may be supplanted with newer technologies. Open channel flows, such as gravity sanitary sewers or storm sewers, and pressure system flows, such as potable water systems, gas mains and sanitary sewer force mains, have often been recorded using continuous paper strip charts, drum charts or circular charts. Analysis of the flows recorded on paper charts requires that operators manually pick data from the charts and tabulate the information. There are many problems associated with using paper chart recorders including loss of data due to ink drying out, data being overwritten if charts aren't replaced at proper intervals, and fading of ink over time.
Data collection and utility system evaluation must include review and confirmation of existing data including utility atlas maps, as-built and record drawings, and field surveys. Surveys are time consuming and expensive, so surveyor activities are normally conducted to rectify datum differences and to verify key data. In order to utilize modern mapping tools, coordinates must be determined for utility structures, labels or structure identifiers must be linked, and spatial relationships must be determined for all entities.
Leading Edge Data Collection Methods
A concern for inspector's safety, increased productivity, inspections where people simply can not go, and reducing the overall cost of information collection has driven the industry to develop more modern inspection methods. Technological advances in microchips and miniaturization, improvements in computers, digital electronic devices and telecommunications, and cost- reductions for electronic devices has changed data collection. Modem inspection and data collection methods includes a wide range of devices and systems.
Digital or VHS TV cameras and SONAR devices are increasingly being used for inspecting structures and conduits. A video tape of a sewer pipe or water line can give invaluable information concerning pipeline alignment and condition.
Digital cameras can be used for capturing still photos of surface conditions and topography, internal conditions and various defects. Digital cameras have improved dramatically recently, and are easy to use and provide high quality computerized images.
Advances are being made for automating or at least streamlining data entry by using specially developed forms and scanning or faxing each form that is subsequently read by and evaluated by companion software. Data taken from the forms can be automatically entered into the databases.
Laptop computers or other handheld devices can be used to enter data directly into the analysis software. These devices can be used in the field which eliminates data entry from paper forms. Laptops and handheld dataloggers can provide direct links from the electronic field forms to evaluation software.
Surveys can be conducted using GPS (Global Positioning System) to obtain location coordinates for each structure. GPS has improved immensely and can be accurate enough for valuable location information and even for design surveys. It is easier to collect more data using electronic tools; more data collected coupled with computerized analytic tools will yield better, more accurate and comprehensive results.
Flow monitoring can be automated by replacing mechanical recorders with electronic devices. Results will be more easily added to databases. Various sensors have been developed to detect mass flow, some are intrusive, being mounted in the flow, others are mounted outside of the stream of flow. Pressure transducers that measure velocity and depth are generally installed at the bottom of a pipe to record gravity flows, while ultra-sonic sensors are mounted above the stream and measure depth. Dopier sensors measure flow in pressurized systems and provide data on mass flow. All of these sensors are attached to data loggers that electronically record measurements at predetermined intervals. The data loggers are interrogated periodically using laptop computers to get the data into evaluation software. Data loggers and office based computers can also be linked by telemetry for real-time, automatic data transfer.
Advantages and Disadvantages
There are several compelling advantages to using paper inspection forms. Retaining field forms assures having the original source document for future reference. This will be helpful in verifying data and assuring reproducibility of analysis results. Inspectors and field technicians are more apt to record comments and draw sketches of field conditions and comment on problems. Sketches may add very important detail for evaluating system conditions. For instance, a field form may identify a crack in a manhole structure as a defect that might not make a rehabilitation schedule for repair. A well drawn sketch of the crack may lead the engineer to understand that the crack is diagonal and is separating and, therefore, a defect that must be addressed.
On the other hand, there are significant disadvantages to using paper inspection forms. First of all, paper forms are bulky and may take up dozens, if not hundreds, of large notebooks or files to hold and organize the documents. Paper forms are subject to being lost, misplaced or inappropriately filed and can be rendered unreadable if handled with dirty hands. In order to report on inspection results, the data from the forms must be summarized by hand or entered into computerized databases. This effort adds a significant labor cost to the project and may increase the chance of mistyping information or overlooking important data. Retrieval of information from paper forms can be difficult and time consuming, and summarizing results or plotting trends is cumbersome at best.
Photos are important for documenting structure locations and important conditions. Using instant film pictures allows the inspector to annotate on the border of the picture as it develops. The pictures can be filed in notebooks with field forms for visual review and confirmation. Instant film photos are expensive, however, costing about $1.00 per picture. On a large project, tens of thousands of dollars may be spent on instant film. Instant photos may fade over time if not protected from sunlight. Digital images won't fade and will last forever, essentially. The only problem is in annotating the pictures; the digital cameras we have tried provide no means of identifying the location or structure so a photo log must be maintained. Pictures can and will be logged incorrectly and images misidentified. After the initial investment for camera and media, pictures cost almost nothing to collect, develop and retain. Printing digital images requires a color inkjet or laser printer.
Collecting data electronically dramatically reduces the time and effort of getting field data into the computer for evaluation and using cameras to inspect structures eliminates the need for inspectors to enter the hazardous environment. Reporting and data summarizing is streamlined and easily customized using common computer applications and custom applications developed specifically for utility problem solving. On the other hand, inspectors are less inclined to write important observations or draw sketches of problems on electronic devices.
Mapping
An important aspect of utility inspection and evaluation is developing and updating location maps. Most utilities have atlas maps of their underground facilities. Many continue to use hand or CAD drafted maps that have served over the years. These atlases are used to locate and identify facilities and structures. Older maps, though technologically outdated, represent an important historic record of utility change over time. These maps are still used to identify unmarked pipes and lines and match utility owners with their facilities.
With the development of powerful yet relatively inexpensive GIS (Geographical Information Systems), more and more utilities and municipalities are moving utility maps to electronic GIS maps. GIS that is available to everyone is a very important development because the crucial information contained on maps is more available for use by locating services, underground utility contractors, engineers, and surveyors. GIS maps are easier to maintain and update for system changes allowing the facility owner to improve map accuracy over time.
What works for us
Our firm specializes in sanitary sewer studies. We provide extensive field inspection services. We have tried most every method available to collect, record, store and evaluate data. Our experience leads us to believe that there are appropriate technologies for inspecting underground utilities and there are inappropriate ones. I will show how we have merged the traditional data collection methods with appropriate new technologies to arrive at an expedient, accurate and safeguarded method of gathering field data. Our industry is not static; we will continue to evaluate technology to assure data we collect and use is the best it can be.
First of all, we enter manhole structures. In spite of the inherent danger associated with confined space entry, we have found that the best information is collected most reliably by having trained inspectors enter sewer facilities. We use only the best confined space entry equipment, keep current and accurate entry records, follow OSHA regulations and guidelines, and have a comprehensive and ongoing safety training program for all our field people. We strongly believe that a well trained inspector can gather the best information when seeing the structure live, in person. We have tried various methods of inspecting manholes that eliminate manned entry, including using video cameras attached to telescopic poles, digital cameras on poles, and top-side visual inspection. For structure evaluation, all of these remote viewing methods have problems, most of which are negated when an inspector enters and physically inspects the structure.
We use paper field inspection forms. Several years ago we developed a handheld data logger application. The application was easy to use and closely followed our field forms to minimize training requirements. The devices we used could download data directly to desktop computers to merge data with existing databases. Laptop computers have also been used to log data in the field. We abandoned the electronic field forms for several reasons. Some field technicians are intimidated by new devices; a problem easily overcome with adequate training. The biggest problem, however, was that it is difficult to sketch on a handheld device or laptop, so diagrams of defects or locations were not drawn. Special comments or observations weren't written down because of space limitations or time constraints.
We use instant film for surface condition photos and digital images for logging many internal defects. The ability to note the location and structure ID on each instant film picture is too important to ignore. Digital cameras work well in locations and environments where it is impossible for men to enter.
We use digital CCTV or digitize VHS tapes of pipes. The analytic software we developed for infrastructure evaluation has the capability of displaying streaming digital images or still images. Images are burned onto CD-ROM and are filed with other project documentation, available to users needing to review images.
We enter data in computers from field forms. We do not use fax or scanned image interpretation software. The accuracy of form interrogation software is too poor to be reliable for data conversion. We have found that these systems will have as much as 20% errors per page, depending, of course, on the care with which the original form was filled out. Trained and experienced data entry personnel can enter as many as 20 forms per hour with a very high rate of accuracy. We have staff engineers and senior technicians review every form for completeness and accuracy on a daily basis. Data entry is checked regularly.
We store field forms in notebooks. Since we use paper forms, project notebooks have become the source document storage means of choice. Notebooks are easy to label and identify. It is convenient to dedicate shelving for easy access. Documents for older projects that have been completed can be kept in the notebooks or scanned and the images archived on CD-ROM or micro-fiche.
We use electronic flow monitors and interrogate the devices with laptop computers. Paper data recording charts are too unreliable and too hard to extract data from. Data collected from electronic data loggers is easily downloaded to software such as spreadsheets or other analytic programs. Using telemetry is a viable option for permanent recorders, but not for temporary installations.
We use ArcView GIS for displaying data. Maintaining paper, hand drafted maps and atlases is too time consuming and expensive. We convert all maps to CAD for drafting and maintenance, and use ArcView for data display and results interpretation. ArcView is inexpensive, relatively powerful, compatible with many database management file formats, and is commonly used by municipalities around the country.
Conclusion
Our goal has been to identify methods of data collection and storage that will provide the quick, accurate access to data and results of analyzing the data while protecting the original source documents and preserving a coherent trail for review and audit. Our approach has been to combine accepted traditional inspection methods with leading edge technological solutions.