Building a solid foundation

Observations from other upgrades
The cost and timeframe to upgrade can be significant when the total cost of the upgrade is considered, including the upgrade itself, consolidating and fixing data from various legacy systems, upgrading landbases and conflating facilities, testing, documentation, training, and organizationwide rollout.

The following list provides common data-related upgrade issues revealed by multiple utilities that have performed major upgrades in the past. These are issues to be considered when developing a data enhancement program.

Utilities almost always feel their data is better than it really is. The people maintaining GIS and facilities data often are too close to their existing system or are politically unable to be objective (it would be a bad career move to say their data isn’t good). Most feel their data is consistent, up-to-date, and accurate but have not researched and quantified whether this is true. In most if not all major upgrade cases, data problems were identified during the migration process.
Data quality is key to user acceptance. In many cases, if users cannot trust the quality of the data, they will not utilize the system to its fullest.
Too much effort is spent on map aesthetics (e.g., pretty curved street names, placement of unimportant land features) and not nearly enough time spent maintaining facilities network and attribute data.
Utilities often converted data initially using the philosophy of start cheap and upgrade the data as part of the ongoing work processes. This is particularly the case with landbases. This philosophy is good in concept, and it allows utilities to get started with GIS, but is very difficult to practice and often leads to some of the endemic data problems listed below.
Most upgrades involve consolidating facilities data from various legacy sources into the new GIS. These different systems often have common/duplicate data, but no processes or software tools have been developed to keep data synchronized. For example, many gas utilities will have leak databases, maintenance and inspection databases, critical valve databases, CP databases, etc., separate from the GIS with no way to tie the data together.
Data maintenance rules and processes, and data requirements change over time. These changes are often implemented without the historical data or databases being upgraded to make them consistent with the new ways of doing business.
Data maintenance rules and processes, and data requirements changes are inconsistent in different areas of the company.
Few automated QA/QC routines.
Too much focus on paper output.
Experts on legacy systems leave and no one at a company really knows the details of the system.
Poor or no documentation. No history of changes to data over time.
Lack of knowledge of the various legacy systems and the overlapping data needs between these systems.

These issues result in many of the following data problems:

Graphic and attribute data not matching
Missing records
Missing required fields
0 length linear features (e.g., gas mains)
Duplicate objects on top of each other
Data not following business rules (e.g., pipes connecting without a fitting or other device)
Lack of unique IDs
Duplicate IDs
Poor or lack of graphic edgematching (tile-based systems), offset data, linear facilities data (pipes, conductors breaking at map boundaries even though they are the same feature)
Overlap between maps resulting in duplicate features
Data inconsistency throughout the entire system. This often results from a lack of consistent data input and maintenance procedures and automated tools to check the data, differences in data depending on who performed the original conversion, and data maintenance standards changing over time and these changes not being applied throughout entire system
Using data fields for multiple purposes or for purposes not originally intended
Lack of use of enumerated values resulting in many different data entries for records that should be the same
Network connectivity handled differently by different systems
Data related by proximity (not snapped)
Substandard data conversion acceptance criteria
Data models allow users to make mistakes (e.g., dimensioning data stored in miscellaneous text fields, no pick lists provided, etc.)

Lessons Learned
Integrating processes and information, across the enterprise, is typically at the top of your corporate agenda. Piecemeal, poorly coordinated data integration efforts can cost a utility significantly in lower productivity and quality of services.

Upgrades will happen as long as you are in business. The software, hardware, databases, networks, applications, etc., will change at an ever-increasing rate, but the data supporting a utility’s business needs will last indefinitely. Therefore, the real underlying value of your GIS is in the data. Without accurate, consistent current data, applications are useless.

Evidence from past upgrades and the level of effort required to fix data during the upgrades points to the need and benefits for starting a well-thought-out data quality enhancement program now knowing that upgrades will happen — it’s only a matter of time. The effort put into this program now will lower the costs and timeframe to upgrade in the future. It will also have the additional benefits of improving user acceptance and improving data analysis results now.

Some simple, low cost steps that can be taken now include:

Know and document your data — what data exists, why is it required, etc.
Assess GIS data problems/issues.
Identify all facilities data sources and determine which sources have the best/most accurate data. Develop and maintain data source matrices.
Prioritize data cleanup tasks and lay out a plan to implement the data cleanup/enhancements.
Research how to extract data from current system and export into standard formats (ASCII, Oracle).
Develop program to synchronize data from different systems (IDs).
Evaluate existing third-party GIS data models and corresponding data migration tools.

More extensive, higher cost steps to take include:

Develop automated QA/QC routines that check data quality.
Understand what processes and data need to be integrated, and how to coordinate phased efforts across multiple business units, data architectures, and applications. In other words, develop an information integration strategy that binds individual integration projects which, left uncoordinated, could introduce even more problems into your information infrastructure.
Define processes/systems/data architectures to keep data consistent across the various legacy systems that contain similar data. Data architectures should eliminate or synchronize the redundancies typically found in legacy architectures. All systems should provide the same answers relative to common data.
Work with management to educate the need for continual data enhancement and cleanup. Try to educate based on examples in the industry where utilities felt a great deal of upgrade and migration pain before getting management’s attention, sponsorship, and funding. Identify and quantify data integration symptoms. Work with senior management to develop a plan of action and launch a comprehensive data integration initiative.

How you respond to integration demands will determine how effectively your company can compete in coming decades. As you contemplate this, consider that building, launching, and sustaining an integration strategy must be woven into the fabric of the enterprise. This is not a one-time project where the employees can pull one more miracle out of their collective hats. Integration of information, within and beyond the enterprise, must, by its very nature, involve multiple business units, executives, IT people, and a growing host of external entities.

Without a comprehensive integration strategy that can bind disparate entities within and outside the enterprise, companies and industries are likely to struggle with individual initiatives that could ultimately be working at cross-purposes. Taking ownership of this effort will demonstrate leadership, but will again be a major challenge. Project leaders from all corners of a given enterprise should band together to demand the requisite levels of leadership, sponsorship, funding, commitment, and participation from management and from any external entities with a stake in an integrated information enterprise.

Integration is not a buzzword, and it is not one more item on an IT task list that can be worked in when time permits. Integration requires a shift in thinking and a change in how various functions within and outside an organization interact. With so much at stake, all relevant and affected parties should step up to the table now to build a data enhancement strategy that can become the backbone for key initiatives going forward.