Building a solid foundation

Building a solid foundation

Adley Harms
Convergent Group, 6399 S. Fiddler’s Green Circle
Suite 600, Greenwood Village, CO 80111

As the old saying goes, nothing’s certain but death, taxes, and upgrades. Software upgrades — the process of migrating an existing system from one version of software to a new version — are inevitable in today’s information technology environment. In the GIS world, upgrades can be brought on by any number of reasons. Some of the most common reasons include:

Losing support for the current version of software
Conforming to a new company standard due to mergers and acquisitions
Having a vendor go out of business
Requiring support of new business initiatives
Upgrading to interfacing systems or underlying software versions (RDBMS, OS, etc.)

Over the past 15 to 20 years, most utilities have implemented a GIS of some sort. The functionality ranges from simple automated mapping functions with the main users being mapping technicians, to a GIS that is fully integrated with a company’s business processes and corporate information technologies with users from all areas of a company.

Needless to say, most utility’s GISs are closer in functionality to the former example than the latter. In the past 5 to 7 years, however, utilities have been increasingly involved in performing large-scale upgrades to their existing GISs to move rapidly toward an integrated business paradigm. In most cases, when major upgrades (e.g., changing GIS vendors, moving to new major software point release, changing data models, etc.) are performed, a utility takes the opportunity to rethink processes and various implementation issues and fix mistakes that they made the first time through. Many of these “If I were to do it all over again” mistakes often revolve around data. The objectives of this paper and subsequent presentation are to evaluate and build on the experience of those who have had a chance to ask the question, “If I were to do it again, I would . . . ” and

Learn what the common data-related mistakes have been.
Identify and prioritize areas for upgrading existing GIS data to support the strategic business needs of the future.
Identify strategies for moving forward — whether it is upgrading an existing GIS or implementing a GIS for the first time.

Startegic Directions
Before one can determine what types of data-related upgrades need to be implemented, two things should happen:

Define, at least at a high-level, the long-term vision for your GIS’s future — how do you envision using GIS in your corporate business processes?
Perform a facilities data quality assessment to understand what data exists and its quality.

In general, the following trends are occurring in the utility industry relative to GIS:

Trends in Utilities Affecting GIS (from left to right)
Weak business process/technology integration	Process centric . data and systems must support a rapidly changing business environment
Departmentalized, non-integrated systems or point-to-point integration	Cross-functional applications based on process models; GISs integrated with many other corporate systems including outage, work management, CIS, network analysis, etc.; increasing use of middleware technologies for integration
Extensive custom software	Configuration of commercial off-the-shelf products or configurable solutions, minimize customization
Islands of information	Consolidation and integration of multiple data sources, treating and managing data as key corporate asset
Limited access to operations data	Corporate wide access, intranet and Internet, still heavily dependent on back-office systems and data quality
GISs primarily mapping and facilities information management systems with access limited to a few people	Wide range of applications supporting all areas of utility with wide access throughout utility
Stand-alone planning functions and technologies	Integrated planning functions
Risk averse	Much more risk tolerant and seeking new business opportunities, business transformation-focused

The trends can be graphically shown as follows:

In general, GISs these days are considered the facility asset registry for utilities. The GISs are no longer being implemented as stand-alone systems supporting mapping and limited facilities management applications. They are becoming mission-critical systems that must integrate with a wide-range of other technologies/systems to support ever-changing business processes.

Observations from other upgrades
The cost and timeframe to upgrade can be significant when the total cost of the upgrade is considered, including the upgrade itself, consolidating and fixing data from various legacy systems, upgrading landbases and conflating facilities, testing, documentation, training, and organizationwide rollout.

The following list provides common data-related upgrade issues revealed by multiple utilities that have performed major upgrades in the past. These are issues to be considered when developing a data enhancement program.

Utilities almost always feel their data is better than it really is. The people maintaining GIS and facilities data often are too close to their existing system or are politically unable to be objective (it would be a bad career move to say their data isn’t good). Most feel their data is consistent, up-to-date, and accurate but have not researched and quantified whether this is true. In most if not all major upgrade cases, data problems were identified during the migration process.
Data quality is key to user acceptance. In many cases, if users cannot trust the quality of the data, they will not utilize the system to its fullest.
Too much effort is spent on map aesthetics (e.g., pretty curved street names, placement of unimportant land features) and not nearly enough time spent maintaining facilities network and attribute data.
Utilities often converted data initially using the philosophy of start cheap and upgrade the data as part of the ongoing work processes. This is particularly the case with landbases. This philosophy is good in concept, and it allows utilities to get started with GIS, but is very difficult to practice and often leads to some of the endemic data problems listed below.
Most upgrades involve consolidating facilities data from various legacy sources into the new GIS. These different systems often have common/duplicate data, but no processes or software tools have been developed to keep data synchronized. For example, many gas utilities will have leak databases, maintenance and inspection databases, critical valve databases, CP databases, etc., separate from the GIS with no way to tie the data together.
Data maintenance rules and processes, and data requirements change over time. These changes are often implemented without the historical data or databases being upgraded to make them consistent with the new ways of doing business.
Data maintenance rules and processes, and data requirements changes are inconsistent in different areas of the company.
Few automated QA/QC routines.
Too much focus on paper output.
Experts on legacy systems leave and no one at a company really knows the details of the system.
Poor or no documentation. No history of changes to data over time.
Lack of knowledge of the various legacy systems and the overlapping data needs between these systems.

These issues result in many of the following data problems:

Graphic and attribute data not matching
Missing records
Missing required fields
0 length linear features (e.g., gas mains)
Duplicate objects on top of each other
Data not following business rules (e.g., pipes connecting without a fitting or other device)
Lack of unique IDs
Duplicate IDs
Poor or lack of graphic edgematching (tile-based systems), offset data, linear facilities data (pipes, conductors breaking at map boundaries even though they are the same feature)
Overlap between maps resulting in duplicate features
Data inconsistency throughout the entire system. This often results from a lack of consistent data input and maintenance procedures and automated tools to check the data, differences in data depending on who performed the original conversion, and data maintenance standards changing over time and these changes not being applied throughout entire system
Using data fields for multiple purposes or for purposes not originally intended
Lack of use of enumerated values resulting in many different data entries for records that should be the same
Network connectivity handled differently by different systems
Data related by proximity (not snapped)
Substandard data conversion acceptance criteria
Data models allow users to make mistakes (e.g., dimensioning data stored in miscellaneous text fields, no pick lists provided, etc.)

Lessons Learned
Integrating processes and information, across the enterprise, is typically at the top of your corporate agenda. Piecemeal, poorly coordinated data integration efforts can cost a utility significantly in lower productivity and quality of services.

Upgrades will happen as long as you are in business. The software, hardware, databases, networks, applications, etc., will change at an ever-increasing rate, but the data supporting a utility’s business needs will last indefinitely. Therefore, the real underlying value of your GIS is in the data. Without accurate, consistent current data, applications are useless.

Evidence from past upgrades and the level of effort required to fix data during the upgrades points to the need and benefits for starting a well-thought-out data quality enhancement program now knowing that upgrades will happen — it’s only a matter of time. The effort put into this program now will lower the costs and timeframe to upgrade in the future. It will also have the additional benefits of improving user acceptance and improving data analysis results now.

Some simple, low cost steps that can be taken now include:

Know and document your data — what data exists, why is it required, etc.
Assess GIS data problems/issues.
Identify all facilities data sources and determine which sources have the best/most accurate data. Develop and maintain data source matrices.
Prioritize data cleanup tasks and lay out a plan to implement the data cleanup/enhancements.
Research how to extract data from current system and export into standard formats (ASCII, Oracle).
Develop program to synchronize data from different systems (IDs).
Evaluate existing third-party GIS data models and corresponding data migration tools.

More extensive, higher cost steps to take include:

Develop automated QA/QC routines that check data quality.
Understand what processes and data need to be integrated, and how to coordinate phased efforts across multiple business units, data architectures, and applications. In other words, develop an information integration strategy that binds individual integration projects which, left uncoordinated, could introduce even more problems into your information infrastructure.
Define processes/systems/data architectures to keep data consistent across the various legacy systems that contain similar data. Data architectures should eliminate or synchronize the redundancies typically found in legacy architectures. All systems should provide the same answers relative to common data.
Work with management to educate the need for continual data enhancement and cleanup. Try to educate based on examples in the industry where utilities felt a great deal of upgrade and migration pain before getting management’s attention, sponsorship, and funding. Identify and quantify data integration symptoms. Work with senior management to develop a plan of action and launch a comprehensive data integration initiative.

How you respond to integration demands will determine how effectively your company can compete in coming decades. As you contemplate this, consider that building, launching, and sustaining an integration strategy must be woven into the fabric of the enterprise. This is not a one-time project where the employees can pull one more miracle out of their collective hats. Integration of information, within and beyond the enterprise, must, by its very nature, involve multiple business units, executives, IT people, and a growing host of external entities.

Without a comprehensive integration strategy that can bind disparate entities within and outside the enterprise, companies and industries are likely to struggle with individual initiatives that could ultimately be working at cross-purposes. Taking ownership of this effort will demonstrate leadership, but will again be a major challenge. Project leaders from all corners of a given enterprise should band together to demand the requisite levels of leadership, sponsorship, funding, commitment, and participation from management and from any external entities with a stake in an integrated information enterprise.

Integration is not a buzzword, and it is not one more item on an IT task list that can be worked in when time permits. Integration requires a shift in thinking and a change in how various functions within and outside an organization interact. With so much at stake, all relevant and affected parties should step up to the table now to build a data enhancement strategy that can become the backbone for key initiatives going forward.