| GISdevelopment.net ---> GITA 2000 ---> User Perspectives |
Measuring the Benefits of an Outage Management System
Morris A. Kemper
Manager, Construction & Operations Support
Public Service Company of Colorado
1123 W. 3rd Ave. Denver, CO 80223
makemper@psco.com
Morris A. Kemper
Manager, Construction & Operations Support
Public Service Company of Colorado
1123 W. 3rd Ave. Denver, CO 80223
makemper@psco.com
It is well known in the utilities industry that customers are becoming more knowledgeable,
increasing their dependence upon electricity, and increasing their expectations for service levels.
Most utilities also have a growing customer base while market pressures are demanding
reductions in operating costs. Finally, risks of not providing high quality emergency response
services include litigation, financial penalties from regulators, and ultimately loss of customers.
Public Service Company of Colorado (PSCo) addresses these challenges by focusing on
customer needs while improving processes. A process improvement methodology prescribed by
Wayne Brunetti in his book Achieving Total Quality, 1993, Quality Resources, was used. It
involves documenting a given process, gathering data about what is happening within the
process, identifying problems in the process, applying proposed solutions in an organized, logical
way, and measuring the results to guide implementation of changes that improve the process.
The method is an iterative, continuous approach.
When process improvement begins, there exist high benefit yet easy to implement alternatives. These “low hanging fruit” are used first. As the cycle is repeated to achieve continual gains, additional improvements can be more challenging. Beyond changing how work is performed is the option of applying technology to a process. PSCo reached this stage in improving its electric distribution outage response process in 1997. Data from customer needs analysis, performance goals established with the Colorado Public Utilities Commission, and experience of executive management with this technology resulted in a directive to obtain an outage management system. This paper will explore the approach PSCo used in selecting an OMS, challenges encountered along the way, and the results to date after getting the system into production.
Business Case
Examples of benefits achieved through use of outage management systems were rare when the project was begun in 1997. Fortunately, Florida Power and Light (FPL) was willing to share its experiences. Key benefits of automated device analysis and inclusion of a mobile data terminal system were covered by Jim Tracey in his GITA paper “Real Time Trouble Call Management Interfaces.” During storm situations FPL demonstrated dramatic reductions in outage analysis and customer minutes interrupted. This was translated into annual cost savings of manpower, fewer phone centers, and fewer trouble offices needed. FPL’s example of applying technology successfully with proven results was an excellent guide for the project team.
Public Service Company’s planned benefits involve customer service, emergency dispatching, emergency response field work, clerical, and electric distribution engineering business areas. Benefits were classified into either reliability improvement, safety for customers/employees, value to customers, or reductions in operating costs. Using a net present value calculation, the break even period for the investment was calculated to be four years. Following is a subset of planned benefits:
A project team of subject matter experts assembled a functional requirements list that contained thirty one categories. These were based upon experience with an internally developed outage management system. A subset of the categories follows:
Receive Trouble Calls from Customer Information System - Customer Information Representatives and the Interactive Voice Response System create electric trouble calls on our Customer Information System (CIS). The proposed system shall receive electric trouble call records from the CIS as they are created.
Provide System Reliability and Performance - The proposed system shall be designed for high availability operation. The proposed system shall handle a large volume of trouble calls per hour, including all normal processing and analysis functions.
Maintain Switch Operating Status – Provide emergency switching functions that allow dispatchers to maintain the operating status of electric distribution switches for automated outage analysis and feedback to our Customer Information System.
Provide Automated Outage Analysis - The system shall analyze all unresolved electric outage trouble calls to determine the number of distinct outages and the probable extent of each outage. The system shall identify the most probable interruption device for each outage. The system shall create a trouble order for each outage.
Return Status Information to Customer Information System - The system shall return trouble information back to the Customer Information System (CIS) to help Customer Information Representatives and the IVR respond to customer service inquiries.
Dispatch Trouble Orders to Mobile Data Terminals - The system shall send electric trouble orders to a Mobile Computing System for dispatch to the assigned trouble crew. A Mobile Data Terminal (MDT) will be used to view the order, enter information, and report current order status. The system shall automatically update orders with information received from the MDT’s.
Use Current Geographic Information System Data - The system shall use land, electric facilities, electrical connectivity, and loadpoint information from our current Geographic Information System (GIS). The system shall update its database as the GIS database is updated. Provide Computer Map System - The system shall provide a map showing streets, electric distribution devices, primary open points, locations of service complaints, location of probable interruption devices, areas of suspected outages, and display the current electric system connectivity by showing each distribution feeder in a distinct color.
Existing Solutions at Evaluation Time
During latter 1997 and into early 1998 when alternatives were being evaluated, three type of OMS solutions existed within the industry. They were mainframe based, Unix based, and Object Oriented systems.
Mainframe based solutions were the most successful outage management system installations. Successful integration with Customer Information and Interactive Voice Response Systems was demonstrated. Impressive results were achieved in how high volume conditions were processed and the accuracy of automated outage device analysis. However, each system was developed inhouse by the utilities. The technologies in place also had limitations such as inability to track emergency switching operations and difficulty in adding other advanced functionality.
Unix based alternatives were beginning to have multiple success stories. These types of systems introduced high performance graphical user interfaces. Emergency switching and more sophisticated summary reporting functionality was also available. These advances resulted in solution alternatives that met a majority of the functional requirements of the team. Once again, significant disadvantages were also found. These included large efforts to integrate with current/proposed systems, limited system enhancements being made by vendors, complete financial responsibility for any system enhancements requested, and potential for system enhancements not becoming part of the basic package offered by the vendor.
Two object oriented solutions were available to the project team. PSCo had previous experience with an object oriented GIS. The team was aware of the related technical benefits of an improved architecture, relatively fast functional development, and much easier development of interfaces with other systems. However, risks associated with limited existing success stories were taken into account.
Evaluation of the alternatives followed on-site demonstrations of four solution alternatives. Each vendor was provided with the list of functional requirements and allowed to respond in writing and then allowed to demonstrate how they would fulfill the stated needs. Contract negotiations followed where basic system, interface, and project costs were covered.
Components Selected
Technical Challenges
Development of interfaces ended up being more of a challenge in determining what was actually needed from a functional basis versus having to build them. The object oriented OMS and offthe- shelf database management system allowed straightforward interface development. It was also valuable when during initial rollout small enhancements were required.
High availability was a critical requirement. This was based on plans for the OMS to be the primary source of operating device status. Paper maps would be plotted periodically for an absolute backup, but the OMS must “always” be available. A three part solution was implemented. First, redundant Unix servers were obtained. Unix was selected over NT servers due to poor experiences with the latter on site and lack of high availability system software at that time. The servers each have a primary and backup system disk internally and are accessing a RAID hard drive for application data storage. Another component was hot fail-over software.
The third was UPS for power to both the server room and the dispatch center. Finally, having multiple NT workstations, including extras that can be easily switched out, in the vicinity of the dispatch center takes care of problems with dispatcher equipment.
Technical support of the large number of components needing to operate in a high availability setting was even more challenging than with the previous OMS. Integration with the mobile data terminal system made diagnosis of problems more complicated. Detailed procedures for responding to system problems were written with contact information for OMS application, MDT application, OMS to MDT interface, OMS/MDT to CIS interface, DBA, hardware, and telecommunications support personnel. Detailed, step by step procedures were established for loading software releases, importing software fixes, nightly updates from the GIS database, nightly hot backups, weekly cold backups, database restorations, and dealing with enhancements/data model changes to the GIS database.
Testing Approach
Test script formats followed the format of the functional specification documents. They were then enhanced after developing a detailed process map. This process map included not only major activities, but also problems that could be encountered. Examples of this include when the predicted device is incorrect, when the customer service department incorrectly codes orders, when “cancelled” orders are sent into the system, and what to do when a crew performing work does not have an operating Mobile Data Terminal.
A variety of types of tests were also performed. Functional tests were first performed with an intent of making sure the functions worked as planned. These were followed by intentional error and volume testing. Volume testing included scenarios of organized and “shotgun” outage call entries. While the former was more realistic, the latter tended to demonstrate clearly where performance bottlenecks could occur.
Summary
The implementation approach and detailed results to date will be discussed at the paper presentation. Benefits achieved to date include:
The method is an iterative, continuous approach.
When process improvement begins, there exist high benefit yet easy to implement alternatives. These “low hanging fruit” are used first. As the cycle is repeated to achieve continual gains, additional improvements can be more challenging. Beyond changing how work is performed is the option of applying technology to a process. PSCo reached this stage in improving its electric distribution outage response process in 1997. Data from customer needs analysis, performance goals established with the Colorado Public Utilities Commission, and experience of executive management with this technology resulted in a directive to obtain an outage management system. This paper will explore the approach PSCo used in selecting an OMS, challenges encountered along the way, and the results to date after getting the system into production.
Business Case
Examples of benefits achieved through use of outage management systems were rare when the project was begun in 1997. Fortunately, Florida Power and Light (FPL) was willing to share its experiences. Key benefits of automated device analysis and inclusion of a mobile data terminal system were covered by Jim Tracey in his GITA paper “Real Time Trouble Call Management Interfaces.” During storm situations FPL demonstrated dramatic reductions in outage analysis and customer minutes interrupted. This was translated into annual cost savings of manpower, fewer phone centers, and fewer trouble offices needed. FPL’s example of applying technology successfully with proven results was an excellent guide for the project team.
Public Service Company’s planned benefits involve customer service, emergency dispatching, emergency response field work, clerical, and electric distribution engineering business areas. Benefits were classified into either reliability improvement, safety for customers/employees, value to customers, or reductions in operating costs. Using a net present value calculation, the break even period for the investment was calculated to be four years. Following is a subset of planned benefits:
- Reduction in Response Time due to Automated Probable Device Prediction &
Outage Order Management - Replacing the use of printed trouble calls will reduce
outage response time even when trouble call volume is high (feeder lockouts or storms).
Known device outages such as feeder lockouts will involve a simple confirmed out
procedure. This functionality will result in a four percent reduction in response time.
- Reduction in On Hold Time for Customers Reporting Outages - What is being
done about an outage, estimated time of restoration, and detailed information from
dispatchers is provided to the Customer Inquiry Representatives who pass the information
to customers. This will avoid lost profits by helping the company meet a performance
goal established by the Colorado Public Utilities Commission.
- Reduction in PUC Complaints by Improving the Accuracy and Quality of Outage
History - Confirming the correct outage devices, use of pull down lists to provide
consistency, automatic creation of the outage history by customer account, and having the
data in easily accessible format improves accuracy and quality. This will allow improved
response to customer inquiries on repeat outages and allow the company to proactively
prevent these problems. This will avoid lost profits by helping the company meet a
performance goal established by the Colorado Public Utilities Commission.
- Improved Storm Management - Electronic distribution of relevant data will improve
resource management decisions and communication to the general public on progress in
dealing with events such as storms.
- Consistent Facility Data Across Business Areas - Safety and efficiency will be supported when Emergency Response Crews, Dispatching, System Operation, Engineering, Reliability Assessment, New Construction, and Maintenance/Permanent Repair personnel are using the same (GIS) database. Coordination during switching will be easier, communication of repair needs will be more accurate, and evaluation of problem areas will be faster.
A project team of subject matter experts assembled a functional requirements list that contained thirty one categories. These were based upon experience with an internally developed outage management system. A subset of the categories follows:
Receive Trouble Calls from Customer Information System - Customer Information Representatives and the Interactive Voice Response System create electric trouble calls on our Customer Information System (CIS). The proposed system shall receive electric trouble call records from the CIS as they are created.
Provide System Reliability and Performance - The proposed system shall be designed for high availability operation. The proposed system shall handle a large volume of trouble calls per hour, including all normal processing and analysis functions.
Maintain Switch Operating Status – Provide emergency switching functions that allow dispatchers to maintain the operating status of electric distribution switches for automated outage analysis and feedback to our Customer Information System.
Provide Automated Outage Analysis - The system shall analyze all unresolved electric outage trouble calls to determine the number of distinct outages and the probable extent of each outage. The system shall identify the most probable interruption device for each outage. The system shall create a trouble order for each outage.
Return Status Information to Customer Information System - The system shall return trouble information back to the Customer Information System (CIS) to help Customer Information Representatives and the IVR respond to customer service inquiries.
Dispatch Trouble Orders to Mobile Data Terminals - The system shall send electric trouble orders to a Mobile Computing System for dispatch to the assigned trouble crew. A Mobile Data Terminal (MDT) will be used to view the order, enter information, and report current order status. The system shall automatically update orders with information received from the MDT’s.
Use Current Geographic Information System Data - The system shall use land, electric facilities, electrical connectivity, and loadpoint information from our current Geographic Information System (GIS). The system shall update its database as the GIS database is updated. Provide Computer Map System - The system shall provide a map showing streets, electric distribution devices, primary open points, locations of service complaints, location of probable interruption devices, areas of suspected outages, and display the current electric system connectivity by showing each distribution feeder in a distinct color.
Existing Solutions at Evaluation Time
During latter 1997 and into early 1998 when alternatives were being evaluated, three type of OMS solutions existed within the industry. They were mainframe based, Unix based, and Object Oriented systems.
Mainframe based solutions were the most successful outage management system installations. Successful integration with Customer Information and Interactive Voice Response Systems was demonstrated. Impressive results were achieved in how high volume conditions were processed and the accuracy of automated outage device analysis. However, each system was developed inhouse by the utilities. The technologies in place also had limitations such as inability to track emergency switching operations and difficulty in adding other advanced functionality.
Unix based alternatives were beginning to have multiple success stories. These types of systems introduced high performance graphical user interfaces. Emergency switching and more sophisticated summary reporting functionality was also available. These advances resulted in solution alternatives that met a majority of the functional requirements of the team. Once again, significant disadvantages were also found. These included large efforts to integrate with current/proposed systems, limited system enhancements being made by vendors, complete financial responsibility for any system enhancements requested, and potential for system enhancements not becoming part of the basic package offered by the vendor.
Two object oriented solutions were available to the project team. PSCo had previous experience with an object oriented GIS. The team was aware of the related technical benefits of an improved architecture, relatively fast functional development, and much easier development of interfaces with other systems. However, risks associated with limited existing success stories were taken into account.
Evaluation of the alternatives followed on-site demonstrations of four solution alternatives. Each vendor was provided with the list of functional requirements and allowed to respond in writing and then allowed to demonstrate how they would fulfill the stated needs. Contract negotiations followed where basic system, interface, and project costs were covered.
Components Selected
Technical Challenges
Development of interfaces ended up being more of a challenge in determining what was actually needed from a functional basis versus having to build them. The object oriented OMS and offthe- shelf database management system allowed straightforward interface development. It was also valuable when during initial rollout small enhancements were required.
High availability was a critical requirement. This was based on plans for the OMS to be the primary source of operating device status. Paper maps would be plotted periodically for an absolute backup, but the OMS must “always” be available. A three part solution was implemented. First, redundant Unix servers were obtained. Unix was selected over NT servers due to poor experiences with the latter on site and lack of high availability system software at that time. The servers each have a primary and backup system disk internally and are accessing a RAID hard drive for application data storage. Another component was hot fail-over software.
The third was UPS for power to both the server room and the dispatch center. Finally, having multiple NT workstations, including extras that can be easily switched out, in the vicinity of the dispatch center takes care of problems with dispatcher equipment.
Technical support of the large number of components needing to operate in a high availability setting was even more challenging than with the previous OMS. Integration with the mobile data terminal system made diagnosis of problems more complicated. Detailed procedures for responding to system problems were written with contact information for OMS application, MDT application, OMS to MDT interface, OMS/MDT to CIS interface, DBA, hardware, and telecommunications support personnel. Detailed, step by step procedures were established for loading software releases, importing software fixes, nightly updates from the GIS database, nightly hot backups, weekly cold backups, database restorations, and dealing with enhancements/data model changes to the GIS database.
Testing Approach
Test script formats followed the format of the functional specification documents. They were then enhanced after developing a detailed process map. This process map included not only major activities, but also problems that could be encountered. Examples of this include when the predicted device is incorrect, when the customer service department incorrectly codes orders, when “cancelled” orders are sent into the system, and what to do when a crew performing work does not have an operating Mobile Data Terminal.
A variety of types of tests were also performed. Functional tests were first performed with an intent of making sure the functions worked as planned. These were followed by intentional error and volume testing. Volume testing included scenarios of organized and “shotgun” outage call entries. While the former was more realistic, the latter tended to demonstrate clearly where performance bottlenecks could occur.
Summary
The implementation approach and detailed results to date will be discussed at the paper presentation. Benefits achieved to date include:
- Demonstrated time savings in high volume situations due to correct device prediction on large outages and feeder lockout functionality.
- Dispatchers are not having to go to wall maps very often
- Status feedback to Customer Service Department has been consistent
- High availability performance of the systems involved has been achieved