Very large area SCADA for water supply

Very large area SCADA for water supply

Ian Wiese
Principal SCADA Planning Officer,
Western Australian Water Corporation
629 Newcastle Street, Leederville,
Western Australia, Australia, 6007
Telephone: +61 8 9420 2610,
Fax: +61 8 9420 2915
E-mail: ian.wiese@watercorporation.com.au

Dr Stephen Beckwith
Senior SCADA Engineer, A.L. Haime and Associates Pty Ltd
6/47 Cambridge Street, West Leederville,
Western Australia, Australia, 6007
Telephone: +61 8 9420 3740,
Fax: +61 8 9420 2915
E-mail: stephen.beckwith@watercorporation.com.au

Introduction
The Western Australian Government established the Water Corporation of Western Australia (WCWA) as a commercial enterprise in January 1996, as part of a program designed to improve the competitiveness of a number of public utilities in Western Australia. The WCWA moved quickly to adopt modern Information Technology (IT) systems extensively for the benefit of the organisation. They have recently installed SAP/R3 as the Management Information System, implemented a sophisticated customer billing system, adopted geographic information system technology and have replaced a mainframe computer infrastructure with PCs connected via a Wide Area Network (WAN) extending across the state to all manned locations.

On the engineering side of the organisation, treatment plants are becoming more sophisticated due to increasingly stringent regulatory requirements. Modern distributed control systems (DCSs) are being used in all but very simple treatment applications to control and optimise the process whilst providing product quality information to regulatory authorities. In many respects, the WCWA's use of technology in the Perth Metropolitan area is similar to that in similar sized cities around the world.

The deployment of SCADA and DCS systems has been extended to cover smaller water distribution systems and waste water collection systems throughout the state. The WCWA has been proactive in installing standalone supervisory control and data acquisition (SCADA) systems that provide control functionality and alarms at these sites which in many cases are very remote. These systems were often used to solve single problems such as reducing power cost, or improving control of a particularly complex operation. The installation of SCADA has subsequently been seen as a means to satisfy a variety of increasing pressures such as consumer demands, regulatory requirements, and to also satisfy the need to reduce operational costs. Consequently, the WCWA is embarking upon a program of widespread installation of SCADA systems to control and monitor all smaller systems.

An important challenge to the commercial success of the organisation is to harness the data collection power of the SCADA and DCS systems to provide a wealth of operational information to all levels of the organisation. Past systems that have been installed throughout the state have failed to meet expectations regarding data availability. This has primarily been attributed to difficulties associated with merging traditional engineering and new IT methodology, and a lack of system openness in data interconnectivity and communications.

The Corporation has recognized the need to more closely link SCADA and DCS systems with Corporate Management Information Systems (MIS) to improve the utilization of information from SCADA systems throughout the organization. To achieve this, the WCWA had to settle upon a standard SCADA and DCS architecture that would allow such data transfer. This architecture had to be compatible with current market trends, meet the particular needs of remote control systems and not lock the WCWA into a single SCADA technology. This approach has since spawned the concept of a centralized integrated SCADA system and has unearthed a wealth of opportunities for enhancing the operational structure of the WCWA. The entire project, including the roll out of SCADA and the integration of such systems with the Corporate IT system, has been named the Integrated Information Monitoring and Control (IIMAC) system project.

Challenges of integration on a grand scale
The challenge facing the IIMAC project is exacerbated by the sheer scope of the enterprise. The WCWA is one of the largest water utilities in the world, providing water related services across a state that makes up nearly a third of the Australian continent - an area approximately the size of the eastern third of continental United States. The WCWA serves a population of nearly 1.8 million people over 639,000 homes and 48,000 businesses in some 230 cities and towns. The majority of these towns are "unmanned" (serviced from nearby regional centres). Some 80% of the population live in and around the capital city, Perth. In an area of 2,525,500 square kilometers, the WCWA operates 243 water treatment plants, 73 dams and reservoirs, 779 water wells in 106 well fields, 107 service reservoirs, 91 wastewater treatment plants, and 635 water tanks and towers. There are also over 28,000 kilometres of water mains, and nearly 11,000 kilometres of sewers. Over the last year, the WCWA supplied consumers with approximately 325,600 Mega-litres of water.

Outside of the capital city of Perth, the WCWA currently uses a variety of technologies to control and operate assets. Due to geographical constraints each individual scheme is separate from other systems. Rural systems are most often unmanned and are designed to run automatically. Individual control solutions have been tailored to meet the particular needs of each system. Communications with these locations (many extremely remote) is difficult and expensive. As a result, existing SCADA systems are small and isolated with little or no data available to central design and investigation personnel, or for operational or regulatory reporting. They are designed and managed only as a tool for day-to-day operations. Maintenance is a particular problem due to the remoteness and availability of skilled staff. In addition, the legacy of a strong regional organizational structure has meant that differing standards and technologies have been applied around the state.

The WCWA therefore faced the challenge of developing a SCADA architecture that would meet local control objectives, solve data interconnectivity issues across the state, reduce the cost of SCADA systems and reduce the amount of maintenance required at remote locations. This led to the concept of appropriate robust technology at remote locations, a highly reliable but low cost communications network for SCADA, a modular design and a centralised SCADA system that would provide a single point for on-line data extraction and supervisory control of remote systems.

Movement towards a single statewide scada system
As time progressed and experience accumulated in installing standalone SCADA systems, the concept of installing "regional" SCADA systems evolved. This involved linking all remote systems in a given region to allow monitoring of all assets from the regional office. This allowed consolidation of management of operations and maintenance to the regional level, and automated regional performance reporting. However, it was important that the control of a remote town's water supply remained at the local level to ensure it operated in the event of communications failure to the central system. The approach was therefore one of distributed system control intelligence, but centralized monitoring and supervisory control.

This regional concept developed further into a plan to integrate all SCADA systems across the state into a single, centrally managed and operated system. This initially was a means by which the physical SCADA equipment and software could be effectively managed by at a central location in the capital city of Perth. As there was already a Corporate IT WAN across the state for business systems users, operator workstations associated with the new SCADA system could be connected to SCADA servers via that WAN. This effectively would allow operator access to the SCADA system from any PC on the IT network, and would avoid the need to establish a second network. This allows flexibility in choosing the location from which to operate an individual system.

Pre requisites for integrated system
There were a number of obstacles that had to be overcome before an integrated system became feasible. WCWA staff set out to identify and resolve those issues, gradually putting in place all the elements of the technology that would allow the WCWA to install an integrated system on this large scale. One of the most pressing issues was to ensure that with such an investment into wide area SCADA, the WCWA was not locked into any particular technology. The WCWA also favored a rollout of a series of individual smaller SCADA systems that could be added to the overall system in a modular fashion. This has the effect of reducing the risks and allowing early delivery of the benefits. The adoption of open programming and communications standards was a prerequisite to this contracting approach ensuring that competitive bids could be sought for each contract.

The development of standards for I/O configurations, control strategies, screen design, database, tag naming, and so on, and the enforcement of these standards is essential to ensure large-scale rollout of the SCADA system. With potentially 3000 sites to cover, custom engineering would not result in a scaleable solution, or one that could be implemented in any useful timeframe. The adoption of standards also has the useful side effect of dramatically reducing costs. The concepts of a local communications network for a particular scheme, a Wide Area Network for SCADA carrying data between the local system and the central SCADA system, and a gateway bridging the two were developed. This lead to a modular structure for the overall system that was scaleable to the size envisaged.

SCADA system architecture - Utilizing open standards
Appropriate technology was needed to meet the particular requirements of the local systems. This primarily meant a philosophy of autonomous local control in that in the event of a loss of communications to the centralized SCADA servers, local control could adequately operate the local system. The central SCADA master server then only has the role of setting control performance targets, and receiving data and alarms. The control philosophy adopted is to ensure the local SCADA remote terminal units (RTUs) operate autonomously and can communicate directly with each other to control the physical system. For example, a tank RTU can directly command a transfer pump station RTU to start a pump, supplying water to the tank.

After investigating market trends, the WCWA settled on the use of DNP3.0 as the local communications protocol on all new SCADA systems. This open protocol allows for peer-topeer communications between sites via a DNP3.0 gateway RTU. This facilitates the local control system, allowing control commands to be passed between local RTUs via peer-topeer requests. The gateway RTU performs the routing of DNP3.0 packets as specified by the address header on the peer-to-peer message. The protocol is also compatible for use with UHF radio links, the preferred local physical communication media. Sites are interconnected at the local level using a talk-through radio repeater, or a series of inter-linked repeaters depending upon the physical distribution of a system.

The identification and adoption of standard RTU configurations for different devices (e.g. transfer pumps, tanks, pressure regulating valves, flow meters, and so on), together with the development of standard techniques for the automatic control of these systems was therefore another important initiative to achieving autonomous local control. The standard technique used for automatic control is to break the system down into independent zones, each of which can be controlled by a set of rules. For example, one zone might be a transfer pump station pumping to a tank, or a set of water wells that pump to a wellfield collector tank. The rule associated with the control of the latter zone could be a set sequence of wells to be brought on line for various levels in the collector tank at various times of the day. This set of rules would be physically located in one of the RTUs in the local control system.

An IEC61131-3 compliant programming language was selected as the standard for RTU programming to create the standard RTU configurations. The objective of adopting the IEC61131-3 standard was to protect the WCWA's investment in RTU software, and to develop a library of configurations for different field devices ensuring consistency of system programming across the state. The software library may be centrally managed and configurations may be downloaded to appropriate devices in the event of loss of local configuration or upon upgrade of system control algorithms.

DNP3.0 over UDP/IP was selected for SCADA wide area communications. This UDP/IP network has been termed the SCADA WAN and it provides the communications link between the central SCADA servers in Perth and the remote autonomous control systems, which can be thousands of kilometres away. Communication can be via a variety of communications media however studies have shown that telco frame relay is the most attractive communications media for all but the most remote area links. For these remote systems, VSAT satellite technology is being tested. The gateway RTU, included in all local autonomous systems, is designed to not only route DNP3.0 packets on the local network, but also to encapsulates DNP3.0 traffic within UDP/IP for transmission over the SCADA WAN. UDP/IP, like TCP/IP, is the communications protocol used by the Internet, and allows general-purpose communications networks and equipment to be used and shared. This has the advantage that the WCWA is able to use its Corporate IT WAN links (suitably enhanced to ensure 99.9% availability) to interconnect local systems with the centralized SCADA servers.

As there appeared to be no real SCADA master software standards, CiTect. was chosen as the SCADA Human Machine Interface (HMI) software as it is widely used and supplied in Australia by a large number of competing suppliers. CiTect. is implemented on the Windows NT platform that has been adopted by the WCWA as the standard operating system for all PCs connected the Corporate IT WAN. CiTect. required implementation of a DNP3.0 driver to communicate with RTUs. Ci Technologies, the developers of CiTect., have recently commercially released this driver. The CiTect. SCADA servers are interconnected with the Corporate IT WAN to allow an operator workstation to be connected to the SCADA system from anywhere in the state connected to the Corporate IT WAN. This allows appropriately empowered operators to see CiTect. as another application on their standard Windows NT PC. Remote operators can log into their local control system, monitor performance or enact manual control, the whole time during which the SCADA system is appropriately managed from a central location.

It should be noted that since the adoption of these SCADA standards by the WCWA, most other major Australian water utilities also saw the need for the water industry to reduce its dependence on proprietary systems and have subsequently adopted the DNP3.0 and IEC61131-3 standards.

Around the clock system availability
An important aspect to the architecture that has evolved is the increasing use of the Corporate IT WAN infrastructure to carry both SCADA data and SCADA operator terminal traffic. The joint use of the communications WAN and PC network removes the need for SCADA to establish a parallel communications networks. However, this also requires that the corporate WAN must be upgraded to provide reliable 24-hour-a-day, 7-day-a-week operation. This represents some fundamental changes in philosophy and design of the Corporate IT WAN.

Considerable effort has to be expended to move key components of the corporate WAN from a "nine to five" office tool to an around-the-clock operations tool. The entire SCADA system is being designed to achieve a desired figure of 99.9% availability through the use of extensive redundant hardware. Accompanying the hardware upgrade of the Corporate IT WAN is a process of review of IT operating procedures and support arrangements. An incident management approach to system failure is now required. In addition it is no longer feasible for the IT support team to schedule parts of the network down for maintenance over the weekend for example. This systematic approach to building a reliable Corporate IT WAN will have dividends for general business users of the network.

Benefits of the IIMAC SCADA system
As previously asserted one of the underlying drivers for this integrated approach is to allow SCADA data to be integrated into Corporate Management Information Systems. Accurate reporting to regulators such as the licensing, environmental, and water resource allocation regulators is vital for the WCWA's continued operation. Automated reporting of operating performance is also an essential tool for better management of operations. Condition monitoring and assessment, and reporting of this data are important tools in improving the WCWA's performance in these areas. As part of the project, SCADA data will be transferred to a data historian for archiving. The data historian facility will be available on-line via the Corporate IT WAN and will provide tools such as links to spreadsheets, and the Corporate MIS, that will enable use of the data in day-to-day planning activities. Centralization of SCADA data at a single point will ease the capture of real-time SCADA data into the data historian software package. This arrangement then allows all suitable WCWA users to access all information required to plan all aspects of the company's engineering business. The WCWA's capital budget for the 1999-2000 financial year is approximately US$300 million, and any information about the performance of existing assets and customer demand patterns is vital to the correct staging and sizing of facilities and equipment upgrades. The WCWA also has sophisticated hydraulic models of each water supply system throughout the state, and actual data from the SCADA systems will be invaluable in calibrating these models.

One of the benefits that has arisen from the architecture that has been established is the ability for the SCADA system to be flexible with organizational change or to even mould such change within the WCWA. Currently the WCWA is structured into separate operational regions that effectively form autonomous business units. This has arisen from geographical constraints associated with the state of Western Australia. Experienced operational staff are located in one of the local regional centres, offering operational and maintenance support to ensure the correct workings of the local water supply, wastewater, drainage or irrigation systems. With recent organization pressures, staff numbers have been reduced at these locations, putting more and more pressure on existing staff to maintain assets. Often such staff are forced to travel large distances to provide such support. With the advent of SCADA and in particular utilization of the Corporate IT WAN as the communications media by which operator workstations are connected to the SCADA master servers, such staff have the option to remotely diagnose problems from any location to which the Corporate IT WAN extends. Further, use of the remote dial-up access capability of the Corporation IT WAN means that staff can literally access the SCADA system from any location with a phone call or by GSM. The flexibility of the system will allow the WCWA to instigate operational organization changes, for example by shifting operational boundaries in the knowledge that the local system can be equally controlled and monitored from any location. The WCWA will also commence moving towards a centralised statewide after-hours alarm monitoring centre and centrally located hydraulic operations support services.

The IIMAC project has within its scope the development of quality systems and documentation systems to support the ongoing operation, maintenance and further expansion of the SCADA system. The IIMAC project allows a systematic, quality assured approach to support and maintenance of SCADA thereby protecting the WCWA's investment. It provides access to technical expertise that would not be available to individual systems. In addition it provides a consistent standard of alarm monitoring, control, and data collection statewide, replacing the current "ad hoc" system-by-system approach. The programmed rollout of SCADA and the adoption of standards will also reduce costs substantially. The IIMAC project can be justified on the basis of reduced costs of the SCADA rollout alone, provided the SCADA rollout is above a minimum rate.

Implementation - Case studies

Perth Metropolitan Wastewater SCADA
This project incorporates the monitoring and supervisory control of a network of 145 wastewater pump stations that are located adjacent to the banks of Perth's river systems. Past wastewater overflows due to a range of factors including power failure have been the catalyst for the project. The community insisted that the WCWA improve monitoring of the performance and status of the pump stations, and reduce the already low incidence of overflows to an almost zero rate. The stations incorporate their own local control systems that operate pumps based on the level of wastewater in a "wet" well. The stations are to be monitored from three locations within the metropolitan area. Two locations are used for the purpose of maintenance staff on-line information (one in the northern areas of Perth, one in the south) and the third location is an after hours alarm monitoring and emergency call out centre.

Because of the open nature of the SCADA technology, the project was let to competitive tender. This produced the advantage that the successful tenderer offered the SCADA system at an extremely competitive price. This system will be one of the lowest cost systems the WCWA has installed. This project, although not complete as of 26th November 1999, forms the basis for the SCADA system that will be developed as part of the IIMAC system project. The project incorporates the provision of a set of SCADA servers, configured as production and test units and will include the provision of hot/standby distributed SCADA servers located in physically separated rooms connected via a high speed optic fibre LAN connection. Each room will contain production and test SCADA servers and production servers will be mirrored to the point that loss of the primary server will mean a seamless take up of SCADA operations by the standby production server. A network of five radio repeater will be used to provide local DNP3.0 communications to the pump stations, and allow a station to inhibit a station upstream of it should the former station experience operational difficulties. Each repeater will include a "Gateway" RTU encapsulating DNP3.0 packets in UDP/IP for transmission to the SCADA servers. With the creation of this system, future SCADA systems around the state may be integrated with the SCADA servers established as part of this project.

Exmouth Water Supply System
Beyond the myriad of benefits to the corporation at large, the move to SCADA system standardization was proven at the local level when Cyclone Vance struck the town of Exmouth in March 1999. The storm's 165 mph winds, the highest ever recorded in mainland Australia, blew away the year-old SCADA system that ran the well field for the town's only water supply. A new SCADA system was urgently required, and within two weeks, a replacement system was up and running, utilising the new SCADA standards described in this paper. The rollout of the new system broke all WCWA records in terms of speed of installation for a SCADA system.

According to Business Support Services Operations Planning Manager Allan O' Neill, the corporation "had a vision of getting to the stage where telemetry and SCADA systems could be set up using modules". The vision became reality when a system developed for another site was brought to Exmouth and assembled in this manner. Although the IIMAC project was not underway at the time of installation of this system the basic concepts were well advanced. In this instance a temporary CiTect. system was installed at Exmouth to provide adequate operator interface. Once the statewide SCADA servers are installed as part of the Perth Wastewater SCADA project, the Exmouth master functions can be shifted to the central location, with an operator workstation installed in the town of Exmouth.

Conclusion
This paper has described the needs of the Water Corporation of Western Australia in terms of expectations of SCADA, and how those needs have driven Water Corporation staff to seek out SCADA technology which is open and fit for purpose. In particular, given the geographical size of the state of Western Australia, it is appropriate that operational control intelligence is distributed to remote control systems. Adequate reporting is required to a centralized SCADA server arrangement to allow operational data to be distributed to all staff in the organization that requires such information. Dissemination of data is achieved through the use of a centralized, widely accessible data historian software package and data links to the Corporate MIS. In addition, centralised SCADA servers form the hub of supervisory manual control of remote systems, allowing any authorised operator to perform control functions on given remote systems over the Corporate IT WAN. These requirements have led to the selection of communications protocols DNP3.0 and UDP/IP, programming language standard IEC1131-3 and the CiTect. HMI software system as the SCADA master software. A large consideration in selecting these standards was to ensure that the WCWA was not locked into particular SCADA technologies. Selection of these standards has led to added benefits in the way in which SCADA may be deployed throughout the state. With the use of UDP/IP as the protocol for the SCADA WAN, the WCWA is able to utilise the existing Corporate IT WAN for use in communicating SCADA data as well as linking operations workstations to the SCADA servers. This obviates the need to construct a parallel SCADA communications network. However it does impose stringent requirements on the availability of the Corporate IT WAN.

Another major benefit described herein is the operational flexibility introduced by such architecture. Operators may monitor and control local systems from any location in the state that is connected to the IT WAN.

Finally, this paper offers two case studies that detail implementation of the chosen SCADA architecture.