Integrating Spatial Architecture with the Enterprise

Integrating Spatial Architecture with the Enterprise

Andre Cassulo
Florida Power and Light 700 Universe Blvd
PO Box 14000,Juno Beach, FL 33408-0420

Brad Sileo
GE Smallworld Inc. 10075 Westmoor Drive
Suite 200, Westminster, CO 80021

Introduction
As spatial systems gain increased importance in the enterprise, system requirements for the architecture of these applications continue to grow. Implementations now require communications with legacy and new systems throughout the enterprise, and even outside the enterprise. These needs drive requirements for an architecture that can scale and adapt to this ever changing and growing environment. The ongoing business to business and business to consumer e-commerce revolution has led to a new set of applications involving spatial systems, and with these new applications come even greater challenges in building architectures which will endure the test of time.

This paper explores options and solutions for building a spatial system that accommodates the need for change, growth, scalability, and stability. It is a case study of the ongoing implementation of the Asset Management System (AMS) at Florida Power and Light (FP&L). This project includes extensive requirements for systems interfaces; fault tolerance and disaster recovery; and a scalable, industry standard-based system which can co-exist and inter-operate with the existing enterprise standards at FP&L.

The intent of this paper is to share the research and experience gained while planning the AMS. As this project is ongoing, this paper does not attempt to provide details on the success or failure of the directions we have chosen. This paper provides a guide for issues and options to consider when implementing a spatial system in an enterprise environment. Primary attention is given to the use of standard operating systems, hardware, data storage systems, software packages, and integration bus technologies.

The Tech21 Project and the Asset Management System The Asset Management System (AMS) will provide a core technical foundation for the Power Systems Business Unit to plan, manage, and analyze its electric infrastructure. Today the basic tasks of managing FP&L’s electric network information are scattered among many processes using disconnected applications, databases, and maps. These existing applications have been built and incrementally modified over the past few decades using various technologies that leave us with an overly complex mix of outdated, redundant systems and interfaces.

The AMS will provide a platform that brings together all the existing electric network data into a common, industry standard, vendor packaged product that provides better common user data access through corporate relational databases. This platform will enable more efficient, reliable interfaces via its object-component, distributed architecture. It will also provide an integration point for additional ‘plug in’ vendor software such as network analysis tools, engineering design tools, field viewing tools, etc. The AMS will be the foundation for other initiatives in the TECH 21 program, such as: SCC, WMS, and OMS.

The AMS will be developed in a phased approach to begin reaping benefits earlier, to ease user training, and to minimize the risk of project delays due to process and/or technology changes. This document defines the scope of Phase 1, and thereby forms our commitment to the Power Systems Business Unit and our vendors as to what is included and excluded from this delivery.

Spatial Architecture in the Enterprise
When implementing a spatial system in a large enterprise, special consideration must be taken to ensure the long-term success of the systems. Large organizations are accustomed to implementing in-house or custom developed systems. However, current trends show that implementing standards-based solutions using open systems leads to lower total cost of ownership. The spatial technologies industry has long been based on custom data storage, development, and interface solutions. The industry as a whole is quickly moving away from this approach, but current systems still require propriety approaches to obtain appropriate functionality and performance. In building architecture for the enterprise, a careful balance between open and custom solutions must be struck. This approach ensures that the existing resources of the enterprise can be leveraged to reduce costs and ensure success. On the other hand, appropriate use of custom technologies is a requirement to guarantee that the system can deliver the functionality and end user experience required for success. The AMS architecture was evaluated based on these criteria.

The AMS Development Environment
The environment used for developing and testing the solution is a key aspect of any IT system. The AMS was constructed as a joint effort between GE-Smallworld staff and Florida Power and Light staff. This means that the development team was spread across sites in Juno Beach, FL; Denver, Colorado; and Pittsburgh, Pennsylvania. Since the project was on a rapid implementation schedule, there was a need to ensure that development, testing, and system feedback could occur simultaneously across all three sites. A custom development environment was established to support this requirement.

To support concurrent work at each site, a set of network connections was needed. An existing network link was available between the GE-Smallworld offices in Denver and Pittsburgh. An additional connection was established between Denver and Juno Beach to allow data to move between the sites. This connection was completed using a virtual private network (VPN) connection through the Internet, based on IPSEC technology. With a permanent network connection in place, the teams were able to establish direct access to Oracle databases and Smallworld databases, fileservers, and web servers. This access was used to allow the entire AMS team to work against a common source code repository. The environment was configured to build new images based on this source code for each site every night. Additionally, scripts were used to replicate the datastores throughout the sites as often as once a week. Finally, a project web site was established to provide a single source for information on the AMS requirements, data model, core, and interface tracks. This site was used extensively throughout the project to promote effective communications.

Figure 1 - AMS Developmemnt Environment Architecture

AMS Production Architecture
The core AMS is based on Smallworld Core Spatial (GIS) technology. In planning for the implementation of this software within Florida Power and Light, a careful review of existing FP&L expertise and Smallworld technologies was completed. The project team wanted to find solutions that would allow AMS to take maximum advantage of existing infrastructure and support groups within the enterprise. This led to a decision to implement the Smallworld Repository on Oracle as the storage container for the main GIS datastores. This technology allowed AMS to use the skills of an existing team or Oracle DBAs inside FP&L to support the application. This group brought experience in deploying and maintaining Oracle systems within FP&L including monitoring, backups, and disaster recovery.

The core system includes access to the full GIS for 100 users during phase 1, as well as access for 400 FP&L staff members using the Smallworld Internet Application Server (SIAS)—a view-only web application. AMS requirements also dictate that the system provide a read-only reporting replica in Oracle, downstream interfaces to legacy FP&L systems, and interfaces to several other Tech21 projects using integration bus technology from STC. Finally, the AMS must be able to recover to a full level of service within two hours of a catastrophic site failure.

Figure 2 - Overall AMS Production Architecture Diagram

Database Servers
The architecture diagram above depicts the hardware systems selected to support database requirements. At the center of this architecture is the primary datastore server. This system is configured with the AIX operating system and the Oracle server. This Oracle environment will host both Smallworld GIS datastores using the Smallworld Repository on Oracle and the AMS reporting replica, which is a conventional set of Oracle tables. This Oracle server will use standard Oracle technologies to backup the Smallworld data on a daily basis. It also implements Oracle Standby server technology to replicate data to the server at the disaster recovery in real-time. This allows for immediate recovery of the system at the standby site with almost no loss of data. In electing to use the Oracle server instead of GE Smallworld's base data storage approach-the Smallworld Master File Server (SWMFS)-the following factors were evaluated:

Management tools - Oracle provides a strong suite of tools for backup,replication, and general data administration.
Existing skill-sets - FP&L has a large staff with extensive skills in Oracle. By selecting this approach, the AMS project could leverage this staff base.
Complexity - The oracle repository provides an increased level of complexity, as it requires that staff be familiar with both Oracle database management and with management for Smallworld databases
Corporate Standards - FP&L corporate standards require that enterprisecritical data be stored in Oracle whenever possible. This solution allowed AMS to meet this requirement.
Performance - Benchmarks indicate that performance is similar for all Smallworld storage approaches. Oracle provides excellent memory management and data caching technologies that accelerate performance. SWMFS provides technologies to implement compression of network traffic, which is not available with Oracle. Initial tests indicate that performance in the FP&L environment is faster using native SMFS storage instead of Oracle, but the use of persistent cache servers in AMS will eliminate most of this impact.
Fault Tolerance - Through the use of the Oracle solution, the AMS system is able to integrate with FP&Ls existing HACMP clusters within the AIX environment. The project was able to leverage existing knowledge of implementing and maintaining this solution with Oracle servers. However, the Smallworld repository does not currently support automatic recover of the database connection due to network or server interruptions. Support for this functionality is coming in a future GE Smallworld release, so it was deemed a worthwhile risk for the short term.

Interface Severs
One of the critical requirements of the AMS system is the ability to interface with other applications throughout FP&L. These interfaces include feeds to legacy systems which are replaced or supplemented by AMS as well as feeds to other systems which are being introduced as part of the Tech21 initiative. The architecture for AMS was designed to support the process of moving data to and from these systems throughout the enterprise. Server and architecture requirements are driven by AMS requirements to provide data in varying formats and varying levels of granularity throughout the enterprise. Some systems, such as the Distribution Management System (DMS), require a full network model from AMS, while others, such as the data warehouse/decision support system, only need tabular data from AMS. When reviewing the architecture for each interface, efforts were made to combine the transport and delivery protocols for each system, which helps to eliminate both development efforts and hardware investment. The interfaces section below provides further details on this integration approach.

The network builds required for DMS are generated using the GE Smallworld NetXchange application. This software is able to analyze complex networks and output the results in a custom format. In this case, the data is output in an XML rendering of the Common Information Model (CIM). The process of analyzing the network can be quite memory intensive and the requirements from DMS dictate that AMS be able to generate large volumes of network data on a nightly basis. To support these needs, a group of three systems are assigned to DMS interface processing. These Windows NT servers are configured to run as high-end clients, with 1 GB of RAM and a high-end disk subsystem to support this rapid processing. An equivalent set of systems is deployed at the disaster recovery site to ensure that interface processing can resume without interruption in the event of a site failure. General Application and Job Servers

In order to promote user productivity and reduce network traffic, AMS was designed with a batch approach to data merge, post, validation, and cleanup operations. This approach allows a user to submit a specific work order for processing and then move on to the next job without waiting for it to complete. In the case of AMS, this approach is call "Real-time batch processing". This means that the system attempts to complete each user request as quickly as possible after it is submitted, and uses the instant messenger capability within the system to notify the user as soon as an operation is complete.

The Job server is responsible for completing real-time batch processing for all AMS users. This server is configured as a high-end client machine, with 512MB of RAM, an 800MHz CPU, and a RAID 5 based disk subsystem to store local cache files. This specification provides adequate performance to process work orders as they are submitted by users and should provide a 5-10 minute turn-around under normal load. Because work order and validation requests fall off in the evening, the job servers also provide general application services for AMS. The primary application function after hours is running the GE Smallworld InSync application that allows a single view of the GIS database to be replicated to Oracle. The server must review all changed data in the GIS and publish these changes to Oracle. It also reviews changes in the Oracle database and returns them to the GIS as part of this process. The resultant Oracle database, which is hosted on the production database server, is used for many interfaces and for access to AMS data from generic reporting tools in the enterprise.

Persistent Cache
In order to ensure high performance for all users, AMS is implemented using persistent cache servers at each user site. These servers store data on local hard drives and are deployed on the LAN such that high-speed connections from each user station are available. Within AMS, each persistent cache sever is configured with 36GB of disk space to allow for storage of the local cache data. Since the cache servers are not fault tolerant, the application will be configured to ensure that the users can operate with degraded performance in the event that a cache server fails. Each cache server will be a basic Windows NT server with a single CPU, 256MB RAM, a 100MB network card, and three 18GB disk drives in a RAID 5 configuration. The selection of RAID 5 for the disk subsystem is primarily to enhance read performance for the cache servers, but also provides fault tolerance on these servers.

Smallworld Internet Application Server Servers
One of the critical requirements for AMS is to provide data access to a group of 400 users at phase one and 1,000 users at phase two. FP&L selected the Smallworld Internet Application Server (SIAS) to meet this requirement and the production architecture for the project includes appropriate systems to implement this software. The SIAS architecture provides fault tolerance and disaster recovery through redundant servers and sites, as shown above. The application is exposed through a single server running Microsoft's Internet Information Server (IIS) as well as the TICS Dispatcher from GE Smallworld, which provides simple load sharing between backend servers. A collection of four SIAS servers is configured to process requests from the users. Each Windows NT server includes an 800MHz CPU, 1GB of RAM, and RAID 5 based disk system for local cache files. This approach provides fault tolerance in the event that any of the SIAS servers should fail. The IIS server provides a single point of failure in this configuration, but the disaster recovery site includes a backup for this equipment and a simple DNS change can redirect subsequent requests to this site in the event of a critical equipment failure.

Fault Tolerance and Disaster Recovery
AMS requirements demand a maximum 2-hour recovery time in the event of a catastrophic site failure. The production architecture was assembled top to bottom to minimize the chance of such a failure ever occurring and to ensure a successful recovery if it should. The production database servers are implemented as part of the existing HACMP clusters at FP&L, providing CPU failure protection. All production disk storage is within a high availability disk subsystem that is maintained by FP&L information services to insure continuous service. Through the use of Oracle Standby server technology, a near real-time copy of the database is maintained at the disaster recovery (DR) site. By using FP&Ls Tivoli infrastructure, the Oracle alias for connecting to the AMS server can be switched throughout the enterprise in the event of a failure, and the users can reconnect to the DR site by simply restarting their client sessions.

In order to exercise all servers in the AMS architecture, the primary and DR sites are rotated every six months. This helps ensure that the DR operations and equipment are functioning as expected and makes maximum use of all hardware in the system.

Interfaces and The Integration Bus
As part of the Tech21 initiative, Florida Power and Light has implemented an integration bus using technologies from STC software. This bus provides the framework and the transport for moving data to and from AMS and other systems in the enterprise. As shown in the diagram below, there are a large number of systems throughout the enterprise that interact with AMS. However, through the use of the integration bus technology, the number of interfaces from AMS can be minimized. There are two renderings of the AMS data required to support all interfaces to the system. The first is a rendering of AMS data in Oracle, using InSync to move the data from GE Smallworld to Oracle. From the Oracle database, standard tools provided by STC are used to transform and deliver the data to a large collection of downstream systems. Applications that require full details of the network connectivity modeled and maintained within AMS can subscribe to the CIM view of the network published by AMS, which includes full object attributes and connectivity information for the network. Because both the network model and the tabular data are delivered and maintained in standard-based systems (Oracle and CIM/XML), adapting future systems to use the data will require minimal effort. This approach also allowed the AMS team to leverage the integration bus and its support and technology team instead of implementing custom transports to support the interface requirements of AMS.

Figure 3 - AMS Interfaces