| GISdevelopment.net ---> GITA 2000 ---> System Architecture |
A Successful System Architecture
Michael B. Waters
Software Engineer / System Architect
U S WEST
1801 California, Room 360
Denver, CO 80202
Phone: (303) 965-5069
Fax: (303) 896-0277
Email : mwaters@uswest.com
Michael B. Waters
Software Engineer / System Architect
U S WEST
1801 California, Room 360
Denver, CO 80202
Phone: (303) 965-5069
Fax: (303) 896-0277
Email : mwaters@uswest.com
Introduction
U S WEST has built a successful GIS system for outside plant engineering design named OSP-FM (Outside Plant - Facilities Management). OSP-FM stores the data for 500 wire centers (a wire center is a local telephone central office) and supports nearly 2 users per wire center. The OSP-FM system architecture must be very robust in order to handle the data volume and user volume, and still maintain acceptable application performance. The system that is in place today is highly successful, but it had to evolve to this level of efficiency. This document will show the evolution of OSP-FM, and the lessons learned during each phase.
A Brief History
U S WEST, like most other telecommunications companies that have been around for a long time, started with paper records of facilities that were hand drawn on non-standard maps and materials. As one might imagine, this is not an effective way to track billions of dollars worth of plant. U S WEST recognized this and when it became technologically possible to make this data electronic, U S WEST transferred the paper records to a CAD (Computer Aided Design) tool. For the time, this was a great advancement. The CAD tool allowed for all of the engineers to pull up any record that they wanted, at any time, just by pointing and clicking. Design engineers were now able to research jobs without shuffling through paper records, thus reducing cycle time and getting the customer phone service faster.
Although the CAD tool was a great improvement over paper records, there were challenges to overcome with the CAD tool. First of all, the data was not intelligent, nor did it use a database to store attribute data. There was no opportunity to automate engineering functions such as network trace. There was no method to enforce rules about concurrency either. If two engineers needed the same drawing, they could both get a copy and create conflicting changes, but the CAD system, like most CAD systems, would not catch the conflict. Another issue with the CAD tool was the hardware needed to store all of the drawings. U S WEST kept nearly every revision ever made to every drawing. This meant that for nearly every edit made to a drawing, a new copy of the drawing had to be stored somewhere (currently U S WEST has multiple terabytes of disk for these storage requirements).
U S WEST believed that a GIS system could solve these issues, and thus OSP-FM was conceived.
The life Cycle of OSP-FM
The Early OSP-FM releases
The early OSP-FM releases were the first releases of OSP-FM to gain broader user at U S WEST. OSP-FM in this phase used an extract-merge back model for editing data. There was a master database for each wire center that was in the OSP-FM system. The design engineers would decide which area they needed to work on and they would initiate an extract of that data from the master database. A local database was created on the engineer's workstation for each work package (job) that the engineer was working on. The user would then edit the local data, and merge the changes back into the master database.
Figure 1: Early OSP-FM Architecture
U S WEST found a lot of value in the data that was put into OSP-FM. With this data engineers could do network traces online, count make-ups, ripple changes downstream automatically, research facilities online, and at the same time, keep the source of record up to date without having to post changes to paper records. U S WEST was one of the first companies to harness this type of power from GIS systems.
U S WEST encountered some problems with this model. First, the extraction process could be lengthy. Once the data was extracted to the workstation, the engineers could encounter problems if they did not extract the right data, or not enough data. When the engineer had extracted all the necessary data to do the required work, putting the data back (merge back) could cause even more problems. For example, if an engineer changed the administrative cable name for a cable, that change needed to ripple though all of the affected cables. If the engineer only extracted the first three sections of a cable that has ten sections, the ripple needed to continue on to the next seven sections when the data was merged back. Detecting that the ripple needed to continue was one hurdle, plus another engineer may have already done a ripple that makes our engineer's data invalid. Issue on top of issue arose from the merge back model.
U S WEST realized that the early OSP-FM releases were not a scalable enterprise solution. The business recognized the potential of a system like OSP-FM and they did not want to give that up. The decision was made to suspend the development of this architecture and do a review of how to make a successful architecture. The business evaluated GIS vendors, alternate architectures, and created some of their own criteria about how the system should look going forward. The re-architecture effort took about a year, and rewrote many key architectural elements.
The Re-Architected OSP-FM releases
The re-architected OSP-FM was a very different system than the previous OSP-FM. The most obvious difference was there was no longer an extract-merge back model. The pendulum had swung all the way from the extract-merge back model to an in-host edit model. Also, the data was actually broken up into four large databases that the users could navigate between nearly seamlessly.
Figure 2: Re-architected OSP-FM Architecture
With the re-architected OSP-FM, all engineers could see all of the data, and everyone edited the master record. This allowed all of the engineers to view the most accurate data that existed. The application ensured that no one could edit the same data at the same time. U S WEST also enhanced the abilities of OSP-FM by including: better network traces, improved conflict management, improved count make-ups, automation of engineering functions, tools to report tax information, creation of a graphics cache, and additional tools that allowed the engineer to do their job better and faster.
The next challenge was upgrading OSP-FM’s performance. During the migration from the old OSP-FM, where engineers dealt with small amounts of data, to this version, where they were engineers handled a vast amount of data, some database tuning was needed. Most of these issues were very easy to mitigate such as basic query tuning, and server tuning.
After the initial releases of this version of OSP-FM, U S WEST started to leverage OSP-FM's open architecture and start interfacing it with other existing systems. Today OSP-FM has the ability to access the older CAD tools' drawings, forecasting systems, systems to track customer orders, systems that track work package (job) information, material ordering systems, LFACS (Loop Facility Assignment and Control System), and links to other miscellaneous U S WEST systems.
Even with the extendibility, and flexibility that this new version of OSP-FM brought U S WEST, the company wanted to expand the systems scalability. U S WEST wanted to offer this tool to as many engineers as possible to help provide premier service to its customers.
OSP-FM Current Release
U S WEST tested OSP-FM to see how it would react to containing the data for 500 wire centers, and found that it was not going to scale as well as U S WEST desired. A team was formed to address scalability issues by reconfiguring the databases into an architecture that would better support the 500 wire centers slated for conversion.
The solution involved providing eight independent databases that would store the OSP-FM data. This solution had many other advantages besides scalability. U S WEST got rid of single points of failure from within the architecture, allowed for incremental releases (only give a new version of OSP-FM to a few users to make sure there are no bugs), and allowed for high application availability. Since there are now eight independent databases, one or more can be down for maintenance and the others are totally unaffected.
Figure 3: Current OSP-FM Architecture
In the current release of OSP-FM U S WEST has also extended and enhanced all of the features that help the engineer get a job out quickly. U S WEST is making OSP-FM even better by enhancing network tools, tuning queries, streamlining work flows, and adding and enhancing a Planning Tool that aids the engineers in knowing where to add facilities in order to plan for future growth. OSP-FM has also automated complex engineering functions, such as de-loading repeaters, load coils, and capacitors into an almost point and click type of operation, which saves the engineers vast amounts of time.
There are a few downsides to the new architecture of OSP-FM; the most obvious is that now users cannot seamlessly go between databases. Most users can be 'turfed' (confined to a state or small set of states) so this is not a huge problem. The other disadvantage is that it is somewhat harder to do queries that span across databases.
Current Hardware Architecture
The current server hardware configuration used in production OSP-FM lends itself to very high system availability (99.9% uptime since the current architecture was implemented), scalability, fault tolerance, application distribution, and maintaining application performance. U S WEST made a substantial investment making sure that the back end of OSP-FM can handle virtually anything the design engineers could throw at it.
At the core of the back end of OSP-FM are two very large database servers. Each database is triple mirrored, with the third mirror being on a remote machine. This third mirror is a very nice capability to have. If there is a production problem, and people need a copy of the production data to track it down, the system administrator simply breaks off the third mirror and brings it up on another database server. This ability lends itself to aid with quick problem resolutions thus helping to limiting user impacts.
U S WEST also uses five other large application servers for running the server side processes for OSP-FM. Four of the servers are used for running processes to keep OSP-FM's graphics cache up to date, as well as running batch server processes. The fifth server runs the server side processes for some interfaces. With this many machines, U S WEST can have multiple hardware outages and still keep OSP-FM up and running. Each of these machines can be reconfigured in a short amount of time to take over processing for another machine. This configuration lends itself to application scalability, flexibility, and fault tolerance. U S WEST is convinced of the importance of providing redundant systems.
Figure 4: OSP-FM Hardware Configuration
Lessons learned about System Architectures
U S WEST has learned lessons about GIS system architecture throughout the life of OSP-FM. Software lessons, hardware lessons, network lessons and business implementation lessons have all brought OSP-FM to the point where it is today. Large GIS systems are new to the computer industry, and sometimes businesses have to experiment with different architectures to get their GIS system set up correctly. These large GIS systems are evolving, and when an architecture change is needed, it should be viewed as a natural step in the evolution of these GIS systems.
U S WEST learned valuable lessons at each phase of the architectural development of OSP-FM. Initial OSP-FM releases showed that an over-complicated the data model (extract, merge back) could have serious repercussions. Telephony has a very complicated data model to begin with, and there is no sense in adding another layer of complication. Another lesson from the early releases is that someone should always ask the question 'Why do it that way?' when setting up a large GIS system like this. When this question is asked, alternatives will be discussed. If this question had been asked before the initial OSP-FM, U S WEST may have been able to skip, or at least shorten, some of the architectural phases.
The re-architected OSP-FM proved that developing a successful system architecture may require learning from experience. Some releases of OSP-FM had single points of failure, as well as new releases of the software had no way to be rolled out to just a few users in the production environment. Architecture issues like these may not be obvious to a company that is setting up a GIS system for the first time. Another lesson learned was that the four databases the users could navigate between seamlessly caused more problems than it solved. The last lesson that the re-architected OSP-FM taught U S WEST was to really watch database sizing. Use the best data available to get accurate initial sizes, but be prepared to make corrections as the data grows.
The current release of OSP-FM is working well. The engineers are saving time and money, while providing the customer with top-notch phone service. OSP-FM also makes life easier for functions that are normally done downstream from engineer (i.e. automatic posting of terminals to LFACS, and automatic generation of a bill of materials.) U S WEST is one of the first telecommunications companies to have this degree of automation among so many systems that deal with outside plant engineering.
Sometimes systems need to go through many iterations before they find the right configuration, and this may be a necessary step in the life cycle of a GIS system. Any business involved should recognize the value of spatial data, and stand behind the project until it finds the right configuration. U S WEST has stood by OSP-FM and the business value that it provides, and is now reaping those benefits that OSP-FM provides.
U S WEST has built a successful GIS system for outside plant engineering design named OSP-FM (Outside Plant - Facilities Management). OSP-FM stores the data for 500 wire centers (a wire center is a local telephone central office) and supports nearly 2 users per wire center. The OSP-FM system architecture must be very robust in order to handle the data volume and user volume, and still maintain acceptable application performance. The system that is in place today is highly successful, but it had to evolve to this level of efficiency. This document will show the evolution of OSP-FM, and the lessons learned during each phase.
A Brief History
U S WEST, like most other telecommunications companies that have been around for a long time, started with paper records of facilities that were hand drawn on non-standard maps and materials. As one might imagine, this is not an effective way to track billions of dollars worth of plant. U S WEST recognized this and when it became technologically possible to make this data electronic, U S WEST transferred the paper records to a CAD (Computer Aided Design) tool. For the time, this was a great advancement. The CAD tool allowed for all of the engineers to pull up any record that they wanted, at any time, just by pointing and clicking. Design engineers were now able to research jobs without shuffling through paper records, thus reducing cycle time and getting the customer phone service faster.
Although the CAD tool was a great improvement over paper records, there were challenges to overcome with the CAD tool. First of all, the data was not intelligent, nor did it use a database to store attribute data. There was no opportunity to automate engineering functions such as network trace. There was no method to enforce rules about concurrency either. If two engineers needed the same drawing, they could both get a copy and create conflicting changes, but the CAD system, like most CAD systems, would not catch the conflict. Another issue with the CAD tool was the hardware needed to store all of the drawings. U S WEST kept nearly every revision ever made to every drawing. This meant that for nearly every edit made to a drawing, a new copy of the drawing had to be stored somewhere (currently U S WEST has multiple terabytes of disk for these storage requirements).
U S WEST believed that a GIS system could solve these issues, and thus OSP-FM was conceived.
The life Cycle of OSP-FM
The Early OSP-FM releases
The early OSP-FM releases were the first releases of OSP-FM to gain broader user at U S WEST. OSP-FM in this phase used an extract-merge back model for editing data. There was a master database for each wire center that was in the OSP-FM system. The design engineers would decide which area they needed to work on and they would initiate an extract of that data from the master database. A local database was created on the engineer's workstation for each work package (job) that the engineer was working on. The user would then edit the local data, and merge the changes back into the master database.
Figure 1: Early OSP-FM Architecture
U S WEST found a lot of value in the data that was put into OSP-FM. With this data engineers could do network traces online, count make-ups, ripple changes downstream automatically, research facilities online, and at the same time, keep the source of record up to date without having to post changes to paper records. U S WEST was one of the first companies to harness this type of power from GIS systems.
U S WEST encountered some problems with this model. First, the extraction process could be lengthy. Once the data was extracted to the workstation, the engineers could encounter problems if they did not extract the right data, or not enough data. When the engineer had extracted all the necessary data to do the required work, putting the data back (merge back) could cause even more problems. For example, if an engineer changed the administrative cable name for a cable, that change needed to ripple though all of the affected cables. If the engineer only extracted the first three sections of a cable that has ten sections, the ripple needed to continue on to the next seven sections when the data was merged back. Detecting that the ripple needed to continue was one hurdle, plus another engineer may have already done a ripple that makes our engineer's data invalid. Issue on top of issue arose from the merge back model.
U S WEST realized that the early OSP-FM releases were not a scalable enterprise solution. The business recognized the potential of a system like OSP-FM and they did not want to give that up. The decision was made to suspend the development of this architecture and do a review of how to make a successful architecture. The business evaluated GIS vendors, alternate architectures, and created some of their own criteria about how the system should look going forward. The re-architecture effort took about a year, and rewrote many key architectural elements.
The Re-Architected OSP-FM releases
The re-architected OSP-FM was a very different system than the previous OSP-FM. The most obvious difference was there was no longer an extract-merge back model. The pendulum had swung all the way from the extract-merge back model to an in-host edit model. Also, the data was actually broken up into four large databases that the users could navigate between nearly seamlessly.
Figure 2: Re-architected OSP-FM Architecture
With the re-architected OSP-FM, all engineers could see all of the data, and everyone edited the master record. This allowed all of the engineers to view the most accurate data that existed. The application ensured that no one could edit the same data at the same time. U S WEST also enhanced the abilities of OSP-FM by including: better network traces, improved conflict management, improved count make-ups, automation of engineering functions, tools to report tax information, creation of a graphics cache, and additional tools that allowed the engineer to do their job better and faster.
The next challenge was upgrading OSP-FM’s performance. During the migration from the old OSP-FM, where engineers dealt with small amounts of data, to this version, where they were engineers handled a vast amount of data, some database tuning was needed. Most of these issues were very easy to mitigate such as basic query tuning, and server tuning.
After the initial releases of this version of OSP-FM, U S WEST started to leverage OSP-FM's open architecture and start interfacing it with other existing systems. Today OSP-FM has the ability to access the older CAD tools' drawings, forecasting systems, systems to track customer orders, systems that track work package (job) information, material ordering systems, LFACS (Loop Facility Assignment and Control System), and links to other miscellaneous U S WEST systems.
Even with the extendibility, and flexibility that this new version of OSP-FM brought U S WEST, the company wanted to expand the systems scalability. U S WEST wanted to offer this tool to as many engineers as possible to help provide premier service to its customers.
OSP-FM Current Release
U S WEST tested OSP-FM to see how it would react to containing the data for 500 wire centers, and found that it was not going to scale as well as U S WEST desired. A team was formed to address scalability issues by reconfiguring the databases into an architecture that would better support the 500 wire centers slated for conversion.
The solution involved providing eight independent databases that would store the OSP-FM data. This solution had many other advantages besides scalability. U S WEST got rid of single points of failure from within the architecture, allowed for incremental releases (only give a new version of OSP-FM to a few users to make sure there are no bugs), and allowed for high application availability. Since there are now eight independent databases, one or more can be down for maintenance and the others are totally unaffected.
Figure 3: Current OSP-FM Architecture
In the current release of OSP-FM U S WEST has also extended and enhanced all of the features that help the engineer get a job out quickly. U S WEST is making OSP-FM even better by enhancing network tools, tuning queries, streamlining work flows, and adding and enhancing a Planning Tool that aids the engineers in knowing where to add facilities in order to plan for future growth. OSP-FM has also automated complex engineering functions, such as de-loading repeaters, load coils, and capacitors into an almost point and click type of operation, which saves the engineers vast amounts of time.
There are a few downsides to the new architecture of OSP-FM; the most obvious is that now users cannot seamlessly go between databases. Most users can be 'turfed' (confined to a state or small set of states) so this is not a huge problem. The other disadvantage is that it is somewhat harder to do queries that span across databases.
Current Hardware Architecture
The current server hardware configuration used in production OSP-FM lends itself to very high system availability (99.9% uptime since the current architecture was implemented), scalability, fault tolerance, application distribution, and maintaining application performance. U S WEST made a substantial investment making sure that the back end of OSP-FM can handle virtually anything the design engineers could throw at it.
At the core of the back end of OSP-FM are two very large database servers. Each database is triple mirrored, with the third mirror being on a remote machine. This third mirror is a very nice capability to have. If there is a production problem, and people need a copy of the production data to track it down, the system administrator simply breaks off the third mirror and brings it up on another database server. This ability lends itself to aid with quick problem resolutions thus helping to limiting user impacts.
U S WEST also uses five other large application servers for running the server side processes for OSP-FM. Four of the servers are used for running processes to keep OSP-FM's graphics cache up to date, as well as running batch server processes. The fifth server runs the server side processes for some interfaces. With this many machines, U S WEST can have multiple hardware outages and still keep OSP-FM up and running. Each of these machines can be reconfigured in a short amount of time to take over processing for another machine. This configuration lends itself to application scalability, flexibility, and fault tolerance. U S WEST is convinced of the importance of providing redundant systems.
Figure 4: OSP-FM Hardware Configuration
Lessons learned about System Architectures
U S WEST has learned lessons about GIS system architecture throughout the life of OSP-FM. Software lessons, hardware lessons, network lessons and business implementation lessons have all brought OSP-FM to the point where it is today. Large GIS systems are new to the computer industry, and sometimes businesses have to experiment with different architectures to get their GIS system set up correctly. These large GIS systems are evolving, and when an architecture change is needed, it should be viewed as a natural step in the evolution of these GIS systems.
U S WEST learned valuable lessons at each phase of the architectural development of OSP-FM. Initial OSP-FM releases showed that an over-complicated the data model (extract, merge back) could have serious repercussions. Telephony has a very complicated data model to begin with, and there is no sense in adding another layer of complication. Another lesson from the early releases is that someone should always ask the question 'Why do it that way?' when setting up a large GIS system like this. When this question is asked, alternatives will be discussed. If this question had been asked before the initial OSP-FM, U S WEST may have been able to skip, or at least shorten, some of the architectural phases.
The re-architected OSP-FM proved that developing a successful system architecture may require learning from experience. Some releases of OSP-FM had single points of failure, as well as new releases of the software had no way to be rolled out to just a few users in the production environment. Architecture issues like these may not be obvious to a company that is setting up a GIS system for the first time. Another lesson learned was that the four databases the users could navigate between seamlessly caused more problems than it solved. The last lesson that the re-architected OSP-FM taught U S WEST was to really watch database sizing. Use the best data available to get accurate initial sizes, but be prepared to make corrections as the data grows.
The current release of OSP-FM is working well. The engineers are saving time and money, while providing the customer with top-notch phone service. OSP-FM also makes life easier for functions that are normally done downstream from engineer (i.e. automatic posting of terminals to LFACS, and automatic generation of a bill of materials.) U S WEST is one of the first telecommunications companies to have this degree of automation among so many systems that deal with outside plant engineering.
Sometimes systems need to go through many iterations before they find the right configuration, and this may be a necessary step in the life cycle of a GIS system. Any business involved should recognize the value of spatial data, and stand behind the project until it finds the right configuration. U S WEST has stood by OSP-FM and the business value that it provides, and is now reaping those benefits that OSP-FM provides.