Getting it right: 99.9% pure OMS data
Kinds of good data
Good data is no accident. There is no substitute for the painstaking effort required to figure
out what makes it good and how to keep it that way. These efforts can be divided into three
areas, systematic, random and automation.
Systematic—underlying software works
A systematic error is one that is caused by a design flaw in the data model or software. For
example, a primary meter might not be modeled correctly, in that the primary tap does not
connect to the primary conductor. This problem will affect every primary meter. In some
cases those responsible for the software may have fixed the cause of the systematic error, but
the data have not been corrected. Sometimes these errors can occur for a very specific set of
criteria, and therefore only affect a few customers. In this case the problem may not even be
detected until the data-cleanup process is well on its way.
Random—users are careful
Random errors are minimized when well-trained users enter data carefully into a well-designed
system. At one successful project I was involved in, the manager deliberately
“overtrained” the people who were making updates. The idea was that they would know the
full range of what was possible, even if they couldn’t use all that they learned. For the most
part, people want to do the right thing, they just get frustrated by not having enough
information.
Automation—software is designed to support users
There’s a lot of poorly designed software in the world. This is not because people don’t care,
it’s more because they don’t know. Good software is hard to design. You have to live with
your users, have lunch with them, follow them around. But once you do that, you can make
the tool fit their needs.
Good software design makes clean updates possible, despite constant changes in the state of
the network. It is always better to take the time to make incorrect data combinations
impossible, rather than figuring out how to fix them after they do happen. Here database
triggers, validators and application logic are all your friends, for with them you can capture
business rules and enforce them.
People are all different, and behave in random ways. This is a good thing. But software is
the interface between the randomness of people, and the precise requirements of application
logic. A user interface must therefore be a filter between randomness and precision. People
should be allowed to press any button, at any time, and the software should only allow
operations that make sense. The software should always respond with meaningful information to every user action. The software should provide the user with an indication of
how long something is taking, or will take.
Steps to clean data
Now that we know what good data are about, we have something to strive for. To get there,
you have to start at the beginning.
Source Data
Most OMS data are derived from the GIS. There are many aspects of GIS data that affect
OMS.
Link to Billing System
An OMS needs a source for customer information. Although some systems may have their
own customer information, most depend on a view, snapshot, copied table or real-time
interface to a billing system. This link gives rise to several issues:
- For non-real-time links, what about customers who are getting power, but their record
hasn’t made it to the OMS?
- Active customers with an incorrect link?
- Seasonal customers?
- Disconnects for non-payment?
Each of these questions gives rise to a potential problem.
Phase Attributes
There are three ways to track phasing, each requiring more data (1) ignore it, i.e. don’t
consider phasing at all, (2) Use A, B, C, AB, AC, BC, ABC, and, (3) consider also the order
of the physical configuration in the field, e.g. AC is not the same as CA. The more detail a
utility captures, the more time data cleanup personnel will spend getting it right.
Connectivity
A map edit that causes a discontinuity in the primary necessarily affects all downstream
customers. The solution is to either not allow such conditions through automation, or
provide personnel who make edits with tools to check continuity (trace routine).
Relational Integrity
Some systems provide built-in checks for data integrity. These tools are very useful for
discovering underlying errors in the source data. If not caught, these problems become
exacerbated once in the OMS.
Creating OMS Data
An OMS requires a separate data structure from the GIS. Therefore, every OMS has a
mechanism to create this data structure from the GIS data. These mechanisms vary widely in
their sophistication and features.
Some generalizations are useful. The process for an initial build is time intensive. Once a
structure is built, source-data (GIS) changes can be incorporated either with an incremental
build for the affected areas, or by repeating the full build. The build can reveal GIS errors, or
errors within the special OMS structure.
Systematic errors
To bridge GIS and OMS, some logic is required. Sometimes, this bridge is built using “glue
code,” i.e. specific logic which tells the OMS about the GIS. However, errors in the glue
code lead to errors in the OMS’s understanding of the GIS. For example, I investigated a
glue-code error where move-in/move-out customers where not being updated properly,
leading to missing links between customers and OMS.
Another problem is a single-phase device on a multi-phase line. If these are not modeled
correctly by the glue code, outages can be overstated.
Incremental
An incremental update process can save a great deal of processing time, however, the update
process may lead to errors. These errors can be unique to the particular incremental update
process. One of our customers found the random problems associated with processing
incremental updates harder to manage than just rerunning the full update each time, and so
abandoned incremental updates.
OMS Data
Once built, OMS data need to be checked in several ways.
Tests for completeness
One of the most important tests is whether the OMS view of the world matches that of the
GIS. Any errors in the network can lead to a few or many thousands of customers not
appearing in OMS. When this occurs before a storm, the OMS application is overwhelmed
with “unlocated” calls. It is critical for a utility to have a mechanism to find and fix areas
where the OMS data are incomplete.
Strange errors
The nature of the OMS data structures make them intolerant to errors. If errors from the GIS
go unchecked into OMS, then strange things can happen. Also, if anything can go wrong in a
production environment, it will. Certain situations within the OMS itself can lead to internal
inconsistencies in the OMS data.
At one client site we discovered two such problems. One involved an anomaly in the
program’s in-memory data structures triggered by bad GIS data. Through analysis of the
error trace, we were able to build a tool to detect these problems in the future. A second
problem resulted from erroneous loops in the primary. These loops, if undetected, caused the
prediction engine to fail, leading to a system failure during a storm.
Measuring Progress
If the people responsible for clean data have a reliable way to measure their progress, they
are going to be a lot more interested in that progress. It is therefore vital to the entire OMS
data effort to first set an achievable goal for data quality, and then provide a means to
measure progress towards that goal.
