GIS Performance AssessmentMarek Polak Gulf Computers Abu Dhabi UAE marek_polak@aster.pl Balaji Krishnamurthy Gulf Computers Abu Dhabi UAE balaji@gulfauh.co.ae 1. Introduction Motto: If you cannot measure the activity, you cannot improve it. What is the sense of GIS in my organization? Is it really benefiting our efficiency and results? Does it go in the right direction? How can I measure its productivity? Should I implement some corrective measures to make my GIS system better corresponding to the needs of my organization? When are we going to achieve the GIS targets established in the original baseline? These are common question that should arise when a GIS system is operational and no one can answer these important questions if the GIS project is not properly evaluated at the beginning and monitored throughout the whole project life. To answer these questions a systematic performance measurement is essential, based on monitoring of quantitative indicators determined in the conceptual phase of the project and showing how well the GIS-related objectives are being met. In our opinion, the market is looking now at GIS as a supportive tool to improve / optimize business processes. Investing in GIS, just to have a new technology on board, is a concept of the past. Also, strange calculations of ROI or TCO for GIS implementations only are being seen now as something very artificial. The real advantage of GIS lies in its enormous capability of improving business processes and that is why the GIS profitability has to be always estimated in a conjunction with business improvement / optimization. As an example, Fig. 1 is showing a case of utility company where majority of key processes (those marked in yellow) is benefiting largely from GIS support.  Fig. 1 GIS as a support for key processes within a utility company 2. GIS project life cycle and systematic performance measurement 2.1. GIS project life cycle phases A standard GIS project, like other projects, has a number of life cycle phases like initiation, planning, realization and close out phases. Well planned GIS project starts from the needs assessment and ex-ante analysis taking into consideration organizational environment, strategic plans and assumed benefits. At this moment the project baseline is established and a number of performance objectives (KPOs) and parameters (KPIs) are being selected. There are many examples of such an approach ([3], [5], [6] and [7]). Extension of the project life cycle to include the results from completion of a project is now a reality for many GIS specialists. The simplest illustration is a contract for GIS implementation with a following “maintenance and support” contract signed for many following years. Thus, project management often encompasses organizational and qualitative change brought on by the successful completion of a project. This can be considered as a post-completion project phase or "project results integration" or "project benefits realization." For all organizations, it is absolutely necessary to integrate the project results into their ongoing business operations.  Fig. 2 GIS project life phases 2.2. GIS Productivity In most of cases, the GIS productivity is clearly appreciated only after the system enters maturity phase. Quite frequently, it leads to some disappointment at the early stages of GIS implementation. The business process “X” supported by GIS (see figure below) is not benefiting from GIS for some time. Then, after the setup reaches full maturity, the benefits (tangible and intangible) are becoming obvious and acceptable to managers and users. In authors’ opinion, the term “GIS productivity” always stays in a very close conjunction with “Business process productivity” and should be always discussed from that angle. Like other supporting technologies (e.g. ERP, CRM, etc.), GIS implementation makes some business processes temporarily less effective. It’s quite normal as the staff, procedures and production regimes are on the learning curve at this stage of GIS implementation. Unfortunately, this phase is also very sensitive for further GIS development. Quite frequently, this is the time when impatient managers and users try to manipulate with earlier established targets and assumptions. As a result, the GIS experiences speed bumps in its initial months/years before organizations recognize how to leverage its full power. Hence, it is important that GIS performance is controlled and measured at this stage to verify negative deviations and to provide sound directives on further progress.  Fig. 3 Business Process Productivity versus GIS support 2.3. What does “GIS productivity” mean? By no means, the term “GIS Productivity” can be limited to the efficiency of system infrastructure and software only. GIS itself is not benefiting the organization unless it’s becoming an important supporting technology for key business processes within the organization. GIS offers the potential to generate many benefits through its flexibility, speed, availability, and processing power. Benefits can be achieved through GIS for a number of objectives. They typically fall into five areas:
Definitely, GIS productivity has a different meaning, depending on type of organization. In case of local government or municipality it can be considered in terms of usability for public or in terms of making the contacts with public easier and more effective. In case of large FMCG corporate business it can be considered in terms of supporting the sales / distribution channels or simplifying fleet management. For a large utility company, GIS will become productive only after it will effectively support their asset management system and maintenance processes. 2.5. Why to control performance at different phases of the GIS project? The multi-functional model of GIS within the organization makes it very difficult to understand and properly assess if the GIS implementation is benefiting the organization and heading for a final success. If, for example, a large corporate organization is using GIS for their business analysis & planning, supply chain logistics, maintenance services and asset inventory, the total picture of costs and benefits resulting from GIS is becoming rather complicated and dim. Such a situation creates quite frequently a set of questions:
The GIS performance has to be measured systematically and also at some very characteristic points of the project life cycle like:
In quality-oriented organizations, responsibility for defining measures / indicators and for measuring performance rests with business process (GIS-supported) owners. In non-quality-oriented organizations, it stays mainly with higher management (COO, CIO, CTO, etc.) 3. Principles of performance measurement 3.1. Performance measurement and project evaluation principles Performance Measurement is the ongoing monitoring and reporting of project accomplishment, particularly progress towards the pre-established goals. Performance measurement focuses on whether a project has achieved its measurable objectives. Performance measurement, because of its ongoing nature, can serve as an early warning system to management and as a vehicle for improving accountability to the public. Project Evaluations are individual studies conducted periodically or on an ad hoc basis to assess how well a project is working. Typically, project evaluations examine a broader range of information on project performance. 3.2. Performance measurement system as integrated part of overall management process Performance measurement constitutes an obligatory element of TQM. Performance measurement and evaluation is a basic tool for quality assurance and quality planning procedures. Performance measurement plays very important role in change management. At many instances it discovers unattended changes and justifies those which are planned and tested. Performance measurement helps enormously in risk management. At one side, it works as a sort of early warning system and, on the other side, it helps to evaluate if the identified risk was properly mitigated. 3.3. Different business objectives and expectations for different types of organizations 3.3.1. Local Government / Large Municipality The approach to GIS in Government has been discussed, among others, by Gartner Research [1] and [2]. Also, Accenture’s e-government valuation model is adding a lot to that problem. Both sources are concluding the same – traditional Return on Investment measures do not work in Government. Following Gartner Research, three following areas have to be evaluated in case of GIS in Government:
3.3.2. Corporate business organization Business organizations are clearly focused on the results and efficiency of their business processes. Sales strategy and targets are determining their approach to GIS. Benefits of GIS support must be obvious to the management in the areas of sales, marketing, customer care and operations. As an example see the figure below where the importance of GIS-based Network Inventory System is shown for the telecom operator.  Fig. 4 GIS-based Network Inventory versus other processes in telecom company (source: Smallworld) 3.3.3. Research institute In case of research institute the most important objective for GIS support can be a spatial visualization of their research works or spatial analyses performed on some scientific series of data. In most cases typical business objectives like support to sales or to customer care are not applied. Quite frequently, in case of R&D, the financial objectives of GIS are not being placed high in the hierarchy. 4. Key Performance Objectives/Outcomes (KPO) and Key Performance Indicators (KPI) applicable at different phases of GIS project life 4.1. Selection of KPI A good set of performance measures must take into consideration three, the most important aspects of business – productivity, total quality and competitiveness. As already mentioned above, the selection of KPOs and KPIs is a very careful process which has to be done at the early stage of project and very well documented. The set of KPOs and KPIs can’t be voluntarily modified and / or removed from the tables. It does not mean, however, that the set of parameters is fixed and untouchable for ever. Modifying and supplementing new parameters is possible if justified by organizational needs. For instance, the organization is merged with another one and its strategy and operations are dramatically changed. Then, it is very likely that original GIS performance parameters have to be adapted to that new situation. Figure 5 illustrates the theoretical process of KPI selection.  Fig. 5 – KPI selection flowchart A good performance indicator should contain the following elements:
Independently of the organization type, the selected measures have to address the following areas:
 5. TCI / ROI versus complex approach to performance evaluation / analysis 5.1. Critical review of profitability measures used for justification of GIS projects Traditionally, quantitative methods to measure benefits versus cost, such as payback period, net present value (NPV), internal rate of return (IRR) and economic added value are being applied for justification of IT projects, including GIS projects. Gartner [2] is questioning this traditional approach not only in case of government but also in the private sector. For instance, how to fix the discount rate used for NPV in governmental GIS expenditures if it comes from taxes or state subsidy? Gartner [2] is also predicting that by 2007 more than 50% of government initiatives in IT, (inclusive eGovernment, GIS, etc.) that have been cost-justified by traditional economic and financial return methods will not achieve their targets. There are many tangible and intangible benefits and the outcomes and benefits from GIS systems in government are mostly intangible. They cannot always be measured in financial terms. Indeed, the majority of governmental GIS programs are primarily focused on social outcomes and are often intended to:
5.2. Profitability analysis as one of elements of performance measurement Like the majority of IT projects, the GIS implementations are, on an average, not clearly documented from profitability point of view. Around 40% of GIS implementations are missing any financial analysis to justify their profitability (Gartner Research 2002). However, it is not clear to many how to prove if GIS is really profitable. The theoretical formula of determining GIS profitability is understood by authors as follows:
 While calculating the investment costs, all the costs including hidden costs and opportunity costs need to be considered, at some percentage points. The opportunity costs of a project, i.e. the amount which a company could earn if the investment was used in another way (like its core business), etc 6. Practical tools applicable in GIS performance measurement 6.1. The baseline matrix established at project start-up It is very essential for further project development that its initial phase is prepared without shortcuts and simplifications. Needs Assessment Study and Performance Measurement Plan constitute together a sort of Bible for well prepared GIS project. At this stage, the baseline, with all major targets has to be worked out. The baseline targets will, from that moment, serve as a reference points for all future performance measurements. Good examples of such an approach can be found in [3], [4], [5] and [7]. 6.2. GIS performance wheel – examples This simple and very intuitive tool is useful for the management to see how the GIS development is becoming closer and closer to the original targets (KPO’s and KPI’s). Below, two examples for two different organizations are being shown in terms of performance progress. The results of successive performance measurements are being recorded on particular axes providing a quick, intuitive overview of the total progress. The number of axes can be adapted to the needs of managers or business processes owners. Some of the axes (black colour) can be assigned to GIS implementation only while remaining ones (blue colour) can be assigned to KPI’s of key business processes.  Fig. 6 Exemplary Performance Wheel – Municipal GIS – current progress  Fig. 7 Exemplary Performance Wheel – Utility Company – current progress 6.3. Internal (intra-organizational) and external surveys In many cases, the objective analysis of situation is possible only through properly designed surveys. This method is recommended especially for organizations with many external relationships and /or obligations. For instance, the opinion and remarks of community, benefiting from automated issue of maps and plans by the local municipality, can be verified in this way. Also, in case of large corporate businesses, with many inter-departmental relationships, such an internal survey can be very useful to estimate the influence of GIS for customer care operations or provisioning process. 6.4. Use of typical project management tools No one should forget that very well known and simple project management tools also constitute an important part of performance measurement. Project tracking and monitoring with use of MS Project or any other similar software is the best example. If systematically analyzed and updated, it will provide a lot of information on critical bottlenecks, resource management or potential threats to the project. 7. Essential elements for managing any successful GIS project While no two projects are ever the same, and every project has its own unique set of challenges, there is a common core of principles that successful GIS project share. As graphically illustrated in [11] and presented with Figure 8 , the essential elements, contributing to success of GIS project are not differing too much from other types of projects and originate from the following four areas:
It has to be clearly underlined that GIS project success may vary with time and different success criteria might be adopted at different project phases. 8. Conclusions
 9. References
|