| GISdevelopment.net ---> GITA 1999 ---> System Architecture |
Project management and process improvement through software metrics
Robert W. Carroll
Director, Sales and Marketing
GeoData Solutions, Inc.
10075 Westmoor Drive, Suite 200
Westminster, CO 80021 USA
Phone: (303) 635-0500
Fax: (303) 635-0510
Email: RCarroll@,GeodataSolutions.com
Robert W. Carroll
Director, Sales and Marketing
GeoData Solutions, Inc.
10075 Westmoor Drive, Suite 200
Westminster, CO 80021 USA
Phone: (303) 635-0500
Fax: (303) 635-0510
Email: RCarroll@,GeodataSolutions.com
Introduction
All software projects begin as a vision. We are presented with a need that can be addressed through a software solution. The project manager's job is to translate this vision into a usable product. The software development process begins with tentative plans and estimates, then expands with the details of what needs to be done and with what resources.
Normally, we think of software development projects in terms of size, effort, complexity, and time. Software metrics are used to quantifiable measure specific attributes of a software project. They assist with project planning, estimation, progress monitoring, evaluation of the final product's quality, defect analysis, and validation of best practices.
Software metrics are defined as "The continuous application of measurement-based techniques to the software development process and its products to supply meaningful and timely management information, together with the use of those techniques to improve that process and its products''(Goodman 1993). Figure 1 illustrates this process.
The Software Engineering Institute (SEI) developed the Capability Maturity Model (CMM) Table 1, a descriptive model of the stages through which organizations progress as they define, implement, evolve, and improve their processes. To progress through the CMM, organizations must quantifi their current state and define how to improve processes through soflware metrics. Software metrics are crucial to reach the Quantitatively Controlled level of the CMM.
| Capability | Level Common Features |
| Continuously Improving | - Improving organizational capability
- Improving process effectiveness |
| Quantitatively Controlled | - Establishing measurable quality goals
- Objectively managing performance |
| Well Defined | - Defining a standard process
- Perform the standard process. |
| Planned and Tracked | - Planning performance
- Disciplined performance - Verifying performance - Tracking performance |
| Performed Informally | - Base practices performed |
Benefits of metrics
There are many practical uses of software metrics. Companies with software metrics programs tend to dominate their industries. Metrics programs insure that GIS development projects are successful in several ways.
Proiect Planning
Traditional project management activities, such as schedule-based resource allocation using a Gantt chart, are enhanced through metric guidelines.
Proiect Sizing
Before a manager can allocate resources to a project, he or she must know how big it is and what resources will be required. Metrics provide a quantifiable estimate for project size.
Proiect Estimating
From the size of the project, estimated costs for the staffing, equipment, and time to complete the project can be calculated through metrics guidelines.
Proiect Tracking
After the project has been scheduled, the manager should not only track whether milestones are met, but also what resources and costs were expended to meet those milestones.
Evaluation of final product
What is the quality of the final product? How much effort will be needed to support and maintain the product? Metrics allow the project manager to analyze the quality of the final product.
Process Improvement throuqh Defect Analvsis
Performing defect analysis and removing major sources of defects offer the greatest short-term potential for improvements. Metrics analysis provides a quantifiable ratio of defects found, corrected, and undiscovered.
Validation of Best Practices
Software metrics allow people to validate new software development practices.
Metrics as planning guidelines
Rules of thumb for projects have been developed over the years into useful guidelines tohelp plan GIS projects. For example, projects invest 18% during specifications/requirements, 19'ZOduring design, 34% during coding, and 29% during testing (Grady, 1992). Project managers can take these guidelines and compare them to their project plans for a "sanity check."
It is important to interpret the guidelines in the same context. For example, reuse projects typically have longer design phases and shorter coding phases than software projects developed from nothing. Guidelines taken from a support and enhancement project will not be relevant to a new development project.
The following guidelines can be used during the planning of a development project:
| Project definition | - Projects created primarily from reused code take about 25'%0the time and resources of those that are new
- Projects invest 18% during requirements, 19!40during design, 34'% during coding, and 29% during testing. |
| Design | - 50-75'XOof all design errors can be found with design inspections
- An average project delivers about 350 non-commented source statements per engineering month. |
| Testing | - Typical testing without measurable code coverage exercises only 55% of the code
this can be raised to 80°/0 with code coverage.
- Projects created from reused software experience about 1/3 the defect density of those that are new |
| Maintenance | - Organizations expend 2-3 times as much effort maintaining and
enhancing code as they expend to create new code
- One post-release defect will be found for every ten pre-release defects. - It takes 4-10 times longer to fix defects in a mature software system than to make fixes around initial release |
What can be measured?
Many aspects of software development projects can be measured including the basic process model of resources, process, and product. Project resources include all of the resources consumed (e.g., personnel, materials, tools, and methods). Processes include time-related activities and events (e.g., requirements analysis, development, and testing). Software products include the project deliverables (e.g., source code, specifications and design documents, test cases, and defect reports).
Each project entity can be measured by multiple attributes. For example, a single code inspection can be measured by amount of code inspected, preparation time, inspection time, number of defects found, and adherence to inspection process standards.
Software metrics can perform four functions. Metrics can help us understand more about our software products, processes and services. Metrics can be used to evaluate software products, processes, and services against established standards and goals. Metrics can provide the information we need to control resources and processes used to produce software. And metrics can be used to predict attributes of software entities in the future.
Goals, questions, and metrics
Many organizations waste time and money by measuring more things than they need to and by measuring the wrong things. It is expensive to design and develop data collection mechanisms, collect and store data, extract data, generate metrics reports, analysis the reports, and, finally, understand what the information means.
Goal/Question/Metric The Goal-Question-Metric (GQM) paradigm, illustrated in Figure 2, Goal 1 Goal 2 developed by Victor Basili and associates at the University of Maryland, provides an excellent mechanism for defining a goal-based measurement program.
Set Goals
Determine how you want to improve your projects and products. Your goal, for example, might be to reduce the number of defects in the soflware, thus minimizing cost and maximizing customer satisfaction. Or you may have specific goals, such as evaluating the effectiveness of a new process or determining if a product is ready for release.
Ask Questions
Determine what questions you need to ask in order to meet your goals. A question might be, "What kind of defects are costing us the most?"
Establish Metrics
Set up metrics that will answer your questions. You might start collecting data on defect types, creation times, detection times, costs to detect, and costs to correct.
For example, if our high-level goal is to "produce defect-free software," this could result in the following questions:
- Are our defect detection techniques effective?
- Is the software adequately tested?
- How many defects are still undetected?
- Are all known defects corrected before release?
The metrics objective statement defines the requirements for the metric. As in any other software project, the requirements must be translated into a design. This section outlines a step-by-step process for defining the design of a software metric.
Step 1 - Clear Definitions
The first step in designing a metric is to agree to a standard definition for the entities and their attributes being measured. When we use terms such as "defect," "problem report," "size," and even "project," other people will interpret these words in their own context with meanings that may differ from our intended definition. These interpretation differences increase when more ambiguous terms such as "quality, " "maintainability," and "user-friendliness" are used.
Step 2 - Define the Model
The next step is to derive a model for the metric. In simple terms, the model defines how we are going to calculate the metric. Most modeling includes an element of simplification. Remember that the model can always be modified to include additional levels of detail in the future. Questions to ask when defining a metrics model include:
- Does the model provide more information than we have now?
- Is the information of practical benefit?
- Does it tell us what we want to know?
Step 3 - Establish Countina Criteria
The next step in designing a metric is to break the model down into its lowest-level metric primitives and define the counting criteria used to measure each primitive. This defines the method for the measurement of each metric primitive.
The importance of the need for defining counting criteria can be illustrated by considering the lines of code metric. Lines of code is one of the most used and most often misused of all software metrics. The problems and variations and anomalies of using lines of code are well documented. However, there is no industry-accepted standard for counting lines of code. Therefore, if you are going to use a lines of code metric, it is critical that specific counting criteria be defined.
Step 4 - Decide What is "Good"
The fourth step in designing a metric is defining what is "good." Once you have decided what to measure and how to measure it, you have to decide what to do with the results. Is 10 too few or 100 too many? Should the trend be up or down? What do the metrics say about whether or not the product is ready to ship?
One of the first places to start looking for "what is good" is the customer/user. Many times, user requirements dictate certain values for some metrics. There may be product reliability levels that must be met. Another source of information is the metrics literature. Research hashas helped establish industry-accepted norms for a few standard measures. For example, software modules should have a McCabe's Cyclomatic Complexity of<= 10.
Once you have decided "what is good," you can determine whether or not action is needed. If you are "meeting or exceeding" desired values, no corrective action is necessary. However, if improvements are needed, goals can be established to help drive and monitor improvement activities. When setting metrics goals, remember four things:
- The goal must be reasonable. It is all right to establish a "stretch" goal, but if the goal is unrealistic, everyone will just ignore it.
- The goal should be associated with a timeframe. To say you want to achieve a 50% backlog reduction without saying "by when" is meaningless.
- The goal should be based on supporting actions. It maybe reasonable to set a goal of 50% backlog reduction for defects if a special team is created to concentrate on fixing problems.
- The goal should be broken down into small incremental pieces. If the "by when" is a long way off, it is only human nature to procrastinate. If there is a year to accomplish the improvement, it will be easy to put it on the back burner in preference of more pressing concerns. If we divide the same goal into 12 smaller end-of-month goals, actions are more likely to be taken now.
The next step is to decide how to report the metric. This includes defining the report format, data extraction and reporting cycle, reporting mechanisms, and distribution and availability.
Step 6 - Additional Qualifiers
The final step in designing a metric is determining the additional metric qualifiers. A good metric is a generic metric. That means that the metric is valid for an entire hierarchy of additional extraction qualifiers. For example, we can talk about the duration of unplanned outages for an entire product line, an individual product, or a specific release of that product. We could look at outages by customer or business segment.
Human factors
No discussion on selecting and designing software metrics would be complete without a look at how measurements affect people and how people affect measurements. Whether a metric is ultimately useful to an organization depends upon the attitudes of the people involved in collecting the data, calculating and reporting the metrics, and using the metric. The simple act of measuring will affect the behavior of the individuals being measured. When something is being measured it is automatically assumed to have importance. People want to look good; therefore, they want the measurements to look good. When creating a metric, always decide what behaviors you want to encourage. Then take a long look at what other behaviors might result from the use or misuse of the metric.
Here are some basic rules for avoiding human factor problems:
Don't measure individuals. The software metric is just not able to account for individual issues. Individual productivity measures are the classic example of this mistake. We give our best people the hardest work and then expect them to mentor others in the group. If we measure productivity in lines of code per hour, these people may concentrate on their own work to the detriment of the team and the project. Or they may come up with unique ways of programming the same function in many extra lines of code. Focus on processes and products, not people.
Never use metrics as a "stick." The first time you use a metric against an individual or a group is the last time you will get valid data.
Don't ignore the data. A sure way to kill a metric program is to ignore the data when making decisions. "Support your people when their reports are backed by data useful to the organization" (Grady 1992). If the goals we establish and communicate do not agree with our actions, then the people in our organization will perform based on our behavior, not on our goals.
Never use only one metric. Software is complex and multifaceted. A metrics program must reflect that complexity. A balance must be maintained among cost, quality, and schedule attributes to meet all of the customers' needs. Focusing on any single metric can cause the attribute being measured to improve at the expense of other attributes.
Select metrics basedon objectives. To have a metrics program that meets our information needs, we have to select the metrics that provide information to answer our questions.
Providefeedback. Providing regular feedback to the team about the data they help collect has several benefits:
- It helps maintain focus on the need to collect the data. When the team sees the data actually being used, they are more likely to consider data collection important.
- If team members are kept informed about the specifics of how the data is used, they are less likely to become suspicious about its use.
- By involving team members in data analysis and process improvement efforts, we benefit from their unique knowledge and experience.
- Feedback on data collection problems and data integrity issues helps educate team members responsible for data collection. The benefit can be more accurate, consistent, and timely data.
Keys to success in using metrics
Here are the keys to success in using metrics:
- Setup a measurement team. Put it in charge of identifying useful metrics and helping projects to measure themselves.
- Track time-accounting data at a fine level of granularity
- Start with a small set of measurements. Select what you want to measure by using the Goals, Questions, and Metrics approach.
- Don't just collect data. Analyze it and provide feedback about it to people whose work it describes.
A metrics program that is based on the goals of the organization will help communicate those goals. People will work to accomplish what they believe to be important. Well designed metrics with documented objectives can help an organization obtain the information it needs to continue to improve its software products, processes, and services while maintaining a focus on what is important to that organization.
References
- Basili, V., 1987. "Tailoring the Software Process to Project Goals and Environments,'' IEEE Proceedings of the Ninth International Conference on Software Engineering, Monterey, CA.
- Florae, W., Park, R., and Carleton, A., 1997. Practical Software Measurement: Measuring for Process Management and Improvement. Hanscom AFB, MA: Carnegie Mellon University.
- Goodman, P., 1993. Practical Implementation of SofMare Metrics. London, UK: McGraw Hill.
- Grady, R., 1992. Practical Software Metrics for Project Management and Process Improvement. Englewood Cliffs, CA: Prentice-Hall.
- McConnell, S., 1996. Rapid Development. Redmond, WA: Microsoft Press.
- Park, R., 1992. "Software Size Measurement: A Framework for Counting Source Statements." Hanscom AFB, MA: Carnegie Mellon University.
- Software Engineering Institute, 1995, "A Systems Engineering Capability Maturity Model SM, Version 1.1" Hanscom AFB, MA: Carnegie Mellon University.