Performance Indicators
H Alegre*, H. , E. Cabrera**
* Chair of the IWA Strategic Asset Management Specialist Group, LNEC, Urban Water Division, Av. Brasil, 101 1700-066 Lisboa – Portugal
** Chair of the IWA Benchmarking and performance indicators Specialist Group, Instituto Tecnológico del Agua, Universidad Politécnica de Valencia, Spain.
Content Table
- Key concepts
- Purposes of the performance indicator systems
- Performance indicators (PI) systems
- PI requirements
- Other elements of a PI system
- How to design a Performance Assessment System (PAS)
- The evolution of performance assessment
- Relevant milestones in the application of performance indicators
- References
- Related Articles
Key concepts
Performance assessment is a widespread activity used in economics, business, sports and in many other areas of life in general, in order to compare and score entities and individuals and make management decisions, Matos et al., 2003, Alegre et al. 2006, Cabrera & Pardo, 2008, Sjovold et al. eds., 2008, ISO 24510, ISO 24511, ISO 24512).
Assessment is defined as a “process, or result of this process, comparing a specified subject matter to relevant references” (ISO 24500).
Performance assessment is therefore any approach that allows evaluation of the efficiency or the effectiveness of a process or activity through the production of performance measures. Performance measures are the specific parameters that are used to inform the assessment. Main types of performance measures include (Sjovold et al. eds., 2008):
- Performance indicators, which are quantitative efficiency or effectiveness measures for the activity of an utility. A performance indicator consists of a value (resulting from the evaluation of the "processing rule") expressed in specific units, and a confidence grade which indicates the quality of the data represented by the indicator. Performance Indicators are typically expressed as ratios between variables; these may be commensurate (e.g. %) or non-commensurate (e.g. $/m3). In the latter case, the denominator shall represent one dimension of the system (e.g. number of service connections; total mains length; annual costs), to allow for comparisons. The use of denominators of variables which may vary substantially from one year to another, particularly if not under the control of the undertaking, should be avoided (e.g. annual consumption, that may be affected by weather or other external reasons), unless the numerator varies in the same proportion. The information provided by a performance indicator is the result of a comparison (to a target value, previous values of the same indicator, or values of the same indicator from other undertakings) (Alegre et al. 2006; ISO 24500, Sjovold et al. eds., 2008).
- Performance indices, which are measures resulting from the combination of more disaggregated performance measures (e.g. weighted average of performance indicators) or from analysis tools (e.g. simulation models, statistical tools, cost efficiency methods). In general, they aim at aggregating several perspectives into in a single measure. Compared to performance indicators, their main advantages are that they can be more aggregated measures and can be used to assess future scenarios (e.g. using simulation results or statistical analyses). However, they have the disadvantages of being more subjective and less auditable (Alegre, 2008, Sjovold et al. eds., 2008).
- Performance levels, which are performance measures of a qualitative nature, expressed in discrete categories (e.g. excellent, good, fair, poor). In general they are adopted when the use of quantitative measures is not appropriate (e.g. evaluation of customer satisfaction by means of surveys) (Alegre, 2008, Sjovold et al. eds., 2008).
Performance indicators may be converted into performance indices when a performance function is applied, or into performance levels when they are compared with reference levels, in order to support interpretation or multi-criteria analyses. These transformations may be particularly useful to represent graphically the results of a set of performance indicators.
Purposes of the performance indicator systems
A large number of performance indicator systems (with an even larger number of PIs) have been developed throughout the world. Some of the PI systems are developed on a company basis while others are aiming to cover national requirements. Some of the PI systems are developed for specific purposes (e.g. leakage reduction) while others aim to cover service in a more holistic view by assessing serviceability. These systems can be developed for different purposes:
- PIs for regulation
- PIs for international statistics
- PIs for global management of the utility
- PIs for thematic use and for use in decision support systems
Performance indicators (PI) systems
The components of PI systems should comply with some key requirements and according to the ISO 24500 standards:
"a performance indicator system comprises a set of the following key components:
- performance indicators
- explanatory factor and variables
In addition specific targets for each indicator can be established and routinely monitored, tracked and adjusted as needed”.
PI requirements
Individually, a performance indicator should comply with the following requirements ( Alegre et al. 2006; ISO 24500):
- be clearly defined, with a concise meaning;
- be measurable;
- be auditable;
- be as universal as possible and provide a measure which is independent from the particular conditions of the utility;
- be simple and easy to understand; and
- be quantifiable so as to provide an objective measurement of the service, avoiding any personal or subjective appraisal.
Collectively, a PI system should comply with the following requirements:
- every PI should provide unique and/or complementary information;
- definitions of the performance indicators should be unequivocal (this requirement is made extensive to its variables);
- only PI which are deemed essential for effective performance evaluation should be established.
Other elements of a PI system
A performance indicator system is based on data elements (Alegre et al. 2006, ISO 24500), as shown in Figure 1. Performance indicators are computed from variables, and interpreted taking into account explanatory factors. An explanatory factor (EF) is any element of the system of performance indicators that can be used to explain PI values, at the analysis stage. This includes PI, variables, context information and other data elements not playing an active role before the analysis stage.
Explanatory factors may depend on short or medium term management policies. Their identification and analysis is crucial for the identification and prioritization of improvement measures. This type of explanatory factor is also known as a “driving factor”. There are other explanatory factors that depend on the “Context”, i.e. on long term management policies (e.g. predominant system materials and age) or on external factors (e.g. climate, economy, topography) that cannot be influenced by the entity. This information is context information (CI). In some cases, the differences in context are so relevant that they prevent any valid PI comparisons (Alegre et al. 2006). Explanatory factors can be expressed qualitatively (aggregated information, e.g. poor rural area, demographics) or by the means of quantitative data (e.g. ground type, rainfall). For example, the following diagram shows how the asset service is influenced by the nature of the asset and the environment:
Each variable should comply with the following requirements:
- definitions should be univocal;
- fit the definition of the PI they are used for;
- be reasonably achievable;
- refer to the same geographical area and the same period of time or reference date as the PI they will be used for;
- be as reliable and accurate as the decisions made based on them require.
Some of the variables in PI systems are often obtained from external data, and their availability, accuracy, reference dates and limits of the corresponding geographical area are generally out of the control of the undertaking. In this case, variables should also comply with the following requirements:
- be collected, whenever possible, from official survey departments;
- be fundamental for the PI assessment or interpretation; and
- collectively, be as few as possible.
Context information and the rest of the data elements in the system (which can be used as explanatory factors) should follow the same general principles as variables and performance indicators. However, the level of detail and confidence grading is usually not considered to be as high as the one required for PI and variables. Consequently, CI and the rest of the data elements should comply with:
- definitions should be univocal;
- be reasonably achievable;
- if external, be collected whenever possible from official survey departments;
- be fundamental for the PI interpretation; and
- collectively, be as few as possible.
Figure 1 –Illustration of components of a performance indicators system ( Alegre et al. 2006).
How to design a Performance Assessment System (PAS)
Setting up a PI system can be time and cost intensive. It is important to prioritise needs, the type of decision that the PIs will be used to support and the potential of current corporate systems and processes to deliver the relevant data.
It is important to monitor and adjust the PI system, once it is in use, to ensure that it is achieving its intended purpose.
Based on current knowledge, there are some learning’s and advice on how to set up a good PAS, these are (Alegre et al., 2008 & 2009):
- You need to clearly define your objectives – the selection of the PI shall be tailored to the objectives and assessment criteria you wish to achieve. PI should allow to establish targets in a quantified way as well as monitoring deviations, based on a Plan-Do-Check_Act approach.
- You need a balanced set of PIs, both in dimensions and level in order to be able to analyse the wide range cause and impact in the complexity we normally face in our utility systems. For example for analysis of water leakage, one should include PI’s both related to the quantity of water (m3) and to the nature of the network (km pipe length and/or service connections). One needs to have sufficient information about the patient to be able to give a diagnosis and administer the right “medicine”.
- Tailor the PMS to the applications intended - For example if the main objective for the application is to reduce water leakage, one should make sure that every indicator, with relevance leakage management, is covered in the PI system, such as information on age and material of water mains, hydraulics, information about consumption, connections etc.
- Use as few PIs as possible - This is not always an easy task, as there are always a lot of measures that can influence the performance of the system. A good advice can be to identify one, or a few key indicators that describe the main objectives, to ease the assessment of goal achievements. Other PI’s can be selected to describe causes and “why”.
- Tailor information to user needs. For management overview, the use of “traffic lights” can be a more useful “instrument” than for more operational applications
- Good quality of data and good routines for data collection is important. Costs for data collection need to reflect the expected benefits.
The evolution of performance assessment
Since the 1990’s, performance assessment has been playing an increasing role in the water industry. Since the establishment of the economic regulator of England and Wales, OFWAT, the number of initiatives using performance measures and comparison in the world has been constantly increasing.
An interesting fact is how most of those initiatives have evolved in a similar way. A natural evolution that reflects the strengths and weaknesses of each of the stages, and how utilities need to start simple, but more complex endeavours are needed to obtain the most valuable results.
To this date, performance assessment usually starts with utilities or associations collecting statistical data. They recognise the importance of assessing the utilities’ performance, but the objectives of these exercises are not always clear.
If the results are positive, regional or national metric benchmarking projects of some sort are undertaken, competitiveness appears and the comparisons become periodical. The best performers are often urged to go abroad and check their performance with some relevant international utilities.
Once utilities realize that someone is doing better in some area, they feel the need to understand the reasons behind that fact (so they can improve and reach at least a similar level). This leads to process benchmarking initiatives, which is the stage where many of the longest running initiatives are now.
In these 20 years of performance assessment of water services, some lessons have been learnt. The following list includes some of the most typical misunderstandings that should be avoided when running a PI project:
- Lack of engagement of the organisation CEO. Performance Indicators systems are useless if data are not reliable or if results achieved are not used to support improvement measures within the organisation. Unless there is tangible support from the top management of the organization, projects are doomed to fail.
- Incorrect selection procedure. The procedure recommended by IWA and in the ISO 24500 standards for the implementation of a PI system starts with the definition of objectives, followed by the establishment of assessment criteria and only then by the selection/definition of performance measures matching to these objectives and criteria. This is not usually the case, and the selection of indicators is in many cases inconsistent, unbalanced and not very useful.
- Temptation of going from zero to a “PI heaven”. When an organisation starts to select and implement performance indicators, there is typically the temptation that every aspect of the management should be covered. It is fundamental to assure that a balanced solution is found, and that the number of indicators is kept as small as possible, so the cost of data collection, validation, archiving and processing can be recovered.
- Temptation to reinvent the wheel. Many organisations feel that they are unique and therefore will need to develop their own performance measures and establish their own systems. This is partially positive and understandable. However, it is important to take benefit of the existing PI systems like the IWA proposal, which have been tested and refined over the years. The use of existing PI systems recognised as international references has the obvious added advantage of allowing comparisons with other organisations adopting the same platform.
- Misuse of concepts. Using the right words and the right tools for each problem is important. A direct measure is not an indicator (length of pipes) and something that can be changed by a management decision is not part of the context. These basic notions, often forgotten, are well documented and easily available in the IWA manual and in the ISO 24500 standards.
- Only best results welcome. Utility leaders are human individuals and tend to easily accept good results (even without sufficient proof), while failing to adequately react on low performance. A common response is to invest a lot of effort trying to justify poor results instead of concentrating on the analysis of potential problems and countermeasures to improve. A fault-positive culture in the utility is crucial for accepting bad performance results as a chance for improvement.
- High short-term expectations. Measuring the company or the sector performance will not provide automatically improved performance. Improvement measures often need some time to make an impact on the performance figures. Benchmarking is per definition a continuous process and its effectiveness cannot be evaluated after one period. Nevertheless, there are many examples for immediate positive response on starting performance measurements, presumably due to the fact that introducing performance thinking in a company automatically drives decisions towards higher efficiency.
Relevant milestones in the application of performance indicators
1989 – OFWAT is established as the economic regulator of the water services in England and Wales. A system of performance indicators, including data quality assessment and auditing, is established.
1995 – The cities of Copenhagen, Oslo, Helsinki, Stockholm, Gothenburg and Malmo decide to start a comparison of their performance by using indicators. The project has evolved through the years into new forms of benchmarking.
1998 – The Water Services Association of Australia (WSAA) starts publishing WSAA Facts (currently National Performance Reports) including audited data from the major urban utilities in the country. WSAA starts running international process benchmarking efforts in 2004.
2001 – A large metric benchmarking project in context with the IWA field-test establishes the concept of voluntary performance assessment in German water supply. A first major application of the IWA indicator system was a project in Bavaria (Germany) with almost 100 small and medium utilities.
2004 – Portuguese utilities begin reporting to the national water services regulator using a performance indicators’ system based in the IWA proposal.
2007 – Publication of the standards ISO 24500: 2007 - Activities relating to drinking water and wastewater services.
2008-2009 – Many PI applications worldwide are reported in the IWA PI08 and PI09 conferences.
References
ALEGRE, H.; HIRNER, W.; BAPTISTA, J.M.; PARENA, R. (2000). Performance indicators for water supply services, 1st edition, Manual of Best Practice Series, IWA Publishing, London, ISBN 1 900222 272 (160 p.).
