Monitoring tools, indicators and data management

Abstract

There are many methods and tools that can be used for monitoring. In choosing and using these, it is important to ensure the information produced is of good quality, useful for the monitoring purpose and consistent over time. These requirements should govern the way you collect and manage your data.

Key messages

There are many different monitoring methods that can be used.
Don’t reinvent the wheel, and think about what other people can do for you.
Indicator selection should reflect the monitoring purpose and how the information will be used.
Don’t get bogged down in detail or feel you need to measure everything.
Data management needs to ensure data is consistent, of high quality, easy to collect and record, and suitable for analysis.

Guidance note

Monitoring tools
Perceptions and making decisions based on uncertainty
Participative monitoring and evaluation
Selecting and using indicators
Interpreting indicator results
Data collection
Managing data and information

Monitoring tools

A wide range of methods and tools can be used for monitoring. While the focus is often placed on quantitative measurement, there are also other ways to monitor whether you are achieving your resource management objectives. Tools could include:
- perception or satisfaction surveys
- sequential visual analysis by comparing photos over time
- participative monitoring and evaluation
- model building and scenario testing – monitoring in close situations where the system of environmental actors, causal chains and results may be readily understood (used for forecasting).

Start small. Pilot schemes can be a good way to build capacity on a small budget. Try techniques and learn from the results in limited situations. Proving capability in this way may help attract more resources and cooperation. The key is never to do more than what you can learn from, and never to promise more than you can achieve.

Don’t reinvent the wheel; explore opportunities to use monitoring carried out by other organisations, or to work with other councils to develop methods for monitoring similar issues.

Think creatively about getting local experts or community groups involved. For example, Auckland Regional Council provides training and support for WaiCare groups to monitor stream health in the Auckland region. (Refer to relevant websites for community-based monitoring).

In deciding whether a particular method is appropriate, you should consider the following questions:

Does it provide a means of measuring what is happening?
Is it repeatable (ie, will the results hold across samples)?
Will it provide information related to management (ie, will a change in the way the resource is managed have any effect on the results of monitoring?)
Will it provide information in the timeframes that you need it?
Is it efficient, or will fewer or less complex measures do?

Develop an understanding of the systemic context of particular issues and indicators (what are the resources, natural and human induced changes in the environment, role of the indicators in understanding these changes), so that the analysis of monitoring results can be accurately analysed and interpreted.

Consider using ‘one-off’ research to supplement ongoing monitoring where you want to fill in specific information gaps or explore an aspect of an issue at a particular time. This can be more cost-effective (and more achievable) than trying to cover everything in your ongoing monitoring programme.

Be wary of systems that include collective ranking across various types of impacts, with a cumulative score of significance. These may not be repeatable from one situation to another because the underlying scales of measurement are unique to each type of impact and are not compatible with other scales. Such scores should be acknowledged to be qualitative, and will need to be described so that people other than the person who decided the score can identify assumptions and make their own findings of significance.

Perceptions and making decisions based on uncertainty

Councils have to deal with a number of uncertainties and manage issues for which there may be no right or wrong answers – but a number of qualitative or perceptive judgements may exist eg, for water quality, amenity issues, etc.
There are times when asking people about their perceptions and views (via qualitative social science research) may assist more quantitative and technical monitoring (eg, is the water perceived to be clean? Or is rural amenity satisfactory?).
It is important to have high-quality social science research (including survey design) if this sort of perception research is to be used to substantiate monitoring and reporting.
There have been many perception surveys carried out in New Zealand. Most of these are satisfaction surveys carried out at a local scale and assessing localised environmental issues (refer to report on a review of previously conducted surveys (PDF 257KB)) ; researchers at Lincoln University also carry out a New Zealand-wide biennial survey on perceptions of the environment.

Participative monitoring and evaluation

Participative monitoring and evaluation recognises that public authorities depend on people in the community to implement their policies and strategies and reproduce it in a myriad of applications. The community is not a passive recipient, but a group of actors.
Monitoring indicators and reporting formats tend to be about the officers who design them and what they want to know, and less about people in the community and what they want to learn.
Participative monitoring and evaluation is explicit about who the project is targeting so we can distinguish their various experiences and who is producing the outputs so we can assist their different needs.
This approach develops indicators and questions that make sense to stakeholders so they can identify significant change in their environment and learn how to improve environmental behaviours or practice. Facilitated reflection is always an element, to make the most of the chance to learn.

Selecting and using indicators

Indicators are an ‘indication’ – they provide a snapshot of meaning that people can easily absorb. The trends revealed by the indicators will need to be explained by further research.

Data, information and indicators figure

Text description of figure:
The diagram shows the different levels of data and information required for different purposes. Scientists (and some planners) need to receive detailed, high-quality data which they can use for analysis (eg, to determine causal relationships). Policy makers do not need all the detailed data, but need sufficient information to make decisions about policy approaches to influence resource pressures and trends. Members of the general public need to receive summary information which is easy to understand and will allow them to make decisions about their own actions.

Before selecting indicators, consider:

What is the purpose of your monitoring (eg, to assess the state of the environment, to assess plan effectiveness, to identify compliance issues)?
What sort of reporting will you be doing (and who will the audience be)?
How will the monitoring feed into review of policy and/or management?

Indicator development should be based on the answers to these questions, rather than on current data collection activity – ie, “why are we doing this?” not “what data do we have?” (Developing a monitoring strategy is a useful way of setting this framework.)

There is lots of information available on developing indicators (eg. refer to example of an indicator development process (PDF 177KB) and relevant websites). There are also useful criteria that can be applied to potential indicators to assess whether they are suitable for use (refer to examples in the RMA Monitoring and Reporting Workshop 2006 Handbook (PDF 328KB)).
Some useful questions to assist in selecting indicators include:
- What are the systemic features of the issue that you seek to understand?

What is the outcome you want to monitor?
What are the pressures/causal factors that affect this outcome?
What sources of information are available to tell you about these?
What information will tell you what effects (if any) your policies and management are having?

Be prepared to reassess your data collection activities in light of the results of your indicator selection process. A stocktake of current data collected in your organisation will help you identify what relevant information you already collect and what data collection activities you might need to modify.
It is important not to commit yourself to more indicators than you have the resources to collect data for, consistently over time. Instead, identify your key priorities and start with these.
Consider the links between the different types of monitoring your organisation undertakes (eg, state of the environment, policy/plan effectiveness, consent/compliance/complaints and community outcomes monitoring). Are there indicators that will be useful for more than one of these purposes?
A wide range of environmental indicators have been developed by various organisations – look at the possibility of adopting some of these rather than reinventing the wheel (refer to examples of monitoring strategies and relevant websites).
Review indicators periodically to determine whether they are providing the information you need to understand pressures, states and responses.

Interpreting indicator results

Consider making data subject to panel review. People with different roles in relation to what is being measured can have important experience that adds meaning to the data. For example, resource consent monitoring frequently produces lists of breaches but less so information about the perceived impact of those breaches, the sensitivity of the receiving environment, or the difficulty of implementing the consent on site due to technical or unforeseen physical issues. The developer, neighbours, consenting planner, monitoring officer or scientist might all make relevant observations that would benefit future practice and put the data in context (eg, in terms of repeatability).
Data collected will not provide useful information for monitoring without a baseline for comparison. Monitoring should always be in reference to the situation before an intervention, and what things would have been like without it. This (predictive) base-line acknowledges that change of some kind often occurs anyway – whether through natural processes or patterns of human resource use – due to economic or social factors. Think about what baseline you need to establish so that you will be able to answer the question, “what real difference has the policy and its methods of implementation achieved?”
Don’t assume that an intervention has actually occurred, or occurred in the way expected when indicators were first written. You may not be able to accurately attribute the data you collect to the logic of your district or regional plan or other programme of interventions. The logic may break down at a number of points. For example, were consent conditions enforced, were conditions a practical and accurate application of plan criteria, and did the criteria reflect the policies? Further, the logic may have made a poor job of understanding causality in the environment in the first place, or the causal chain may have been too complex or subject to many external factors.

Data collection

When you collect data you should be confident that people want to and will be able to use and understand it. Much effort has been expended collecting data that no one looks at. Think about who will use it: who stands to gain most from it? Whose behaviours might change as a result of the lessons learnt from it? This will lead you to consider what level of detail you need to record.
Data collection can be made more efficient by enlisting the help of people who are working in the field (eg, consent planners, building officers, technicians). If you make it easy for people to collect and record data as they are doing their jobs, it helps with consistency and reduces the chances of getting gaps in the information. It is a good idea to get input on design of data collection forms from the people who will collect the data (refer to example of data collection forms used by Matamata-Piako District Council for plan monitoring (PDF 28KB) and state of the environment monitoring (PDF 34KB)).
Find out what information is collected by other agencies that you can use. There may also be opportunities to share the costs of your data collection with other agencies who also want the information.
Think about how often you need to collect the data to provide information about trends for reporting and review. If change in the values of a particular indicator will only be apparent over several years, you might not need to measure it annually.

Managing data and information

Just collecting data is not enough – to have confidence in its quality, consistency, and accessibility over time means developing clear procedures for its measurement, recording, and storage and security as well as descriptions about how these things change. To ensure it will be accessible for future users, it must also be stored and described in suitable ways.
Metadata is information about the data – such as what, how, where, when, how often and by whom data is collected, as well as how it is recorded, stored and analysed. This information allows you to determine whether or not data sets collected at different times are truly compatible and so able to be combined to build accurate time series. It preserves a clear understanding of the data over the years despite staff turnover. It can set out criteria for significance when interpreting the data, as well as make clear any limitations in the measuring techniques applied or samples taken [[refer to geospatial metadata standard and examples of metadata templates in the RMA Monitoring and Reporting Workshop 2006 Handbook (PDF 328KB)].
Metadata acts like a library catalogue. It describes topics, how they can be accessed, and where to direct enquiries. In this way it assists the public, other agencies, and your own colleagues to locate all available data in a field of interest. This helps to prevent duplication of effort and to share knowledge. Shared knowledge can translate to shared commitment. An effective approach is to have your metadata on the internet for ease of access. Environment Waikato has a good example of this.
Key matters to consider in designing a system to manage data are:
- How would a person find relevant data?

What makes the data fit for purpose?
How will the data be shared and with whom?
How will it be secured for future use?

Consider whether there is any opportunity to link or combine your council databases to include all monitoring information relevant to state of environment, policy/ plan effectiveness and compliance and complaints monitoring. Integrated data storage can occur at different levels – metadata, indicators, underlying data.
Consider having a data review process as quality of data is critical.
Refer to monitoring strategies and monitoring forums for examples of how councils are dealing with data management issues.

Relevant publications

Ministry for the Environment Indicators
Published by the Ministry for the Environment – May 2007

The Ministry has recently confirmed a set of core national environmental indicators that will measure and report on:

the pressures that are being put on the environment,
the current and historical state of the environment, the effectiveness of any responses made to protect or repair the environment.

The indicator topics include:

air quality
water quality
water allocation
greenhouse gases
ozone levels
solid waste
contaminated sites
land cover and land use
indigenous biodiversity
fish stocks
transport
energy
household consumption
ecological footprint.

Ministry for the Environment Environmental Classifications
Classification systems have been developed to define and describe types of environments, allowing similar environments to be compared. The Ministry for the Environment website has information about classification systems for land, river and marine environments and land cover.

Core Environmental Indicators for Reporting on the State of the Environment
Published by Australia New Zealand Environmental Conservation Council – January 2000.

This report presents a core set of environmental indicators for reporting on the state of the environment. Ministers in the Australia New Zealand Environment and Conservation Council endorsed these indicators in December 1999. The indicators cover:

the atmosphere,
biodiversity,
the land,
inland waters,
estuaries and the sea
human settlements.

Indicators for cultural heritage are under development.

Environmental Health Indicators for New Zealand 2002 (PDF 110 KB)
Published by Ministry of Health - October 2002
This report was commissioned by the Ministry of Health, with the goal being to develop a national core set of environmental health indicators.

Heritage indicators

The Historic Places Trust has recently produced two reports on development of indicators for historic heritage.

‘Wellington Region: SER Historic Heritage – Technical Report’ (PDF 1.92MB) (Historic Places Trust, April 2005) provided information for state of the environment reporting by Greater Wellington Regional Council.
A draft historic heritage guideline for state of the environment monitoring (PDF 316KB) and reporting has also been produced (December 2006) for consultation as part of a series of guidelines for sustainable management of historic heritage. This report has yet to be finalised.

Māori cultural indicators

The following publications present sets of indicators for Māori cultural concerns in relation to the health of water bodies. These have been developed in collaboration with tangata whenua.

Coordinated monitoring of New Zealand wetlands: Māori environmental performance indicators for wetland condition and trend (Published by Landcare Research, 2002).
Using the cultural health index: how to assess the health of streams and waterways (Published by Ministry for the Environment – February 2006).

Hamilton City Council Sustainability Indicators

In 1998, community representatives and Hamilton City Council staff began working together to develop a set of sustainability indicators (PDF 53KB) to measure Hamilton’s progress towards becoming a more sustainable city. More than 150 people representing a wide cross-section of Hamilton’s community were involved. A parallel process was conducted by Māori, for Māori.

In 2004, Hamilton City Council established the My Hamilton website as a tool for reporting on the sustainability indicators. This website has information on air quality, water quality, water usage, soil health, noise, urban trees, energy and solid waste.

This website is also a tool for linking this information with actions of what individuals can do to help achieve better outcomes in these areas. This approach reflects that small individual actions as well as community involvement in council all contribute to sustainable development.

Statistics New Zealand: Environmental monitoring and reporting
In 2002 Statistics New Zealand produced a report ‘Monitoring Progress Towards a Sustainable New Zealand.’ This publication provides a selection and analysis of information related to sustainable development in New Zealand.

Also see other monitoring examples.

Relevant websites

Community-based monitoring

Indicator development

Choosing Futures – development of common indicators for councils in the Waikato region and links to other information about indicators
Statistics New Zealand indicators for sustainable development
Environment Canada: Guidelines for the Development of Sustainability Indicators
South Australia Department for Environment and Heritage links to information about environment and sustainability indicators
OECD development of environmental indicators (PDF 469KB)
US Environmental Protection Agency environmental indicators
UK Government Sustainable Development Indicators
UK Department of Communities and Local Government – indicators for monitoring local development frameworks.

Current challenges in practice

Developing monitoring systems and indicators
Most councils are developing indicators, but there has been some confusion about what they are and how to use them, eg,

the use of quantitative and/or qualitative approaches
the relationship with the national environmental indicators
who decides what indicators to use
linking indicators to the purpose of monitoring and reporting
linking them to outcomes that are meaningful.

Developing multi-purpose indicators
Local authorities have to monitor community outcomes under the Local Government Act as well as environmental outcomes under the Resource Management Act. While there are often connections between the two, outcomes defined in plans under the respective acts are often pitched at quite different levels. It can be a challenge to identify indicators that will serve the purposes of both.
Perception monitoring
Making indicators mean something and ensuring they are useful for management is a challenge. An area that is a challenge for state of the environment monitoring and reporting is perception monitoring – or how to take account of different people’s views on what is an acceptable level of environmental risk and performance.

Related guidance notes

The following guidance notes are related:

Within this page