Policy and plan effectiveness monitoring

Abstract

Monitoring the efficiency and effectiveness of policies, rules or other methods in policy statements or plans (policy and plan monitoring) is an important part of the resource management planning process. It is an ongoing activity throughout the planning cycle to assess how well the plan is working. Policy and plan effectiveness monitoring helps determine the need for further action, and possible changes and improvements in policy statements and plans, or in actions taken to implement them.

The Resource Management Act (RMA) requires local authorities to monitor the efficiency and effectiveness of policies, rules, or other methods in its policy statements or plans; to take appropriate action when monitoring indicates that this is necessary and to prepare a report at least every five years on the results of section 35(2)(b) monitoring.

Monitoring closes the loop in the ‘plan – do – monitor – review’ cycle and informs decision-makers of the consequences of actions and changes in the environment. Policy and plan effectiveness monitoring provides a means for determining how well plans are working in practice. It both builds on and provides information for state of the environment monitoring and can be assisted by monitoring resource consents, compliance and complaints. It is important to have an integrated approach.

Guidance note

Start plan monitoring early
Have a clear purpose for policy and plan monitoring
Be systematic and apply a consistent approach
What and how to monitor
Have a strategic and integrated approach
Who to involve
Share ideas and allow for training
Determining causality
What to do when plan effectiveness is established

Start plan monitoring early

Think about monitoring at the start of the policy and plan development process. Do not wait until the plan is fully operative or useful information may be missed.
Develop a monitoring/evaluation programme at the same time as plan provisions are developed but recognise things may change.
Monitor the implementation of policy and plans. It is essential to know that plan methods are being implemented before checking that anticipated outcomes are being achieved (refer to best practice examples).
Remember monitoring is a systematic and ongoing process rather than a one-off task. This means there will be a continuous review process for policies and plans.
Be focused; monitor priority matters first. Many councils have started by monitoring and reviewing the environmental results expected (EREs) in policy and plans. This helps to determine if EREs are being achieved and whether or not they need to be more focused (and rewritten) to be measurable [refer to guidance on writing EREs]. This is all part of the monitoring process.
Consider the plan’s context and external factors, eg, population growth or decline.
Establish the links with monitoring the state of the environment and the monitoring of resource consents, compliance and complaints.

Have a clear purpose for policy and plan monitoring

A lot of resources have been invested into developing policy and planning documents so it is important to have a means of checking that this investment has been worthwhile. Policy and plan monitoring is more than a statutory requirement. It is a useful management tool to evaluate and review the effectiveness of policy provisions and plans.

Have a clear purpose for policy and plan monitoring. Is it for:

accountability to the community (to show you have provided a means of managing what you said you would manage and achieved the plan’s environmental goals – such as required by section 32 of the Resource Management Act)? OR
continuous improvement of your organisation? OR
both? (which is likely to be the most useful approach).

Consider summarising the approach in a monitoring strategy.

Policy and plan effectiveness monitoring is systematic and involves tracking and evaluating whether and how well policy or plan implementation is resolving the issues raised in plans.

Is the policy or plan achieving its objectives? How do you know?
Are the implementing agencies delivering on anticipated outcomes?
Have the environmental outcomes have been achieved? This has strong links to state of the environment monitoring and reporting (Refer to the state of the environment monitoring guidance note.)
How effective have policy or plan preparation and implementation processes been?
Does the policy or plan cover the most important things? Are there emerging issues that are not being addressed? (This is an extension of the section 32 process) This also links closely to state of the environment monitoring.

Be systematic and apply a consistent approach

There are few well-established systems for policy and plan monitoring and reporting, but there is a lot of monitoring and evaluation research to build on (refer to key presentations from 2001 workshop in relevant publications). The monitoring process includes:

being clear about the purpose and goals
stating what will be monitored and why
developing indicators
consistent collection of data
analysing, interpreting and presenting information
reviewing the plan or delivery of implementation programmes as a result of this process and triggering a continuous review and reporting cycle
making policy changes and adjustments as necessary ie, taking action.

There is no one right approach to policy and plan effectiveness monitoring. The approach taken should be flexible.
Build on other agencies’ policy and plan monitoring efforts.
Ensure the approach used is relevant to your situation.
Include both quantitative and qualitative assessments (use multiple methods and triangulate the results for greater confidence when making inferences).
Include questions about WHY, HOW, WHO, WHAT, WHERE, WHEN to monitor.
Develop a commonly used terminology and language.
Have a good project management process.

What and how to monitor

Most councils monitor environmental results expected from policy statements and plans to assess outcomes. Methods are also important as they determine the implementation programme.
It is important to monitor whether the objectives of the plan or policy statement are being met. If the environmental results expected are not clearly focused on meeting the plan’s objectives, then just monitoring EREs will not give you the information, it may be necessary to rewrite the EREs in the plan.
If your policy/plan structure is simple and clear and your provisions clear and focused, monitoring will be more manageable (refer to Development of the policy framework Guidance Note).
Ensure key issues, outcomes, processes, impacts and implementation are monitored (this means you need an integrated approach with other monitoring like state of the environment and consents monitoring).
Think about what information you need to decide whether any change in the environment is due to the effect of the policy statement/plan or to other factors beyond this.
Think about how to monitor and review permitted activities (eg, links to complaint, state of the environment monitoring, etc).
Develop a system to trigger reviews of policy statements, plans and implementation programmes in response to the results of policy and plan monitoring.

Have a strategic and integrated approach

Develop criteria for prioritising monitoring so it is cost-effective. Refer to Integrated Monitoring: A Manual for Practitioners (PDF 484KB) Ruth Beanland and Beat Huser (1999).
Develop indicators to assess the means and ends of the plan, and how effective implementation of policies and methods was/is.
Make the most of the data you already collect and supplementary information from different agencies. Consider links between:

other policy objectives within council such as non-Resource Management Act requirements including pest management, reserve, land transport strategies, Long Term Council Community Plans and so on
territorial local authorities, regional councils and others
state of environment, consents, complaints, compliance monitoring
different environmental media and other agencies’ work.

Combine your council databases to include all useful existing information (state of environment, compliance and complaints monitoring) and feedback to policy and other plan provisions.

Who to involve

Corporate commitment is needed to monitor plan effectiveness.
Involve integrated council teams to ensure policy/plan implementation goals are well understood, for example include planners, scientists, environmental education staff, database managers, engineers, consent and compliance staff.
Work with other councils in your area on monitoring methodologies for similar issues (eg, amenity values/cross-boundary issues).
Where possible form or join a Monitoring Forum to share ideas and information or set up other partnerships and review processes.
Involve iwi as they are more than another affected party.
Involve other members of the community as appropriate and find out what community outcomes are wanted and ensure relevance of your policy and plan provisions. Link this to your requirements under the Local Government Act for Long Term Council Community Plans.
Work in with other agencies such as regional councils, territorial local authorities, Department of Conservation, Ministry for the Environment, Statistics NZ, and build on existing information if possible.

Share ideas and allow for training

Think about setting up plan effectiveness workshops and focus groups; sharing ideas is helpful and can make monitoring easier.
Develop partnerships as appropriate (refer to Environment Southland partnership agreement) and consider the use of a Regional Monitoring Forum to share ideas.
Train people and develop learning environments for staff.
Get external guidance where necessary.

Determining causality

Establish cause-and-effect relationships where possible and illustrate and report on attribution. What has changed, and how does this relate to the policy statement or plan?
It is important to monitor process, policy and plan implementation, outputs, impacts and outcomes. If all of these things are monitored (rather than just focusing on process) then we will be a step closer to determining causality.
Be flexible and ask strategic questions like WHY, HOW, WHO, WHAT, WHERE, and WHEN to monitor (known as programme logic). In relation to each issue or group of linked issues:

does the plan make a difference? Is it effective?
was it well implemented?
what worked and didn’t work?
what are the risks of not taking any action, or of taking certain types of action?

Consider a range of methods to help assess what difference the policies and plan has made on the ground (such as using perception or satisfaction surveys, decision trees, monitoring letters to the editor, sequential visual analysis (eg, by comparing photos over time), comparative risk analysis, and triangulation of approaches).
The key challenge is to move beyond measuring only processes and outputs, and start focussing on the impacts and outcomes achieved by the policies and plans.

What to do once plan effectiveness is established

Record the information from your monitoring and evaluation.
Analyse the results of monitoring and interpret what they mean.
Report on the results of monitoring and on the effectiveness of the plan/policy statement (for an example, see Wellington City Council’s 2006 District Plan Effectiveness Report (PDF 971KB)).
Assess where changes may be required to your policy provisions and plans because of the results of your monitoring.
Review your plan provisions as a result of monitoring and evaluation of effectiveness.
Make recommendations for action (which usually involves reporting on monitoring).
Take action and trigger reviews as appropriate.
Provide feedback to the council and any others involved in plan development and the monitoring process.
Ensure there is a good written record of this process and

the results of monitoring
evaluation of effectiveness
recommendations for action and
actual actions and reviews undertaken.

Examples of linking plan effectiveness monitoring to review of sections of a district plan or regional policy statement have been provided by Dunedin City Council (PDF 29KB) and Environment Waikato (PDF 806KB).

Best practice examples

The following best practice examples are the templates provided by local authorities that have been active in the policy and plan monitoring area. They illustrate best practice in particular aspects of plan preparation, as described in the guidance note.

Policy and Plan Effectiveness Monitoring: Environment Canterbury 2003 (PDF 24 KB)
Published by Environment Canterbury - June 2003
This is the summary of Environment Canterbury's policy and plan monitoring.

Policy and Plan Effectiveness Monitoring: Eurobodalla Shire, New South Wales, Australia 2003 (PDF 137 KB)
Published June 2003
This is the summary of Eurobodalla Shire's policy and plan monitoring.

Policy and Plan Effectiveness Monitoring: Manukau City Council 2003 (PDF 90 KB)
Published by Manukau City Council - June 2003
This is a summary of Manukau City Council's policy and plan monitoring.

Policy and Plan Effectiveness Monitoring: Taranaki Regional Council 2003 (PDF 26 KB)
Published by Taranaki Regional Council - June 2003
This is the summary of Taranaki Regional Council's policy and plan monitoring.

Policy and Plan Effectiveness Monitoring: Tasman District Council 2003 (PDF 42 KB)
Published by Tasman District Council - June 2003
This is the summary of Tasman District Council's policy and plan monitoring.

Policy and Plan Effectiveness Monitoring: Waikato District Council 2003 (PDF 26 KB)
Published by Environment Waikato - June 2003
This is the summary of Waikato District Council's policy and plan monitoring.

Policy and Plan Effectiveness Monitoring: Environment Waikato (PDF 45 KB)
Published by Environment Waikato - 2007
This template provides a summary of Environment Waikato’s policy and plan monitoring approaches. To be read in conjunction with the Monitoring and Reporting One-Stop-Shop guidance note on the Quality Planning website.

Policy and Plan Effectiveness Monitoring: Matamata Piako District Council (PDF 30 KB)
Published by Matamata-Piako District Council - 2007
This template provides a summary of Matamata Piako District Council’s policy and plan monitoring. To be read in conjunction with the Monitoring and Reporting One-Stop-Shop guidance note on the Quality Planning website.

Policy and Plan Effectiveness Monitoring: Waitakere City Council (PDF 27 KB)
Published by Waitakere City Council - 2007
This template provides a summary of Waitakere City Council’s policy and plan effectiveness monitoring. To be read in conjunction with the Monitoring and Reporting One-Stop-Shop guidance note on the Quality Planning website.

Policy and Plan Effectiveness Monitoring: Whangarei District Council (PDF 27 KB)
Published by Whangarei District Council - 2007
This template provides a summary of Whangarei District Council’s policy and plan monitoring. To be read in conjunction with the Monitoring and Reporting One-Stop-Shop guidance note on the Quality Planning website.

Managing Change in North Shore’s Residential 3 Built Heritage Zone: An Evaluation of Resource Consent Outcomes 2005 (PDF 2.74 MB)
Mason, G. and McEwan, A. A Report commissioned by North Shore City Council.

Plan Effectiveness Monitoring: Built Heritage in Wellington City. February 2005 (PDF 2.23 MB)
Mason, G. and McEwan A. A report commissioned by Wellington City Council.

RMA provisions

Section 35 specifies the duty to gather information, monitor and keep records. In particular section 35(2)(b) requires every local authority to monitor the efficiency and effectiveness of policies, rules or other methods in its policy statement or plan for its region or district.

Under section 35(2A) local authorities are required prepare a report at least every five years on the results of their monitoring under section 35(2)(b) for policy and plan efficiency and effectiveness. This may be some form of integrated policy and plans, consents and compliance, complaints and state of the environment reporting.

Section 35(3) requires every local authority to keep reasonably available at it principal office, information which is relevant to the administration of policy statements and plans, the monitoring of resource consents, and current issues relating to the environment in the area, to enable the public –

to be better informed of their duties and of the functions, powers and duties of the local authority
to participate effectively under the Act.

Section 62(1)(j) requires that a regional policy statement must state... ‘the procedures used to monitor the efficiency and effectiveness of the policies or methods contained in the statement’.

Section 67(2)(e), as amended in 2005, notes that a regional plan may state... ‘the procedures for monitoring the efficiency and effectiveness of the policies and methods’ contained in the plan.

Section 75(2)(e), as amended in 2005, notes that a district plan may state... ‘the procedures for monitoring the efficiency and effectiveness of the policies and methods’ contained in the plan.

The Local Government Act 2002

The Local Government Act (LGA) also requires monitoring. Under the LGA local authorities must prepare Long Term Council Community Plans (LTCCPs). LTCCPs must state how the local authority will monitor and report on the community’s progress towards achieving the community outcomes (including environmental outcomes), not less that once every three years.

Section 98(2) also requires local authorities to compare annually their actual activities and performance with their intended activities and level of performance as set out in the Long Term Council Community Plan and the annual plan.

Related guidance notes

The following guidance notes are related:

Relevant publications

Environmental Program Evaluation: A Primer
Published by University of Illinois Press – March 1997
This volume explores the possibilities for a more comprehensive approach to evaluating environmental programmes. Experts in a variety of fields including biology, ecology, sociology, economics and public administration provide integrated perspectives on the task of environmental programme evaluation.

New Zealand Planning Institute Conference Workshop (May 2003)
‘Closing the Loop – Effective Monitoring for Plan Development’ by Paula Rolfe, of Matamata-Piako District Council

Review of the New Zealand Coastal Policy Statement (PDF 786KB)
Published by Department of Conservation – August 2004
This is the first independent review of the New Zealand Coastal Policy Statement.

Turning the Tide: A Review of the First Five Years of the New Zealand Biodiversity Strategy, the Synthesis Report (PDF 378KB)
Published by Department of Conservation – November 2005
This is the first review of progress on implementation of the New Zealand Biodiversity Strategy which was launched in 2000.

Creating Great Places to Live + Work + Play: Liveable Urban Environments: Process, Strategy, Action
Published by Ministry for the Environment – June 2002
This urban amenity guide includes a summary document and CD Rom with examples and templates that could assist with consultation processes with communities.

How Does Government Use Monitoring? (PDF 10 KB)
Published by Greater Wellington - February 2001
Discusses the use of monitoring by central government stakeholders such as the use of strategic national monitoring as well as emerging problems.

Monitoring and Evaluation (PDF 8 KB)
Published by Greater Wellington - February 2001
Discusses OECD issues in improving plan evaluation practices.

Monitoring the NZCPS (PDF 19 KB)
Published by Greater Wellington - February 2001
This presentation discusses DoC's intention to monitor the effectiveness of the NZCPS in achieving the purpose of the RMA.

District Plan Monitoring: A Guide to Getting Started (PDF 1 MB)
Published by Ministry for the Environment - April 2000
Monitoring is critical to the District planning process. Without the benefit of robust and appropriate monitoring, a Council cannot judge its progress towards sustainable management.

Towards Effective District Plans (PDF 168 KB)
Published by New Zealand Planning Institute - December 1998
This article discusses the proposal to improve the monitoring of district plans nationally.

RMA Survey of Local Authorities
Every two years, the Ministry for the Environment carries out a survey of resource management processes in local authorities.

Current challenges in practice

Starting policy and plan monitoring as soon as possible.
Monitoring should ideally start when the plan is being developed – but this is a busy time for practitioners, and resources are often limited.
Deciding specifically what to monitor. This decision can be difficult due to different ideas about priorities. Some agencies have monitored objectives, some environmental results expected, performance indicators, some methods (ie, rules) and others a combination of provisions.
Measuring the outcomes and determining causality
It is important but challenging to measure the environmental outcomes and to attribute causality between policies and plan provisions and environmental performance and results on the ground.
Measuring efficiency
Measuring efficiency involves an assessment of the costs of developing and implementing policies, methods and rules. This may be difficult to determine. It also requires a decision about how broadly costs should be considered. Is it sufficient to have a measure based on comparing the relative efficiency of alternatives, rather than attempting to measure absolute efficiency?
Ensuring good integration and project management.
The links between state of the environment, policy and plan and resource consents monitoring and reporting are important but can be confusing. How does everything tie in? How do we ensure an integrated approach?
Deciding on how to monitor and having a good technical process.
It is important to follow a systematic process and ensure there is good technical backing for monitoring. But there is a lack of a clear process for monitoring policy and plan effectiveness.

Within this page