Developing an evaluation framework

Introduction

There is no one-size-fits-all process for evaluating community engagement or any other government activity. Each community engagement program requires an evaluation framework tailored to the purpose for which the evaluation will be used, the intended audience of the evaluation, the types of community engagement activities and the scale and signifi cance of the activities.

Scale and significance refers to:

• the amount of resources committed to the community engagement activities (and/or may be committed in the future)
• the significance of the activities to government, including the degree of political support and attention
• the nature of public expectations surrounding the activities.

Developing an evaluation framework involves four main steps (see Figure 2). Although the steps are sequential, they are also interlinked. Completing later steps can necessitate revisions to earlier steps. This is illustrated by the arrows moving in both directions.

Figure 2. Steps in developing a community engagement evaluation framework

Step 1: Clearly articulate the community engagement program to be evaluated, including:

• the individual community engagement activities to be undertaken
• the critical success factors for these activities
• the intended short, medium and long-term outcomes of the activities
• the potential external factors that may affect the process and outcomes of engagement

Step 2: Establish the purpose for the evaluation and who to involve

• Decide why you want to evaluate your community engagement program
• Identify who the audience for the evaluation will be
• Decide who will conduct the evaluation and who should participate in the evaluation

Step 3: Identify key evaluation questions and information requirements

• Write a list of the questions that will guide the evaluation
• Identify the types of information required to answer the evaluation questions including performance criteria and indicators

Step 4: Identify data sources and methods

• Identify existing data sources
• Decide how and when new data will be collected and analysed
• Decide whether the research plan can be implemented
• Consider the requirements for privacy and ethics approval
• Develop tools for data collection

Key principles for the development of evaluation frameworks

Five key principles can be used to guide the development of frameworks for the evaluation community engagement programs and activities.

1. Evaluation should be an integral part of the planning and management of community engagement activities.
   

   Ideally, a framework for evaluation should be designed at the time of community engagement planning and be a part of the on-going management and refinement of the community engagement activities.
   
   However, the steps described in this section can be applied to designing an evaluation framework at any time before, during or after a community engagement program.

2. Evaluation should be a structured and planned process
   

   While informal judgements about the success of community engagement activities occur throughout programs, these are not the same as evaluation. Good evaluations need to:

   • be purposeful and focused
   • be based on clear performance criteria derived from the goals, objectives and anticipated outcomes of the activity
   • apply systematic procedures to gather trustworthy evidence
   • weigh the evidence against the criteria carefully and objectively in the process of evaluating the program’s success².

3. The scale and scope of the evaluation should reflect the purpose, audience, and the scale and signifi cance of the community engagement activities.

Evaluations of small-scale community engagement activities with limited political significance do not require costly and extensive evaluation programs. In many cases, all that may be required is a
feedback questionnaire or interviews with participants based on a few key evaluation questions of interest and a plan to use and share that information. This might include questions such as: whether the target community received information, whether the information was easy to understand, or whether an activity was accessible and comfortable to participants.

On the other hand, evaluating a major community engagement program often requires detailed planning and someone with skills and experience in evaluation to implement the evaluation. In this case, it may be necessary to bring in evaluation expertise from another part of the agency, another agency, or through an external provider.

No matter what the scale and scope of the community engagement program, it is important that the objectives of the program and how it is expected to function are clear. It is also important to clearly articulate the purpose of the evaluation and key evaluation questions. The steps outlined in this guide can assist in defining these for all types of community engagement programs.

4. Evaluation should, whenever possible, be a participatory activity
   

   The practice of evaluation has evolved from an emphasis on measurement and judgment undertaken by experts to a collaborative learning process aimed at improving activities as they are on-going. Whenever possible, evaluation design and implementation should involve representatives from the key stakeholder groups, including:

   • public sector staff who engage directly with the community
   • managers who make decisions about community engagement or use the information collected through community engagement
   • community participants and other stakeholders.

   The benefits of participatory evaluation are discussed further on page 17.

5. It is important to undertake an opportunities and risk assessment of all evaluation decisions
   

   One of the first steps in designing an evaluation is to carefully assess the context of the community engagement program and determine the opportunities and risks of different approaches to  evaluation (see tip box on page 11).
   
   Careful consideration needs to be given to evaluation design in situations where:

• community engagement is associated with government activities that are highly controversial
• problems are already known, or
• there are pre-existing tensions between the communities concerned and the government.

In particular, it is important to consider:

• how the activity of evaluation might raise expectations about improvements
• what performance criteria and indicators are politically, culturally and socially appropriate
• the time people may have available to participate in an evaluation
• the willingness of a community to participate in an evaluation.

However, the identification of risks should not be an excuse not to evaluate. Evaluations can almost always be modifi ed to avoid potential risks. For example, in situations where there are known problems and major improvements or policy changes are not possible, it is prudent to consider how to focus the evaluation process on learning and areas where changes might be possible.

Box 1 summarises the key elements of a good evaluation framework based on the principles above and the steps that will be reviewed in this guide.