Introduction to evaluation

What is evaluation?

Evaluation is a process of assessment used to generate information about the way in which an activity or program of activities is undertaken (process) and the results of the activity or program (outcomes).

Evaluation involves comparing aspects of process and outcomes to performance standards or expectations to judge the success of the activity or program, taking into consideration that external factors may also have had an affect.

The results of evaluation are used to inform and improve planning and decision-making about the activity or program, or future similar activities, and to report on practice. Evaluation is also used to develop understanding of the links between actions, context and outcomes in the activity or program.

This process can range from:

• a small scale evaluation by those conducting and/or participating in the activity or program based on a few key evaluation questions and simple data collection methods such as participant feedback questionnaires, to
• a large scale evaluation study conducted by external evaluators utilising multiple methods based on a detailed evaluation framework and project plan.

As is discussed further in this guide, the scale and scope of an evaluation should reflect the purpose, audience and the scale and significance of the activity or program to be evaluated.

How can evaluation be used for community engagement activities?

The evaluation of community engagement activities can take place at the project level such as when a one-off community engagement activity is conducted. It can also take place at the program level with the evaluation of a sequence of community engagement activities that contribute to the development or delivery of a government policy, program or service.

In this guide, the evaluation of community engagement is discussed in terms of ‘community engagement programs’, in recognition of the fact that most community engagement involves a set of linked activities, rather than one-off stand-alone activities. Nonetheless, the steps discussed can be used for the evaluation of single activities.

The types of activities discussed include the spectrum of community engagement practice ranging from information provision (for example, the production and distribution of a brochure on a government initiative), to consultation (for example, a public meeting on a new project), to active participation (for example, a deliberative process in which citizens are invited to contribute to the planning of a new initiative).

The evaluation of community engagement activities can focus on short, medium and/or longer-term outcomes¹. However, the longerterm the outcome, the more likely it is that it will have been affected
by factors external to the community engagement activities being evaluated and the more challenging evaluation becomes.

This guide has a major focus on evaluating the quality of the process of community engagement and the short to medium-term outcomes of community engagement.

However, the evaluation of the longer-term outcomes of community engagement, while more challenging, is also important. The guide provides a brief discussion of some of the issues related to evaluating the longer-term outcomes of community engagement in Part 4.

The benefits of evaluating community engagement

Effective evaluation of community engagement activities can provide considerable benefits to Queensland Government agencies.

Benefits include:

• improving practice by identifying and articulating lessons and achievements
• contributing to setting best-practice standards and benchmarks
• helping to build an evidence base for innovative approaches to community engagement
• contributing to engagement capability development by providing feedback on performance
• presenting opportunities for further citizen involvement in the evaluation process
• building support for client and community involvement in government planning and decision-making by providing evidence of how effective engagement works.

The purpose of evaluating community engagement in government

There are three key roles or purposes for undertaking community engagement evaluation in government, which are outlined in Table 1 and also illustrated diagrammatically in Figure 1. These roles are not mutually exclusive, rather a good evaluation process should attempt to recognise and provide for all three functions.

Figure 1. The roles of community engagement evaluation

What does evaluation involve?

There are many different methods and approaches to evaluation, however, all involve three main activities:

• developing an evaluation framework and data collection tools
• collecting and analysing data
• interpreting, sharing, reporting and responding to results.

The scale and scope of these activities will vary depending on the scale and scope of the community engagement program and the purpose of the evaluation. At one end of the scale, evaluation may involve developing, administering, analysing and reporting on a feedback questionnaire of participants in a  consultation workshop based on a few key evaluation questions. At the other, evaluation may involve a three year multi-method evaluation of a major community engagement program involving several community engagement activities.

Managers should expect to budget 2%-10% of the overall community engagement project budget for an effective evaluation.

Larger evaluations may also require setting up new or concurrent project management arrangements including:

• an evaluation steering committee, which includes key government and community representatives, to direct the design of the evaluation framework and the interpretation of the results
• project plans for data collection, analysis and reporting, including timelines, deadlines and budgets
• project and tender briefs for outside consultants or contractors
• systems for reporting the results of the evaluations – to internal and external stakeholders and into knowledge bases
• systems for the on-going review of projects, programs or policies.

Part 2 of this resource provides guidance on the fi rst of the three activities involved in evaluation – developing an evaluation framework and data collection tools. It draws from and adapts several different approaches to and methods for evaluation taken from a variety of disciplines to present a suggested methodology for developing an evaluation framework for community engagement. The methodology presented is based on a ‘program logic’ approach to evaluation.

Part 3 of the guide will briefly discuss issues related to interpreting, sharing, reporting and responding to results.

Part 4 of the guide will discuss the more complex issues with evaluating the longer-term outcomes of community engagement. This section is important for those public managers and offi cers involved in evaluating large-scale community engagement programs or divisional or departmental-level community engagement strategies.

Detailed guidance on collecting and analysing data is not provided, however, a list of resources is provided in Part 5 - References and Further Resources.