Evaluating the longer-term outcomes of community engagement
Introduction
The involvement of clients and citizens in the business of government planning and decision-making is important for ensuring the responsiveness, quality and effectiveness of government programs and services. Therefore, community engagement can be considered as a process used to improve policies, programs and services or a ‘means to an end’.
However, ‘engagement’ is also an outcome characterized by effective information exchange, consultation and participation between government and the community that strengthens our system of democracy and often leads to building communities’ trust and confidence in government.
Furthermore, research evidence shows that community engagement also makes critical contributions to broader social outcomes such as building social capital and community capacity, and increasing people’s quality of life (see Box 4 for further information on these concepts).
These outcomes, which are commonly associated with developing ‘active citizens’ or ‘engaged communities’, may be thought of as community engagement as an ‘end-in-itself’.
Thus community engagement can be both an important part of the processes of government (a means to an end) and also an outcome (end-in-itself).
This section discusses some of the key decisions and challenges with measuring the longer-term outcomes of community engagement, in light of its role as both a process and an outcome. It is not intended to be a comprehensive tool for addressing these issues.
The section discusses these issues in relation to two key aspects of evaluating the longer-term outcomes of community engagement:
- identifying what the longer-term outcomes of a community engagement activity might be
- identifying how to measure the longer-term outcomes of a community engagement activity.
The example scenario introduced on page 13 is used to explore ways of approaching these decisions.
Identifying the longer-term outcomes of community engagement
The first aspect of evaluating the longer-term outcomes of a community engagement program is to identify what those outcomes should be. This step was introduced in Part 2 of the guide in terms of articulating the community engagement program’s Program Logic (Step 1).
In order to identify what the longer-term outcomes of a community engagement program might be, there are two major points to consider:
- is it a fundamental aspect of the overall policy/program/service delivery development or implementation process it is part of and should it be measured by the outcomes of that process (a meansto-an-end)? and/or
- is it likely to contribute signifi cantly to the building of community capacity and other social outcomes and should it be measured against these outcomes (an end-in-itself)?
These decisions involve two main questions:
- a practical question about what can realistically be expected of the community engagement activity being evaluated based on its scale and scope
- a conceptual or policy question around what longer-term social outcomes is the community engagement program or overall agency community engagement strategy trying to achieve.
In answer to the fi rst question, signifi cant longer-term outcomes can only be expected from community engagement programs of a reasonably large scale and scope in terms of the number of participants involved, the time and intensity of involvement and/or the degree of infl uence participants have over decisions and/or implementation.
To answer the second question think about the original objectives of the community engagement program – did it envisage that the community engagement would lead to broader social outcomes in the communities engaged? Also consider any policies or strategies your agency has developed around community engagement and the social outcomes for community engagement they describe.
Finally, reflect on what you consider the longer-term outcomes of the community engagement program should be. Conceptual frameworks that outline some of these outcomes include those related to social capital, community capacity and quality of life (see Box 4).
Measuring the longer-term outcomes of community engagement
The next aspect of evaluating the longer-term outcomes of a community engagement activity is to decide how the identified outcomes can be measured. This step was introduced in Part 2 of the guide in steps 3 and 4.
When deciding how to measure the longer-term outcomes of community engagement two common and interrelated challenges with measurement become more signifi cant. These are:
- finding appropriate measures or indicators for outcomes (Step 3)
- identifying appropriate data collection and analysis methods (Step 4).
Finding appropriate measures or indicators for longer-term outcomes
As discussed in Step 3 on page 23, the measurement of community engagement outcomes usually involves the use of indicators.
In the case of measuring the longer-term outcomes of community engagement, indicators can be used to measure both the longerterm policy/program/service delivery outcomes and the potential social outcomes, such as community capacity, which community engagement may contribute to. The diffi culty lies in finding measures that reliably ‘indicate’ that these outcomes have been achieved, taking into consideration what is possible in terms of data collection. This is made diffi cult by the lack of standard, agreed upon measuresof outcomes such as ‘successful policies’ or ‘increased community capacity’.
However, in the case of social outcomes such as social capital, there are a considerable number of resources that provide example indicators. For example, the Australian Bureau of Statistics’ information paper Measuring Social Capital: An Australian Framework and Indicators (2004). Other references are provided in the Resources and References section.
Identifying appropriate data collection and analysis methods
Step 4 in Part 2 discussed decisions related to identifying data collection and analysis methods. With measuring the longer-term outcomes of community engagement, data collection and analysis becomes more difficult because:
- the increasing diffi culties of establishing causality
- the time frame and resource requirements for data collection.
Community engagement generally operates within complex social and political systems where a number of different variables are interacting. Such variables include the nature of the communities and individuals concerned, and the participants’ exposure to other information and experiences during the time of the community engagement program.
When evaluating the longer-term outcomes of community engagement, it is often diffi cult to understand the degree to which the community engagement program, rather than other variables, infl uenced or caused any changes in the outcome indicators being measured. This is referred to as establishing causality (see discussion page 27).
One way that evaluators have tried to address this is through the use of experimental and quasi-experimental approaches which can take account of (often referred to as ‘control for’) as many of these outside influences as possible. However, these approaches can never fully control these influences in natural social settings.
In addition, there are significant limitations on the use of experimental and quasi-experimental approaches in the public sector. Firstly and most signifi cantly, true experimental approaches are often impractical in policy settings for ethical and political reasons. Secondly, these approaches often have signifi cant data collection requirements, including the need to collect baseline data, data from a comparable (control) setting and time series data. This level of data collection is often beyond the resources available for a project.
In many cases with longer-term outcomes, causality cannot be established in a reliable way. Instead, the evaluation of longer-term outcomes often relies on the measurement of perceptions of cause and effect. This requires the identifi cation of whose perceptions should be measured. Ideally, a full range of participants’ and stakeholders’ perceptions should be gathered through a mix of quantitative and qualitative methods. This is referred to as ‘triangulation’.
While this approach is less scientifi cally accurate it often can present a compelling and reasonably accurate picture or ‘story’ of change, including its perceived significance. These stories can often carry as much political weight as more scientifi cally rigorous results.
Ideally, with data collection around longer-term outcomes, there should be a mix of more scientifically rigorous quantitative methods and illustrative exploratory qualitative methods depending on the resources and skills available.
However, with whatever approach is chosen, the timeframe for data collection will often be lengthy. This is a significant challenge for government as the research timeframe often falls outside of normal budget cycles and must be considered when designing evaluation frameworks.
Example Scenario: The Koala Shire Enterprising Futures Program
Identifying the longer-term outcomes
In the example scenario, the community engagement program was fairly large in terms of its scale and scope because of the number of participants reached by the information provision and consultation components, and the degree of infl uence the participants in the partnership process had over the development and implementation of the program.
It would be reasonable to assume, based on the scale and scope of this program, that the community engagement would be a signifi cant contributor to the overall outcomes of the Enterprising Futures (EF) program. Therefore, it would be reasonable to evaluate the success of the community engagement in terms of the overall outcomes of the EF program.
Furthermore, given the level of involvement by many of the community members in the program, either as members of the program steering group or as community volunteers in the program, and the small size of the community, the program could be expected to result in signifi cant social outcomes.
This was reflected in the outcomes hierarchy presented on page 12, in which the longer-term outcomes of the program included:
- increasing the employment levels of young people in the community
- reducing incidents of anti-social behaviour among young people in the community
- trust and confi dence in government improves
- the community has an increased capacity to develop solutions to local problems.
Measuring the longer-term outcomes
The first two of the outcomes in the list above could be measured using traditional indicators of employment levels and crime. The third could be asked directly in a survey. The last requires ‘capacity’ to be defined more precisely.
For example, a common element of ‘community capacity’ is ‘perceptions of community effi cacy’. An indicator that has been developed to measure this is ‘the proportion of people who feel confident that people would work together to solve problems in the community’7.
Identifying appropriate data collection and analysis methods
In the Enterprising Futures example, the evaluation design process did not begin until after the program had begun. As a result, experimental or quasi-experimental approaches could not be used. There was also no option to collect before and after data for a time series; however, employment and crime statistics would have been collected regularly for the area.
For this scenario, the approach to evaluating longer-term outcomes would need to use existing data sources and qualitative approaches that examine perceptions of change in social indicators from the points of view of a range of participants.
This might involve interviews and focus groups with the range of participants in the program including: young people, the community volunteers, the business people who supported to program, the workshop facilitators and state and local government stakeholders.
These interviews and focus groups might ask the participants to discuss:
- how the program changed their knowledge, skills, attitudes, perceptions and actions
- why it did or did not result in changes to these elements
- how they perceive these changes have had fl ow-on effects in their businesses/ communities
- other changes they have observed in their businesses/communities that they believe have been influenced by the program.
