Stakeholder engagement in evaluation of Value for Investment

ACER researchers Dr Kerry Elliott and Dr Hilary Hollingsworth joined Dr Julian King at the Australian Evaluation Society 2023 International Evaluation Conference to share their experiences from 2 projects: one focusing on school leadership and the other on teacher professional learning.  

Their presentation highlighted how 'participatory evaluation' can provide a more comprehensive understanding of programs' value. It also introduced the Value for Investment approach as a practical and participatory tool for guiding such evaluations, and engaging stakeholders in the process. 

The importance of participatory approaches in evaluation 

The presenters began by emphasising the importance of involving the right stakeholders and balancing 'voices' in the evaluation process, arguing it can lead to more valid, credible, and useful evaluations. While cost-benefit analysis is a common method for assessing value for money, the researchers argue Value for Investment provides greater opportunity to understand initiatives and their impact more deeply. This perspective aligns with the principles of the General Logic of Evaluation and New Zealand’s professional standards for ethical evaluation practice. 

Engaging stakeholders in the evaluation process 

The project clients recognised the limitations of using cost-benefit analysis alone to understand the value of the initiatives. To address this challenge, the researchers introduced new ways of working that required participants to engage actively in the evaluation process. They focused on building deep stakeholder input into various aspects of the evaluation, including key evaluation questions, context analysis, program theory, data gathering methods, and reporting formats. This produced agreed-upon frameworks to evaluate the evidence. 

The Value for Investment approach 

The researchers used the Value for Investment approach to bring clarity to their evaluations. Developed through Dr King's doctoral research, this approach helps make the evaluation process practical and intuitive, particularly for stakeholders who may not have been involved in such evaluations before. 

The Value for Investment approach involves an 8-step process: 

1. understanding the program and context, including posing clear evaluative questions
2. building a rubric or ‘prism’ by identifying performance criteria 
3. defining the standards for levels of performance
4. determining the mix of evidence needed
5. gathering credible evidence
6. analysing each stream of evidence
7. reaching evaluative judgments
8. preparing the evaluation report. 

This process allows for a holistic understanding of the program and context within which the program is implemented, and a shift to analytic thinking. 

Challenges and benefits of a participatory approach 

Drs King, Elliott and Hollingsworth acknowledge that a participatory approach can be challenging, as it requires time to build relationships and trust. Additionally, external factors such as political pressures or changes in management can hinder participatory evaluation efforts. Overcoming these challenges requires patience and persistence. 

The researchers consider evaluation as ‘a collaborative, social practice’ and aim to incorporate stakeholder input into every step, including the evaluation design and conduct. This inclusivity supports the credibility and validity of the findings and ensures that the evaluation will be useful and used by those involved. 

Conclusion 

King, Elliott and Hollingsworth describe their journey through participatory evaluation and the application of the Value for Investment approach as ‘transformative’. Involving stakeholders in the evaluation process enriched the quality of the evaluations and empowered those most affected by the programs. They emphasise that a relational approach to ‘value for money’ is positive and possible when stakeholders are actively engaged and included. 

Find out more: 

Read the full article, ‘Making meaning through co-constructed lenses’ by Julian King, Kerry Elliott and Hilary Hollingsworth.