Blog

Systems thinking is a “buzz word” we often hear in evaluation. To help us understand it better, seasoned evaluator and thinker Bob Williams has produced a really helpful workbook, read more here from Bob:

“Systems thinking is a field, not a single methodology. Indeed, just like the evaluation field, the systems field is bursting with methodologies. Some of them are especially relevant to evaluation. Critical Systems Heuristics (CSH) is one of them, primarily because it focuses explicitly on the value and worth of an intervention. It helps identify what is (or was) the right thing to do, reveals the power that underpins that claim to rightness, and what makes, or made, an intervention legitimate and sustainable. It also helps identify appropriate evaluation criteria and questions. Over the years, it has had a powerful impact on my evaluation practice.

I’ve just published Values in Action: A Critical Systems Workbook, which takes you through a process of learning how to use CSH. This includes the design and analysis of evaluation, determining evaluation criteria and evaluation questions, stakeholder analyses and the construction and investigation of Theories of Change.

It is donationware. In other words, you can download the 120-page workbook (complete with exercises) for free or donate to the cost of design and production.“

You can download the workbook here>>

Integrating communications into the evaluation has been an interest of mine since I left my work as a communicator to take on the topsy-turvy world of evaluation – and I have recently contributed a chapter to the Evaluation Handbook on this subject: “Communicating with Interest Holders“.

I’ve also summarised the main points of this chapter in this presentation.

The Global Communication Report from USC Annenberg has been published. An annual global survey of communication professionals, normally the report has a strong focus on communication measurement and evaluation. This year’s edition “Mind the Gap” only has one mention on measurement – it found that 24% of communication professionals thought that the ability to measure the impact of public relations will greatly improve because of AI (the smaller numbers in the chart represent demographic splits from Millennials to Boomers).

That’s only one quarter of communicators – personally I think it will improve our ability to measure the impact of communications – perhaps not “greatly” though – that remains to be proved!

Download the report here (download is free but you need to register)

Here is an excellent article from Prof. Jim Macnamara on communication evaluation at WHO. He was part of a team which worked with WHO over a number of years to improve their communication evaluation, also over the period of the COVID-19 pandemic. it provides useful insights for anyone interested in communication evaluation and the challenges (and solutions) in this area for large global organisations today.
View the article here>>

When thinking about communicating evaluation findings, an often overlooked medium is to use video, as Better Evaluation says “When produced well, videos provide an excellent means to convey messages coming out of an evaluation.”

Creating a short video can be an excellent way to communicate key findings to audiences. But they do require some preparation and planning, notably:
– Think about what will be the topic and story of the video – what messages of the evaluation findings will the video focus on?
– A video is a great opportunity for stakeholders and beneficiaries to speak up and have their voices heard – of course you need their consent to use their images / voices.
– Think of what images can illustrate well the intervention being evaluated – as it is a visual medium! For example, if it is a community-based activity, film the activity and the community.
-These days you don’t need a big budget and expensive equipment – you can film a lot on a smart phone – but if you are not skilled in editing, a small budget for editing would be needed.

Also important is that the evaluation commissioner must be onboard – as ideally the evaluation team would do some filming during data collection. The challenge for the evaluation team is that is rarely in an evaluation ToR to do a video – the team has to be bold and propose it!

I’ve had the chance to use video several times for evaluation and would always like to do more.

Here is an example where we had a simple approach; filming testimonies of staff for an evaluation of an intervention for the INGO WaterAid – then weaving them together with a narrative – all under 6 minutes:

And in another example, we used graphics, interviews and testimonies for a climate change evaluation to present a comprehensive overview of the evaluation findings:

And with nearly a 1000 views on YouTube, it’s certainly more than read the report!

Here is a more professionally produced example from the Global Environment Facility; a great summary of the evaluation findings in under 5 minutes:

For further tips and hints for using video in evaluation, read this post from Better Evaluation.

An excellent post from Riccardo Polastro on how to foster a culture of evaluation within organizations.

He highlights five recommended practices:

1. Embed evaluation and build trust to support learning and decision making
2. Co-create a shared vision for evaluation
3. Invest in resources to support independent evaluation
4. Strengthen capacity for credible and useful evaluations
5. Align evaluations with user needs and intended impact
Read the full post here.

I think the last recommended practice is key; as an evaluator we often see evaluations commissioned that no one really knows what are their purposes and intended use; this only discredits evaluation and makes staff think twice next time they are asked to cooperate for an evaluation.

The European Commission has an interesting 1-pager from 2022 on possible indicators for communication activities- splitting indicators by type of activities (e.g. websites, events, social media, etc.) – it’s a handy guide if you are looking for indicators for specific communication activities.

View the 1-pager here>>

I came across this guide on evaluating “convening” – events, conferences, seminars, etc from the TCC Group; really worth reading if you are interested to learn further about evaluation in this field.

View the guide here (pdf) >>

The Equitable Communications Guide is a great new resource for evaluators. Developed by Innovation Network, the Equitable Communications Guide is designed for evaluators in the social sector, but has relevant lessons for anyone looking to improve their communications! The guide explores how to communicate equitably, center the experiences of others, and convey the meaning behind key messages. Download the free guide here: https://innonet.org/news-insights/resources/equitable-communications-guide/

November is communication evaluation month – AMEC, the professional association is hosting a range of free events, open to all, see the full agenda here.