The Owl RE team writes and speaks regularly on evaluation and related areas. Our philosophy is to share as much as possible our know-how and beliefs on the exciting and topsy-turvy world of evaluation. We also encourage our clients to share their evaluation reports. On this page you will find all our available resources. Please feel free to share and use.
Here is an excellent article from Prof. Jim Macnamara on communication evaluation at WHO. He was part of a team which worked with WHO over a number of years to improve their communication evaluation, also over the period of the COVID-19 pandemic. it provides useful insights for anyone interested in communication evaluation and the challenges (and solutions) in this area for large global organisations today.
View the article here>>
When thinking about communicating evaluation findings, an often overlooked medium is to use video, as Better Evaluation says “When produced well, videos provide an excellent means to convey messages coming out of an evaluation.”
Creating a short video can be an excellent way to communicate key findings to audiences. But they do require some preparation and planning, notably:
– Think about what will be the topic and story of the video – what messages of the evaluation findings will the video focus on?
– A video is a great opportunity for stakeholders and beneficiaries to speak up and have their voices heard – of course you need their consent to use their images / voices.
– Think of what images can illustrate well the intervention being evaluated – as it is a visual medium! For example, if it is a community-based activity, film the activity and the community.
-These days you don’t need a big budget and expensive equipment – you can film a lot on a smart phone – but if you are not skilled in editing, a small budget for editing would be needed.
Also important is that the evaluation commissioner must be onboard – as ideally the evaluation team would do some filming during data collection. The challenge for the evaluation team is that is rarely in an evaluation ToR to do a video – the team has to be bold and propose it!
I’ve had the chance to use video several times for evaluation and would always like to do more.
Here is an example where we had a simple approach; filming testimonies of staff for an evaluation of an intervention for the INGO WaterAid – then weaving them together with a narrative – all under 6 minutes:
And in another example, we used graphics, interviews and testimonies for a climate change evaluation to present a comprehensive overview of the evaluation findings:
And with nearly a 1000 views on YouTube, it’s certainly more than read the report!
Here is a more professionally produced example from the Global Environment Facility; a great summary of the evaluation findings in under 5 minutes:
For further tips and hints for using video in evaluation, read this post from Better Evaluation.
An excellent post from Riccardo Polastro on how to foster a culture of evaluation within organizations.
He highlights five recommended practices:
1. Embed evaluation and build trust to support learning and decision making
2. Co-create a shared vision for evaluation
3. Invest in resources to support independent evaluation
4. Strengthen capacity for credible and useful evaluations
5. Align evaluations with user needs and intended impact
Read the full post here.
I think the last recommended practice is key; as an evaluator we often see evaluations commissioned that no one really knows what are their purposes and intended use; this only discredits evaluation and makes staff think twice next time they are asked to cooperate for an evaluation.
The European Commission has an interesting 1-pager from 2022 on possible indicators for communication activities- splitting indicators by type of activities (e.g. websites, events, social media, etc.) – it’s a handy guide if you are looking for indicators for specific communication activities.
I came across this guide on evaluating “convening” – events, conferences, seminars, etc from the TCC Group; really worth reading if you are interested to learn further about evaluation in this field.
The Equitable Communications Guide is a great new resource for evaluators. Developed by Innovation Network, the Equitable Communications Guide is designed for evaluators in the social sector, but has relevant lessons for anyone looking to improve their communications! The guide explores how to communicate equitably, center the experiences of others, and convey the meaning behind key messages. Download the free guide here: https://innonet.org/news-insights/resources/equitable-communications-guide/
November is communication evaluation month – AMEC, the professional association is hosting a range of free events, open to all, see the full agenda here.
I asked ChatGPT, the artificial intelligence tool a simple question on communication evaluation: “How to evaluate a communication campaign?“. The response is copied in below.
It is interesting in that covers many good practices of campaign evaluation, such as setting KPIs, carrying out a baseline, ongoing monitoring and learning from the results. But it mixes up methods and measures and does not address one important aspect – assessing contribution/attribution of any results found. Also it fails to distinguish between the different levels of results, from output to outcomes to impact. Although it does end with a nice statement that I fully agree with:
“Remember that the evaluation process should be ongoing and iterative. It’s not just about measuring the success of a single campaign but also about improving your overall communication efforts over time.“
My conclusion is that AI has not yet eliminated us evaluators!
Q. How to evaluation a communication campaign?
Answer from ChatGPT:
Evaluating a communication campaign is crucial to determine its effectiveness and make informed decisions for future campaigns. Here’s a step-by-step guide on how to evaluate a communication campaign:
Set Clear Objectives and Key Performance Indicators (KPIs):
Before launching a campaign, establish specific, measurable, and achievable objectives. Define KPIs that align with these objectives, such as increased website traffic, higher social media engagement, or improved brand awareness.
Pre-Campaign Benchmarking:
Gather baseline data related to your KPIs before starting the campaign. This will provide a reference point for measuring the campaign’s impact.
Real-time Monitoring:
Monitor the campaign’s progress in real-time. Track metrics such as website visits, social media mentions, email open rates, and any other relevant data. Tools like Google Analytics, social media analytics platforms, and email marketing software can be valuable for this purpose.
Audience Feedback:
Collect feedback from your target audience through surveys, focus groups, or social media comments. This helps gauge how well the campaign is resonating with the intended audience.
Media Analysis:
Evaluate the media coverage your campaign generates. Are there positive or negative mentions in the press? Analyze the sentiment and tone of media coverage to understand public perception.
ROI Analysis:
Calculate the return on investment (ROI) of your campaign. Compare the costs of running the campaign to the revenue or value it generated. This can include both quantitative (sales, revenue) and qualitative (brand perception) aspects.
Website and Social Media Analytics:
Analyze website traffic, bounce rates, conversion rates, and social media engagement metrics during the campaign period. Assess whether there’s a significant increase in these metrics compared to the pre-campaign period.
Content Performance:
Evaluate the performance of individual pieces of content (e.g., blog posts, videos, infographics). Determine which content formats and topics resonated most with your audience.
Message Recall and Recognition:
Test the audience’s recall and recognition of your campaign’s key messages. Conduct surveys or interviews to assess whether the intended messages are getting across effectively.
Competitive Analysis:
Compare your campaign’s performance with that of your competitors. Analyze how your campaign stands out and whether it’s achieving its objectives in comparison.
Post-Campaign Benchmarking:
After the campaign ends, gather data similar to what you collected before the campaign. This will allow you to compare the post-campaign results to the pre-campaign benchmarks.
Report and Analysis:
Compile all the data and insights into a comprehensive report. Summarize the key findings, including successes, challenges, and areas for improvement.
Learn and Adjust:
Use the insights gained from the evaluation to inform future campaigns. Adjust your strategies, messaging, and targeting based on what worked and what didn’t.
Iterate and Optimize:
Apply the lessons learned from the evaluation to refine your communication strategies continuously. Implement improvements to maximize the effectiveness of future campaigns.
Share Results Internally and Externally:
Communicate the results of the evaluation both within your organization and, if relevant, to external stakeholders. Transparency can build trust and help secure support for future campaigns.
Remember that the evaluation process should be ongoing and iterative. It’s not just about measuring the success of a single campaign but also about improving your overall communication efforts over time.
Here is an interesting post from the Better Evaluation blog on writing evaluation reports, with their key points summarised as follows:
In summary, the post highlights as key: “Evaluators need to communicate better and foster the utilisation of evaluation findings through clear and engaging writing“.
Often it’s useful in an evaluation during data collection to define or check what the given intervention sought to achieve in terms of outcomes – both intended and unintended. One approach that has become popular is outcome monitoring or harvesting. The World Bank has produced a very useful guide in this area that provides practical guidance – do take a look!
