Blog

I asked ChatGPT, the artificial intelligence tool a simple question on communication evaluation: “How to evaluate a communication campaign?“. The response is copied in below.

It is interesting in that covers many good practices of campaign evaluation, such as setting KPIs, carrying out a baseline, ongoing monitoring and learning from the results. But it mixes up methods and measures and does not address one important aspect – assessing contribution/attribution of any results found. Also it fails to distinguish between the different levels of results, from output to outcomes to impact. Although it does end with a nice statement that I fully agree with:
“Remember that the evaluation process should be ongoing and iterative. It’s not just about measuring the success of a single campaign but also about improving your overall communication efforts over time.“

My conclusion is that AI has not yet eliminated us evaluators!

Q. How to evaluation a communication campaign?
Answer from ChatGPT:
Evaluating a communication campaign is crucial to determine its effectiveness and make informed decisions for future campaigns. Here’s a step-by-step guide on how to evaluate a communication campaign:

Set Clear Objectives and Key Performance Indicators (KPIs):
Before launching a campaign, establish specific, measurable, and achievable objectives. Define KPIs that align with these objectives, such as increased website traffic, higher social media engagement, or improved brand awareness.

Pre-Campaign Benchmarking:
Gather baseline data related to your KPIs before starting the campaign. This will provide a reference point for measuring the campaign’s impact.

Real-time Monitoring:
Monitor the campaign’s progress in real-time. Track metrics such as website visits, social media mentions, email open rates, and any other relevant data. Tools like Google Analytics, social media analytics platforms, and email marketing software can be valuable for this purpose.

Audience Feedback:
Collect feedback from your target audience through surveys, focus groups, or social media comments. This helps gauge how well the campaign is resonating with the intended audience.

Media Analysis:
Evaluate the media coverage your campaign generates. Are there positive or negative mentions in the press? Analyze the sentiment and tone of media coverage to understand public perception.

ROI Analysis:
Calculate the return on investment (ROI) of your campaign. Compare the costs of running the campaign to the revenue or value it generated. This can include both quantitative (sales, revenue) and qualitative (brand perception) aspects.

Website and Social Media Analytics:
Analyze website traffic, bounce rates, conversion rates, and social media engagement metrics during the campaign period. Assess whether there’s a significant increase in these metrics compared to the pre-campaign period.

Content Performance:
Evaluate the performance of individual pieces of content (e.g., blog posts, videos, infographics). Determine which content formats and topics resonated most with your audience.

Message Recall and Recognition:
Test the audience’s recall and recognition of your campaign’s key messages. Conduct surveys or interviews to assess whether the intended messages are getting across effectively.

Competitive Analysis:
Compare your campaign’s performance with that of your competitors. Analyze how your campaign stands out and whether it’s achieving its objectives in comparison.

Post-Campaign Benchmarking:
After the campaign ends, gather data similar to what you collected before the campaign. This will allow you to compare the post-campaign results to the pre-campaign benchmarks.

Report and Analysis:
Compile all the data and insights into a comprehensive report. Summarize the key findings, including successes, challenges, and areas for improvement.

Learn and Adjust:
Use the insights gained from the evaluation to inform future campaigns. Adjust your strategies, messaging, and targeting based on what worked and what didn’t.

Iterate and Optimize:
Apply the lessons learned from the evaluation to refine your communication strategies continuously. Implement improvements to maximize the effectiveness of future campaigns.

Share Results Internally and Externally:
Communicate the results of the evaluation both within your organization and, if relevant, to external stakeholders. Transparency can build trust and help secure support for future campaigns.

Remember that the evaluation process should be ongoing and iterative. It’s not just about measuring the success of a single campaign but also about improving your overall communication efforts over time.

Here is an interesting post from the Better Evaluation blog on writing evaluation reports, with their key points summarised as follows:

• Get to the point

• Develop the best structure for your report
• Report strong findings
• Write for your users
• Be aware of how the evaluation’s scope can affect its useability
• Allocate time for writing, editing, and quality assurance processes

In summary, the post highlights as key: “Evaluators need to communicate better and foster the utilisation of evaluation findings through clear and engaging writing“.

Often it’s useful in an evaluation during data collection to define or check what the given intervention sought to achieve in terms of outcomes – both intended and unintended. One approach that has become popular is outcome monitoring or harvesting. The World Bank has produced a very useful guide in this area that provides practical guidance – do take a look!

There has been a lot of talk about “nudge theory”- basically people being guided, encouraged and nudged towards the right decision, rather than being told – and now the evaluation unit of the UN agency, WIPO has produced a guide on how nudge theory can apply to evaluation – interesting reading!

I am happy to announce a series of e-learning modules I designed with em4 for social impact organisations that would like to learn more about monitoring and evaluation:

• Module 1: Monitoring and evaluation(M&E): The key concepts of M&E in a social impact setting

• Module 2: Integration of monitoring and evaluation within the project cycle

• Module 3: Monitoring and evaluation in organisational operations, procedures and systems

For volunteers, staff and board members working towards social impact, whether it be in health, environment, community initiatives or other focuses, these three modules will be of great interest and use. Each module is stand-alone and self-paced; taking about an hour per module to complete.

Learn more about the e-learning modules>>

The Institute for Public Relations has produced an excellent new guide: “The communicators guide to research, analysis and evaluation“.

The guide highlights:

“The importance of communication research, analysis, and evaluation continues to grow due to the pace of change in the marketplace and the corresponding need to make smarter and better decisions at the speed of business. The ability for an organization to successfully transform during this time of dynamic change requires the active leadership of the Chief Communication Officer and the application of talent and technology to perform the work, measure its impact, and improve at each stage of the public relations process.”

View the guide here>>

I am very happy to be able to share a report I co-researched and authored with my colleague Dr Sarah Grosso for the World Health Organization on “Effective risk communication for environment and health” (pdf).

The report provides an overview of the latest trends, theories and concepts of risk communication for environment and health (EH), and key challenges and good practices are identified. The report’s findings are complemented by three cases studies: promoting indoor air quality in schools in Hungary; water contamination in the Veneto region, Italy; and heat health action in Styria, Austria.

View the report (pdf)>>

As part of AMEC’s Measurement Month, I featured on the Smoke Signals Podcast with Paul Cheal, discussing about the practices and challenges of evaluating communications.

Here is an interesting post summarizing a 2021 roundtable discussion from the US-based Institute of Public Relations on “Are PR Pros [communicators] measuring the right things?”. Some highlights from the discussion:

Read the full post here>>

“Katie Paine (Paine Publishing): All the things that we have traditionally measured don’t seem as valuable today.–Long ago, the “Grammy of Communications” went to the person who got the most impressions and reported the most hits or clicks or whatever. That’s not going to happen today. I don’t know a lot of organizations that measure on that. Generally, they’re measuring on some kind of change, whether it’s increased awareness, or increased traffic, or saving lives.“

“Elizabeth Rector (Cisco): I think it’s an “and.” I think some of the standard metrics aren’t going away, it’s just multiplied, and then we are asking how they come together. That’s blowing up in my world. And finding comms people…finding people who can have that mindset and that blend, because most of my kind of comms insights people have been very much traditional comms, and some of the outcome, like the reputation and brand and all of that. But as we’re getting into thought leadership, and driving to the website, and views, and paid media that’s all now under comms’ umbrella.”

“Joseph Czabovsky (University of North Carolina): I don’t think the fundamentals have changed. As you were saying Katie, it’s still the different elements of funnel, or whether we’re doing awareness, attitude, or behavior, essentially. I think that’s still the core of what we do and what we measure“.

Read the full post here>>

Here is an interesting case study presenting an evaluation of a European-wide smoking cessation campaign (in other words, trying to get people to stop smoking) that I was involved with. Some interesting findings on campaigns and results in this challenging area.