There are many promises about AI. As a pitch consultant, I hear them from agencies, tech vendors, platforms, and others. But for marketers, how can you be sure it will actually deliver once you've made the commitment?
The latest Gartner Hype Cycle for Artificial Intelligence indicates that AI agents and AI-ready data are at the peak of inflated expectations, while technologies like generative AI are in the trough of disillusionment. Yet for marketers and their agencies, the promise of faster, better, and cheaper delivery at scale remains at an all-time high.
The critical question we all need to ask is, what are the key criteria or KPIs that we should measure and benchmark for AI solutions in marketing and advertising? Our view is that fewer are better, provided they are the right ones. Across a range of clients, we have been benchmarking agency AI solutions against four key criteria: productivity improvement, speed to market, scalability, and, of course, the quality of output and outcome.
Let’s examine each of these separately. The first is productivity. A recent MIT research report found that 95% of generative AI implementations within organisations had no demonstrable impact on the organisation's profit and loss (P&L) performance.
So, how can we genuinely assess if this promise is being honoured? Considering that this industry usually has paid for services based on who, what, and how much—such as resources, roles, hours of work, and related costs—without much or any regard for what is actually delivered.
This situation poses a challenge to marketers, their agencies, and procurement teams alike. When an agency claims savings of 50% or more, how can a marketer and procurement assess this if there's no apparent connection to what is being cut? The shift to output-based pricing has made most procurement rate benchmarking ineffective in offering meaningful insights in an AI-driven marketing landscape.
When setting a benchmark, the first step is establishing a solid baseline. Traditionally, many marketers rely on salary or hourly rates for each role. However, this approach needs to be expanded to include benchmarks for human resource levels, seniority mix, delivery times, and scale efficiencies, offering a comprehensive understanding of the financial cost model of the agency’s outputs, not just the costs of the inputs.
The same applies to speed to market. Every agency production person will tell you the client wanted it in the market yesterday or the day before. But with much of the time taken up by approval processes, how much faster will that AI automation be? And the point is not just for a one-off situation, but every time?
This requires establishing a strong baseline of delivery times for each stage of the advertising process to assess performance after AI implementation. Most focus is on one-off demand, with little attention to typical delivery times.
Next is the second most crucial benchmark – scale. The scalability of any process, whether in-house or outsourced to an agency, is essential. Our data, supported by Michael Farmer at Farmer & Co, shows that in the first two decades of this century, workload demands on advertising agencies increased from 250 pieces per brand to over 3,000, driven by the requirements of digital and social media.
Of course, the manual, hands-on tinkering approach of traditional agencies cannot scale to meet this demand without significant cost impacts. But how far can the AI-powered process scale, and what compromises are made along the way? Empty promises of scalability can be persuasive until the volume overwhelms the process due to the system's limitations.
Finally, it is the most critical metric, and possibly the most subjective, if not the hardest to quantify. Quality. There are currently two approaches here. The first is a subjective assessment of the quality of the outputs based on the buyers’ (marketers') level of satisfaction. However, increasingly there is an alternative measure of quality, which is the outcomes or the impact that the outputs have on achieving the marketing and business goals and performance.
Productivity and performance, efficiency and effectiveness, go hand in hand. There’s no point in improving one without the other. Suppose marketing and advertising are regarded as a cost of doing business. In that case, productivity and efficiency become more crucial than when they are viewed as a growth investment, where performance and effectiveness take priority.
Either way, the AI promise, which is both generative and agentic, emphasises increasing productivity and efficiency. Therefore, it is crucial to have a solid baseline of data to verify not only that you are delivering on the promise but also that you are doing so without sacrificing, and ideally enhancing, effectiveness and performance. Otherwise, the cost of implementation becomes just another marketing folly.
