As marketers, we talk a lot about “optimizing spend,” but too often that means trimming what looks bad in a platform dashboard. In a recent roundtable with Rick Bruner and Professor John Chandler, we explored a more important question: how do we uncover the upside our current measurement can’t see?
From “Waste” to Hidden Upside
Rick shared a case that stuck with me: a Fortune 100 advertiser spending about $100 million a year on branded search. Platform data suggested it didn’t do much for online sales: classic “waste.”
But when Rick’s team ran a geo-randomized test across U.S. DMAs and tied exposure to total CRM sales, including phone orders, they found something very different:
- Around 80% of sales came by phone, not online.
- When the branded search ad appeared above the fold, it drove roughly 3% of all transactions.
For a $55 billion company, that 3% translated to roughly $1.5 billion in sales, put at risk in pursuit of $50 million in “savings.” That’s the core idea: bad or incomplete measurement can hide enormous upside. John noted that 25–50% incremental ROI from better experimentation is not unusual in his work.
Why Geo-Experiments Are Back
The old dream of perfect user-level measurement has run into reality: privacy changes, tracking gaps, and biased samples. The result is that Rick and John lean heavily on geo-experiments:
- Randomizing DMAs into test and control
- Measuring real outcomes (sales, CRM) instead of isolated platform events
This allows marketers and CFOs to compare channels on a true apples-to-apples basis across the whole market, not just the slice the platforms can see.
When true randomization isn’t possible, they turn to quasi-experiments like synthetic controls; essentially match-market tests “on statistical steroids.” For smaller advertisers, even an on/off test over time (“the Dilbert test”) can reveal meaningful lift.
Brand as a Performance Driver
Coming from search, I’ve long seen that strong brands get a discount in auction media: better quality scores, better CTRs, better response overall. That’s the brand halo, and it connects upper-funnel work directly to lower-funnel performance.
Rick’s point is that correctly measuring brand strength is more achievable than many assume, especially for big advertisers. A large share of their sales is baseline demand built over years, but the 2–5% incremental lift from current campaigns is still:
- Real
- Material in dollars
- Measurable via well-powered geo-experiments and media mix models (MMM)
John emphasized that, thanks to open-source tools, MMM that used to be bleeding-edge is now accessible even for mid-market advertisers.
AI, Incentives, and Who Gets to Measure
We also touched on how AI assistants and chatbots may be diverting some research behavior into “zero-click” journeys, making traditional signals like branded search volume less reliable as leading indicators.
A recurring theme was incentives. Agencies paid on a percentage of spend, internal channel teams, and the platforms themselves all have reasons to favor bigger budgets and flattering dashboards. Rick was clear that platforms shouldn’t be allowed to “grade their own homework” by running closed-loop MMM based only on their own data.
John, speaking as an educator, put it plainly: “grading your own homework” is not a recipe for accurate learning, even if it does produce great scores.
The alternative is independent, causal measurement, whether that sits in-house or with specialized partners, and a culture where experimentation is planned, not feared. One global CPG Rick works with now bakes experiments into annual planning, so tests are expected, not disruptive surprises.
Discussion points include:
- How a Fortune 100 branded search test surfaced $1.5B in at-risk revenue
- Why focusing on upside can be more valuable than chasing “waste”
- The role of geo-experiments and synthetic controls in modern measurement
- How the brand halo improves lower-funnel efficiency
- Using experiments + Bayesian MMM to get a truer view of incrementality
- The impact of AI-driven, zero-click journeys on traditional leading indicators
- Why platforms shouldn’t be the sole source of truth on their own performance
- How to bake experimentation into annual planning instead of bolting it on