Marketing Mix Modelling & Incremental Experiments – How to Test New Marketing Activity

An Incremental Marketing Experiment Method

So, if the variance in your data is not rich enough, your marketing scientist should tell you that you need to conduct an experiment to understand the value of a specific channel or activity in your marketing mix.

Say we would like to understand if adding a new channel into the mix adds value, but profit margins & targets are super tight and we can’t be taking any risks. Here is one way to run an incrementality test, keeping the methods and example as basic as possible:

1. Estimate the smallest budget amount to invest which can make an impact on all sales

We first estimate a model of diminishing returns to spending on all marketing.

We then determine what the smallest reduction in spend would be to feel an impact on our top line sales. We predict that if we drop spend by that amount then sales will fall but efficiency (ROAS or CAC) will improve. We are going to see if spending this amount into some new activity instead will be as effective. (Note in the chart that at the new spend level, the upper bound (of our confidence interval) of predicted new customers is below the lower bound for predicted customers at the current spend level)

2. Estimate how many days of results we need to collect to get a good read

We must determine what the minimum number of days we should spend at this new level should be, so that the aggregated results from our experiment are likely not due to random variance. (get a large enough sample).

We also make an assumption about “ad stock”. We know that spend on day 1 likely does not just lead to sales on day 1, but instead there is a time lag between hitting prospective customers with advertising and them going on to make a purchase. We need an assumption on our existing spend (it may be possible to estimate this using historic data but if not we can make a conservative guess). We also need one for the new activity we want to test.

3. The experiment plan

We first drop spend by the given amount for the recommended time period. If sales and CAC land where we predicted, this is a very strong validation of our model! Any efficiency gains we made from spending less and having better costs can also be reinvested into our new test activity, or this can go towards the next test.

Now we are here, we can invest the amount of budget we cut into our activity we are testing. If sales then do not pick up, we can infer the activity we tested does not add enough value to us. 

If sales go back to where they were, great – the activity has added sales and we can estimate an incremental return on ad spend for this new activity (in this case at a similar incremental ROAS to our existing marketing). If results are at least this good then we can take a read on how results look in the ad platform or our website tracking data, and create benchmarks for what good performance looks like for the new activity. 

If we have even better incremental gains then we have generated genuine, efficient scaling of the business. We now need to increase spend as we are ahead of targets. We can choose to continue to scale our new activity, or increase spend in our existing marketing activities back to the pre-experiment level (will the incremental gain be the same, or is there an overarching market saturation at play?).

When judging success, we can keep track of a rolling average of new customer volume, based on the number of days we estimated are needed to get a good read, and map out the threshold for success:

This is a great way to work when under strict profitability goals. In the worst case scenario, we lose some volume of new customers in return for better efficiency, whilst our profit targets stay intact. It also works well to explain the concept! We could just shuffle our marketing budget and not go into a period of lower spend. If we have a specific target we want to hit and some funding to leverage we may instead want to model how much incremental value the new activity needs to bring in order to hit our goals, and then plan the parameters for our experiment accordingly. It is important, though, that different businesses have different levels of variance in daily sales and different diminishing returns on marketing spend, and we need to take account for that when planning experiments.

In Summary:

Taking an experimental approach to media planning is something everyone can start doing more of. Doing so generates the data you need to understand marketing effectiveness. Whether you are running some statistics or not, engaging in a push and pull of budget into different strategies is a great way to achieve growth.

Just modelling how growth scales with marketing spend whilst accounting for external factors is a problem which needs to be solved for a good granular MMM system to work, and it isn’t something we would feel comfortable allowing a machine to completely automate (at this point in time at least 🤖). 

Solid results from experiments inform benchmarks on what our KPIs for ad platform or Google Analytics results should be. Sense check all of this with incremental lift experiments and then we have a solid, modern triangulation method for digital marketing attribution.