Acting on MMM Insights: Re-activating Tik Tok for Just Bee Honey

The Approach

Year on year, the overall blended CAC for Just Bee was running higher than expected. The most obvious difference was that Tik Tok ads had been live the previous year. In-platform results did not look good, but overall the business was healthier when the ads were live.

Research shows that many social media platform users are influenced to buy products even if they never engage with the content. Without engagements, ad platforms and website analytics simply cannot track a lot of conversions.

Most attribution platforms and MMM providers cannot account for how the same level of spend produces different returns at different times of the year. Our bespoke MMM analysis is designed to solve this.

Our MMM analysis identified TikTok as undervalued in the media mix, so we reactivated it, and measured the impact.

The Method

Marketing Mix Modelling (How we model)

We used our own custom modelling using OLS. The advantage of our in-house process is that we can model interaction effects, and the models are calculated quickly, enabling us to explore how our models respond to different variable selections and back-testing.

We firmly believe that understanding how to validate models, and knowing how to select what variables to include in the model and how, are a lot more important than what algorithm or package you use to make a Marketing Mix Model. Our marketing science team’s background is in Life course epidemiology, so we are highly experienced in modelling to explain cause and effect.

Causal Validation

We were confident in the causal validity of our model prior to re-launching Tik Tok because:

1. We did not have inter-correlated variables in our model.
2. Our model reflected well how we believe things work in reality, where marketing budgets can be scaled during periods of high demand.
3. We demonstrated strong predictive validity to a a period in time which our model was blinded to, and during which the marketing budget mix changed significantly.

Intercorrelation

One challenge we encountered was that Meta and Google spends were highly intercorrelated. Trying to model them separately produced multiple viable models with conflicting views on each channel’s performance (including Tik Tok).

To explain this further: when we “fit” such a model, we are, in one way or another, iterating through different values we could have for each channel’s saturation shape, ad stock and overall effect. The result of all these processes could lead to one very clear answer, or there may be several different settings, which are very comparable in terms of how well they predict.

Most SaaS MMM products gloss over or straight up ignore these issues, and present the model which gives the best fit to the data, or heavily influence the modelling process with prior assumptions about how performant they expect marketing channels to be, and what sort of speed of diminishing returns they expect. Calibrating models with experiment results also does not make this issue disappear.

Rather than defaulting to the model with the best-looking fit, we grouped Meta and Google together, and treated them as one channel.

This resolved the intercorrelation problem and produced consistent findings across all models and algorithms. It is a more valid approach than forcing granular channel separation where the data does not support it. And since our main goal was to measure what Tik Tok ads to the mix, this model is preferable to one which tries to incorporate more marketing channel lines.

A model that fits historical data well does not guarantee it has the channel-level detail right. Say we have a bunch of models with different views on the channel mix, which all have a good fit to our training data – If the media mix in the hold out period is the same as before, all competing models will look accurate. Change the budget mix, and you find out quickly which model was correct.

We were able to show predictive validity to periods where budgets changed, giving us strong confidence to take action based on our models.

Seasonality as an Interaction Effect

Our bespoke MMM models the interplay between marketing spend and external effects, allowing us to simulate scenarios and forecast outcomes with confidence. What that means is, that in our models the return from a given budget change is dependent on the time of year, temperature, or some other important effects.

To measure the true impact of ad spend on new customer acquisition for Just Bee, we modelled seasonality as a moderating interaction effect: the expected return from a given level of spend is amplified when underlying demand is stronger. Modelling it this way significantly improved forecast accuracy and gave us a clear basis for adjusting acquisition targets and budgets throughout the year.

The Results

We measured performance across the period immediately after TikTok was re-activated, before the Starter Pack launched. We calculated two counterfactuals:

Given our TikTok spend, what did we predict versus what actually happened?

If we had not invested in TikTok, what would we have expected?

Overall, we estimated TikTok’s CPA was 60% lower than what was reported in-platform. Our modelling uncovered the true value, and we confirmed this with action.

Get in touch to find out how we can do this for your brand.