CCTMB #001 Can Bayesian Methods Help Us Learn More from Studies Within a Trial (SWATs)?

By Suzie Cro, Associate Professor in Medical Statistics and Clinical Trials

Randomised controlled trials (RCTs) are the gold standard for testing new medical treatments. But what about testing how we run trials?

That’s where a Study Within A Trial (SWAT) comes in. A SWAT is a small study embedded inside a larger host trial to test ways of improving trial processes, such as recruitment or retention strategies.

SWATs are increasingly popular. But there’s a challenge, they are often small, with the sample size dictated by the main host trial. And small SWATs analysed in the usual frequentist way can leave us with frustratingly inconclusive answers.

Our recent proof-of-concept study explored whether Bayesian methods combined with a tool called ACCEPT analysis can help us extract more meaningful insights from SWATs.

Why is this needed?

Most SWATs are analysed using traditional frequentist statistics. This typically produces alongside an estimate of the intervention effect:

-        A 95% confidence interval

-        A p-value

-        A binary “significant” or “not significant” conclusion

But p-values can be difficult to interpret, especially for non-statisticians. Furthermore, when SWATs are small (which they often are), they may lack the statistical power to detect meaningful effects. That means potentially useful trial process strategies might be dismissed simply because the SWAT wasn’t large enough.

What did we do?

We re-analysed two SWATs that tested whether adding a video to traditional trial patient information improved trial recruitment:

1)     An earlier SWAT by Du et al (2009), which suggested a possible benefit of a video but was statistically non-significant.

2)     A later SWAT by Mattock et al (2020) found that recruitment was actually lower in the video group.

Instead of focusing on p-values, we used Bayesian methods, which allow us to answer more intuitive questions like:

“What is the probability this intervention actually works?”

Bayesian methods also allow us to combine current data with evidence from previous similar studies, express uncertainty in terms of probabilities and readily explore how likely an intervention is to be effective at different meaningful thresholds.

What is ACCEPT analysis?

We also used ACCEPT (ACceptability Curve Estimation using Probability above Threshold) analysis.

Rather than asking simply “Does it work?”, ACCEPT asks:

“How likely is the intervention to achieve an effect of this size?”

It produces a curve showing the probability that the intervention effect exceeds various thresholds. See the example ACCEPT curve in the figure below for the primary Bayesian analysis of the Du et al 2009 SWAT. This avoids rigid “yes/no” conclusions and gives stakeholders a richer picture.

Different stakeholders might have different views about what size of improvement is worthwhile to adopt a trial process in practice. ACCEPT lets everyone see the probabilities and make informed judgments.

Figure 1 – Bayesian ACCEPT analysis for Du et al 2009. OR represents the odds of recruitment for the video intervention relative to standard patient information. The curve shows the probability that the intervention effect exceeds various OR thresholds.

What did we find?

For the earlier Du et al SWAT, the Bayesian analysis suggested an 86% probability that the video improved recruitment, even though the original frequentist analysis reported a non-significant result.

For the latter, Mattock et al SWAT, the Bayesian analysis showed an extremely low probability that the video was helpful. Even when incorporating evidence from earlier studies, the probability of benefit remained small.

So in both cases, Bayesian methods gave clear, intuitive probability statements and allowed previous evidence to be incorporated transparently.

Why does this matter?

SWATs are often small, underpowered and/or conducted in isolation. Traditionally, we wait years for enough SWATs to accumulate before conducting a meta-analysis.

Bayesian methods allow us to:

-        Incorporate prior evidence immediately

-        Update conclusions as new SWATs emerge

-        Quantify uncertainty more directly

-        Move beyond binary “significant / not significant” thinking

For trialists making practical decisions about whether to use a recruitment or other trial process strategy, probability-based summaries from Bayesian analyses may be more useful than p-values.

Looking ahead

We found no other published examples of SWATs using Bayesian analysis or ACCEPT curves. Yet these methods are particularly well-suited to the realities of SWAT research.

Greater use of Bayesian approaches could:

-        Strengthen the evidence base on how to conduct trials

-        Reduce wasted effort

-        Help us learn more from every embedded study

-        Make findings more accessible to non-statisticians

If we want trials to be more efficient, we believe it’s time to rethink how we analyse the studies that improve them.

Key takeaways

1. Bayesian methods can make SWATs, particularly small SWATs, more informative.

2. ACCEPT analysis shows how likely an intervention is to have a meaningful benefit for different thresholds of benefit.

3. Bayesian approaches can also combine past and new evidence to support better trial decisions overall.

Read the full paper here:  The use of Bayesian methods for the analysis of Studies Within A Trial: a proof-of-concept case study | Trials | Springer Nature Link