Bayesian Credibility Analysis
(an online calculator)

Introduction:

This page allows the credibility of a clinical trial finding to be assessed in the light of current knowledge.

Many clinical trials produce "statistically significant" evidence for the efficacy of some new therapy. However, statistical significance is a notoriously poor indicator of the credibility of a finding - that is, the extent to which it provides convincing evidence for efficacy - as it fails to take full account of the size of the trial, or of pre-existing insights.

This page takes the outcome of a given clinical trial - as summarised by the published Odds Ratio (OR) together with its 95 % confidence interval (CI) - and uses a Bayesian method due to Matthews (2001) to determine the credibility of the finding in the light of what is already known. Specifically, it shows whether the new finding, when combined with existing knowledge, can be taken to have credibly demonstrated efficacy at the 95% confidence level. Such a finding is said to be credible at the 95% confidence level.

For a given result to meet this standard, the prior evidence for efficacy must exceed a specific level; this level is captured by the Critical Odds Ratio (COR), the quantity calculated by this page. The assessment of a given clinical trial result can then proceed as follows:

  1. If the stated 95% confidence interval excludes an OR of 1.00 (corresponding to no effect), the result can be deemed statistically significant at the 95% level;
  2. If ORs at least as impressive as that indicated by the COR can be justified by existing knowledge, the result can also be deemed credible at the 95% level

The COR produced by this page reflects both the effect-size and the statistical power of the clinical trial under study. Thus trials involving large numbers of patients and/or large effect sizes will typically produce undemanding CORs, which are easily justified on the basis of current knowledge. On the other hand, trials involving relatively few patients, and/or small effect sizes will - because they possess relatively little evidential weight - produce CORs that demand a substantial amount of prior evidence before the new result can be deemed credible.

Example: The Grampian Region Early Anistreplase (GREAT) study (1992) found that patients with suspected acute myocardial infarction had lower 3-month mortality if given anistreplase early. The finding was summarised as an OR of 0.47, with a 95% CI of (0.23, 0.97). As the latter excludes an OR of 1.00, the result is statistically significant at the 95% level. However, entering this 95% CI into the window below leads to a COR of 0.087. Thus the result can be deemed credible at the 95% level only if ORs for mortality even more impressive (i.e. lower) than around 0.09 can be justified in the light of current knowledge about the effectiveness of anistreplase. Pocock and Spiegelhalter (1992) found that extant knowledge indicated that ORs below 0.4 were implausible. Thus ORs below the calculated COR level of 0.087 are not plausible, and the findings of the GREAT study may be deemed statistically significant but not credible at the 95% level. Subsequent experience has borne out this assessment.

NB: The CRO is based on the equipoise assumption of ethical randomised clinical trials (RCTs), according to which patients allocated to either arm of the trial are deemed equally likely to benefit. The CRO can thus be used to assess the credibility of meta-analyses of RCTs, as well as with individual trials. It can also be used to assess evidence from observational studies using the concept of the "fair minded sceptic", which implies a prior distribution identical to that based on the equipoise assumption.

Online Calculator for Bayesian Credibility Assessment:

1. Enter the published 95% confidence interval for the odds ratio (OR) of the clinical trial result:
Lower Limit = , Upper Limit =

2. Click this button:

3. Results ("Critical Odds Ratio"):

If ORs less than 1.0 are indicative of efficacy, then this trial is credible only if ORs less than are already plausible in the light of existing knowledge.

If ORs greater than 1.0 are indicative of efficacy, then this trial is credible only if ORs greater than are already plausible in the light of existing knowledge.

Note 1: Clinical trial findings that fail to produce evidence of efficacy at the 95% confidence level (i.e. whose results are "not statistically significant") cannot be credible at the 95% level. Entering the confidence limits for such findings will produce the result "NaN" ("Not a Number"). If one of the confidence limits is exactly 1.0, then one of the results will be "Inf" ("Infinity").

Note 2: There's nothing special about the 95% confidence level. If you enter the 99% confidence interval, you will get the 99% critical odds ratio.

References:

Matthews, R.A.J. 2001 Methods for assessing the credibility of clinical trial outcomes Drug Information Journal vol 35 (4) 1469-1478 Available online at:
http://bit.ly/CredibilityAnalysisPaper

Spiegelhalter, D.J. Abrams, K.R., Myles, J.P. Bayesian Approaches to Clinical Trials and Health-care Evaluation (Wiley: Chichester, 2004) Chapter 3.

GREAT Group 1992 Feasibility, safety and efficacy of domiciliary thrombolysis by general practitioners: Grampian Region Early Anistreplase Trial BMJ 305 548

Pocock, S. J., Spiegelhalter D. J. 1992 Letter (untitled) BMJ 305 1015

If you have questions about the Bayesian statistical concepts embodied in this method, send e-mail to Robert Matthews at rajm@physics.org

If you have questions about the operation of this web page, send e-mail to John C. Pezzullo at statpages.org@gmail.com

Return to the Interactive Statistics page or to the JCP Home Page