This page allows the credibility of a clinical trial finding to be assessed in the light of current knowledge.
Many clinical trials produce "statistically significant" evidence for the efficacy of some new therapy. However, statistical significance is a notoriously poor indicator of the credibility of a finding - that is, the extent to which it provides convincing evidence for efficacy - as it fails to take full account of the size of the trial, or of pre-existing insights.
This page takes the outcome of a given clinical trial - as summarised by the published Odds Ratio (OR) together with its 95 % confidence interval (CI) - and uses a Bayesian method due to Matthews (2001) to determine the credibility of the finding in the light of what is already known. Specifically, it shows whether the new finding, when combined with existing knowledge, can be taken to have credibly demonstrated efficacy at the 95% confidence level. Such a finding is said to be credible at the 95% confidence level.
For a given result to meet this standard, the prior evidence for efficacy must exceed a specific level; this level is captured by the Critical Odds Ratio (COR), the quantity calculated by this page. The assessment of a given clinical trial result can then proceed as follows:
The COR produced by this page reflects both the effect-size and the statistical power of the clinical trial under study. Thus trials involving large numbers of patients and/or large effect sizes will typically produce undemanding CORs, which are easily justified on the basis of current knowledge. On the other hand, trials involving relatively few patients, and/or small effect sizes will - because they possess relatively little evidential weight - produce CORs that demand a substantial amount of prior evidence before the new result can be deemed credible.
Example: The Grampian Region Early Anistreplase (GREAT) study (1992) found that patients with suspected acute myocardial infarction had lower 3-month mortality if given anistreplase early. The finding was summarised as an OR of 0.47, with a 95% CI of (0.23, 0.97). As the latter excludes an OR of 1.00, the result is statistically significant at the 95% level. However, entering this 95% CI into the window below leads to a COR of 0.087. Thus the result can be deemed credible at the 95% level only if ORs for mortality even more impressive (i.e. lower) than around 0.09 can be justified in the light of current knowledge about the effectiveness of anistreplase. Pocock and Spiegelhalter (1992) found that extant knowledge indicated that ORs below 0.4 were implausible. Thus ORs below the calculated COR level of 0.087 are not plausible, and the findings of the GREAT study may be deemed statistically significant but not credible at the 95% level. Subsequent experience has borne out this assessment.
NB: The CRO is based on the equipoise assumption of ethical randomised clinical trials (RCTs), according to which patients allocated to either arm of the trial are deemed equally likely to benefit. The CRO can thus be used to assess the credibility of meta-analyses of RCTs, as well as with individual trials. It can also be used to assess evidence from observational studies using the concept of the "fair minded sceptic", which implies a prior distribution identical to that based on the equipoise assumption.
Matthews, R.A.J. 2001 Methods for assessing the credibility of clinical
trial outcomes Drug Information Journal vol 35 (4) 1469-1478 Available
online at:
http://bit.ly/CredibilityAnalysisPaper
Spiegelhalter, D.J. Abrams, K.R., Myles, J.P. Bayesian Approaches to Clinical Trials and Health-care Evaluation (Wiley: Chichester, 2004) Chapter 3.
GREAT Group 1992 Feasibility, safety and efficacy of domiciliary thrombolysis by general practitioners: Grampian Region Early Anistreplase Trial BMJ 305 548
Pocock, S. J., Spiegelhalter D. J. 1992 Letter (untitled) BMJ 305 1015
If you have questions about the Bayesian statistical concepts embodied in this method, send e-mail to Robert Matthews at rajm@physics.org
If you have questions about the operation of this web page, send e-mail to John C. Pezzullo at statpages.org@gmail.com