• David Sidebotham, Felicity Dominick, Carolyn Deng, Jake Barlow, and Philip M Jones.
• Department of Cardiothoracic and ORL Anaesthesia, Auckland City Hospital, Auckland, New Zealand; Cardiothoracic and Vascular Intensive Care Unit, Auckland City Hospital, Auckland, New Zealand; Department of Anaesthesiology, Faculty of Health Sciences, University of Auckland, New Zealand. Electronic address: dsidebotham@adhb.govt.nz.
• Br J Anaesth. 2024 Jan 1; 132 (1): 116123116-123.

BackgroundThe American Statistical Association has highlighted problems with null hypothesis significance testing and outlined alternative approaches that may 'supplement or even replace P-values'. One alternative is to report the false positive risk (FPR), which quantifies the chance the null hypothesis is true when the result is statistically significant.MethodsWe reviewed single-centre, randomised trials in 10 anaesthesia journals over 6 yr where differences in a primary binary outcome were statistically significant. We calculated a Bayes factor by two methods (Gunel, Kass). From the Bayes factor we calculated the FPR for different prior beliefs for a real treatment effect. Prior beliefs were quantified by assigning pretest probabilities to the null and alternative hypotheses.ResultsFor equal pretest probabilities of 0.5, the median (inter-quartile range [IQR]) FPR was 6% (1-22%) by the Gunel method and 6% (1-19%) by the Kass method. One in five trials had an FPR ≥20%. For trials reporting P-values 0.01-0.05, the median (IQR) FPR was 25% (16-30%) by the Gunel method and 20% (16-25%) by the Kass method. More than 90% of trials reporting P-values 0.01-0.05 required a pretest probability >0.5 to achieve an FPR of 5%. The median (IQR) difference in the FPR calculated by the two methods was 0% (0-2%).ConclusionsOur findings suggest that a substantial proportion of single-centre trials in anaesthesia reporting statistically significant differences provide limited evidence of real treatment effects, or, alternatively, required an implausibly high prior belief in a real treatment effect.Clinical Trial RegistrationPROSPERO (CRD42023350783).Copyright © 2023 The Author(s). Published by Elsevier Ltd.. All rights reserved.

13 keywords...

hide…