In this new paper I merged and developed a bit further the 2 IJSPP papers on 1) the stats that changed my life and 2) some personal thoughts on chasing the 0.2 (i., making an impact) in an elite setting.
The value and importance of sport science varies greatly between elite clubs and federations. Among the different components of effective sport science support, the three most important elements are likely the following:
- Appropriate understanding and analysis of the data; i.e. using the most important and useful metrics only and using magnitude-based inferences as statistics. In fact, traditional null hypothesis significance testing (P values) is neither appropriate to answer the types of questions that arise from the field (i.e. assess magnitude of effects and examine small sample sizes) nor to assess changes in individual performances.
- Attractive and informative reports via improved data presentation/ visualisation (‘simple but powerful’).
- Appropriate communication skills and personality traits that help to deliver data and reports to coaches and athletes. Developing such an individual profile requires time, effort and most importantly, humility
A few tweets, re-tweets and emails from colleagues have caught my attention within the last 24 hrs, all pointing toward a new study showing improvements in sprinting and high-intensity intermittent running performance after dietary nitrate supplementation (beetroot shots) (1). In the 36 team-sports players (training 5-10h a week) who volunteered for the study, significant “improvements” in 5- (2.3%), 10- (1.6%) and 20-m (1.2%) sprint times and a 3.9% “increase” in high-intensity intermittent performance were reported, after no longer than 5 days of supplementation! (1)
To all practitioners who may read both the article (1) and the present blog post, the topic is obviously highly relevant; we are all looking for various ways to improve our players’ running performance – even better if these improvements can be gained legally (no doping) and without (physical) efforts. If you can convince yourself to commit to drink daily an awful 70-ml beetroot shot for 5 days before an important competition, then you may have found a really cool and lazy way to get faster and fitter!!
However, before I began to tell (again) every player at the club (who would systematically pass on beetroot because of its taste) to finally commit themselves to drink this stuff, because it really works, I wished to make sure it would be worth the effort, both for them and me. After a deeper read of the paper, a closer look at the study design, the data analysis and the stat approach, I realized that in fact, beetroot supplementation, within the context of the present study, may not be as promising as it could be understood while only reading the title of the paper. This for at least two important reasons: 1) the somewhat limited magnitude of the “changes”, although significant and 2) the questionable study design/data analysis that doesn’t allow individual responses to be clearly accounted for and analyzed.
- The magnitude of the “improvement” may not be large enough to be meaningful. When considering the magnitude of the smallest worthwhile changes for different sprint distances (SWC, i.e., the minimum improvement likely to have an impact on the field, such as that required to be 20 cm ahead of an opponent to win a ball) (2), the changes reported in the present study are in fact either smaller (5 m: study 2.3% vs SWC ̴ 4%, 10 m: study 1.6% vs SWC ̴2%) or just similar to (20 m: study 1.2% vs SWC ̴1%) (2). Even for 20-m time, which magnitudes equals the SWC, chances for the “improvement” to be substantial may be no more than 50% at the individual level (when considering a typical error of the measurement (TE) of the same magnitude than the SWC – while in fact the TE may actually be twice as large as the SWC for such a distance (2), decreasing further the likelihood of a substantial change) (2). The same reasoning applies to the “increase” in Yo-YoIR1 performance (+3.9%), which SWC is generally twice larger (̴ +8% (3), +7% as 0.2xSD in the present study). In conclusion, the comparison of the reported changes, although significant, to their specific SWC directly questions the practical impact and in turn, the usefullness of beetroot supplementation in the context of the present study. These data illustrate once again that the use of null hypothesis significance testing (NHST) is clearly limited to assess the actual performance benefit of a supplement or an intervention (4, see the blog on the topic) – in the present case the significant P value likely results from the large sample size (n=36) – different conclusions (and probably less misleading in the present case) would be drawn with lower samples (i.e., n<15).
- The data analysis doesn’t allow individual responses to be clearly accounted for/analyzed. In fact, the authors simply chose to compare the sprints/YoYoIR1 performances following beetroot supplementation to these following the placebo drink (Post beetroot – Post placebo, via paired-samples t-tests)!? While it is not clear why such a limited approach was chosen, the proper way to analyze these data would be to look first at within-group changes, and more importantly, to compare these within-group changes (i.e., between-group differences in the changes – typical crossover design, as ‘post beetroot – pre beetroot’ compared with ‘post placebo – pre placebo’). This latter approach is way more powerful and allows the understanding of i) the effect of each treatment per se (within-group effect, in relation to the SWC), ii) the variability of the response within each treatment (SD of the change, which has important implications when using supplementation with athletes – some will respond, some not !! – and how many and by which magnitude?), iii) compare the efficacy of the treatments (differences in the magnitude of the changes) and even more importantly, iiii) compare the magnitude of the individual responses between each treatment (i.e., which treatment shows the greater variability in response). Unfortunately, all these relevant information for practitioners are missing in the manuscript.
That being said, I am happy to keep beetroot shots on the supplement table for the moment (for players that can cope with the taste… at least it hasn’t been shown to be detrimental). I may, however, not use the present study to advertise the benefit of beetroot to the players – if we want to keep our legitimacy and maintain the trust that the players put on us, I believe it is important to come to them with the right message – and in that case, applying some appropriate stats surely helps!
- Thompson C, Vanhatalo A, Jell H, Fulford J, Carter c, Nyman L, Bailey SJ and Jones AM. Dietary nitrate supplementation improves sprint and high-intensity intermittent running performance. Nitric Oxide 61 (2016) 55-61.
- Haugen T, Buchheit M. Sprint running performance monitoring: methodological and practical considerations. Sports Med. 2016;46(5):641.
- Bangsbo J, Iaia FM, Krustrup P. The Yo-Yo Intermittent Recovery Test: a useful tool for evaluation of physical performance in intermittent sports. Sports Med. 2008;38:37–51.
- Buchheit M. The Numbers Will Love You Back in Return—I Promise. Int J Sports Physiol & Perf, 2016, 11, 551 -554.
Buchheit M. The numbers will love you back in return – I promise. IJSPP 2016, 11, 551 – 554
Full text here
Abstract: The first sport science-oriented and comprehensive paper on magnitude-based inferences (MBI) was published 10 years ago in the first issue of this journal. While debate continues, MBI is today well-established in sports science and in other fields, particularly clinical medicine where practical/clinical significance often takes priority over statistical significance. In this commentary, some reasons why both academics and sport scientists should abandon null hypothesis significance testing (NHST) and embrace MBI are reviewed. Apparent limitations and future areas of research are also discussed. The following arguments are presented: P values and in turn, study conclusions, are sample-size dependent, irrespective of the size of the effect; significance doesn’t inform on magnitude of effects, yet magnitude is what matters the most; MBI allows authors to be honest with their sample size and better acknowledge trivial effects; the examination of magnitudes per se helps provide better research questions; MBI can be applied to assess changes in individuals; MBI improves data visualisation; and lastly, MBI is supported by spreadsheets freely available on the internet. Finally, recommendations to define the smallest important effect and improve the presentation of standardized effects are presented.
Keywords: magnitude-based inferences; null hypothesis significance testing; sample size; trivial effect; smallest important effect.
Figure 3. Differences in various anthropometric, physiological and performance measures between two groups of young soccer players differing by their maturity status (0.9 ± 0.3 vs. -0.2 ± 0.4 years from predicted peak height velocity)30 when expressed in percentages (A), using Cohen’s effect size principle (B) and as a factor of variable-specific smallest worthwhile differences (SWD) (C):28 0.2 x between-athletes SD for height, MAS and matches tracking data; performance-related changes for HRR and MSS (723 and 222%, respectively). The numbers of * indicate the likelihood for the between-group differences to be substantial, with 1 symbols referring to possible difference, 2 to likely, 3 to very likely and 4 to almost certain differences. Note that that magnitude of the between-group differences and their likelihood varies between the panels. My suggestion is to use the method used in panel C (with a variable-specific SWD). MSS: maximal sprinting speed, MAS: maximal aerobic speed, HRR: heart rate recovery after submaximal exercise, D>16 km/h: distance ran above 16 km/h during matches, #HIR: number of high intensity runs during matches.