A lot of serious statistical errors are made in psychological academic research. Dutch researchers have not sufficiently mastered elementary statistical methods and make statements that are more definite than they can prove. This has been revealed by research conducted by Rink Hoekstra, who will be awarded a PhD by the University of Groningen on 8 October.

Psychologists try to understand human behaviour and its underlying mental processes. In order to be able to make general statements about human behaviour, practical considerations usually mean that this behaviour is studied among a limited number of people (a random sample), and this random data is then generalized to the population in which the researcher is interested. There are various ways of conducting this generalization process, of which the significance test is by far the most popular.

Old problem

The use of the significance test has been under discussion for decades. Hoekstra: ‘Imagine you are researching the heights of men and women. One question you could ask is whether men are taller than women. After a random sample test where you measure the heights of ten men and ten women, you would be able to state that men are significantly taller than women. But is that interesting? And does it tell you anything? A better idea would be to ask how much the difference in height is. That would not only provide insight into the question of whether men are taller than women, but also the extent of the height difference.’ It appears not only to be useful to answer the question of whether there is an effect (which is what a significance test is often used for), but also how large that effect is precisely, which is expressed at random sample level by the effect size. A reliability interval helps to generalize the effect size of the sample to the entire population.

Ignorance and unawareness

According to international regulations, in psychological research a significance test is not required although effect size is crucial and reliability intervals are strongly recommended. Hoekstra’s research has revealed that this nuancing of the significance test in psychological academic research is usually not applied, or that it is conducted on a pro forma basis but ignored when conclusions are drawn. Hoekstra: ‘If you look with a very critical eye, you can find things that are not accurate in virtually every published article. And that includes renowned journals.’ Interviews with and tests by thirty PhD students at seven Dutch universities have revealed that there is no question of lack of desire, but mainly ignorance and unawareness. Hoekstra: ‘The mistakes have become so common that no-one sees them any more.’

‘It sounds better’

However, statistical errors can sometimes also suit researchers better, suspects Hoekstra. ‘Scientists are searching for the truth, but in practice they also have to concentrate on presenting their research in the best possible way. My research results suggest that researchers are inclined to think in black and white and claim things more definitely than they can actually prove. It sounds better if you call something significant. It’s also possible that journal editors will be more interested in your article – after all, they prefer to publish interesting stories.’ The quality of statistics training must improve, thinks Hoekstra. ‘Students are smart enough to understand that the truth is not black and white. Lecturers must dare to tell their students a nuanced story. By doing so they’ll not only be doing their students a favour, but also future research.’

Curriculum vitae

Rink Hoekstra (Drachten, 1977) studied Psychology and Technical Cognitive Sciences at the University of Groningen. He conducted his PhD research at the Department of Statistics and Methodology of the Faculty of Behavioural and Social Sciences of the same university. His supervisors were Prof. H.A.L. Kiers and Prof. A. Johnson. Hoekstra currently works as lecturer and researcher in the Department of Theory of Education of the Faculty of Behavioural and Social Studies. His thesis is entitled ‘The Use and Usability of Inferential Techniques’.

Note for the press

Contact: Rink Hoekstra, tel.: 050-3637024, email: r.hoekstra@rug.nl