Threats of a replication crisis in empirical computer science - Laboratoire Interdisciplinaire des Sciences du Numérique Accéder directement au contenu
Article Dans Une Revue Communications of the ACM Année : 2020

Threats of a replication crisis in empirical computer science

Andy Cockburn
  • Fonction : Auteur
  • PersonId : 1005483
Pierre Dragicevic
Lonni Besançon

Résumé

Many areas of computer science research (e.g., performance analysis, software engineering, artificial intelligence, and human-computer interaction) validate research claims by using statistical significance as the standard of evidence. A loss of confidence in statistically significant findings is plaguing other empirical disciplines, yet there has been relatively little debate of this issue and its associated 'replication crisis' in computer science. We review factors that have contributed to the crisis in other disciplines, with a focus on problems stemming from an over-reliance on-and misuse of-null hypothesis significance testing. Computer science research can be greatly improved by following the steps taken by other disciplines, such as using more sophisticated evidentiary criteria, and showing greater openness and transparency through experimental preregistration and data/artifact repositories.
Fichier principal
Vignette du fichier
Computer_Science_Research_and_the_Replication_Crisis_preprint.pdf (394.23 Ko) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)
Loading...

Dates et versions

hal-02907143 , version 1 (27-07-2020)

Identifiants

Citer

Andy Cockburn, Pierre Dragicevic, Lonni Besançon, Carl Gutwin. Threats of a replication crisis in empirical computer science. Communications of the ACM, 2020, 63 (8), pp.70-79. ⟨10.1145/3360311⟩. ⟨hal-02907143⟩
204 Consultations
518 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More