Analysis of the AutoML Challenge series 2015-2018

Isabelle Guyon (1, 2) , Lisheng Sun-Hosoya (1) , Marc Boullé (3) , Hugo Jair Escalante (4) , Sergio Escalera (5) , Zhengying Liu (1, 6) , Damir Jajetic (7) , Bisakha Ray (8) , Mehreen Saeed (9) , Michèle Sebag (10) , Alexander Statnikov (11) , Wei-Wei Tu (12) , Evelyne Viegas (13)

1 TAU - TAckling the Underspecified
2 Chalearn
3 Orange Labs [Lannion]
4 INAOE - Instituto Nacional de Astrofísica, Óptica y Electrónica
5 CVC - Computer Vision Center (Centre de visio per computador)
6 UP11 - Université Paris-Sud - Paris 11
7 IN2 [Zagreb]
8 NYU Langone Medical Center - New York University Langone Medical Center
9 NUCES | FAST Karachi - Foundation for Advancement of Science and Technology
10 LRI - Laboratoire de Recherche en Informatique
11 NYU - New York University School of Medicine
12 4Paradigm
13 Microsoft Corporation [Redmond]

The ChaLearn AutoML Challenge 1 (NIPS 2015-ICML 2016) consisted of six rounds of a machine learning competition of progressive difficulty, subject to limited computational resources. It was followed by one round of AutoML challenge (PAKDD 2018). The AutoML setting differs from former model selection/hyper-parameter selection challenges, such as the one we previously organized for NIPS 2006: the participants aim to develop fully automated and computationally efficient systems, capable of being trained and tested without human intervention, with code submission. This paper analyzes the results of these competitions and provides details about the datasets, which were not revealed to the participants. The solutions of the winners are systematically benchmarked over all datasets of all rounds and compared with canonical machine learning algorithms available in scikit-learn. All materials discussed in this paper (data and code) have been made publicly available at http://automl.chalearn.org/.

Domaines: Machine Learning [stat.ML]

Isabelle Guyon, Lisheng Sun-Hosoya, Marc Boullé, Hugo Jair Escalante, Sergio Escalera, et al.. Analysis of the AutoML Challenge series 2015-2018. Frank Hutter; Lars Kotthoff; Joaquin Vanschoren. AutoML: Methods, Systems, Challenges, Springer Verlag, inPress, The Springer Series on Challenges in Machine Learning. ⟨hal-01906197⟩