An experimental analysis of Noise-Contrastive Estimation: the noise distribution matters

Matthieu Labeau (1) , Alexandre Allauzen (2)

1 LTCI - Laboratoire Traitement et Communication de l'Information
2 LIMSI - Laboratoire d'Informatique pour la Mécanique et les Sciences de l'Ingénieur

Noise Contrastive Estimation (NCE) is a learning procedure that is regularly used to train neural language models, since it avoids the computational bottleneck caused by the output softmax. In this paper , we attempt to explain some of the weaknesses of this objective function, and to draw directions for further developments. Experiments on a small task show the issues raised by the unigram noise distribution, and that a context dependent noise distribution, such as the bigram distribution , can solve these issues and provide stable and data-efficient learning.

Domaines: Informatique [cs]; Statistiques [stat]

Matthieu Labeau, Alexandre Allauzen. An experimental analysis of Noise-Contrastive Estimation: the noise distribution matters. 15th Conference of the European Chapter of the Association for Computational Linguistics:, Apr 2017, Valencia, Spain. pp.15 - 20. ⟨hal-02912384⟩