An experimental analysis of Noise-Contrastive Estimation: the noise distribution matters - Archive ouverte HAL Access content directly
Conference Papers Year :

An experimental analysis of Noise-Contrastive Estimation: the noise distribution matters

Abstract

Noise Contrastive Estimation (NCE) is a learning procedure that is regularly used to train neural language models, since it avoids the computational bottleneck caused by the output softmax. In this paper , we attempt to explain some of the weaknesses of this objective function, and to draw directions for further developments. Experiments on a small task show the issues raised by the unigram noise distribution, and that a context dependent noise distribution, such as the bigram distribution , can solve these issues and provide stable and data-efficient learning.
Fichier principal
Vignette du fichier
E17-2003.pdf (255.31 Ko) Télécharger le fichier
Origin : Publisher files allowed on an open archive
Loading...

Dates and versions

hal-02912384 , version 1 (05-08-2020)

Identifiers

  • HAL Id : hal-02912384 , version 1

Cite

Matthieu Labeau, Alexandre Allauzen. An experimental analysis of Noise-Contrastive Estimation: the noise distribution matters. 15th Conference of the European Chapter of the Association for Computational Linguistics:, Apr 2017, Valencia, Spain. pp.15 - 20. ⟨hal-02912384⟩
22 View
56 Download

Share

Gmail Facebook Twitter LinkedIn More