RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publicación
Contribución de congreso (ConferenceOutput)
Filtering of Noisy Parallel Corpora Based on Hypothesis Generation
[EN] The filtering task of noisy parallel corpora in WMT2019 aims to challenge participants to create filtering methods to be useful for training machine translation systems. In this work, we introduce a noisy parallel corpora filtering system based on generating hypotheses by means of a translation model. We train translation models in both language pairs: Nepali English and Sinhala English using provided parallel corpora. To create the best possible translation model, we first join all provided parallel corpora (Nepali, Sinhala and Hindi to English) and after that, we applied bilingual
cross-entropy selection for both language pairs (Nepali English and Sinhala English). Once
the translation models are trained, we translate the noisy corpora and generate a hypothesis for each sentence pair. We compute the smoothed BLEU score between the target sentence and generated hypothesis. In addition, we apply several rules to discard very noisy or inadequate sentences which can lower the translation score. These heuristics are based on sentence length, source and target similarity and source language detection. We compare our results with the baseline published on the shared task website, which uses the Zipporah model, over which we achieve significant improvements in one of the conditions in the shared task. The designed filtering system is domain independent and all experiments are conducted using neural machine translation., Work partially supported by MINECO under grant
DI-15-08169 and by Sciling under its R+D programme. The authors would like to thank NVIDIA
for their donation of Titan Xp GPU that allowed to
conduct this research.