Skip to main content

Resultados relacionados

Resultados totales (Incluyendo duplicados): 8
Encontrada(s) 1 página(s)

Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

Reference bias in monolingual machine translation evaluation

  • Fomicheva, Marina
  • Specia, Lucia
Comunicació presentada a la 54th Annual Meeting of the Association for Computational Linguistics, celebrada del 7 al 10 d'agost de 2016 a Berlín, Alemanya., In the translation industry, human translations are assessed by comparison with the source texts. In the Machine Translation (MT) research community, however, it is a common practice to perform quality assessment using a reference translation instead of the source text. In this paper we show that this practice has a serious issue – annotators are strongly biased by the reference translation provided, and this can have a negative impact on the assessment of MT quality., Marina Fomicheva was supported by funding from IULA (UPF) and the FI-DGR grant program of the Generalitat de Catalunya. Lucia Specia was supported by the QT21 project (H2020 No. 645452).
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

CobaltF: a fluent metric for MT evaluation

  • Fomicheva, Marina
  • Bel Rafecas, Núria
  • Specia, Lucia
  • da Cunha Fanego, Iria
  • Malinovskiy, Anton
Comunicació presentada a la First Conference on Machine Translation (WMT), que es va dur a terme durant el 54th Annual Meeting of the Association for Computational Linguistics, els dies 7 a 12 d'agost de 2016 a Berlín, Alemanya., The vast majority of Machine Translation (MT) evaluation approaches are based on the idea that the closer the MT output is to a human reference translation, the higher its quality. While translation quality has two important aspects, adequacy and fluency, the existing referencebased metrics are largely focused on the former. In this work we combine our metric UPF-Cobalt, originally presented at the WMT15 Metrics Task, with a number of features intended to capture translation fluency. Experiments show that the integration of fluency-oriented features significantly improves the results, rivalling the best-performing evaluation metrics on the WMT15 data., This work was partially funded by TUNER (TIN2015-65308-C5-5-R) and MINECO/FEDER, UE. Marina Fomicheva was supported by funding from the FI-DGR grant program of the Generalitat de Catalunya. Iria da Cunha was supported by a Ramon y Cajal contract (RYC-2014-16935). Lucia Specia was supported by the QT21 project (H2020 No. 645452).
Addi. Archivo Digital para la Docencia y la Investigación
Publication Contribución de congreso (ConferenceOutput)

Semeval-2017 task 1: Semantic textual similarity-multilingual and cross-lingual focused evaluation

  • Cer, Daniel
  • Diab, Mona
  • Agirre Bengoa, Eneko
  • López Gazpio, Iñigo
  • Specia, Lucia
Semantic Textual Similarity (STS) measures the meaning similarity of sentences. Applications include machine translation (MT), summarization, generation, question answering (QA), short answer grading, semantic search, dialog and conversational systems. The STS shared task is a venue for assessing the current state-of-the-art. The 2017 task focuses on multilingual and cross-lingual pairs with one sub-track exploring MT quality estimation (MTQE) data. The task obtained strong participation from 31 teams, with 17 participating in all language tracks. We summarize performance and review a selection of well performing methods. Analysis highlights common errors, providing insight into the limitations of existing models. To support ongoing work on semantic representations, the STS Benchmark is introduced as a new shared training and evaluation set carefully selected from the corpus of English STS shared task data (2012-2017)., This material is based in part upon work supported by QNRF-NPRP 6 - 1020-1-199 OPTDIAC that funded Arabic translation, and by a grant from the Spanish MINECO (projects TUNER TIN2015-65308-C5-1-R and MUSTER PCIN-2015-226 cofunded by EU FEDER) that funded STS label annotation and by the QT21 EU project (H2020 No. 645452) that funded STS labels and data preparation for machine translation pairs. I˜nigo Lopez-Gazpio is supported by the Spanish MECD. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of QNRF-NPRP, Spanish MINECO, QT21 EU, or the Spanish MECD.
RUA. Repositorio Institucional de la Universidad de Alicante
Publication Otros (Other)

Towards Optimizing MT for Post-Editing Effort: Can BLEU Still Be Useful?

  • Forcada, Mikel L.
  • Sánchez-Martínez, Felipe
  • Esplà-Gomis, Miquel
  • Specia, Lucia
  • Esplà-Gomis, Miquel|||/items/6614b776-cf96-43c9-b300-de4c8cf17c43
  • Sánchez-Martínez, Felipe|||/items/117378d0-8f7d-40ef-9b4a-5646b1af1999
We propose a simple, linear-combination automatic evaluation measure (AEM) to approximate post-editing (PE) effort. Effort is measured both as PE time and as the number of PE operations performed. The ultimate goal is to define an AEM that can be used to optimize machine translation (MT) systems to minimize PE effort, but without having to perform unfeasible repeated PE during optimization. As PE effort is expected to be an extensive magnitude (i.e., one growing linearly with the sentence length and which may be simply added to represent the effort for a set of sentences), we use a linear combination of extensive and pseudo-extensive features. One such pseudo-extensive feature, 1–BLEU times the length of the reference, proves to be almost as good a predictor of PE effort as the best combination of extensive features. Surprisingly, effort predictors computed using independently obtained reference translations perform reasonably close to those using actual post-edited references. In the early stage of this research and given the inherent complexity of carrying out experiments with professional post-editors, we decided to carry out an automatic evaluation of the AEMs proposed rather than a manual evaluation to measure the effort needed to post-edit the output of an MT system tuned on these AEMs. The results obtained seem to support current tuning practice using BLEU, yet pointing at some limitations. Apart from this intrinsic evaluation, an extrinsic evaluation was also carried out in which the AEMs proposed were used to build synthetic training corpora for MT quality estimation, with results comparable to those obtained when training with measured PE efforts.
RUA. Repositorio Institucional de la Universidad de Alicante
Publication Otros (Other)

Evaluation of Terminology Translation in Instance-Based Neural MT Adaptation

  • Farajian, M. Amin
  • Bertoldi, Nicola
  • Negri, Matteo
  • Turchi, Marco
  • Federico, Marcello
We address the issues arising when a neural machine translation engine trained on generic data receives requests from a new domain that contains many specific technical terms. Given training data of the new domain, we consider two alternative methods to adapt the generic system: corpus-based and instance-based adaptation. While the first approach is computationally more intensive in generating a domain-customized network, the latter operates more efficiently at translation time and can handle on-the-fly adaptation to multiple domains. Besides evaluating the generic and the adapted networks with conventional translation quality metrics, in this paper we focus on their ability to properly handle domain-specific terms. We show that instance-based adaptation, by fine-tuning the model on-the-fly, is capable to significantly boost the accuracy of translated terms, producing translations of quality comparable to the expensive corpus-based method.
RUA. Repositorio Institucional de la Universidad de Alicante
Publication Otros (Other)

SRL for low resource languages isn’t needed for semantic SMT

  • Beloucif, Meriem
  • Wu, Dekai
Previous attempts at injecting semantic frame biases into SMT training for low resource languages failed because either (a) no semantic parser is available for the low resource input language; or (b) the output English language semantic parses excise relevant parts of the alignment space too aggressively. We present the first semantic SMT model to succeed in significantly improving translation quality across many low resource input languages for which no automatic SRL is available —consistently and across all common MT metrics. The results we report are the best by far to date for this type of approach; our analyses suggest that in general, easier approaches toward including semantics in training SMT models may be more feasible than generally assumed even for low resource languages where semantic parsers remain scarce. While recent proposals to use the crosslingual evaluation metric XMEANT during inversion transduction grammar (ITG) induction are inapplicable to low resource languages that lack semantic parsers, we break the bottleneck via a vastly improved method of biasing ITG induction toward learning more semantically correct alignments using the monolingual semantic evaluation metric MEANT. Unlike XMEANT, MEANT requires only a readily-available English (output language) semantic parser. The advances we report here exploit the novel realization that MEANT represents an excellent way to semantically bias expectation-maximization induction even for low resource languages. We test our systems on challenging languages including Amharic, Uyghur, Tigrinya and Oromo. Results show that our model influences the learning towards more semantically correct alignments, leading to better translation quality than both the standard ITG or GIZA++ based SMT training models on different datasets.
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Reference bias in monolingual machine translation evaluation

  • Fomicheva, Marina
  • Specia, Lucia
Comunicació presentada a la 54th Annual Meeting of the Association for Computational Linguistics, celebrada del 7 al 10 d'agost de 2016 a Berlín, Alemanya., In the translation industry, human translations are assessed by comparison with the source texts. In the Machine Translation (MT) research community, however, it is a common practice to perform quality assessment using a reference translation instead of the source text. In this paper we show that this practice has a serious issue – annotators are strongly biased by the reference translation provided, and this can have a negative impact on the assessment of MT quality., Marina Fomicheva was supported by funding from IULA (UPF) and the FI-DGR grant program of the Generalitat de Catalunya. Lucia Specia was supported by the QT21 project (H2020 No. 645452).
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

CobaltF: a fluent metric for MT evaluation

  • Fomicheva, Marina
  • Bel Rafecas, Núria
  • Specia, Lucia
  • da Cunha Fanego, Iria
  • Malinovskiy, Anton
Comunicació presentada a la First Conference on Machine Translation (WMT), que es va dur a terme durant el 54th Annual Meeting of the Association for Computational Linguistics, els dies 7 a 12 d'agost de 2016 a Berlín, Alemanya., The vast majority of Machine Translation (MT) evaluation approaches are based on the idea that the closer the MT output is to a human reference translation, the higher its quality. While translation quality has two important aspects, adequacy and fluency, the existing referencebased metrics are largely focused on the former. In this work we combine our metric UPF-Cobalt, originally presented at the WMT15 Metrics Task, with a number of features intended to capture translation fluency. Experiments show that the integration of fluency-oriented features significantly improves the results, rivalling the best-performing evaluation metrics on the WMT15 data., This work was partially funded by TUNER (TIN2015-65308-C5-5-R) and MINECO/FEDER, UE. Marina Fomicheva was supported by funding from the FI-DGR grant program of the Generalitat de Catalunya. Iria da Cunha was supported by a Ramon y Cajal contract (RYC-2014-16935). Lucia Specia was supported by the QT21 project (H2020 No. 645452).