Pasar al contenido principal

TRANSCRIPCION DE DOCUMENTOS CON PLATAFORMAS INTERACTIVAS UBICUAS MULTIMODALES

RTI2018-095645-B-C22

•
Nombre agencia financiadora Agencia Estatal de Investigación
Acrónimo agencia financiadora AEI
Programa Programa Estatal de I+D+i Orientado a los Retos de la Sociedad
Convocatoria Proyectos I+D+i Retos de Investigación
Año convocatoria 2018
Plan Estatal 2017-2020 Plan Estatal de Investigación Científica y Técnica y de Innovación
Centro beneficiario UNIVERSITAT POLITÈCNICA DE VALÈNCIA

Resultados relacionados

Resultados totales (Incluyendo duplicados): 5
Encontrada(s) 1 página(s)

RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publicación Artículo científico (JournalArticle)

On the Use of Mouse Actions at the Character Level

  • Navarro-Martínez, Ángel|||0000-0002-5957-8152
  • Casacuberta Nolla, Francisco|||0000-0002-8497-5598
[EN] Neural Machine Translation (NMT) has improved performance in several tasks up to human parity. However, many companies still use Computer-Assisted Translation (CAT) tools to achieve perfect translation, as well as other tools. Among these tools, we find Interactive-Predictive Neural Machine Translation (IPNMT) systems, whose main feature is facilitating machine-human interactions. In the most conventional systems, the human user fixes a translation error by typing the correct word, sending this feedback to the machine which generates a new translation that satisfies it. In this article, we remove the necessity of typing to correct translations by using the bandit feedback obtained from the cursor position when the user performs a Mouse Action (MA). Our system generates a new translation that fixes the error using only the error position. The user can perform multiple MAs at the same position if the error is not fixed, each of which increases the correction probability. One of the main objectives in the IPNMT field is reducing the required human effort, in order to optimize the translation time. With the proposed technique, an 84% reduction in the number of keystrokes performed can be achieved, while still generating perfect translations. For this reason, we recommend the use of this technique in IPNMT systems., This work received funds from the Comunitat Valenciana under project EU-FEDER (ID-IFEDER/2018/025), Generalitat Valenciana under project ALMAMATER (PrometeoII/2014/030), and Ministerio de Ciencia e Investigacion/Agencia Estatal de Investigacion/10.13039/501100011033/, "FEDER Una manera de hacer Europa" under project MIRANDA-DocTIUM (RTI2018-095645-B-C22), and Universitat Politecnica de Valencia under the program (PAID-01-21).
RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publicación Artículo científico (JournalArticle)

Neural Models for Measuring Confidence on Interactive Machine Translation Systems

  • Navarro-Martínez, Ángel|||0000-0002-5957-8152
  • Casacuberta Nolla, Francisco|||0000-0002-8497-5598
[EN] Reducing the human effort performed with the use of interactive-predictive neural machine translation (IPNMT) systems is one of the main goals in this sub-field of machine translation (MT). Prior works have focused on changing the human¿machine interaction method and simplifying the feedback performed. Applying confidence measures (CM) to an IPNMT system helps decrease the number of words that the user has to check through the translation session, reducing the human effort needed, although this supposes losing a few points in the quality of the translations. The effort reduction comes from decreasing the number of words that the translator has to review¿it only has to check the ones with a score lower than the threshold set. In this paper, we studied the performance of four confidence measures based on the most used metrics on MT. We trained four recurrent neural network (RNN) models to approximate the scores from the metrics: Bleu, Meteor, Chr-f, and TER. In the experiments, we simulated the user interaction with the system to obtain and compare the quality of the translations generated with the effort reduction. We also compare the performance of the four models between them to see which of them obtains the best results. The results achieved showed a reduction of 48% with a Bleu score of 70 points¿a significant effort reduction to translations almost perfect., This work received funds from the Comunitat Valenciana under project EU-FEDER (ID-IFEDER/2018/025), Generalitat Valenciana under project ALMAMATER (PrometeoII/2014/030), and Ministerio de Ciencia e Investigacion/Agencia Estatal de Investigacion/10.13039/501100011033/and "FEDER Una manera de hacer Europa" under project MIRANDA-DocTIUM (RTI2018-095645-B-C22).
RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publicación Artículo científico (JournalArticle)

Reading order detection on handwritten documents

  • Quirós, Lorenzo
  • Vidal, Enrique|||0000-0003-4579-5196
[EN] Recent advances in Handwritten Text Recognition and Document Layout Analysis have made it possible to convert digital images of manuscripts into electronic text. However, providing this text with the correct structure and context is still an open problem that needs to be solved to actually enable extracting the relevant information conveyed by the text. The most important structure needed for a set of text elements is their reading order. Most of the studies on the reading order problem are rule-based approaches and focus on printed documents. Much less attention has been paid so far to handwritten text documents, where the problem becomes particularly important-and challenging. In this work, we propose a new approach to automatically determine the reading order of text regions and lines in handwritten text documents. The task is approached as a sorting problem where the order-relation operator is automatically learned from examples. We experimentally demonstrate the effectiveness of our method on three different datasets at different hierarchical levels., The authors want to thank to the Centre de Recerca d'Història Rural, the National Archives of Finland and Déjean Hervé for facilitating the datasets used in this work, and to Juan Miguel Vilar for the enlighment comments. Also, this work was partially supported by Universitat Politècnica de València under grant FPI-II/900, by Generalitat Valenciana under the EU-FEDER Comunitat Valenciana 2014-2020 grant IDIFEDER/2018/025 'Sistemas de fabricación inteligente para la indústria 4.0', by the Ministerio de Ciencia, Innovación y Universidades project DocTIUM (Ref. RTI2018-095645-B-C22), by the BBVA Foundation through the 2019 Digital Humanities research grant 'HistWeather ' Dos Siglos de Datos Climáticos' and by the Agencia Estatal de Investigación under project SimancasSearch (PID2020-116813RB-I00).Open Access funding provided thanks to the CRUE-CSIC agreement with Springer Nature.
RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publicación Tesis doctoral (DoctoralThesis)

Contributions to Automatic Lipreading for Spanish

  • Gimeno-Gómez, David|||0000-0002-7375-9515
[ES] Inspirados por la naturaleza multisensorial de la percepción del habla humana, ha habido un creciente interés en el desarrollo de sistemas automáticos de lectura de labios. Esta desafiante tarea tiene como objetivo interpretar el habla analizando únicamente los movimientos labiales del hablante sin depender de señales acústicas. Entre la amplia variedad de aplicaciones ofrecidas por esta tecnología, destacamos su prometedor papel en el avance de las interfaces de habla silenciosas a través del diseño de soluciones no invasivas que den soporte a la comunicación de personas que han perdido su capacidad de hablar. La eficacia de esta tecnología, sin embargo, se ve comprometida por numerosos factores, entre los que se incluyen las ambigüedades visuales a la hora de intentar distinguir entre sonidos distintos pero visualmente similares, y la dificultad a la que se enfrentan los modelos multi-hablantes para generalizar y abordar las variaciones inter-personales entre hablantes. Aparte de estos desafíos inherentes a la lectura de labios en sí misma, existen cuestiones más técnicas, como son la escasez de datos audiovisuales para el habla continua y natural en idiomas distintos del inglés y la actual dependencia de modelos de alta complejidad para lograr un rendimiento sólido. En relación con el desarrollo de interfaces de habla silenciosa, también observamos que la investigación sobre enfoques dependientes del hablante se ha limitado en su mayoría a entornos controlados y protocolos experimentales restringidos, pasando por alto la impor\-tancia de explorar este tipo de estudios dependientes del hablante en condiciones más naturalistas, en particular con habla continua. Esta tesis no sólo aborda esta brecha de investigación en la dependencia del hablante, sino que también investiga el diseño de sistemas automáticos de lectura labial y su integración con señales acústicas del habla para el español continuo natural. Nuestro enfoque integral ofrece una perspectiva holística del problema, comprendiendo la recopilación y el procesamiento de datos audiovisuales, la transición de los métodos tradicionales basados en modelos ocultos de Markov a arquitecturas profundas de extremo a extremo de última generación, y la integración de señales del habla audiovisuales para el reconocimiento robusto de la voz. Más específicamente, proponemos estrategias de entrenamiento intermodal en el contexto de decodificadores tradicionales, evaluamos y analizamos la eficacia de las arquitecturas de extremo a extremo actuales y su adaptación a hablantes específicos, realizamos estudios comparativos sistemáticos entre estos dos distintos paradigmas de decodificación en diversas condiciones de entrenamiento, y exploramos las contribuciones de las señales visuales cuando se integran con señales acústicas para el reconocimiento de voz audiovisual bajo una perspectiva de eficiencia. Como resultado, nuestra investigación establece un punto de referencia de lectura labial en español, con el objetivo de garantizar que las tecnologías de lectura labial se puedan estudiar de manera efectiva para este idioma en diversas condiciones de datos. Un factor común en todas estas contribuciones es nuestro particular énfasis en la eficiencia de esta tecnología y su aplicación en situaciones del mundo real. Al abordar la heterogeneidad y la falta ocasional de datos que sufren la mayoría de los idiomas en el contexto de las tecnologías de habla audiovisual, creemos firmemente que nuestro trabajo promueve una mayor investigación en idiomas subrepresentados y entornos de bajos recursos., [CA] Inspirats per la naturalesa multisensorial de la percepció de la parla humana, hi ha hagut un interès creixent en el desenvolupament de sistemes automàtics de lectura de llavis. Aquesta desafiant tasca té com a objectiu interpretar el parlar analitzant únicament els moviments labials del parlant sense dependre de senyals acústics. Entre l'àmplia varietat d'aplicacions ofertes per aquesta tecnologia, en destaquem el paper prometedor en l'avenç de les interfícies de parla silencioses a través del disseny de solucions no invasives que donen suport a la comunicació de persones que han perdut la capacitat de parlar. L'eficàcia d'aquesta tecnologia, però, es veu compromesa per nombrosos factors, entre els quals s'inclouen les ambigüitats visuals a l'hora d'intentar distingir entre sons diferents però visualment similars, i la dificultat a què s'enfronten els models multiparlants per generalitzar i abordar les variacions interpersonals entre parlants. A banda d'aquests desafiaments inherents a la lectura de llavis en si mateixa, hi ha qüestions més tècniques, com són l'escassetat de dades audiovisuals per a la parla continua i natural en idiomes diferents de l'anglès i l'actual dependència en models d'alta complexitat per aconseguir un rendiment sòlid. En relació amb el desenvolupament d'interfícies de parla silenciosa, observem que la investigació sobre enfocaments dependents del parlant s'ha limitat majoritàriament a entorns controlats i protocols experimentals restringits, passant per alt la importància d'explorar aquest tipus d'estudis dependents del parlant en condicions més naturalistes, com ara la parla contínua. Aquesta tesi no sols aborda aquesta bretxa de recerca en la dependència del parlant, sinó que també investiga el disseny de sistemes automàtics de lectura labial i la seva integració amb senyals acústics de la parla per a l'espanyol continu natural. El nostre enfocament integral ofereix una perspectiva holística del problema, comprenent la recopilació i el processament de dades audiovisuals, la transició dels mètodes tradicionals basats en models ocults de Markov a arquitectures profundes d'extrem a extrem d'última generació, i la integració de senyals de la parla audiovisuals per al reconeixement robust de la veu. Més específicament, proposem estratègies d'entrenament intermodal en el context de descodificadors tradicionals, avaluem i analitzem l'eficàcia de les arquitectures d'extrem a extrem actuals i la seua adaptació a parlants específics, realitzem estudis comparatius sistemàtics entre aquests dos diferents paradigmes de descodificació en diverses condicions de entrenament, i explorem les contribucions dels senyals visuals quan s'integren amb senyals acústics per al reconeixement de veu audiovisual sota una perspectiva d'eficiència. Com a resultat, la nostra investigació estableix un punt de referència de lectura labial en espanyol, per tal de garantir que les tecnologies de lectura labial es puguen estudiar de manera efectiva per a aquest idioma en diverses condicions de dades. Un factor comú en totes aquestes contribucions és el nostre èmfasi particular en l'eficiència d'aquesta tecnologia i la seua aplicació en situacions del món real. En abordar l'heterogeneïtat i la manca ocasional de dades que pateixen la majoria dels idiomes en el context de les tecnologies de parla audiovisual, creiem fermament que el nostre treball promou una major investigació en idiomes subrepresentats i entorns de baixos recursos., [EN] Inspired by the multi-sensory nature of human speech perception, there has been a growing interest in developing automatic lipreading systems. This challenging task aims to interpret speech solely by analyzing the speaker's lip movements, without relying on acoustic cues. Among the wide variety of applications offered by this lipreading technology, we highlight its promising role in advancing silent speech interfaces through the design of non-invasive solutions that support communication for individuals who have lost the ability to speak. The efficacy of this technology, however, is compromised by numerous factors, including visual ambiguities when attempting to distinguish between distinct but visually similar sounds, and the difficulty multi-speaker models face in generalizing and handling interpersonal variations across speakers. In addition to these challenges inherent to lipreading itself, more technical issues --- such as the scarcity of available audio-visual data for continuous and natural speech in languages other than English, and the current reliance on model complexity for achieving strong performance --- also hinder progress. Regarding the development of silent speech interfaces, we also noted that research on speaker-dependent approaches was mostly limited to indoor controlled settings and constrained experimental protocols, overlooking the importance of exploring this type of speaker-dependent studies under more naturalistic conditions, particularly with continuous speech. This thesis not only addresses this research gap in speaker dependency, but also investigates the design of automatic lipreading systems and their integration with acoustic speech cues for natural continuous Spanish. Our comprehensive approach offers a holistic perspective on the problem, encompassing the collection and processing of audio-visual data, the transition from traditional Hidden Markov Model (HMM)-based methods to state-of-the-art deep end-to-end architectures, and the integration of audio-visual speech cues for robust speech recognition. More specifically, we propose cross-modal training strategies in the context of traditional decoders, evaluate and analyze the efficacy of current end-to-end architectures and their adaptation to specific speakers, conduct systematic comparative studies between these two distinct decoding paradigms under diverse training conditions, and explore the contributions of visual cues when integrated with acoustic cues for audio-visual speech recognition from a parameter-efficient perspective. As a result, our research establishes a Spanish lipreading benchmark, aiming to ensure that lipreading technologies can be effectively studied for this language under diverse data conditions. A common theme throughout these contributions is our particular emphasis on the efficiency of this technology and its application to real-world scenarios. By addressing the heterogeneity and occasional significant lack of data that most languages suffer from in the context of audiovisual speech technologies, we strongly believe our work promotes further research in underrepresented languages and low-resource settings., The research leading to the results of this thesis was partially supported by Generalitat Valenciana under project DeepPattern (PROMETEO/2019/121), by Ministerio de Ciencia under project MIRANDA-DocTIUM (RTI2018-095645-B-C22), by the Grant CIACIF/2021/295 funded by Generalitat Valenciana, and by the Grant PID2021-124719OB-I00 under project LLEER (PID2021-124719OB-100) funded by MCIN/AEI/10.13039/501100011033/ and by ERDF, EU A way of making Europe. The mobility for the 3-month research stay at INESC-ID (Lisbon, Portugal) supervised by Dr. Alberto Abad was supported by Generalitat Valenciana under reference CIBEFP/2023/167. We extend our special thanks to RTVE and Cátedra RTVE-UZ for providing us with access to audio-visual data materials.
RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publicación Artículo científico (JournalArticle)

Study of the influence of lexicon and language restrictions on computer assisted transcription of historical manuscripts

  • GRANELL, EMILIO|||0000-0001-5782-7568
  • Martínez-Hinarejos, Carlos-D.|||0000-0002-6139-2891
  • Romero, Verónica
[EN] State-of-the-art Handwritten Text Recognition (HTR) systems allow transcribers to speed-up the transcription of handwritten text images. These systems provide transcribers an initial draft transcription that can be corrected with less effort than transcribing the handwritten text images from scratch. Currently, even the draft transcriptions offered by the most advanced HTR systems contain errors. Therefore, the supervision of this draft by a human transcriber is still necessary to obtain the correct transcription of the handwritten text images. This supervision can be eased by using interactive and assistive transcription systems, where the transcriber and the automatic system cooperate in the amending process. In this paper, the draft transcription is provided by an HTR system based on Convolutional and Recurrent Neural Networks with Bidirectional Long-Short Term Memory units, and the assistive system is fed by lattices generated by using Weighted Finite State Transducers. The influence of the lexicon and language restrictions on the performance of our computer assisted transcription system is evaluated on three historical manuscripts. The transcriptions offered by the proposed HTR system present very low error rates for the studied historical manuscripts. However, our assistive transcription system without lexicon or language restrictions is able to provide an additional reduction on the human effort required to correct the transcriptions in more than 50% over the transcriptions offered by the HTR system. (C) 2020 Elsevier B.V. All rights reserved., Work partially supported by the BBVA Foundation through the 2017-2018 Digital Humanities research grant "Carabela" and the grant "Ayudas Fundacion BBVA a equipos de investigacion cientifica 2018"(PR[18]_HUM_C2_0087), by the Generalitat Valenciana under the EU-FEDER Comunitat Valenciana 2014-2020 grant IDIFEDER/2018/025 "Sistemas de fabricacion inteligente para la industria 4.0"and the grant PROMETEO/2019/121 (DeepPattern), and by the Ministerio de Ciencia/AEI/FEDER/EU through the MIRANDA-DocTIUM project (RTI2018-095645-B-C22)