Pasar al contenido principal

TRANSCRIPCION MULTIMODAL DE PARTITURAS

PID2020-118447RA-I00

•
Nombre agencia financiadora Agencia Estatal de Investigación
Acrónimo agencia financiadora AEI
Programa Programa Estatal de I+D+i Orientado a los Retos de la Sociedad
Convocatoria Proyectos de I+D+i (Generación de Conocimiento y Retos Investigación)
Año convocatoria 2020
Plan Estatal 2017-2020 Plan Estatal de Investigación Científica y Técnica y de Innovación
Centro beneficiario UNIVERSIDAD DE ALICANTE

Resultados relacionados

Resultados totales (Incluyendo duplicados): 25
Encontrada(s) 1 página(s)

Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Multilabel prototype generation for data reduction in k-nearest neighbour classification

  • Valero-Mas, Jose J.
  • Gallego, Antonio Javier
  • Alonso-Jiménez, Pablo
  • Serra, Xavier
Prototype Generation (PG) methods are typically considered for improving the efficiency of the k-Nearest Neighbour (kNN) classifier when tackling high-size corpora. Such approaches aim at generating a reduced version of the corpus without decreasing the classification performance when compared to the initial set. Despite their large application in multiclass scenarios, very few works have addressed the proposal of PG methods for the multilabel space. In this regard, this work presents the novel adaptation of four multiclass PG strategies to the multilabel case. These proposals are evaluated with three multilabel kNN-based classifiers, 12 corpora comprising a varied range of domains and corpus sizes, and different noise scenarios artificially induced in the data. The results obtained show that the proposed adaptations are capable of significantly improving—both in terms of efficiency and classification performance—the only reference multilabel PG work in the literature as well as the case in which no PG method is applied, also presenting statistically superior robustness in noisy scenarios. Moreover, these novel PG strategies allow prioritising either the efficiency or efficacy criteria through its configuration depending on the target scenario, hence covering a wide area in the solution space not previously filled by other works., This research was partially funded by the Spanish Ministerio de Ciencia e Innovación through the MultiScore (PID2020-118447RA-I00) and DOREMI (TED2021-132103A-I00) projects. The first author is supported by grant APOSTD/2020/256 from “Programa I+D+i de la Generalitat Valenciana”.
RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publicación Artículo científico (JournalArticle)

End-to-End page-Level assessment of handwritten text recognition

  • Vidal, Enrique|||0000-0003-4579-5196
  • Toselli, Alejandro Héctor|||0000-0001-6955-9249
  • Ríos-Vila, Antonio
  • Calvo-Zaragoza, Jorge
[EN] The evaluation of Handwritten Text Recognition (HTR) systems has traditionally used metrics based on the edit distance between HTR and ground truth (GT) transcripts, at both the character and word levels. This is very adequate when the experimental protocol assumes that both GT and HTR text lines are the same, which allows edit distances to be independently computed to each given line. Driven by recent ad-vances in pattern recognition, HTR systems increasingly face the end-to-end page-level transcription of a document, where the precision of locating the different text lines and their corresponding reading order (RO) play a key role. In such a case, the standard metrics do not take into account the inconsistencies that might appear. In this paper, the problem of evaluating HTR systems at the page level is introduced in detail. We analyse the convenience of using a two-fold evaluation, where the transcription accuracy and the RO goodness are considered separately. Different alternatives are proposed, analysed and empir-ically compared both through partially simulated and through real, full end-to-end experiments. Results support the validity of the proposed two-fold evaluation approach. An important conclusion is that such an evaluation can be adequately achieved by just two simple and well-known metrics: the Word Error Rate (WER), that takes transcription sequentiality into account, and the here re-formulated Bag of Words Word Error Rate (bWER), that ignores order. While the latter directly and very accurately assess intrin-sic word recognition errors, the difference between both metrics (AWER) gracefully correlates with the Normalised Spearman's Foot Rule Distance (NSFD), a metric which explicitly measures RO errors associ-ated with layout analysis flaws. To arrive to these conclusions, we have introduced another metric called Hungarian Word Word Rate (hWER), based on a here proposed regularised version of the Hungarian Al-gorithm. This metric is shown to be always almost identical to bWER and both bWER and hWER are also almost identical to WER whenever HTR transcripts and GT references are guarantee to be in the same RO., This paper is part of the I+D+i projects: PID2020-118447RA-I00 (MultiScore) and PID2020-116813RB-I00a (SimancasSearch), funded by MCIN/AEI/10.13039/501100011033. The first author research was developed in part with the Valencian Graduate School and Research Network of Artificial Intelligence (valgrAI, co-funded by Generalitat Valenciana and the European Union). The second author is supported by a Maria Zambrano grant from the Spanish Ministerio de Universidades and the European Union NextGenerationEU/PRTR. The third author is supported by grant ACIF/2021/356 from the "Programa I+D+i de la Generalitat Valenciana".
RUA. Repositorio Institucional de la Universidad de Alicante
Publicación Otros (Other)

Few-Shot Symbol Classification via Self-Supervised Learning and Nearest Neighbor

  • Alfaro-Contreras, María
  • Ríos-Vila, Antonio
  • Valero-Mas, Jose J.
  • Calvo-Zaragoza, Jorge
  • Calvo-Zaragoza, Jorge|||/items/a4f3686d-f0fe-4317-a73c-cd711ceb1dde
  • Ríos-Vila, Antonio|||/items/f47307e4-fa30-4f8d-b90d-b6d71e26d89b
  • Valero-Mas, Jose J.|||/items/8ad448df-2d0f-41e8-8403-cd85202b7263
The recognition of symbols within document images is one of the most relevant steps involved in the Document Analysis field. While current state-of-the-art methods based on Deep Learning are capable of adequately performing this task, they generally require a vast amount of data that has to be manually labeled. In this paper, we propose a self-supervised learning-based method that addresses this task by training a neural-based feature extractor with a set of unlabeled documents and performs the recognition task considering just a few reference samples. Experiments on different corpora comprising music, text, and symbol documents report that the proposal is capable of adequately tackling the task with high accuracy rates of up to 95% in few-shot settings. Moreover, results show that the presented strategy outperforms the base supervised learning approaches trained with the same amount of data that, in some cases, even fail to converge. This approach, hence, stands as a lightweight alternative to deal with symbol classification with few annotated data.
RUA. Repositorio Institucional de la Universidad de Alicante
Publicación Otros (Other)

End-to-End Page-Level Assessment of Handwritten Text Recognition

  • Vidal, Enrique
  • Toselli, Alejandro H.
  • Ríos-Vila, Antonio
  • Calvo-Zaragoza, Jorge
  • Calvo-Zaragoza, Jorge|||/items/a4f3686d-f0fe-4317-a73c-cd711ceb1dde
  • Ríos-Vila, Antonio|||/items/f47307e4-fa30-4f8d-b90d-b6d71e26d89b
The evaluation of Handwritten Text Recognition (HTR) systems has traditionally used metrics based on the edit distance between HTR and ground truth (GT) transcripts, at both the character and word levels. This is very adequate when the experimental protocol assumes that both GT and HTR text lines are the same, which allows edit distances to be independently computed to each given line. Driven by recent advances in pattern recognition, HTR systems increasingly face the end-to-end page-level transcription of a document, where the precision of locating the different text lines and their corresponding reading order (RO) play a key role. In such a case, the standard metrics do not take into account the inconsistencies that might appear. In this paper, the problem of evaluating HTR systems at the page level is introduced in detail. We analyse the convenience of using a two-fold evaluation, where the transcription accuracy and the RO goodness are considered separately. Different alternatives are proposed, analysed and empirically compared both through partially simulated and through real, full end-to-end experiments. Results support the validity of the proposed two-fold evaluation approach. An important conclusion is that such an evaluation can be adequately achieved by just two simple and well-known metrics: the Word Error Rate (WER), that takes transcription sequentiality into account, and the here re-formulated Bag of Words Word Error Rate (bWER), that ignores order. While the latter directly and very accurately assess intrinsic word recognition errors, the difference between both metrics (ΔWER) gracefully correlates with the Normalised Spearman’s Foot Rule Distance (NSFD), a metric which explicitly measures RO errors associated with layout analysis flaws. To arrive to these conclusions, we have introduced another metric called Hungarian Word Word Rate (hWER), based on a here proposed regularised version of the Hungarian Algorithm. This metric is shown to be always almost identical to bWER and both bWER and hWER are also almost identical to WER whenever HTR transcripts and GT references are guarantee to be in the same RO.
RUA. Repositorio Institucional de la Universidad de Alicante
Publicación Otros (Other)

Decoupling music notation to improve end-to-end Optical Music Recognition

  • Alfaro-Contreras, María
  • Ríos-Vila, Antonio
  • Valero-Mas, Jose J.
  • Iñesta, José M.
  • Calvo-Zaragoza, Jorge
  • Calvo-Zaragoza, Jorge|||/items/a4f3686d-f0fe-4317-a73c-cd711ceb1dde
  • Iñesta, José M.|||/items/4c7c78fb-09ed-43c3-a8ea-0c16445669d0
  • Ríos-Vila, Antonio|||/items/f47307e4-fa30-4f8d-b90d-b6d71e26d89b
  • Valero-Mas, Jose J.|||/items/8ad448df-2d0f-41e8-8403-cd85202b7263
Inspired by the Text Recognition field, end-to-end schemes based on Convolutional Recurrent Neural Networks (CRNN) trained with the Connectionist Temporal Classification (CTC) loss function are considered one of the current state-of-the-art techniques for staff-level Optical Music Recognition (OMR). Unlike text symbols, music-notation elements may be defined as a combination of (i) a shape primitive located in (ii) a certain position in a staff. However, this double nature is generally neglected in the learning process, as each combination is treated as a single token. In this work, we study whether exploiting such particularity of music notation actually benefits the recognition performance and, if so, which approach is the most appropriate. For that, we thoroughly review existing specific approaches that explore this premise and propose different combinations of them. Furthermore, considering the limitations observed in such approaches, a novel decoding strategy specifically designed for OMR is proposed. The results obtained with four different corpora of historical manuscripts show the relevance of leveraging this double nature of music notation since it outperforms the standard approaches where it is ignored. In addition, the proposed decoding leads to significant reductions in the error rates with respect to the other cases.
RUA. Repositorio Institucional de la Universidad de Alicante
Publicación Otros (Other)

End-To-End Full-Page Optical Music Recognition of Monophonic Documents via Score Unfolding

  • Ríos-Vila, Antonio
  • Iñesta, José M.
  • Calvo-Zaragoza, Jorge
  • Calvo-Zaragoza, Jorge|||/items/a4f3686d-f0fe-4317-a73c-cd711ceb1dde
  • Iñesta, José M.|||/items/4c7c78fb-09ed-43c3-a8ea-0c16445669d0
  • Ríos-Vila, Antonio|||/items/f47307e4-fa30-4f8d-b90d-b6d71e26d89b
Full Page Optical Music Recognition (OMR) systems typically consist of multi-step workflows. However, the fine-tuning of these systems tends to be costly. We present the first layout analysis-free full-page OMR model that receives a page image and directly outputs its transcription in a single step. This model requires only the annotations of full score pages during training. The model has been tested with early-notation monophonic music scores, for which the presented approach is especially beneficial. Results show that this methodology provides a solution with promising results and establishes a new line of research for end-to-end music transcription.
RUA. Repositorio Institucional de la Universidad de Alicante
Publicación Otros (Other)

Exploring recursive neural networks for compact handwritten text recognition models

  • Mas Candela, Enrique
  • Calvo-Zaragoza, Jorge
  • Calvo-Zaragoza, Jorge|||/items/a4f3686d-f0fe-4317-a73c-cd711ceb1dde
  • Mas Candela, Enrique|||/items/41000b26-081c-4d3c-ba51-54e0ca95db6b
This paper addresses the challenge of deploying recognition models in specific scenarios in which memory size is relevant, such as in low-cost devices or browser-based applications. We specifically focus on developing memory-efficient approaches for Handwritten Text Recognition (HTR) by leveraging recursive networks. These networks reuse learned weights across successive layers, thus enabling the maintenance of depth, a critical factor associated with model accuracy, without an increase in memory footprint. We apply neural recursion techniques to models typically used in HTR that contain convolutional and recurrent layers. We additionally study the impact of kernel scaling, which allows the activations of these recursive layers to be modified for greater expressiveness with little cost to memory. Our experiments on various HTR benchmarks demonstrate that recursive networks are, indeed, a good alternative. It is noteworthy that these recursive networks not only preserve but in some instances also enhance accuracy, making them a promising solution for memory-efficient HTR applications. This research establishes the utility of recursive networks in addressing memory constraints in HTR models. Their ability to sustain or improve accuracy while being memory-efficient positions them as a promising solution for practical deployment, especially in contexts where memory size is a critical consideration, such as low-cost devices and browser-based applications.
RUA. Repositorio Institucional de la Universidad de Alicante
Publicación Otros (Other)

Multilabel Prototype Generation for data reduction in K-Nearest Neighbour classification

  • Valero-Mas, Jose J.
  • Gallego, Antonio-Javier
  • Alonso-Jiménez, Pablo
  • Serra, Xavier
  • Gallego, Antonio-Javier|||/items/8f865afc-8f63-4e0d-9408-bebc7ba836d0
  • Valero-Mas, Jose J.|||/items/8ad448df-2d0f-41e8-8403-cd85202b7263
Prototype Generation (PG) methods are typically considered for improving the efficiency of the k-Nearest Neighbour (kNN) classifier when tackling high-size corpora. Such approaches aim at generating a reduced version of the corpus without decreasing the classification performance when compared to the initial set. Despite their large application in multiclass scenarios, very few works have addressed the proposal of PG methods for the multilabel space. In this regard, this work presents the novel adaptation of four multiclass PG strategies to the multilabel case. These proposals are evaluated with three multilabel kNN-based classifiers, 12 corpora comprising a varied range of domains and corpus sizes, and different noise scenarios artificially induced in the data. The results obtained show that the proposed adaptations are capable of significantly improving—both in terms of efficiency and classification performance—the only reference multilabel PG work in the literature as well as the case in which no PG method is applied, also presenting statistically superior robustness in noisy scenarios. Moreover, these novel PG strategies allow prioritising either the efficiency or efficacy criteria through its configuration depending on the target scenario, hence covering a wide area in the solution space not previously filled by other works.
RUA. Repositorio Institucional de la Universidad de Alicante
Publicación Otros (Other)

Encoding Polyphony from Medieval Manuscripts Notated in Mensural Notation

  • Desmond, Karen
  • Pugin, Laurent
  • Regimbal, Juliette
  • Rizo, David
  • Sapp, Craig
  • Thomae, Martha E.
  • Rizo, David|||/items/447626c0-6412-451c-8f26-b4a244418da2
This panel submission for the 2021 Music Encoding Conference brings together five short papers that focus on the making of computer-readable encodings of polyphony in the notational style – mensural notation – in which it was originally copied. Mensural notation was used in the medieval West to encode polyphony from the late thirteenth to sixteenth centuries. The Measuring Polyphony (MP) Online Editor, funded by an NEH Digital Humanities Advancement Grant, is a software that enables non-technical users to make Humdrum and MEI encodings of mensural notation, and links these encodings to digital images of the manuscripts in which these compositions were first notated. Topics explored by the authors include: the processes of, and the goals informing, the linking of manuscript images to music encodings; choices and compromises made in the development process of the MP Editor in order to facilitate its rapid deployment; and the implications of capturing dual encodings – a parts-based encoding that reflects the layout of the original source, and a score-based encoding. Having two encodings of the music data is useful for a variety of activities, including performance and analysis, but also within the editorial process, and for sharing data with other applications. The authors present two case studies that document the possibilities and potential in the interchange of music data between the MP Editor and other applications, specifically, MuRET, an optical music recognition (OMR) tool, and Humdrum analysis tools.
RUA. Repositorio Institucional de la Universidad de Alicante
Publicación Otros (Other)

End-to-end Optical Music Recognition Beyond Staff-Level Transcription

  • Ríos-Vila, Antonio
  • Ríos-Vila, Antonio|||/items/f47307e4-fa30-4f8d-b90d-b6d71e26d89b
El Reconocimiento Óptico de Música (Optical Music Recognition, OMR) es un campo de investigación que estudia cómo leer computacionalmente la notación musical presente en documentos y almacenarla en un formato digital estructurado. Los enfoques tradicionales de OMR suelen estructurarse en torno a un proceso de varias etapas: (i) preprocesamiento de imágenes, donde se abordan cuestiones relacionadas con el proceso de escaneado y la calidad del papel, (ii) segmentación y clasificación de símbolos, donde se detectan y etiquetan los distintos elementos de la imagen, (iii) reconstrucción de la notación musical, una fase de postprocesamiento del proceso de reconocimiento, y (iv) codificación de resultados, donde los elementos reconocidos se almacenan en un formato simbólico adecuado. Estos sistemas logran tasas de reconocimiento competitivas a costa de utilizar determinadas heurísticas, adaptadas a los casos para los que fueron diseñados. En consecuencia, la escalabilidad se convierte en una limitación importante, ya que para cada colección o tipo notacional es necesario diseñar un nuevo conjunto de heurísticas. Además, otro inconveniente de estos enfoques tradicionales es la necesidad de un etiquetado detallado, a menudo obtenido manualmente. Dado que cada símbolo se reconoce individualmente, se requieren las posiciones exactas de cada uno de ellos, junto con sus correspondientes etiquetas musicales. La integración del Aprendizaje Profundo (Deep Learning, DL) en el campo del OMR ha marcado un punto de inflexión hacia la adopción de sistemas holísticos o de extremo a extremo. Estos sistemas, fundamentados en la inteligencia artificial y las redes neuronales profundas, abordan la segmentación y la clasificación de símbolos musicales como un proceso unificado, en lugar de fraccionarlo en múltiples etapas discretas. La metodología permite que el aprendizaje de la extracción de características y la clasificación se realice de manera simultánea, eliminando la necesidad de desarrollar y ajustar procedimientos específicos para cada tarea. La clave de este enfoque radica en el uso de conjuntos de datos compuestos por imágenes de partituras y sus transcripciones correspondientes, obviando la necesidad de marcar la posición exacta de cada símbolo. Así, el avance simplifica significativamente el proceso de transcripción musical, al permitir que las características relevantes para la clasificación sean aprendidas directamente de los datos, sin intervención manual detallada en el etiquetado de elementos individuales. El paradigma de procesamiento de extremo a extremo ha sido objeto de análisis en investigaciones recientes. Estos trabajos, si bien avanzan bajo la premisa de que una fase de preprocesamiento específica ya ha llevado a cabo la segmentación de los pentagramas en las partituras, centran su atención en a recuperación de secuencias de símbolos musicales a partir de imágenes de pentagramas. En este ámbito, las Redes Neuronales Convolucionales Recurrentes (CRNN) son la solución más popular. En estas, el componente convolucional se dedica a la extracción de características significativas de las imágenes, mientras que las capas recurrentes se encargan de interpretar estas características como secuencias de símbolos musicales. Los resultados actuales de OMR han demostrado una gran precisión para transcribir partituras musicales, incluso en los casos más complejos. Estos avances permiten el planteamiento de metas más ambiciosas. Una línea de trabajo destacable es la del OMR universal. Un sistema de transcripción universal de música es aquel capaz de transcribir el contenido de cualquier documento musical. Esto significa que, independientemente de las características y la notación de dicho documento, el modelo es capaz transcribir, en una notación adecuada, y generar la versión digital del mismo. El OMR universal es un modelo ideal por diversas razones. La primera es práctica, ya que facilita el trabajo de los usuarios finales, quienes precisan actualmente de herramientas específicas para cada tipo de partitura musical. La producción de un transcriptor universal permitiría juntar estos programas en herramientas genéricas capaces de cubrir todo el espectro de necesidades de los usuarios, lo cual reduce el coste de procesamiento y mantenimiento de los documentos musicales. Desde un punto de vista científico, esta técnica desbloquearía el potencial de los modelos basados en aprendizaje automático para leer e interpretar documentos musicales, ya que lo harían desde un conocimiento genérico. El logro permite abordar tareas más complejas que necesitan de esta información, pero van más allá de ella. Algunas de estas tareas serían la detección de patrones de autor, la estimación de la dificultad de una partitura o la clasificación por época. Sin embargo, el estado de la cuestión de OMR no es capaz de abordar tal objetivo todavía, debido a una serie de limitaciones. En esta tesis, se proponen trabajos que avanzan el estado de la cuestión de OMR hacia ese objetivo. En primer lugar, se proponen contribuciones para completar los sistemas de OMR, los cuales no son capaces de exportar sus resultados en formatos compatibles con las herramientas musicológicas más comunes. Una vez obtenido un sistema de OMR completo, se proponen trabajos para abordar los problemas de Aligned Music Notation & Lyrics Transcription y polifonía, los cuales son retos relevantes que la literatura no ha abordado (por dificultad). De esta forma, mediante adaptaciones de los sistemas actuales, se avanza el estado de la cuestión en estos temas. Finalmente, se abordan los sistemas libres de segmentación para transcribir páginas musicales, liberando así a los modelos OMR de su estructura secuencial de segmentación y transcripción. En concreto, las investigaciones se enfocan hacia el Sheet Music Transformer, un modelo de transcripción basado en tecnologías de vanguardia para obtener la transcripción de una partitura directamente desde la imagen de su página.
RUA. Repositorio Institucional de la Universidad de Alicante
Publicación Otros (Other)

Domain adaptation for staff-region retrieval of music score images

  • Castellanos, Francisco J.
  • Gallego, Antonio-Javier
  • Calvo-Zaragoza, Jorge
  • Fujinaga, Ichiro
  • Calvo-Zaragoza, Jorge|||/items/a4f3686d-f0fe-4317-a73c-cd711ceb1dde
  • Castellanos, Francisco J.|||/items/6b515109-60a8-4a00-a12d-0726355ae966
  • Gallego, Antonio-Javier|||/items/8f865afc-8f63-4e0d-9408-bebc7ba836d0
Optical music recognition (OMR) is the field that studies how to automatically read music notation from score images. One of the relevant steps within the OMR workflow is the staff-region retrieval. This process is a key step because any undetected staff will not be processed by the subsequent steps. This task has previously been addressed as a supervised learning problem in the literature; however, ground-truth data are not always available, so each new manuscript requires a preliminary manual annotation. This situation is one of the main bottlenecks in OMR, because of the countless number of existing manuscripts , and the associated manual labeling cost. With the aim of mitigating this issue, we propose the application of a domain adaptation technique, the so-called Domain-Adversarial Neural Network (DANN), based on a combination of a gradient reversal layer and a domain classifier in the inference neural architecture. The results from our experiments support the benefits of our proposed solution, obtaining improvements of approximately 29% in the F-score.
RUA. Repositorio Institucional de la Universidad de Alicante
Publicación Otros (Other)

Few-Shot Music Symbol Classification via Self-Supervised Learning and Nearest Neighbor

  • Alfaro-Contreras, María
The recognition of music symbols within score images represents one of the main stages in Optical Music Recognition systems. While current state-of-the-art methods based on Deep Learning are capable of adequately performing this task, they generally require a vast amount of data that has to be manually labeled. Such a particularity generally limits their applicability when addressing historical manuscripts with early music notation, for which annotated data is considerably scarce. In this paper, we propose a self-supervised learning-based method that addresses this task by training a neural-based feature extractor with a set of unlabeled documents and performs the recognition task considering just a few reference samples. Experiments on a reference early music corpus report that the proposal outperforms the contemplated baseline strategies even with a remarkably reduced number of labeled examples for the classification task.
RUA. Repositorio Institucional de la Universidad de Alicante
Software de investigación Software (Software)

Sheet Music Transformer, SMT

  • Ríos-Vila, Antonio
  • Calvo-Zaragoza, Jorge
  • Rizo, David
  • Paquet, Thierry
  • Ríos-Vila, Antonio|||/items/f47307e4-fa30-4f8d-b90d-b6d71e26d89b
  • Calvo-Zaragoza, Jorge|||/items/a4f3686d-f0fe-4317-a73c-cd711ceb1dde
  • Rizo, David|||/items/447626c0-6412-451c-8f26-b4a244418da2
The Sheet Music Transfomrmer (SMT) is a novel model for Optical Music Recognition (OMR) beyond monophonic level transcription. Unlike traditional approaches that primarily leverage monophonic transcription techniques for complex score layouts, the SMT model overcomes these limitations by offering a robust image-to-sequence solution for transcribing polyphonic musical scores directly from images.
RUA. Repositorio Institucional de la Universidad de Alicante
Publicación Otros (Other)

Optical music recognition for homophonic scores with neural networks and synthetic music generation

  • Alfaro-Contreras, María
  • Iñesta, José M.
  • Calvo-Zaragoza, Jorge
  • Calvo-Zaragoza, Jorge|||/items/a4f3686d-f0fe-4317-a73c-cd711ceb1dde
  • Iñesta, José M.|||/items/4c7c78fb-09ed-43c3-a8ea-0c16445669d0
The recognition of patterns that have a time dependency is common in areas like speech recognition or natural language processing. The equivalent situation in image analysis is present in tasks like text or video recognition. Recently, Convolutional Recurrent Neural Networks (CRNN) have been broadly applied to solve these tasks in an end-to-end fashion with successful performance. However, its application to Optical Music Recognition (OMR) is not so straightforward due to the presence of different elements sharing the same horizontal position, disrupting the linear flow of the timeline. In this paper, we study the ability of the state-of-the-art CRNN approach to learn codes that represent this disruption in homophonic scores. In our experiments, we study the lower bounds in the recognition task of real scores when the models are trained with synthetic data. Two relevant conclusions are drawn: (1) Our serialized ways of encoding the music content are appropriate for CRNN-based OMR; (2) the learning process is possible with synthetic data, but there exists a glass ceiling when recognizing real sheet music.
RUA. Repositorio Institucional de la Universidad de Alicante
Software de investigación Software (Software)

MuRET v1.0, Aplicación online de reconocimiento óptico de música y transcripción

  • Rizo, David
  • Calvo-Zaragoza, Jorge
  • Calvo-Zaragoza, Jorge|||/items/a4f3686d-f0fe-4317-a73c-cd711ceb1dde
  • Rizo, David|||/items/447626c0-6412-451c-8f26-b4a244418da2
MuRET es el acrónimo de “Music, recognition, encoding, and transcription”. El software que registramos consiste en una aplicación web que ayuda al usuario a convertir un conjunto de imágenes que contienen notación musical en una partitura digital codificada en formato MEI (Music Encoding Initiative). No se trata sólo de un OMR (optical music recognition) sino que incluye todas las funcionalidades necesarias para obtener una transcripción completa de una obra a partir de partes separadas en ficheros de entrada.
RUA. Repositorio Institucional de la Universidad de Alicante
Publicación Otros (Other)

Insights into end-to-end audio-to-score transcription with real recordings: A case study with saxophone works

  • Martínez Sevilla, Juan Carlos
  • Alfaro-Contreras, María
  • Valero-Mas, Jose J.
  • Calvo-Zaragoza, Jorge
  • Calvo-Zaragoza, Jorge|||/items/a4f3686d-f0fe-4317-a73c-cd711ceb1dde
  • Martínez Sevilla, Juan Carlos|||/items/f70a7dc5-bace-42e3-b5bd-6affe80a6e53
  • Valero-Mas, Jose J.|||/items/8ad448df-2d0f-41e8-8403-cd85202b7263
Neural end-to-end Audio-to-Score (A2S) transcription aims to retrieve a score that encodes the music content of an audio recording in a single step. Due to the recentness of this formulation, the existing works have exclusively addressed controlled scenarios with synthetic data that fail to provide conclusions applicable to real-world cases. In response to this gap in the literature, this work introduces a novel assortment of real saxophone recordings---together with their digital scores---and poses several experimental scenarios involving real and synthetic data. The obtained results confirm the adequacy of this A2S framework to deal with real data as well as proving the relevance of leveraging synthetic interpretations to improve the recognition rate in scenarios with real-data scarcity.
RUA. Repositorio Institucional de la Universidad de Alicante
Publicación Otros (Other)

Music Encoding Conference Proceedings 2021, 19–22 July, 2021 University of Alicante (Spain): Onsite & Online

  • Münnich, Stefan
  • Rizo, David
  • Rizo, David|||/items/447626c0-6412-451c-8f26-b4a244418da2
Este documento incluye los artículos y pósters presentados en el Music Encoding Conference 2021 realizado en Alicante entre el 19 y el 22 de julio de 2022.
RUA. Repositorio Institucional de la Universidad de Alicante
Publicación Otros (Other)

Region-based layout analysis of music score images

  • Castellanos, Francisco J.
  • Garrido Muñoz, Carlos
  • Ríos-Vila, Antonio
  • Calvo-Zaragoza, Jorge
  • Calvo-Zaragoza, Jorge|||/items/a4f3686d-f0fe-4317-a73c-cd711ceb1dde
  • Castellanos, Francisco J.|||/items/6b515109-60a8-4a00-a12d-0726355ae966
  • Garrido Muñoz, Carlos|||/items/e3da0db8-e4ef-4921-ba71-04b76ae521e3
  • Ríos-Vila, Antonio|||/items/f47307e4-fa30-4f8d-b90d-b6d71e26d89b
The Layout Analysis (LA) stage is of vital importance to the correct performance of an Optical Music Recognition (OMR) system. It identifies the regions of interest, such as staves or lyrics, which must then be processed in order to transcribe their content. Despite the existence of modern approaches based on deep learning, an exhaustive study of LA in OMR has not yet been carried out with regard to the performance of different models, their generalization to different domains or, more importantly, their impact on subsequent stages of the pipeline. This work focuses on filling this gap in the literature by means of an experimental study of different neural architectures, music document types, and evaluation scenarios. The need for training data has also led to a proposal for a new semi-synthetic data-generation technique that enables the efficient applicability of LA approaches in real scenarios. Our results show that: (i) the choice of the model and its performance are crucial for the entire transcription process; (ii) the metrics commonly used to evaluate the LA stage do not always correlate with the final performance of the OMR system, and (iii) the proposed data-generation technique enables state-of-the-art results to be achieved with a limited set of labeled data.
RUA. Repositorio Institucional de la Universidad de Alicante
Publicación Otros (Other)

Rotations Are All You Need: A Generic Method For End-To-End Optical Music Recognition

  • Ríos-Vila, Antonio
  • Ríos-Vila, Antonio|||/items/f47307e4-fa30-4f8d-b90d-b6d71e26d89b
End-to-end Optical Music Recognition traditionally involves multi-step processes to address complex documents, where single stave isolation is performed. These state-of-the art methods often fall short in handling diverse music textures, such as polyphony or Aligned Music Notation and Lyrics Transcription (AMNLT). We introduce a generic end-to-end OMR approach compatible with monophonic, polyphonic, and AMNLT systems. By leveraging score rotations and multi-line transcription unfolding, this model only requires input system annotations during training. Experimental evidence suggests this approach offers a competitive solution with encouraging outcomes, paving the way for future end-to-end music transcription research.
RUA. Repositorio Institucional de la Universidad de Alicante
Publicación Otros (Other)

Late multimodal fusion for image and audio music transcription

  • Alfaro-Contreras, María
  • Valero-Mas, Jose J.
  • Iñesta, José M.
  • Calvo-Zaragoza, Jorge
  • Calvo-Zaragoza, Jorge|||/items/a4f3686d-f0fe-4317-a73c-cd711ceb1dde
  • Iñesta, José M.|||/items/4c7c78fb-09ed-43c3-a8ea-0c16445669d0
  • Valero-Mas, Jose J.|||/items/8ad448df-2d0f-41e8-8403-cd85202b7263
Music transcription, which deals with the conversion of music sources into a structured digital format, is a key problem for Music Information Retrieval (MIR). When addressing this challenge in computational terms, the MIR community follows two lines of research: music documents, which is the case of Optical Music Recognition (OMR), or audio recordings, which is the case of Automatic Music Transcription (AMT). The different nature of the aforementioned input data has conditioned these fields to develop modality-specific frameworks. However, their recent definition in terms of sequence labeling tasks leads to a common output representation, which enables research on a combined paradigm. In this respect, multimodal image and audio music transcription comprises the challenge of effectively combining the information conveyed by image and audio modalities. In this work, we explore this question at a late-fusion level: we study four combination approaches in order to merge, for the first time, the hypotheses regarding end-to-end OMR and AMT systems in a lattice-based search space. The results obtained for a series of performance scenarios–in which the corresponding single-modality models yield different error rates–showed interesting benefits of these approaches. In addition, two of the four strategies considered significantly improve the corresponding unimodal standard recognition frameworks.
RUA. Repositorio Institucional de la Universidad de Alicante
Publicación Otros (Other)

End-to-end optical music recognition for pianoform sheet music

  • Ríos-Vila, Antonio
  • Rizo, David
  • Iñesta, José M.
  • Calvo-Zaragoza, Jorge
  • Calvo-Zaragoza, Jorge|||/items/a4f3686d-f0fe-4317-a73c-cd711ceb1dde
  • Iñesta, José M.|||/items/4c7c78fb-09ed-43c3-a8ea-0c16445669d0
  • Ríos-Vila, Antonio|||/items/f47307e4-fa30-4f8d-b90d-b6d71e26d89b
  • Rizo, David|||/items/447626c0-6412-451c-8f26-b4a244418da2
End-to-end solutions have brought about significant advances in the field of Optical Music Recognition. These approaches directly provide the symbolic representation of a given image of a musical score. Despite this, several documents, such as pianoform musical scores, cannot yet benefit from these solutions since their structural complexity does not allow their effective transcription. This paper presents a neural method whose objective is to transcribe these musical scores in an end-to-end fashion. We also introduce the GrandStaff dataset, which contains 53,882 single-system piano scores in common western modern notation. The sources are encoded in both a standard digital music representation and its adaptation for current transcription technologies. The method proposed in this paper is trained and evaluated using this dataset. The results show that the approach presented is, for the first time, able to effectively transcribe pianoform notation in an end-to-end manner.
RUA. Repositorio Institucional de la Universidad de Alicante
Publicación Otros (Other)

MUSCAT: A Multimodal mUSic Collection for Automatic Transcription of Real Recordings and Image Scores

  • Galán Cuenca, Alejandro
  • Valero-Mas, Jose J.
  • Martínez Sevilla, Juan Carlos
  • Hidalgo Centeno, Antonio
  • Pertusa, Antonio
  • Calvo-Zaragoza, Jorge
  • Galán Cuenca, Alejandro|||/items/d81019ac-5845-4e4f-8af5-9c00dc6d1467
  • Valero-Mas, Jose J.|||/items/8ad448df-2d0f-41e8-8403-cd85202b7263
  • Martínez Sevilla, Juan Carlos|||/items/f70a7dc5-bace-42e3-b5bd-6affe80a6e53
  • Pertusa, Antonio|||/items/3bf633ff-4d49-4aaa-a004-abcf6fed3540
  • Calvo-Zaragoza, Jorge|||/items/a4f3686d-f0fe-4317-a73c-cd711ceb1dde
  • Hidalgo Centeno, Antonio|||/items/e6f2276c-913f-4a5f-97ae-2b4c768fa99d
Multimodal audio-image music transcription has been recently posed as a means of retrieving a digital score representation by leveraging the individual estimations from Automatic Music Transcription (AMT)---acoustic recordings---and Optical Music Recognition (OMR)---image scores---systems. Nevertheless, while proven to outperform single-modality recognition rates, this approach has been exclusively validated under controlled scenarios---monotimbral and monophonic synthetic data---mainly due to a lack of collections with symbolic score-level annotations for both recordings and graphical sheets. To promote research on this topic, this work presents the Multimodal mUSic Collection for Automatic Transcription (MUSCAT) assortment of acoustic recordings, image sheets, and their score-level annotations in several notation formats. This dataset comprises almost 80 hours of real recordings with varied instrumentation and polyphony degrees---ranging from piano to orchestral music---, 1251 scanned sheets, and 880 symbolic scores from 37 composers, which may also be used in other tasks involving metadata such as instrument identification or composer recognition. A fragmented subset of this collection solely focused on acoustic data for score-level AMT---the MUSic Collection for aUtomatic Transcription - fragmented Subset (MUSCUTS) assortment---is also presented together with a baseline experimentation, concluding the need to foster research on this field with real recordings. Finally, a web-based service is also provided to increase the size of the collections collaboratively.
RUA. Repositorio Institucional de la Universidad de Alicante
Publicación Otros (Other)

Multimodal image and audio music transcription

  • Fuente Torres, Carlos de la
  • Valero-Mas, Jose J.
  • Castellanos, Francisco J.
  • Calvo-Zaragoza, Jorge
  • Calvo-Zaragoza, Jorge|||/items/a4f3686d-f0fe-4317-a73c-cd711ceb1dde
  • Castellanos, Francisco J.|||/items/6b515109-60a8-4a00-a12d-0726355ae966
  • Valero-Mas, Jose J.|||/items/8ad448df-2d0f-41e8-8403-cd85202b7263
Optical Music Recognition (OMR) and Automatic Music Transcription (AMT) stand for the research fields that aim at obtaining a structured digital representation from sheet music images and acoustic recordings, respectively. While these fields have traditionally evolved independently, the fact that both tasks may share the same output representation poses the question of whether they could be combined in a synergistic manner to exploit the individual transcription advantages depicted by each modality. To evaluate this hypothesis, this paper presents a multimodal framework that combines the predictions from two neural end-to-end OMR and AMT systems by considering a local alignment approach. We assess several experimental scenarios with monophonic music pieces to evaluate our approach under different conditions of the individual transcription systems. In general, the multimodal framework clearly outperforms the single recognition modalities, attaining a relative improvement close to 40% in the best case. Our initial premise is, therefore, validated, thus opening avenues for further research in multimodal OMR-AMT transcription.
RUA. Repositorio Institucional de la Universidad de Alicante
Publicación Otros (Other)

Neural Audio-To-Score Music Transcription For Unconstrained Polyphony Using Compact Output Representations

  • Arroyo, Victor
  • Valero-Mas, Jose J.
  • Calvo-Zaragoza, Jorge
  • Pertusa, Antonio
  • Calvo-Zaragoza, Jorge|||/items/a4f3686d-f0fe-4317-a73c-cd711ceb1dde
  • Pertusa, Antonio|||/items/3bf633ff-4d49-4aaa-a004-abcf6fed3540
  • Valero-Mas, Jose J.|||/items/8ad448df-2d0f-41e8-8403-cd85202b7263
Neural Audio-to-Score (A2S) Music Transcription systems have shown promising results with pieces containing a fixed number of voices. However, they still exhibit fundamental limitations that constrain their applicability in wider scenarios. This work aims at tackling two of them: we introduce a novel output representation which addresses shortcomings related to the sequence-based A2S recognition framework and we report a first approximation to dealing with unconstrained polyphony. This is validated on a Convolutional Recurrent Neural Network (CRNN) with Connectionist Temporal Classification (CTC) A2S scheme using synthetic audio from string quartets and piano sonatas with intricate polyphonic mixtures. Our results, which improve fixed-polyphony state-of-the-art rates, may be considered a reference for future A2S works dealing with an unconstrained number of voices.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Multilabel prototype generation for data reduction in k-nearest neighbour classification

  • Valero-Mas, Jose J.
  • Gallego, Antonio Javier
  • Alonso-Jiménez, Pablo
  • Serra, Xavier
Prototype Generation (PG) methods are typically considered for improving the efficiency of the k-Nearest Neighbour (kNN) classifier when tackling high-size corpora. Such approaches aim at generating a reduced version of the corpus without decreasing the classification performance when compared to the initial set. Despite their large application in multiclass scenarios, very few works have addressed the proposal of PG methods for the multilabel space. In this regard, this work presents the novel adaptation of four multiclass PG strategies to the multilabel case. These proposals are evaluated with three multilabel kNN-based classifiers, 12 corpora comprising a varied range of domains and corpus sizes, and different noise scenarios artificially induced in the data. The results obtained show that the proposed adaptations are capable of significantly improving—both in terms of efficiency and classification performance—the only reference multilabel PG work in the literature as well as the case in which no PG method is applied, also presenting statistically superior robustness in noisy scenarios. Moreover, these novel PG strategies allow prioritising either the efficiency or efficacy criteria through its configuration depending on the target scenario, hence covering a wide area in the solution space not previously filled by other works., This research was partially funded by the Spanish Ministerio de Ciencia e Innovación through the MultiScore (PID2020-118447RA-I00) and DOREMI (TED2021-132103A-I00) projects. The first author is supported by grant APOSTD/2020/256 from “Programa I+D+i de la Generalitat Valenciana”.