Pasar al contenido principal

INTELIGENCIA ARTIFICIAL DE SOPORTE A EXPERIENCIAS MUSICALES: HACIA UNA APROXIMACION BASADA EN DATOS Y CENTRADA EN EL USUARIO

PID2019-111403GB-I00

•
Nombre agencia financiadora Agencia Estatal de Investigación
Acrónimo agencia financiadora AEI
Programa Programa Estatal de I+D+i Orientado a los Retos de la Sociedad
Convocatoria Proyectos de I+D+i (Generación de Conocimiento y Retos Investigación)
Año convocatoria 2019
Plan Estatal 2017-2020 Plan Estatal de Investigación Científica y Técnica y de Innovación
Centro beneficiario UNIVERSITAT POMPEU FABRA CCT

Resultados relacionados

Resultados totales (Incluyendo duplicados): 15
Encontrada(s) 1 página(s)

Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico antes de ser publicado, versión del autor (Preprint)

Bottlenecks and solutions for audio to score alignment research

  • Morsi, Alia
  • Serra, Xavier
This work has been accepted at the 23rd International Society for Music Information Retrieval Conference (ISMIR 2022), at Bengaluru, India. December 4-8, 2022., Although audio to score alignment is a classic Music Information Retrieval problem, it has not been defined uniquely with the scope of musical scenarios representing its core. The absence of a unified vision makes it difficult to pinpoint its state-of-the-art and determine directions for improvement. To get past this bottleneck, it is necessary to consolidate datasets and evaluation methodologies to allow comprehensive benchmarking. In our review of prior work,we demonstrate the extent of variation in problem scope, datasets, and evaluation practices across audio to score alignment research. To circumvent the high cost of creating large-scale datasets with various instruments, styles, performance conditions, and musician proficiency from scratch, the research community could generate ground truth approximations from non-audio to score alignment datasets which include a temporal mapping between a music score and its corresponding audio. We show a methodology for adapting the Aligned Scores and Performances dataset, created originally for beat tracking and music transcription. We filter the dataset semi-automatically by applying a set of Dynamic Time Warping based Audio to Score Alignment methods using out-of-the-box Chroma and Constant-Q Transform extraction algorithms, suitable for the characteristics of the piano performances of the dataset. We use the results to discuss the limitations of the generated ground truths and data adaptation method. While the adapted dataset does not provide the necessary diversity for solving the initial problem, we conclude with ideas for expansion, and identify future directions for curating more comprehensive datasets through data adaptation, or synthesis., This research was carried out under the project Musical AI - PID2019-111403GB-I00/AEI/10.13039/501100011033, funded by the Spanish Ministerio de Ciencia e Innovación and the Agencia Estatal de Investigación.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico antes de ser publicado, versión del autor (Preprint)

Score-Informed MIDI Velocity Estimation for Piano Performance by FiLM Conditioning

  • Kim, Hyon
  • Miron, Marius
  • Serra, Xavier
This work has been accepted at the SMC Sound and Music Computing Conference 2023, at Stockholm, Sweden. June 15-17, 2023., Piano is one of the most popular instruments among people that learn to play music. When playing the piano, the level of loudness is crucial for expressing emotions as well as manipulating tempo. These elements convey the expressiveness of music performance. Detecting the loudness of each note could provide more valuable feedback for music students, helping to improve their performance dynamics. This can be achieved by visualizing the loudness levels not only for self-learning purposes but also for effective communication between teachers and students. Also, given the polyphonic nature of piano music, which often involves parallel melodic streams, determining the loudness of each note is more informative than analyzing the cumulative loudness of a specific time frame. This research proposes a method using Deep Neural Network (DNN) with score information to estimate note-level MIDI velocity of piano performances from audio input. In addition, when score information is available, we condition the DNN with score information using a Feature-wise Linear Modulation (FiLM) layer. To the best of our knowledge, this is the first attempt to estimate the MIDI velocity using a neural network in an end to end fashion. The model proposed in this study achieved improved accuracy in both MIDI velocity estimation and estimation error deviation, as well as higher recall accuracy for note classification when compared to the DNN model that did not use score information, This research was carried out under the project Musical AI - PID2019- 111403GB-I00/AEI/10.13039/501100011033, funded by the Spanish Ministerio de Ciencia e Innovacion ́ and the Agencia Estatal de Investigacion.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico antes de ser publicado, versión del autor (Preprint)

A diffusion-inspired training strategy for singing voice extraction in the waveform domain

  • Plaja-Roglans, Genís
  • Miron, Marius
  • Serra, Xavier
This work has been accepted at the 23rd International Society for Music Information Retrieval Conference (ISMIR 2022), at Bengaluru, India. December 4-8, 2022., Notable progress in music source separation has been achieved using multi-branch networks that operate on both temporal and spectral domains. However, such networks tend to be complex and heavy-weighted. In this work, we tackle the task of singing voice extraction from polyphonic music signals in an end-to-end manner using an approach inspired by the training procedure of denoising diffusion models. We perform unconditional signal modelling to gradually convert an input mixture signal to the corresponding singing voice or accompaniment. We use fewer parameters than the state-of-the-art models while operating on the waveform domain, bypassing phase-related problems. More concisely, we train a non-causal WaveNet using a diffusion-inspired strategy improving the said network for singing voice extraction and obtaining performance comparable to the end-to-end state-of-the-art on MUSDB18. We further report results on a non-MUSDB-overlapping version of MedleyDB and the multi-track audio of the Saraga Carnatic dataset showing good generalization, and run perceptual tests of our approach. Code, models, and audio examples are made available., This work was carried out under the projects Musical AI - PID2019-111403GB-I00/AEI/10.13039/501100011033 and NextCore - RTC2019-007248-7 funded by the Spanish Ministerio de Ciencia, Innovación y Universidades (MCIU) and the Agencia Estatal de Investigación (AEI).
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico antes de ser publicado, versión del autor (Preprint)

TapTamDrum: a dataset for dualized drum patterns

  • Haki, Behzad
  • Kotowski, Błażej
  • Lee, Cheuk Lun Isaac
  • Jordà Puig, Sergi
This work has been accepted at the 24th International Society for Music Information Retrieval Conference (ISMIR 2023), at Milan, Italy. October 5-9, 2023., Drummers spend extensive time practicing rudiments to develop technique, speed, coordination, and phrasing. These rudiments are often practiced on "silent" practice pads using only the hands. Additionally, many percussive instruments across cultures are played exclusively with the hands. Building on these concepts and inspired by Einstein's probably apocryphal quote, "Make everything as simple as possible, but not simpler," we hypothesize that a dual-voice reduction could serve as a natural and meaningful compressed representation of multi-voiced drum patterns. This representation would retain more information than its corresponding monotonic representation while maintaining relative simplicity for tasks such as rhythm analysis and generation. To validate this potential representation, we investigate whether experienced drummers can consistently represent and reproduce the rhythmic essence of a given drum pattern using only their two hands. We present TapTamDrum: a novel dataset of repeated dualizations from four experienced drummers, along with preliminary analysis and tools for further exploration of the data., This research was partly funded by the Ministry of Science and Innovation of the Spanish Government. Agencia Estatal de Investigación (AEI). (Reference: PID2019- 111403GB-I00).
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico antes de ser publicado, versión del autor (Preprint)

DiffVel: note-level MIDI velocity estimation for piano performance by a double conditioned diffusion model

  • Kim, Hyon
  • Serra, Xavier
This work has been accepted at the CMMR2023, the 16th International Symposium on Computer Music Multidisciplinary Research, at Tokyo, Japan. November 13-17, 2023., In any piano performance, expressiveness is paramount for effectively conveying the intent of the performer, and one of the most significant aspects of expressiveness is the loudness at the individual key or note level. However, accurately detecting note-level loudness poses a considerable technical challenge due to the polyphonic nature of piano performances, wherein multiple notes are played simultaneously, as well as the similarity of harmonic elements. MIDI velocity is crucial for indicating loudness in piano notes. This study conducted experiments for estimating a note-level MIDI velocity expanding the DiffRoll model: the Diffusion Model for piano performance transcription. By adopting double conditioning—audio and score information—and implementing noise removal as a post-processing, our findings highlight the model’s potential in estimating note level MIDI velocity., This research was carried out under the project Musical AI - PID2019- 111403GBI00/AEI/10.13039/501100011033, funded by the Spanish Ministerio de Ciencia e Innovación and the Agencia Estatal de Investigación.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico antes de ser publicado, versión del autor (Preprint)

Note level midi velocity estimation for piano performance

  • Kim, Hyon
  • Miron, Marius
  • Serra, Xavier
Piano is one of the most popular music instruments. During the piano performance, loudness is an important factor for expressiveness, alongside tempo, changes in dynamics play with expectation, convey various emotions, and render expressiveness. Due to the polyphonic characteristics and with the goal of better analysing the expressiveness of performance of piano with multiple notes playing simultaneously, it is more useful to find loudness for each note than looking at accumulated loudness for a single time frame. Most of the research in this topic uses Non-negative Matrix Factorization (NMF) techniques to find note level loudness. In contrast, we propose to use Deep Neural Networks (DNNs) conditioned with score information to estimate the loudness based on MIDI velocity for each note performed by piano players. To our best knowledge, this is a novel research for note level MIDI velocity estimation by a DNN model in end to end fashion having FiLM conditioning., This research was carried out under the project Musical AI - PID2019- 111403GB-I00/AEI/10.13039/501100011033, funded by the Spanish Ministerio de Ciencia e Innovación and the Agencia Estatal de Investigación.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico antes de ser publicado, versión del autor (Preprint)

High-resolution violin transcription using weak labels

  • Tamer, Nazif Can
  • Özer, Yigitcan
  • Müller, Meinard
  • Serra, Xavier
This work has been accepted at the 24th International Society for Music Information Retrieval Conference (ISMIR 2023), at Milan, Italy. October 5-9, 2023., A descriptive transcription of a violin performance requires detecting not only the notes but also the fine-grained pitch variations, such as vibrato. Most existing deep learning methods for music transcription do not capture these variations and often need frame-level annotations, which are scarce for the violin. In this paper, we propose a novel method for high-resolution violin transcription that can leverage piece-level weak labels for training. Our conformer-based model works on the raw audio waveform and transcribes violin notes and their corresponding pitch deviations with 5.8 ms frame resolution and 10-cent frequency resolution. We demonstrate that our method (1) outperforms generic systems in the proxy tasks of violin transcription and pitch estimation, and (2) can automatically generate new training labels by aligning its feature representations with unseen scores. We share our model along with 34 hours of score-aligned solo violin performance dataset, notably including the 24 Paganini Caprices., This research is funded by the project Musical AI- PID2019-111403GB-I00/AEI/10.13039/501100011033 funded by the Spanish Ministerio de Ciencia, Innovación y Universidades (MCIU) and the Agencia Estatal de Investigación (AEI), and by the German Research Foundation(DFG MU 2686/10-2).
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico antes de ser publicado, versión del autor (Preprint)

Carnatic singing voice separation using cold diffusion on training data with bleeding

  • Plaja-Roglans, Genís
  • Miron, Marius
  • Shankar, Adithi
  • Serra, Xavier
This work has been accepted at the 24th International Society for Music Information Retrieval Conference (ISMIR 2023), at Milan, Italy. October 5-9, 2023., Supervised music source separation systems using deep learning are trained by minimizing a loss function between pairs of predicted separations and ground-truth isolated sources. However, open datasets comprising isolated sources are few, small, and restricted to a few music styles. At the same time, multi-track datasets with source bleeding are usually found larger in size, and are easier to compile. In this work, we address the task of singing voice separation when the ground-truth signals have bleeding and only the target vocals and the corresponding mixture are available. We train a cold diffusion model on the frequency domain to iteratively transform a mixture into the corresponding vocals with bleeding. Next, we build the final separation masks by clustering spectrogram bins according to their evolution along the transformation steps. We test our approach on a Carnatic music scenario for which solely datasets with bleeding exist, while current research on this repertoire commonly uses source separation models trained solely with Western commercial music. Our evaluation on a Carnatic test set shows that our system improves Spleeter on interference removal and it is competitive in terms of signal distortion. Code is open sourced., This work was carried out under the projects Musical AI - PID2019-111403GB-I00/AEI/10.13039/501100011033 funded by the Spanish Ministerio de Ciencia, Innovación y Universidades (MCIU) and the Agencia Estatal de Investigación (AEI).
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico antes de ser publicado, versión del autor (Preprint)

TAPE: An End-to-End Timbre-Aware Pitch Estimator

  • Tamer, Nazif C
  • Özer, Yigitcan
  • Müller, Meinard
  • Serra, Xavier
This work has been accepted at the 2023 IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP 2023), at Rhodes, Greece. June 4-10, 2023., Pitch estimation of a target musical source within a multi-source polyphonic signal is of great interest for music performance analysis. One possible approach for extracting the pitch of a target source is to first perform source separation and then estimate the pitch of the separated track. However, as we will show, this typically leads to poor results. As an alternative to this approach, we introduce a timbre-aware pitch estimator (TAPE), which estimates the pitch of a target source in an end-to-end manner without the need for an explicit source separation step. Opposed to existing approaches that assume the predominance of a lead voice, our approach builds upon other cues that only rely on the timbral characteristics. Our results on real violin-piano duets show that, without any pre-processing step, TAPE outperforms the sequential procedure of source separation and pitch estimation under many settings, even if the target source is not predominant., This research is funded by the project Musical AI - PID2019-111403GB-I00/AEI/10.13039/501100011033 funded by the Spanish Ministerio de Ciencia, Innovacion y Universidades (MCIU) and the Agencia Estatal de Investigacion (AEI), and by the German Research Foundation (DFG MU 2686/10-2).
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico antes de ser publicado, versión del autor (Preprint)

MUSAV: a dataset of relative arousal-valence annotations for validation of audio models

  • Bogdanov, Dmitry
  • Lizarraga Seijas, Xavier
  • Alonso-Jiménez, Pablo
  • Serra, Xavier
We present MusAV, a new public benchmark dataset for comparative validation of arousal and valence (AV) regression models for audio-based music emotion recognition. To gather the ground truth, we rely on relative judgments instead of absolute values to simplify the manual annotation process and improve its consistency. We build MusAV by gathering comparative annotations of arousal and valence on pairs of tracks, using track audio previews and metadata from the Spotify API. The resulting dataset contains 2,092 track previews covering 1,404 genres, with pairwise relative AV judgments by 20 annotators and various subsets of the ground truth based on different levels of annotation agreement. We demonstrate the use of the dataset in an example study evaluating nine models for AV regression that we train based on state-of-the-art audio embeddings and three existing datasets of absolute AV annotations. The results on MusAV offer a view of the performance of the models complementary to the metrics obtained during training and provide insights into the impact of the considered datasets and embeddings on the generalization abilities of the models., This research was carried out under the project Musical AI - PID2019-111403GB-I00/AEI/10.13039/501100011033, funded by the Spanish Ministerio de Ciencia e Innovación and the Agencia Estatal de Investigación.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico antes de ser publicado, versión del autor (Preprint)

Predicting performance difficulty from piano sheet music images

  • Ramoneda, Pedro
  • Valero-Mas, Jose J.
  • Jeong, Dasaem
  • Serra, Xavier
This work has been accepted at the 24th International Society for Music Information Retrieval Conference (ISMIR 2023), at Milan, Italy. October 5-9, 2023., Estimating the performance difficulty of a musical score is crucial in music education for adequately designing the learning curriculum of the students. Although the Music Information Retrieval community has recently shown interest in this task, existing approaches mainly use machinereadable scores, leaving the broader case of sheet music images unaddressed. Based on previous works involving sheet music images, we use a mid-level representation, bootleg score, describing notehead positions relative to staff lines coupled with a transformer model. This architecture is adapted to our task by introducing an encoding scheme that reduces the encoded sequence length to oneeighth of the original size. In terms of evaluation, we consider five datasets—more than 7500 scores with up to 9 difficulty levels—, two of them particularly compiled for this work. The results obtained when pretraining the scheme on the IMSLP corpus and fine-tuning it on the considered datasets prove the proposal’s validity, achieving the bestperforming model with a balanced accuracy of 40.34% and a mean square error of 1.33. Finally, we provide access to our code, data, and models for transparency and reproducibility., This work is funded by the Spanish Ministerio de Ciencia, Innovación y Universidades (MCIU) and the Agencia Estatal de Investigación (AEI) within the Musical AI Project – PID2019-111403GBI00/AEI/10.13039/501100011033 and the Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Korea Government (MSIT) (NRF-2022R1F1A1074566).
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico antes de ser publicado, versión del autor (Preprint)

Efficient supervised training of audio transformers for music representation learning

  • Alonso Jiménez, Pablo
  • Serra, Xavier
  • Bogdanov, Dmitry
This work has been accepted at the 24th International Society for Music Information Retrieval Conference (ISMIR 2023), at Milan, Italy. October 5-9, 2023., In this work, we address music representation learning using convolution-free transformers. We build on top of existing spectrogram-based audio transformers such as AST and train our models on a supervised task using patchout training similar to PaSST. In contrast to previous works, we study how specific design decisions affect downstream music tagging tasks instead of focusing on the training task. We assess the impact of initializing the models with different pre-trained weights, using various input audio segment lengths, using learned representations from different blocks and tokens of the transformer for downstream tasks, and applying patchout at inference to speed up feature extraction. We find that 1) initializing the model from ImageNet or AudioSet weights and using longer input segments are beneficial both for the training and downstream tasks, 2) the best representations for the considered downstream tasks are located in the middle blocks of the transformer, and 3) using patchout at inference allows faster processing than our convolutional baselines while maintaining superior performance. The resulting models, MAEST, 1 are publicly available and obtain the best performance among open models in music tagging tasks., This work has been supported by the Musical AI project - PID2019-111403GB-I00/AEI/10.13039/501100011033, funded by the Spanish Ministerio de Ciencia e Innovación and the Agencia Estatal de Investigación.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico antes de ser publicado, versión del autor (Preprint)

Violin etudes: a comprehensive dataset for f0 estimation and performance analysis

  • Tamer, Nazif C
  • Ramoneda, Pedro
  • Serra, Xavier
This work has been accepted at the 23rd International Society for Music Information Retrieval Conference (ISMIR 2022), at Bengaluru, India. December 4-8, 2022., Violin performance analysis requires accurate and robust f0 estimates to give feedback on the playing accuracy. Despite the recent advancements in data-driven f0 estimators, their application to performance analysis remains a challenge due to style-specific and dataset-induced biases. In this paper, we address this problem by introducing Violin Etudes, a 27.8-hours violin performance dataset constructed with domain knowledge in instrument pedagogy and a novel automatic f0-labeling paradigm. Experimental results on unseen datasets show that the CREPE f0 estimator trained on Violin Etudes outperforms the widely-used pre-trained version trained on multiple manually-labeled datasets. Further preliminary findings suggest that (i) existing data-driven f0 estimators may overfit to equal temperament, and (ii) iterative re-labeling regularized by our novel Constrained Harmonic Resynthesis method can simultaneously enhance datasets and f0 estimators. Our dataset curation methodology is easily scalable to other instruments owing to the quantity of pedagogical data online. It also supports a range of MIR research directions thanks to the performance difficulty labels from educational institutions., We would like to thank Esteban Maestre for his valuable insights on the violin resonance structure. This research was carried out under the project Musical AI - PID2019- 111403GB-I00/AEI/10.13039/501100011033, funded by the Spanish Ministerio de Ciencia e Innovación and the Agencia Estatal de Investigación.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico antes de ser publicado, versión del autor (Preprint)

Music representation learning based on editorial metadata from discogs

  • Alonso-Jiménez, Pablo
  • Serra, Xavier
  • Bogdanov, Dmitry
This work has been accepted at the 23rd International Society for Music Information Retrieval Conference (ISMIR 2022), at Bengaluru, India. December 4-8, 2022., This paper revisits the idea of music representation learning supervised by editorial metadata, contributing to the state of the art in two ways. First, we exploit the public editorial metadata available on Discogs, an extensive community-maintained music database containing information about artists, releases, and record labels. Second, we use a contrastive learning setup based on COLA, different from previous systems based on triplet loss. We train models targeting several associations derived from the metadata and experiment with stacked combinations of learned representations, evaluating them on standard music classification tasks. Additionally, we consider learning all the associations jointly in a multi-task setup. We show that it is possible to improve the performance of current self-supervised models by using inexpensive metadata commonly available in music collections, producing representations comparable to those learned on classification setups. We find that the resulting representations based on editorial metadata outperform a system trained with music style tags available in the same large-scale dataset, which motivates further research using this type of supervision. Additionally, we give insights on how to preprocess Discogs metadata to build training objectives and provide public pre-trained models., This research was carried out under the project Musical AI - PID2019-111403GB-I00/AEI/10.13039/501100011033, funded by the Spanish Ministerio de Ciencia e Innovación and the Agencia Estatal de Investigación.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico antes de ser publicado, versión del autor (Preprint)

Essentia API: a web API for music audio analysis

  • Correya, Albin Andrew
  • Bogdanov, Dmitry
  • Alonso Jiménez, Pablo
  • Serra, Xavier
We present Essentia API, a web API to access a collection of state-of-the-art music audio analysis and description algorithms based on Essentia, an open-source library and machine learning (ML) models for audio and music analysis. We are developing it as part of a broader project in which we explore strategies for the commercial viability of technologies developed at Music Technology Group (MTG) following open science and open source practices, which involves finding licensing schemes and building custom solutions. Currently, the API supports music auto-tagging and classification algorithms (for genre, instrumentation, mood/emotion, danceability, approachability, and engagement), and algorithms for musical key, tempo, loudness, and many more. In the future, we envision expanding it with new machine learning models developed by the MTG and our collaborators to facilitate their access for a broader community of users.