Skip to main content

INTELIGENCIA ARTIFICIAL DE SOPORTE A EXPERIENCIAS MUSICALES: HACIA UNA APROXIMACION BASADA EN DATOS Y CENTRADA EN EL USUARIO

PID2019-111403GB-I00

•
Funding agency name Agencia Estatal de Investigación
Funding agency acronym AEI
Program Programa Estatal de I+D+i Orientado a los Retos de la Sociedad
Call Proyectos de I+D+i (Generación de Conocimiento y Retos Investigación)
Call year 2019
State Plan 2017-2020 Plan Estatal de Investigación Científica y Técnica y de Innovación
Beneficiary centre UNIVERSITAT POMPEU FABRA CCT

Resultados relacionados

Resultados totales (Incluyendo duplicados): 28
Encontrada(s) 1 página(s)

Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

A diffusion-inspired training strategy for singing voice extraction in the waveform domain

  • Plaja-Roglans, Genís
  • Miron, Marius
  • Serra, Xavier
Comunicació presentada a 23nd International Society for Music Information Retrieval Conference (ISMIR 2022), celebrat del 4 al 8 de desembre de 2022 a Bangalore, Índia., Notable progress in music source separation has been achieved using multi-branch networks that operate on both temporal and spectral domains. However, such networks tend to be complex and heavy-weighted. In this work, we tackle the task of singing voice extraction from polyphonic music signals in an end-to-end manner using an approach inspired by the training and sampling process of denoising diffusion models. We perform unconditional signal modelling to gradually convert an input mixture signal to the corresponding singing voice or accompaniment. We use fewer parameters than the state-of-the-art models while operating on the waveform domain, bypassing the phase estimation problem. More concisely, we train a non-causal WaveNet using a diffusion-inspired strategy while improving the said network for singing voice extraction and obtaining performance comparable to the end-to-end stateof-the-art on MUSDB18. We further report results on a non-MUSDB-overlapping version of MedleyDB and the multi-track audio of Saraga Carnatic showing good generalization, and run perceptual tests of our approach. Code, models, and audio examples are made available., This work was carried out under the projects Musical AI - PID2019-111403GB-I00/AEI/10.13039/501100011033 and NextCore - RTC2019-007248-7 funded by the Spanish Ministerio de Ciencia, Innovación y Universidades (MCIU) and the Agencia Estatal de Investigación (AEI).
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

Music representation learning based on editorial metadata from Discogs

  • Alonso-Jiménez, Pablo
  • Serra, Xavier
  • Bogdanov, Dmitry
Comunicació presentada a 23nd International Society for Music Information Retrieval Conference (ISMIR 2022), celebrat del 4 al 8 de desembre de 2022 a Bangalore, Índia., This paper revisits the idea of music representation learning supervised by editorial metadata, contributing to the state of the art in two ways. First, we exploit the public editorial metadata available on Discogs, an extensive community-maintained music database containing information about artists, releases, and record labels. Second, we use a contrastive learning setup based on COLA, different from previous systems based on triplet loss. We train models targeting several associations derived from the metadata and experiment with stacked combinations of learned representations, evaluating them on standard music classification tasks. Additionally, we consider learning all the associations jointly in a multi-task setup. We show that it is possible to improve the performance of current self-supervised models by using inexpensive metadata commonly available in music collections, producing representations comparable to those learned on classification setups. We find that the resulting representations based on editorial metadata outperform a system trained with music style tags available in the same large-scale dataset, which motivates further research using this type of supervision. Additionally, we give insights on how to preprocess Discogs metadata to build training objectives and provide public pre-trained models., This research was carried out under the project Musical AI - PID2019-111403GB-I00/AEI/10.13039/501100011033, funded by the Spanish Ministerio de Ciencia e Innovación and the Agencia Estatal de Investigación.
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

Multi-type outer product-based fusion of respiratory sounds for detecting COVID-19

  • Mallol Ragolta, Adrià
  • Cuesta, Helena
  • Gómez Gutiérrez, Emilia, 1975-
  • Schuller, Björn
Comunicació presentada a Interspeech 2022, celebrat del 18 al 22 de setembre de 2022 a Inchon, Corea del Sud., This work presents an outer product-based approach to fuse the embedded representations learnt from the spectrograms of cough, breath, and speech samples for the automatic detection of COVID-19. To extract deep learnt representations from the spectrograms, we compare the performance of specific Convolutional Neural Networks (CNNs) trained from scratch and ResNet18- based CNNs fine-tuned for the task at hand. Furthermore, we investigate whether the patients’ sex and the use of contextual attention mechanisms are beneficial. Our experiments use the dataset released as part of the Second Diagnosing COVID-19 using Acoustics (DiCOVA) Challenge. The results suggest the suitability of fusing breath and speech information to detect COVID-19. An Area Under the Curve (AUC) of 84.06 % is obtained on the test partition when using specific CNNs trained from scratch with contextual attention mechanisms. When using ResNet18-based CNNs for feature extraction, the baseline model scores the highest performance with an AUC of 84.26 %., This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No. 826506 (sustAGE), and from the Spanish Ministry of Science and Innovation under the Musical AI project (PID2019-111403GB-I00).
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

PYWDF: an open source library for prototyping and simulating wave filter circuits in Python

  • Anthon, Gustav
  • Lizarraga Seijas, Xavier
  • Font Corbera, Frederic
Comunicació presentada a 26th International Conference on Digital Audio Effects (DAFx-23), celebrada del 4 al 7 desetembre de 2023 a Copenhaguen, Dinamarca., This paper introduces a new open-source Python library for the modeling and simulation of wave digital filter (WDF) circuits. The library, called pwydf, allows users to easily create and analyze WDF circuit models in a high-level, object-oriented manner. The library includes a variety of built-in components, such as voltage sources, capacitors, diodes etc., as well as the ability to create custom components and circuits. Additionally, pywdf includes a variety of analysis tools, such as frequency response and transient analysis, to aid in the design and optimization of WDF circuits. We demonstrate the library’s efficacy in replicating the nonlinear behavior of an analog diode clipper circuit, and in creating an allpass filter that cannot be realized in the analog world. The library is well-documented and includes several examples to help users get started. Overall, pywdf is a powerful tool for anyone working with WDF circuits, and we hope it can be of great use to researchers and engineers in the field., This research was carried out under the project Musical AI -PID2019- 111403GB-I00/AEI/10.13039/501100011033, funded by the Spanish Ministerio de Ciencia e Innovación and the Agencia Estatal de Investigación.
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

Bottlenecks and solutions for audio to score alignment research

  • Morsi, Alia
  • Serra, Xavier
Comunicació presentada a 23nd International Society for Music Information Retrieval Conference (ISMIR 2022), celebrat del 4 al 8 de desembre de 2022 a Bangalore, Índia., Although audio to score alignment is a classic Music Information Retrieval problem, it has not been defined uniquely with the scope of musical scenarios representing its core. The absence of a unified vision makes it difficult to pinpoint its state-of-the-art and determine directions for improvement. To get past this bottleneck, it is necessary to consolidate datasets and evaluation methodologies to allow comprehensive benchmarking. In our review of prior work, we demonstrate the extent of variation in problem scope, datasets, and evaluation practices across audio to score alignment research. To circumvent the high cost of creating large-scale datasets with various instruments, styles, performance conditions, and musician proficiency levels from scratch, the research community could generate ground truth approximations from non-audio to score alignment datasets which include a temporal mapping between a music score and its corresponding audio. We show a methodology for adapting the Aligned Scores and Performances dataset, created originally for beat tracking and music transcription. We filter the dataset semi-automatically by applying a set of Dynamic Time Warping based Audio to Score Alignment methods using out-of-the-box Chroma and Constant-Q Transform extraction algorithms, suitable for the characteristics of the piano performances of the dataset. We use the results to discuss the limitations of the generated ground truths and data adaptation method. While the adapted dataset does not provide the necessary diversity for solving the initial problem, we conclude with ideas for expansion, and identify future directions for curating more comprehensive datasets through data adaptation, or synthesis., This research was carried out under the project Musical AI - PID2019-111403GB-I00/AEI/10.13039/501100011033, funded by the Spanish Ministerio de Ciencia e Innovación and the Agencia Estatal de Investigación.
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

Analysis of musical dynamics in vocal performances

  • Narang, Jyoti
  • Miron, Marius
  • Lizarraga Seijas, Xavier
  • Serra, Xavier
Comunicació presentada a: The 15th International Symposium on Computer Music Multidisciplinary Research celebrat del 15 al 19 de novembre de 2021 de manera virtual., Dynamics are one of the fundamental tools of expressivity in a performance. While the usage of this tool is highly subjective, a systematic methodology to derive loudness markings based on a performance can be highly beneficial. With this goal in mind, this paper is a first step towards developing a methodology to automatically transcribe dynamic markings from vocal rock and pop performances. To this end, we make use of commercial recordings of some popular songs followed by source separation and compare them to the karaoke versions of the same songs. The dynamic variations in the original commercial recordings are found to be structurally very similar to the aligned karaoke/multi-track versions of the same tracks. We compare and show the differences between tracks using statistical analysis, with an eventual goal to use the transcribed markings as guiding tools, to help students adapt with a specific interpretation of a given piece of music. We perform a qualitative analysis of the proposed methodology with the teachers in terms of informativeness and accuracy., Part of this research is funded by the projects Musical AI (PID2019-111403GB-I00/AEI/10.13039/501100011033 funded by the Spanish Ministerio de Ciencia, Innovación y Universidades (MCIU) and the Agencia Estatal de Investigación (AEI)) and NextCore (RTC2019-007248-7 funded by the Spanish Ministerio de Ciencia, Innovación y Universidades (MCIU) and the Agencia Estatal de Investigación (AEI)).
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

Violin etudes: a comprehensive dataset for f0 estimation and performance analysis

  • Can Tamer, Nazif
  • Ramoneda, Pedro
  • Serra, Xavier
Comunicació presentada a 23nd International Society for Music Information Retrieval Conference (ISMIR 2022), celebrat del 4 al 8 de desembre de 2022 a Bangalore, Índia., Violin performance analysis requires accurate and robust f0 estimates to give feedback on the playing accuracy. Despite the recent advancements in data-driven f0 estimators, their application to performance analysis remains a challenge due to style-specific and dataset-induced biases. In this paper, we address this problem by introducing Violin Etudes, a 27.8-hours violin performance dataset constructed with domain knowledge in instrument pedagogy and a novel automatic f0-labeling paradigm. Experimental results on unseen datasets show that the CREPE f0 estimator trained on Violin Etudes outperforms the widely-used pre-trained version trained on multiple manually-labeled datasets. Further preliminary findings suggest that (i) existing data-driven f0 estimators may overfit to equal temperament, and (ii) iterative re-labeling regularized by our novel Constrained Harmonic Resynthesis method can simultaneously enhance datasets and f0 estimators. Our dataset curation methodology is easily scalable to other instruments owing to the quantity of pedagogical data online. It also supports a range of MIR research directions thanks to the performance difficulty labels from educational institutions., We would like to thank Esteban Maestre for his valuable insights on the violin resonance structure. This research was carried out under the project Musical AI - PID2019- 111403GB-I00/AEI/10.13039/501100011033, funded by the Spanish Ministerio de Ciencia e Innovación and the Agencia Estatal de Investigación.
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

In search of Sañcāras: tradition-informed repeated melodic pattern recognition in carnatic music

  • Nuttall, Thomas
  • Plaja-Roglans, Genís
  • Pearson, Lara
  • Serra, Xavier
Comunicació presentada a 23nd International Society for Music Information Retrieval Conference (ISMIR 2022), celebrat del 4 al 8 de desembre de 2022 a Bangalore, Índia., Carnatic Music is a South Indian art and devotional musical practice in which melodic patterns (motifs and phrases), known as sañcaras Å , play a crucial structural and expressive role. We demonstrate how the combination of transposition invariant features learnt by a Complex Autoencoder (CAE) and predominant pitch tracks extracted using a Frequency-Temporal Attention Network (FTANet) can be used to annotate and group regions of variable-length, repeated, melodic patterns in audio recordings of multiple Carnatic Music performances. These models are trained on novel, expert-curated datasets of hundreds of Carnatic audio recordings and the extraction process tailored to account for the unique characteristics of sañcaras Å in Carnatic Music. Experimental results show that the proposed method is able to identify 54% of all sañcaras Å annotated by a professional Carnatic vocalist. Code to reproduce and interact with these results is available online., This research work was carried out under the project Musical AI - PID2019- 111403GBI00/AEI/10.13039/501100011033, funded by the Spanish Ministerio de Ciencia e Innovación and the Agencia Estatal de Investigación.
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

An overview of automatic piano performance assessment within the music education context

  • Kim, Hyon
  • Ramoneda, Pedro
  • Miron, Marius
  • Serra, Xavier
Comunicació presentada a la 14th International Conference on Computer Supported Education, celebrada del 22 a 24 d'abril de 2021 de manera virtual., Piano is one of the most popular instruments among music learners. Technologies to evaluate piano performances have been researched and developed in recent years rapidly, including data driven methods using machine learning. Despite the demand from people and speed of the development, there are still gaps between the methods and the pedagogical setup for real use case scenarios due to lack of accuracy of methods, insufficient amount of training data or the biases in training machine learning models, ignoring actual use case of the technology and such. In this paper, we first propose a feedback approach in piano performance education and review methods for Automated Piano Performance Assessment (APPA). After that, we discuss about gaps between a feedback approach and current methods, emphasizing their music education application. As a future work we propose a potential approach to overcome the gaps., This project is supported by Project Musical AI funded by the Spanish Ministerio de Ciencia, Innovacion y Universidades (MCIU) and the Agencia Estatal de Investigacion (AEI).
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

The Matrix profile for motif discovery in audio - an example application in Carnatic music

  • Nuttall, Thomas
  • Plaja-Roglans, Genís
  • Pearson, Lara
  • Serra, Xavier
Comunicació presentada a: The 15th International Symposium on Computer Music Multidisciplinary Research celebrat del 15 al 19 de novembre de 2021 de manera virtual., We present here a pipeline for the automated discovery of repeated motifs in audio. Our approach relies on state-of-the-art source separation, pre- dominant pitch extraction and time series motif detection via the matrix profile. Owing to the appropriateness of this approach for the task of motif recognition in the Carnatic musical style of South India, and with access to the recently re- leased Saraga Dataset of Indian Art Music, we provide an example application on a recording of a performance in the Carnatic r ̄aga, R ̄itigaul.a, finding 56 dis- tinct patterns of varying lengths that occur at least 3 times in the recording. The authors include a discussion of the potential musicological significance of this motif finding approach in relation to the particular tradition and beyond., This research was funded by the MUSICAL AI project (PID2019- 111403GB-I00) granted by the Ministry of Science and Innovation of the Spanish Gov- ernment. We also thank Rafael Caro Repetto for his continued guidance and input.
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

Diversity in the music listening experience: insights from focus group interviews

  • Porcaro, Lorenzo
  • Gómez Gutiérrez, Emilia, 1975-
  • Castillo, Carlos
Comunicació presentada a: CHIIR '22: ACM SIGIR Conference on Human Information Interaction and Retrieval, celebrat del 14 al 18 de març de 2022 a Regensburg, Alemanya., Music listening in today’s digital spaces is highly characterized by the availability of huge music catalogues, accessible by people all over the world. In this scenario, recommender systems are designed to guide listeners in finding tracks and artists that best fit their requests, having therefore the power to influence the diversity of the music they listen to. Albeit several works have proposed new techniques for developing diversity-aware recommendations, little is known about how people perceive diversity while interacting with music recommendations. In this study, we interview several listeners about the role that diversity plays in their listening experience, trying to get a better understanding of how they interact with music recommendations. We recruit the listeners among the participants of a previous quantitative study, where they were confronted with the notion of diversity when asked to identify, from a series of electronic music lists, the most diverse ones according to their beliefs. As a follow-up, in this qualitative study we carry out semi-structured interviews to understand how listeners may assess the diversity of a music list and to investigate their experiences with music recommendation diversity. We report here our main findings on 1) what can influence the diversity assessment of tracks and artists’ music lists, and 2) which factors can characterize listeners’ interaction with music recommendation diversity., This work is partially supported by the European Commission under the TROMPA project (H2020 - grant agreement No. 770376). It is also part of the project Musical AI - PID2019-111403GB-I00/AEI/10.13039/501100011033 funded by the Spanish Ministerio de Ciencia, Innovación y Universidades (MCIU) and the Agencia Estatal de Investigación (AEI). This work is also partially supported by the HUMAINT programme (Human Behaviour and Machine Intelligence), Joint Research Centre, European Commission. The project leading to these results received funding from "la Caixa" Foundation (ID 100010434), under the agreement LCF/PR/PR16/51110009.
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

Personalized musically induced emotions of not-so-popular Colombian music

  • Gómez Cañón, Juan Sebastián
  • Herrera Boyer, Perfecto, 1964-
  • Cano, Estefanía
  • Gómez Gutiérrez, Emilia, 1975-
Comunicació presentada al workshop Human Centered AI inclòs a: 35th Conference on Neural Information Processing Systems (NeurIPS 2021) celebrat el 13 de desembre de manera virtual., This work presents an initial proof of concept of how Music Emotion Recognition (MER) systems could be intentionally biased with respect to annotations of musically-induced emotions in a political context. In specific, we analyze traditional Colombian music containing politically-charged lyrics of two types: (1) vallenatos and social songs from the “left-wing” guerrilla Fuerzas Armadas Revolucionarias de Colombia (FARC) and (2) corridos from the “right-wing” paramilitaries Autodefensas Unidas de Colombia (AUC). We train personalized machine learning models to predict induced emotions for three users with diverse political views – we aim at identifying the songs that may induce negative emotions for a particular user, such as anger and fear. To this extent, a user’s emotion judgements could be interpreted as problematizing data – subjective emotional judgments could in turn be used to influence the user in a human-centered machine learning environment. In short, highly desired “emotion regulation” applications could potentially deviate to “emotion manipulation” – the recent discredit of emotion recognition technologies might transcend ethical issues of diversity and inclusion., The research work conducted at the Universitat Pompeu Fabra is partially supported by the Eu- ropean Commission under the TROMPA project (H2020 770376) and the Project Musical AI - PID2019-111403GB-I00/AEI/10.13039/501100011033 funded by the Spanish Ministerio de Ciencia, Innovación y Universidades (MCIU) and the Agencia Estatal de Investigación (AEI).
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

MusAV: a dataset of relative arousal-valence annotations for validation of audio models

  • Bogdanov, Dmitry
  • Lizarraga Seijas, Xavier
  • Alonso-Jiménez, Pablo
  • Serra, Xavier
Comunicació presentada a 23nd International Society for Music Information Retrieval Conference (ISMIR 2022), celebrat del 4 al 8 de desembre de 2022 a Bangalore, Índia., We present MusAV, a new public benchmark dataset for comparative validation of arousal and valence (AV) regression models for audio-based music emotion recognition. To gather the ground truth, we rely on relative judgments instead of absolute values to simplify the manual annotation process and improve its consistency. We build MusAV by gathering comparative annotations of arousal and valence on pairs of tracks, using track audio previews and metadata from the Spotify API. The resulting dataset contains 2,092 track previews covering 1,404 genres, with pairwise relative AV judgments by 20 annotators and various subsets of the ground truth based on different levels of annotation agreement. We demonstrate the use of the dataset in an example study evaluating nine models for AV regression that we train based on state-of-the-art audio embeddings and three existing datasets of absolute AV annotations. The results on MusAV offer a view of the performance of the models complementary to the metrics obtained during training and provide insights into the impact of the considered datasets and embeddings on the generalization abilities of the models., This research was carried out under the project Musical AI - PID2019-111403GB-I00/AEI/10.13039/501100011033, funded by the Spanish Ministerio de Ciencia e Innovación and the Agencia Estatal de Investigación. We also thank Juan Sebastián Gómez Cañón for his suggestions and all participating annotators.
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

NeuralMidiFx: a wrapper template for deploying neural networks as VST3 plugins

  • Haki, Behzad
  • Lenz, Julian
  • Jordà Puig, Sergi
Comunicació presentada a AIMC 2023, celebrada del 30 d'agost al 1 de setembre de 2023 a Sussex, Anglaterra., Proper research, development and evaluation of AI-based generative systems of music that focus on performance or composition require active user-system interactions. To include a diverse group of users that can properly engage with a given system, researchers should provide easy access to their developed systems. Given that many users (i.e. musicians) are non-technical to the field of AI and the development frameworks involved, the researchers should aim to make their systems accessible within the environments commonly used in production/composition workflows (e.g. in the form of plugins hosted in digital audio workstations). Unfortunately, deploying generative systems in this manner is highly expensive. As such, researchers with limited resources are often unable to provide easy access to their works, and subsequently, are not able to properly evaluate and encourage active engagement with their systems. Facing these limitations, we have been working on a solution that allows for easy, effective and accessible deployment of generative systems. To this end, we propose a wrapper/template called NeuralMidiFx, which streamlines the deployment of neural network based symbolic music generation systems as VST3 plugins. The proposed wrapper is intended to allow researchers to develop plugins with ease while requiring minimal familiarity with plugin development., This research was partly funded by the Ministry of Science and Innovation of the Spanish Government. Agencia Estatal de Investigación (AEI). (Reference: PID2019-111403GB-I00)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Bottlenecks and solutions for audio to score alignment research

  • Morsi, Alia
  • Serra, Xavier
Comunicació presentada a 23nd International Society for Music Information Retrieval Conference (ISMIR 2022), celebrat del 4 al 8 de desembre de 2022 a Bangalore, Índia., Although audio to score alignment is a classic Music Information Retrieval problem, it has not been defined uniquely with the scope of musical scenarios representing its core. The absence of a unified vision makes it difficult to pinpoint its state-of-the-art and determine directions for improvement. To get past this bottleneck, it is necessary to consolidate datasets and evaluation methodologies to allow comprehensive benchmarking. In our review of prior work, we demonstrate the extent of variation in problem scope, datasets, and evaluation practices across audio to score alignment research. To circumvent the high cost of creating large-scale datasets with various instruments, styles, performance conditions, and musician proficiency levels from scratch, the research community could generate ground truth approximations from non-audio to score alignment datasets which include a temporal mapping between a music score and its corresponding audio. We show a methodology for adapting the Aligned Scores and Performances dataset, created originally for beat tracking and music transcription. We filter the dataset semi-automatically by applying a set of Dynamic Time Warping based Audio to Score Alignment methods using out-of-the-box Chroma and Constant-Q Transform extraction algorithms, suitable for the characteristics of the piano performances of the dataset. We use the results to discuss the limitations of the generated ground truths and data adaptation method. While the adapted dataset does not provide the necessary diversity for solving the initial problem, we conclude with ideas for expansion, and identify future directions for curating more comprehensive datasets through data adaptation, or synthesis., This research was carried out under the project Musical AI - PID2019-111403GB-I00/AEI/10.13039/501100011033, funded by the Spanish Ministerio de Ciencia e Innovación and the Agencia Estatal de Investigación.
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Analysis of musical dynamics in vocal performances

  • Narang, Jyoti
  • Miron, Marius
  • Lizarraga Seijas, Xavier
  • Serra, Xavier
Comunicació presentada a: The 15th International Symposium on Computer Music Multidisciplinary Research celebrat del 15 al 19 de novembre de 2021 de manera virtual., Dynamics are one of the fundamental tools of expressivity in a performance. While the usage of this tool is highly subjective, a systematic methodology to derive loudness markings based on a performance can be highly beneficial. With this goal in mind, this paper is a first step towards developing a methodology to automatically transcribe dynamic markings from vocal rock and pop performances. To this end, we make use of commercial recordings of some popular songs followed by source separation and compare them to the karaoke versions of the same songs. The dynamic variations in the original commercial recordings are found to be structurally very similar to the aligned karaoke/multi-track versions of the same tracks. We compare and show the differences between tracks using statistical analysis, with an eventual goal to use the transcribed markings as guiding tools, to help students adapt with a specific interpretation of a given piece of music. We perform a qualitative analysis of the proposed methodology with the teachers in terms of informativeness and accuracy., Part of this research is funded by the projects Musical AI (PID2019-111403GB-I00/AEI/10.13039/501100011033 funded by the Spanish Ministerio de Ciencia, Innovación y Universidades (MCIU) and the Agencia Estatal de Investigación (AEI)) and NextCore (RTC2019-007248-7 funded by the Spanish Ministerio de Ciencia, Innovación y Universidades (MCIU) and the Agencia Estatal de Investigación (AEI)).
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Personalized musically induced emotions of not-so-popular Colombian music

  • Gómez Cañón, Juan Sebastián
  • Herrera Boyer, Perfecto, 1964-
  • Cano, Estefanía
  • Gómez Gutiérrez, Emilia, 1975-
Comunicació presentada al workshop Human Centered AI inclòs a: 35th Conference on Neural Information Processing Systems (NeurIPS 2021) celebrat el 13 de desembre de manera virtual., This work presents an initial proof of concept of how Music Emotion Recognition (MER) systems could be intentionally biased with respect to annotations of musically-induced emotions in a political context. In specific, we analyze traditional Colombian music containing politically-charged lyrics of two types: (1) vallenatos and social songs from the “left-wing” guerrilla Fuerzas Armadas Revolucionarias de Colombia (FARC) and (2) corridos from the “right-wing” paramilitaries Autodefensas Unidas de Colombia (AUC). We train personalized machine learning models to predict induced emotions for three users with diverse political views – we aim at identifying the songs that may induce negative emotions for a particular user, such as anger and fear. To this extent, a user’s emotion judgements could be interpreted as problematizing data – subjective emotional judgments could in turn be used to influence the user in a human-centered machine learning environment. In short, highly desired “emotion regulation” applications could potentially deviate to “emotion manipulation” – the recent discredit of emotion recognition technologies might transcend ethical issues of diversity and inclusion., The research work conducted at the Universitat Pompeu Fabra is partially supported by the Eu- ropean Commission under the TROMPA project (H2020 770376) and the Project Musical AI - PID2019-111403GB-I00/AEI/10.13039/501100011033 funded by the Spanish Ministerio de Ciencia, Innovación y Universidades (MCIU) and the Agencia Estatal de Investigación (AEI).
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

MusAV: a dataset of relative arousal-valence annotations for validation of audio models

  • Bogdanov, Dmitry
  • Lizarraga Seijas, Xavier
  • Alonso-Jiménez, Pablo
  • Serra, Xavier
Comunicació presentada a 23nd International Society for Music Information Retrieval Conference (ISMIR 2022), celebrat del 4 al 8 de desembre de 2022 a Bangalore, Índia., We present MusAV, a new public benchmark dataset for comparative validation of arousal and valence (AV) regression models for audio-based music emotion recognition. To gather the ground truth, we rely on relative judgments instead of absolute values to simplify the manual annotation process and improve its consistency. We build MusAV by gathering comparative annotations of arousal and valence on pairs of tracks, using track audio previews and metadata from the Spotify API. The resulting dataset contains 2,092 track previews covering 1,404 genres, with pairwise relative AV judgments by 20 annotators and various subsets of the ground truth based on different levels of annotation agreement. We demonstrate the use of the dataset in an example study evaluating nine models for AV regression that we train based on state-of-the-art audio embeddings and three existing datasets of absolute AV annotations. The results on MusAV offer a view of the performance of the models complementary to the metrics obtained during training and provide insights into the impact of the considered datasets and embeddings on the generalization abilities of the models., This research was carried out under the project Musical AI - PID2019-111403GB-I00/AEI/10.13039/501100011033, funded by the Spanish Ministerio de Ciencia e Innovación and the Agencia Estatal de Investigación. We also thank Juan Sebastián Gómez Cañón for his suggestions and all participating annotators.
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Diversity in the music listening experience: insights from focus group interviews

  • Porcaro, Lorenzo
  • Gómez Gutiérrez, Emilia, 1975-
  • Castillo, Carlos
Comunicació presentada a: CHIIR '22: ACM SIGIR Conference on Human Information Interaction and Retrieval, celebrat del 14 al 18 de març de 2022 a Regensburg, Alemanya., Music listening in today’s digital spaces is highly characterized by the availability of huge music catalogues, accessible by people all over the world. In this scenario, recommender systems are designed to guide listeners in finding tracks and artists that best fit their requests, having therefore the power to influence the diversity of the music they listen to. Albeit several works have proposed new techniques for developing diversity-aware recommendations, little is known about how people perceive diversity while interacting with music recommendations. In this study, we interview several listeners about the role that diversity plays in their listening experience, trying to get a better understanding of how they interact with music recommendations. We recruit the listeners among the participants of a previous quantitative study, where they were confronted with the notion of diversity when asked to identify, from a series of electronic music lists, the most diverse ones according to their beliefs. As a follow-up, in this qualitative study we carry out semi-structured interviews to understand how listeners may assess the diversity of a music list and to investigate their experiences with music recommendation diversity. We report here our main findings on 1) what can influence the diversity assessment of tracks and artists’ music lists, and 2) which factors can characterize listeners’ interaction with music recommendation diversity., This work is partially supported by the European Commission under the TROMPA project (H2020 - grant agreement No. 770376). It is also part of the project Musical AI - PID2019-111403GB-I00/AEI/10.13039/501100011033 funded by the Spanish Ministerio de Ciencia, Innovación y Universidades (MCIU) and the Agencia Estatal de Investigación (AEI). This work is also partially supported by the HUMAINT programme (Human Behaviour and Machine Intelligence), Joint Research Centre, European Commission. The project leading to these results received funding from "la Caixa" Foundation (ID 100010434), under the agreement LCF/PR/PR16/51110009.
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

The Matrix profile for motif discovery in audio - an example application in Carnatic music

  • Nuttall, Thomas
  • Plaja-Roglans, Genís
  • Pearson, Lara
  • Serra, Xavier
Comunicació presentada a: The 15th International Symposium on Computer Music Multidisciplinary Research celebrat del 15 al 19 de novembre de 2021 de manera virtual., We present here a pipeline for the automated discovery of repeated motifs in audio. Our approach relies on state-of-the-art source separation, pre- dominant pitch extraction and time series motif detection via the matrix profile. Owing to the appropriateness of this approach for the task of motif recognition in the Carnatic musical style of South India, and with access to the recently re- leased Saraga Dataset of Indian Art Music, we provide an example application on a recording of a performance in the Carnatic r ̄aga, R ̄itigaul.a, finding 56 dis- tinct patterns of varying lengths that occur at least 3 times in the recording. The authors include a discussion of the potential musicological significance of this motif finding approach in relation to the particular tradition and beyond., This research was funded by the MUSICAL AI project (PID2019- 111403GB-I00) granted by the Ministry of Science and Innovation of the Spanish Gov- ernment. We also thank Rafael Caro Repetto for his continued guidance and input.
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

PYWDF: an open source library for prototyping and simulating wave filter circuits in Python

  • Anthon, Gustav
  • Lizarraga Seijas, Xavier
  • Font Corbera, Frederic
Comunicació presentada a 26th International Conference on Digital Audio Effects (DAFx-23), celebrada del 4 al 7 desetembre de 2023 a Copenhaguen, Dinamarca., This paper introduces a new open-source Python library for the modeling and simulation of wave digital filter (WDF) circuits. The library, called pwydf, allows users to easily create and analyze WDF circuit models in a high-level, object-oriented manner. The library includes a variety of built-in components, such as voltage sources, capacitors, diodes etc., as well as the ability to create custom components and circuits. Additionally, pywdf includes a variety of analysis tools, such as frequency response and transient analysis, to aid in the design and optimization of WDF circuits. We demonstrate the library’s efficacy in replicating the nonlinear behavior of an analog diode clipper circuit, and in creating an allpass filter that cannot be realized in the analog world. The library is well-documented and includes several examples to help users get started. Overall, pywdf is a powerful tool for anyone working with WDF circuits, and we hope it can be of great use to researchers and engineers in the field., This research was carried out under the project Musical AI -PID2019- 111403GB-I00/AEI/10.13039/501100011033, funded by the Spanish Ministerio de Ciencia e Innovación and the Agencia Estatal de Investigación.
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Violin etudes: a comprehensive dataset for f0 estimation and performance analysis

  • Can Tamer, Nazif
  • Ramoneda, Pedro
  • Serra, Xavier
Comunicació presentada a 23nd International Society for Music Information Retrieval Conference (ISMIR 2022), celebrat del 4 al 8 de desembre de 2022 a Bangalore, Índia., Violin performance analysis requires accurate and robust f0 estimates to give feedback on the playing accuracy. Despite the recent advancements in data-driven f0 estimators, their application to performance analysis remains a challenge due to style-specific and dataset-induced biases. In this paper, we address this problem by introducing Violin Etudes, a 27.8-hours violin performance dataset constructed with domain knowledge in instrument pedagogy and a novel automatic f0-labeling paradigm. Experimental results on unseen datasets show that the CREPE f0 estimator trained on Violin Etudes outperforms the widely-used pre-trained version trained on multiple manually-labeled datasets. Further preliminary findings suggest that (i) existing data-driven f0 estimators may overfit to equal temperament, and (ii) iterative re-labeling regularized by our novel Constrained Harmonic Resynthesis method can simultaneously enhance datasets and f0 estimators. Our dataset curation methodology is easily scalable to other instruments owing to the quantity of pedagogical data online. It also supports a range of MIR research directions thanks to the performance difficulty labels from educational institutions., We would like to thank Esteban Maestre for his valuable insights on the violin resonance structure. This research was carried out under the project Musical AI - PID2019- 111403GB-I00/AEI/10.13039/501100011033, funded by the Spanish Ministerio de Ciencia e Innovación and the Agencia Estatal de Investigación.
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Music representation learning based on editorial metadata from Discogs

  • Alonso-Jiménez, Pablo
  • Serra, Xavier
  • Bogdanov, Dmitry
Comunicació presentada a 23nd International Society for Music Information Retrieval Conference (ISMIR 2022), celebrat del 4 al 8 de desembre de 2022 a Bangalore, Índia., This paper revisits the idea of music representation learning supervised by editorial metadata, contributing to the state of the art in two ways. First, we exploit the public editorial metadata available on Discogs, an extensive community-maintained music database containing information about artists, releases, and record labels. Second, we use a contrastive learning setup based on COLA, different from previous systems based on triplet loss. We train models targeting several associations derived from the metadata and experiment with stacked combinations of learned representations, evaluating them on standard music classification tasks. Additionally, we consider learning all the associations jointly in a multi-task setup. We show that it is possible to improve the performance of current self-supervised models by using inexpensive metadata commonly available in music collections, producing representations comparable to those learned on classification setups. We find that the resulting representations based on editorial metadata outperform a system trained with music style tags available in the same large-scale dataset, which motivates further research using this type of supervision. Additionally, we give insights on how to preprocess Discogs metadata to build training objectives and provide public pre-trained models., This research was carried out under the project Musical AI - PID2019-111403GB-I00/AEI/10.13039/501100011033, funded by the Spanish Ministerio de Ciencia e Innovación and the Agencia Estatal de Investigación.
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

An overview of automatic piano performance assessment within the music education context

  • Kim, Hyon
  • Ramoneda, Pedro
  • Miron, Marius
  • Serra, Xavier
Comunicació presentada a la 14th International Conference on Computer Supported Education, celebrada del 22 a 24 d'abril de 2021 de manera virtual., Piano is one of the most popular instruments among music learners. Technologies to evaluate piano performances have been researched and developed in recent years rapidly, including data driven methods using machine learning. Despite the demand from people and speed of the development, there are still gaps between the methods and the pedagogical setup for real use case scenarios due to lack of accuracy of methods, insufficient amount of training data or the biases in training machine learning models, ignoring actual use case of the technology and such. In this paper, we first propose a feedback approach in piano performance education and review methods for Automated Piano Performance Assessment (APPA). After that, we discuss about gaps between a feedback approach and current methods, emphasizing their music education application. As a future work we propose a potential approach to overcome the gaps., This project is supported by Project Musical AI funded by the Spanish Ministerio de Ciencia, Innovacion y Universidades (MCIU) and the Agencia Estatal de Investigacion (AEI).
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

A diffusion-inspired training strategy for singing voice extraction in the waveform domain

  • Plaja-Roglans, Genís
  • Miron, Marius
  • Serra, Xavier
Comunicació presentada a 23nd International Society for Music Information Retrieval Conference (ISMIR 2022), celebrat del 4 al 8 de desembre de 2022 a Bangalore, Índia., Notable progress in music source separation has been achieved using multi-branch networks that operate on both temporal and spectral domains. However, such networks tend to be complex and heavy-weighted. In this work, we tackle the task of singing voice extraction from polyphonic music signals in an end-to-end manner using an approach inspired by the training and sampling process of denoising diffusion models. We perform unconditional signal modelling to gradually convert an input mixture signal to the corresponding singing voice or accompaniment. We use fewer parameters than the state-of-the-art models while operating on the waveform domain, bypassing the phase estimation problem. More concisely, we train a non-causal WaveNet using a diffusion-inspired strategy while improving the said network for singing voice extraction and obtaining performance comparable to the end-to-end stateof-the-art on MUSDB18. We further report results on a non-MUSDB-overlapping version of MedleyDB and the multi-track audio of Saraga Carnatic showing good generalization, and run perceptual tests of our approach. Code, models, and audio examples are made available., This work was carried out under the projects Musical AI - PID2019-111403GB-I00/AEI/10.13039/501100011033 and NextCore - RTC2019-007248-7 funded by the Spanish Ministerio de Ciencia, Innovación y Universidades (MCIU) and the Agencia Estatal de Investigación (AEI).
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

NeuralMidiFx: a wrapper template for deploying neural networks as VST3 plugins

  • Haki, Behzad
  • Lenz, Julian
  • Jordà Puig, Sergi
Comunicació presentada a AIMC 2023, celebrada del 30 d'agost al 1 de setembre de 2023 a Sussex, Anglaterra., Proper research, development and evaluation of AI-based generative systems of music that focus on performance or composition require active user-system interactions. To include a diverse group of users that can properly engage with a given system, researchers should provide easy access to their developed systems. Given that many users (i.e. musicians) are non-technical to the field of AI and the development frameworks involved, the researchers should aim to make their systems accessible within the environments commonly used in production/composition workflows (e.g. in the form of plugins hosted in digital audio workstations). Unfortunately, deploying generative systems in this manner is highly expensive. As such, researchers with limited resources are often unable to provide easy access to their works, and subsequently, are not able to properly evaluate and encourage active engagement with their systems. Facing these limitations, we have been working on a solution that allows for easy, effective and accessible deployment of generative systems. To this end, we propose a wrapper/template called NeuralMidiFx, which streamlines the deployment of neural network based symbolic music generation systems as VST3 plugins. The proposed wrapper is intended to allow researchers to develop plugins with ease while requiring minimal familiarity with plugin development., This research was partly funded by the Ministry of Science and Innovation of the Spanish Government. Agencia Estatal de Investigación (AEI). (Reference: PID2019-111403GB-I00)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Multi-type outer product-based fusion of respiratory sounds for detecting COVID-19

  • Mallol Ragolta, Adrià
  • Cuesta, Helena
  • Gómez Gutiérrez, Emilia, 1975-
  • Schuller, Björn
Comunicació presentada a Interspeech 2022, celebrat del 18 al 22 de setembre de 2022 a Inchon, Corea del Sud., This work presents an outer product-based approach to fuse the embedded representations learnt from the spectrograms of cough, breath, and speech samples for the automatic detection of COVID-19. To extract deep learnt representations from the spectrograms, we compare the performance of specific Convolutional Neural Networks (CNNs) trained from scratch and ResNet18- based CNNs fine-tuned for the task at hand. Furthermore, we investigate whether the patients’ sex and the use of contextual attention mechanisms are beneficial. Our experiments use the dataset released as part of the Second Diagnosing COVID-19 using Acoustics (DiCOVA) Challenge. The results suggest the suitability of fusing breath and speech information to detect COVID-19. An Area Under the Curve (AUC) of 84.06 % is obtained on the test partition when using specific CNNs trained from scratch with contextual attention mechanisms. When using ResNet18-based CNNs for feature extraction, the baseline model scores the highest performance with an AUC of 84.26 %., This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No. 826506 (sustAGE), and from the Spanish Ministry of Science and Innovation under the Musical AI project (PID2019-111403GB-I00).
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

In search of Sañcāras: tradition-informed repeated melodic pattern recognition in carnatic music

  • Nuttall, Thomas
  • Plaja-Roglans, Genís
  • Pearson, Lara
  • Serra, Xavier
Comunicació presentada a 23nd International Society for Music Information Retrieval Conference (ISMIR 2022), celebrat del 4 al 8 de desembre de 2022 a Bangalore, Índia., Carnatic Music is a South Indian art and devotional musical practice in which melodic patterns (motifs and phrases), known as sañcaras Å , play a crucial structural and expressive role. We demonstrate how the combination of transposition invariant features learnt by a Complex Autoencoder (CAE) and predominant pitch tracks extracted using a Frequency-Temporal Attention Network (FTANet) can be used to annotate and group regions of variable-length, repeated, melodic patterns in audio recordings of multiple Carnatic Music performances. These models are trained on novel, expert-curated datasets of hundreds of Carnatic audio recordings and the extraction process tailored to account for the unique characteristics of sañcaras Å in Carnatic Music. Experimental results show that the proposed method is able to identify 54% of all sañcaras Å annotated by a professional Carnatic vocalist. Code to reproduce and interact with these results is available online., This research work was carried out under the project Musical AI - PID2019- 111403GBI00/AEI/10.13039/501100011033, funded by the Spanish Ministerio de Ciencia e Innovación and the Agencia Estatal de Investigación.