Pasar al contenido principal

INTELIGENCIA ARTIFICIAL DE SOPORTE A EXPERIENCIAS MUSICALES: HACIA UNA APROXIMACION BASADA EN DATOS Y CENTRADA EN EL USUARIO

PID2019-111403GB-I00

•
Nombre agencia financiadora Agencia Estatal de Investigación
Acrónimo agencia financiadora AEI
Programa Programa Estatal de I+D+i Orientado a los Retos de la Sociedad
Convocatoria Proyectos de I+D+i (Generación de Conocimiento y Retos Investigación)
Año convocatoria 2019
Plan Estatal 2017-2020 Plan Estatal de Investigación Científica y Técnica y de Innovación
Centro beneficiario UNIVERSITAT POMPEU FABRA CCT

Resultados relacionados

Resultados totales (Incluyendo duplicados): 64
Encontrada(s) 1 página(s)

Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Audio and music analysis on the web using Essentia.js

  • Correya, Albin Andrew
  • Marcos Fernández, Jorge
  • Joglar-Ongay, Luis
  • Alonso Jiménez, Pablo
  • Serra, Xavier
  • Bogdanov, Dmitry
Open-source software libraries have a significant impact on the development of Audio Signal Processing and Music Information Retrieval (MIR) systems. Despite the abundance of such tools, there is a lack of an extensive and easy-to-use reference library for audio feature extraction on Web clients. In this article, we present Essentia.js, an open-source JavaScript (JS) library for audio and music analysis on both web clients and JS engines. Along with the Web Audio API, it can be used for both offline and real-time audio feature extraction on web browsers. Essentia.js is modular, lightweight, and easy-to-use, deploy, maintain, and integrate into the existing plethora of JS libraries and web technologies. It is powered by a WebAssembly back end cross-compiled from the Essentia C++ library, which facilitates a JS interface to a wide range of low-level and high-level audio features, including signal processing MIR algorithms as well as pre-trained TensorFlow.js machine learning models. It also provides a higher-level JS API and add-on MIR utility modules along with extensive documentation, usage examples, and tutorials. We benchmark the proposed library on two popular web browsers and the Node.js engine, and four devices, including mobile Android and iOS, comparing it to the native performance of Essentia and the Meyda JS library., The work on Essentia.js has been partially funded by the Ministry of Science and Innovation of the Spanish Government under the grant agreement PID2019-111403GB-I00 (Musical AI).
Repositorio Digital de la UPF
Publicación Contribución de congreso (ConferenceOutput)

A diffusion-inspired training strategy for singing voice extraction in the waveform domain

  • Plaja-Roglans, Genís
  • Miron, Marius
  • Serra, Xavier
Comunicació presentada a 23nd International Society for Music Information Retrieval Conference (ISMIR 2022), celebrat del 4 al 8 de desembre de 2022 a Bangalore, Índia., Notable progress in music source separation has been achieved using multi-branch networks that operate on both temporal and spectral domains. However, such networks tend to be complex and heavy-weighted. In this work, we tackle the task of singing voice extraction from polyphonic music signals in an end-to-end manner using an approach inspired by the training and sampling process of denoising diffusion models. We perform unconditional signal modelling to gradually convert an input mixture signal to the corresponding singing voice or accompaniment. We use fewer parameters than the state-of-the-art models while operating on the waveform domain, bypassing the phase estimation problem. More concisely, we train a non-causal WaveNet using a diffusion-inspired strategy while improving the said network for singing voice extraction and obtaining performance comparable to the end-to-end stateof-the-art on MUSDB18. We further report results on a non-MUSDB-overlapping version of MedleyDB and the multi-track audio of Saraga Carnatic showing good generalization, and run perceptual tests of our approach. Code, models, and audio examples are made available., This work was carried out under the projects Musical AI - PID2019-111403GB-I00/AEI/10.13039/501100011033 and NextCore - RTC2019-007248-7 funded by the Spanish Ministerio de Ciencia, Innovación y Universidades (MCIU) and the Agencia Estatal de Investigación (AEI).
Repositorio Digital de la UPF
Publicación Contribución de congreso (ConferenceOutput)

Music representation learning based on editorial metadata from Discogs

  • Alonso-Jiménez, Pablo
  • Serra, Xavier
  • Bogdanov, Dmitry
Comunicació presentada a 23nd International Society for Music Information Retrieval Conference (ISMIR 2022), celebrat del 4 al 8 de desembre de 2022 a Bangalore, Índia., This paper revisits the idea of music representation learning supervised by editorial metadata, contributing to the state of the art in two ways. First, we exploit the public editorial metadata available on Discogs, an extensive community-maintained music database containing information about artists, releases, and record labels. Second, we use a contrastive learning setup based on COLA, different from previous systems based on triplet loss. We train models targeting several associations derived from the metadata and experiment with stacked combinations of learned representations, evaluating them on standard music classification tasks. Additionally, we consider learning all the associations jointly in a multi-task setup. We show that it is possible to improve the performance of current self-supervised models by using inexpensive metadata commonly available in music collections, producing representations comparable to those learned on classification setups. We find that the resulting representations based on editorial metadata outperform a system trained with music style tags available in the same large-scale dataset, which motivates further research using this type of supervision. Additionally, we give insights on how to preprocess Discogs metadata to build training objectives and provide public pre-trained models., This research was carried out under the project Musical AI - PID2019-111403GB-I00/AEI/10.13039/501100011033, funded by the Spanish Ministerio de Ciencia e Innovación and the Agencia Estatal de Investigación.
Repositorio Digital de la UPF
Publicación Contribución de congreso (ConferenceOutput)

Multi-type outer product-based fusion of respiratory sounds for detecting COVID-19

  • Mallol Ragolta, Adrià
  • Cuesta, Helena
  • Gómez Gutiérrez, Emilia, 1975-
  • Schuller, Björn
Comunicació presentada a Interspeech 2022, celebrat del 18 al 22 de setembre de 2022 a Inchon, Corea del Sud., This work presents an outer product-based approach to fuse the embedded representations learnt from the spectrograms of cough, breath, and speech samples for the automatic detection of COVID-19. To extract deep learnt representations from the spectrograms, we compare the performance of specific Convolutional Neural Networks (CNNs) trained from scratch and ResNet18- based CNNs fine-tuned for the task at hand. Furthermore, we investigate whether the patients’ sex and the use of contextual attention mechanisms are beneficial. Our experiments use the dataset released as part of the Second Diagnosing COVID-19 using Acoustics (DiCOVA) Challenge. The results suggest the suitability of fusing breath and speech information to detect COVID-19. An Area Under the Curve (AUC) of 84.06 % is obtained on the test partition when using specific CNNs trained from scratch with contextual attention mechanisms. When using ResNet18-based CNNs for feature extraction, the baseline model scores the highest performance with an AUC of 84.26 %., This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No. 826506 (sustAGE), and from the Spanish Ministry of Science and Innovation under the Musical AI project (PID2019-111403GB-I00).
Repositorio Digital de la UPF
Publicación Contribución de congreso (ConferenceOutput)

PYWDF: an open source library for prototyping and simulating wave filter circuits in Python

  • Anthon, Gustav
  • Lizarraga Seijas, Xavier
  • Font Corbera, Frederic
Comunicació presentada a 26th International Conference on Digital Audio Effects (DAFx-23), celebrada del 4 al 7 desetembre de 2023 a Copenhaguen, Dinamarca., This paper introduces a new open-source Python library for the modeling and simulation of wave digital filter (WDF) circuits. The library, called pwydf, allows users to easily create and analyze WDF circuit models in a high-level, object-oriented manner. The library includes a variety of built-in components, such as voltage sources, capacitors, diodes etc., as well as the ability to create custom components and circuits. Additionally, pywdf includes a variety of analysis tools, such as frequency response and transient analysis, to aid in the design and optimization of WDF circuits. We demonstrate the library’s efficacy in replicating the nonlinear behavior of an analog diode clipper circuit, and in creating an allpass filter that cannot be realized in the analog world. The library is well-documented and includes several examples to help users get started. Overall, pywdf is a powerful tool for anyone working with WDF circuits, and we hope it can be of great use to researchers and engineers in the field., This research was carried out under the project Musical AI -PID2019- 111403GB-I00/AEI/10.13039/501100011033, funded by the Spanish Ministerio de Ciencia e Innovación and the Agencia Estatal de Investigación.
Repositorio Digital de la UPF
Publicación Contribución de congreso (ConferenceOutput)

Bottlenecks and solutions for audio to score alignment research

  • Morsi, Alia
  • Serra, Xavier
Comunicació presentada a 23nd International Society for Music Information Retrieval Conference (ISMIR 2022), celebrat del 4 al 8 de desembre de 2022 a Bangalore, Índia., Although audio to score alignment is a classic Music Information Retrieval problem, it has not been defined uniquely with the scope of musical scenarios representing its core. The absence of a unified vision makes it difficult to pinpoint its state-of-the-art and determine directions for improvement. To get past this bottleneck, it is necessary to consolidate datasets and evaluation methodologies to allow comprehensive benchmarking. In our review of prior work, we demonstrate the extent of variation in problem scope, datasets, and evaluation practices across audio to score alignment research. To circumvent the high cost of creating large-scale datasets with various instruments, styles, performance conditions, and musician proficiency levels from scratch, the research community could generate ground truth approximations from non-audio to score alignment datasets which include a temporal mapping between a music score and its corresponding audio. We show a methodology for adapting the Aligned Scores and Performances dataset, created originally for beat tracking and music transcription. We filter the dataset semi-automatically by applying a set of Dynamic Time Warping based Audio to Score Alignment methods using out-of-the-box Chroma and Constant-Q Transform extraction algorithms, suitable for the characteristics of the piano performances of the dataset. We use the results to discuss the limitations of the generated ground truths and data adaptation method. While the adapted dataset does not provide the necessary diversity for solving the initial problem, we conclude with ideas for expansion, and identify future directions for curating more comprehensive datasets through data adaptation, or synthesis., This research was carried out under the project Musical AI - PID2019-111403GB-I00/AEI/10.13039/501100011033, funded by the Spanish Ministerio de Ciencia e Innovación and the Agencia Estatal de Investigación.
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Voice assignment in vocal quartets using deep learning models based on pitch salience

  • Cuesta, Helena
  • Gómez Gutiérrez, Emilia, 1975-
This paper deals with the automatic transcription of four-part, a cappella singing, audio performances. In particular, we exploit an existing, deep-learning based, multiple F0 estimation method and complement it with two neural network architectures for voice assignment (VA) in order to create a music transcription system that converts an input audio mixture into four pitch contours. To train our VA models, we create a novel synthetic dataset by collecting 5381 choral music scores from public-domain music archives, which we make publicly available for further research. We compare the performance of the proposed VA models on different types of input data, as well as to a hidden Markov model-based baseline system. In addition, we assess the generalization capabilities of these models on audio recordings with differing pitch distributions and vocal music styles. Our experiments show that the two proposed models, a CNN and a ConvLSTM, have very similar performance, and both of them outperform the baseline HMM-based system. We also observe a high confusion rate between the alto and tenor voice parts, which commonly have overlapping pitch ranges, while the bass voice has the highest scores in all evaluated scenarios., This work is partially supported by the European Commission under the TROMPA project (H2020 770376), the Spanish Ministry of Science and Innovation under the Musical AI project (PID2019-111403GB-I00), and by AGAUR (Generalitat de Catalunya) through an FI Predoctoral Grant (2018FI-B01015).
Repositorio Digital de la UPF
Publicación Contribución de congreso (ConferenceOutput)

Analysis of musical dynamics in vocal performances

  • Narang, Jyoti
  • Miron, Marius
  • Lizarraga Seijas, Xavier
  • Serra, Xavier
Comunicació presentada a: The 15th International Symposium on Computer Music Multidisciplinary Research celebrat del 15 al 19 de novembre de 2021 de manera virtual., Dynamics are one of the fundamental tools of expressivity in a performance. While the usage of this tool is highly subjective, a systematic methodology to derive loudness markings based on a performance can be highly beneficial. With this goal in mind, this paper is a first step towards developing a methodology to automatically transcribe dynamic markings from vocal rock and pop performances. To this end, we make use of commercial recordings of some popular songs followed by source separation and compare them to the karaoke versions of the same songs. The dynamic variations in the original commercial recordings are found to be structurally very similar to the aligned karaoke/multi-track versions of the same tracks. We compare and show the differences between tracks using statistical analysis, with an eventual goal to use the transcribed markings as guiding tools, to help students adapt with a specific interpretation of a given piece of music. We perform a qualitative analysis of the proposed methodology with the teachers in terms of informativeness and accuracy., Part of this research is funded by the projects Musical AI (PID2019-111403GB-I00/AEI/10.13039/501100011033 funded by the Spanish Ministerio de Ciencia, Innovación y Universidades (MCIU) and the Agencia Estatal de Investigación (AEI)) and NextCore (RTC2019-007248-7 funded by the Spanish Ministerio de Ciencia, Innovación y Universidades (MCIU) and the Agencia Estatal de Investigación (AEI)).
Repositorio Digital de la UPF
Publicación Contribución de congreso (ConferenceOutput)

Violin etudes: a comprehensive dataset for f0 estimation and performance analysis

  • Can Tamer, Nazif
  • Ramoneda, Pedro
  • Serra, Xavier
Comunicació presentada a 23nd International Society for Music Information Retrieval Conference (ISMIR 2022), celebrat del 4 al 8 de desembre de 2022 a Bangalore, Índia., Violin performance analysis requires accurate and robust f0 estimates to give feedback on the playing accuracy. Despite the recent advancements in data-driven f0 estimators, their application to performance analysis remains a challenge due to style-specific and dataset-induced biases. In this paper, we address this problem by introducing Violin Etudes, a 27.8-hours violin performance dataset constructed with domain knowledge in instrument pedagogy and a novel automatic f0-labeling paradigm. Experimental results on unseen datasets show that the CREPE f0 estimator trained on Violin Etudes outperforms the widely-used pre-trained version trained on multiple manually-labeled datasets. Further preliminary findings suggest that (i) existing data-driven f0 estimators may overfit to equal temperament, and (ii) iterative re-labeling regularized by our novel Constrained Harmonic Resynthesis method can simultaneously enhance datasets and f0 estimators. Our dataset curation methodology is easily scalable to other instruments owing to the quantity of pedagogical data online. It also supports a range of MIR research directions thanks to the performance difficulty labels from educational institutions., We would like to thank Esteban Maestre for his valuable insights on the violin resonance structure. This research was carried out under the project Musical AI - PID2019- 111403GB-I00/AEI/10.13039/501100011033, funded by the Spanish Ministerio de Ciencia e Innovación and the Agencia Estatal de Investigación.
Repositorio Digital de la UPF
Publicación Contribución de congreso (ConferenceOutput)

In search of Sañcāras: tradition-informed repeated melodic pattern recognition in carnatic music

  • Nuttall, Thomas
  • Plaja-Roglans, Genís
  • Pearson, Lara
  • Serra, Xavier
Comunicació presentada a 23nd International Society for Music Information Retrieval Conference (ISMIR 2022), celebrat del 4 al 8 de desembre de 2022 a Bangalore, Índia., Carnatic Music is a South Indian art and devotional musical practice in which melodic patterns (motifs and phrases), known as sañcaras Å , play a crucial structural and expressive role. We demonstrate how the combination of transposition invariant features learnt by a Complex Autoencoder (CAE) and predominant pitch tracks extracted using a Frequency-Temporal Attention Network (FTANet) can be used to annotate and group regions of variable-length, repeated, melodic patterns in audio recordings of multiple Carnatic Music performances. These models are trained on novel, expert-curated datasets of hundreds of Carnatic audio recordings and the extraction process tailored to account for the unique characteristics of sañcaras Å in Carnatic Music. Experimental results show that the proposed method is able to identify 54% of all sañcaras Å annotated by a professional Carnatic vocalist. Code to reproduce and interact with these results is available online., This research work was carried out under the project Musical AI - PID2019- 111403GBI00/AEI/10.13039/501100011033, funded by the Spanish Ministerio de Ciencia e Innovación and the Agencia Estatal de Investigación.
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Repertoire-specific vocal pitch data generation for improved melodic analysis of carnatic music

  • Plaja-Roglans, Genís
  • Nutall, Thomas
  • Pearson, Lara
  • Serra, Xavier
  • Miron, Marius
Deep Learning methods achieve state-of-the-art in many tasks, including vocal pitch extraction. However, these methods rely on the availability of pitch track annotations without errors, which are scarce and expensive to obtain for Carnatic Music. Here we identify the tradition-related challenges and propose tailored solutions to generate a novel, large, and open dataset, the Saraga-Carnatic-Melody-Synth (SCMS), comprising audio mixtures and time-aligned vocal pitch annotations. Through a cross-cultural evaluation leveraging this novel dataset, we show improvements in the performance of Deep Learning vocal pitch extraction methods on Indian Art Music recordings. Additional experiments show that the trained models outperform the currently used heuristic-based pitch extraction solutions for the computational melodic analysis of Carnatic Music and that this improvement leads to better results in the musicologically relevant task of repeated melodic pattern discovery when evaluated using expert annotations. The code and annotations are made available for reproducibility. The novel dataset and trained models are also integrated into the Python package compIAM1 which allows them to be used out-of-the-box., This work is funded by the Spanish Ministerio de Ciencia, Innovación y Universidades (MCIU) and the Agencia Estatal de Investigación (AEI) within the Musical AI Project – PID2019-111403GB-I00/AEI/10.13039/501100011033.
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Personalisation and profiling using algorithms and not-so-popular Colombian music: goal-directed mechanisms in music emotion recognition

  • Gómez-Cañon, Juan Sebastián
  • Lennie, Thomas Magnus
  • Eerola, Tuomas
  • Aragón, Pablo
  • Cano, Estefanía
  • Herrera, Perfecto
  • Gómez, Emilia
This work investigates how personalised Music Emotion Recognition (MER) systems may lead to sensitive profiling when applied to musically induced emotions in politically charged contexts. We focus on traditional Colombian music with explicit political content, including (1) vallenatos and social songs aligned with the left-wing guerrilla Fuerzas Armadas Revolucionarias de Colombia (FARC), and (2) corridos linked to sympathisers of the right-wing paramilitary group Autodefensas Unidas de Colombia (AUC). Using data from 49 participants with diverse political leanings, we train personalised machine learning models to predict induced emotional responses -particularly negative emotions. Our findings reveal that political identity plays a significant role in shaping emotional experiences of music with explicit political content, and that emotion recognition models can capture this variation to a certain extent. These results raise critical concerns about the potential misuse of emotion recognition technologies. What is often framed as a tool for wellbeing and emotional regulation could, in politically sensitive contexts, be repurposed for user profiling. This work highlights the ethical risks of deploying AI-driven emotion analysis without safeguards, particularly among populations that are politically or socially vulnerable. We argue that subjective emotional responses may constitute sensitive personal data, and that failing to account for their sociopolitical context could amplify harm and exclusion., JSGC's contribution was partially supported by the European Commission through the TROMPA project (H2020 770376) and the project Musical AI (PID2019-111403GB-I00/AEI/10.13039/501100011033), funded by the Spanish Ministerio de Ciencia, Innovación y Universidades (MCIU) and the Agencia Estatal de Investigación (AEI). TML was supported by the Arts and Humanities Research Council (AHRC) through a Northern Bridge Doctoral Training Partnership scholarship (grant code: AH/L503927/1).
Repositorio Digital de la UPF
Publicación Contribución de congreso (ConferenceOutput)

An overview of automatic piano performance assessment within the music education context

  • Kim, Hyon
  • Ramoneda, Pedro
  • Miron, Marius
  • Serra, Xavier
Comunicació presentada a la 14th International Conference on Computer Supported Education, celebrada del 22 a 24 d'abril de 2021 de manera virtual., Piano is one of the most popular instruments among music learners. Technologies to evaluate piano performances have been researched and developed in recent years rapidly, including data driven methods using machine learning. Despite the demand from people and speed of the development, there are still gaps between the methods and the pedagogical setup for real use case scenarios due to lack of accuracy of methods, insufficient amount of training data or the biases in training machine learning models, ignoring actual use case of the technology and such. In this paper, we first propose a feedback approach in piano performance education and review methods for Automated Piano Performance Assessment (APPA). After that, we discuss about gaps between a feedback approach and current methods, emphasizing their music education application. As a future work we propose a potential approach to overcome the gaps., This project is supported by Project Musical AI funded by the Spanish Ministerio de Ciencia, Innovacion y Universidades (MCIU) and the Agencia Estatal de Investigacion (AEI).
Repositorio Digital de la UPF
Publicación Contribución de congreso (ConferenceOutput)

The Matrix profile for motif discovery in audio - an example application in Carnatic music

  • Nuttall, Thomas
  • Plaja-Roglans, Genís
  • Pearson, Lara
  • Serra, Xavier
Comunicació presentada a: The 15th International Symposium on Computer Music Multidisciplinary Research celebrat del 15 al 19 de novembre de 2021 de manera virtual., We present here a pipeline for the automated discovery of repeated motifs in audio. Our approach relies on state-of-the-art source separation, pre- dominant pitch extraction and time series motif detection via the matrix profile. Owing to the appropriateness of this approach for the task of motif recognition in the Carnatic musical style of South India, and with access to the recently re- leased Saraga Dataset of Indian Art Music, we provide an example application on a recording of a performance in the Carnatic r ̄aga, R ̄itigaul.a, finding 56 dis- tinct patterns of varying lengths that occur at least 3 times in the recording. The authors include a discussion of the potential musicological significance of this motif finding approach in relation to the particular tradition and beyond., This research was funded by the MUSICAL AI project (PID2019- 111403GB-I00) granted by the Ministry of Science and Innovation of the Spanish Gov- ernment. We also thank Rafael Caro Repetto for his continued guidance and input.
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

A deep-learning based framework for source separation, analysis, and synthesis of choral ensembles

  • Chandna, Pritish
  • Cuesta, Helena
  • Petermann, Darius
  • Gómez Gutiérrez, Emilia, 1975-
Choral singing in the soprano, alto, tenor and bass (SATB) format is a widely practiced and studied art form with significant cultural importance. Despite the popularity of the choral setting, it has received little attention in the field of Music Information Retrieval. However, the recent publication of high-quality choral singing datasets as well as recent developments in deep learning based methodologies applied to the field of music and speech processing, have opened new avenues for research in this field. In this paper, we use some of the publicly available choral singing datasets to train and evaluate state-of-the-art source separation algorithms from the speech and music domains for the case of choral singing. Furthermore, we evaluate existing monophonic F0 estimators on the separated unison stems and propose an approximation of the perceived F0 of a unison signal. Additionally, we present a set of applications combining the proposed methodologies, including synthesizing a single singer voice from the unison, and transposing and remixing the separated stems into a synthetic multi-singer choral signal. We finally conduct a set of listening tests to perform a perceptual evaluation of the results we obtain with the proposed methodologies., This work is partially supported by the European Commission under the TROMPA project (H2020 770376), and the project Musical AI (PID 2019-111403GB-I00/AEI/10.13039/501100011033) funded by the Spanish Ministerio de Ciencia, Innovación y Universidades (MCIU) and the Agencia Estatal de Investigación (AEI).
Repositorio Digital de la UPF
Publicación Contribución de congreso (ConferenceOutput)

Diversity in the music listening experience: insights from focus group interviews

  • Porcaro, Lorenzo
  • Gómez Gutiérrez, Emilia, 1975-
  • Castillo, Carlos
Comunicació presentada a: CHIIR '22: ACM SIGIR Conference on Human Information Interaction and Retrieval, celebrat del 14 al 18 de març de 2022 a Regensburg, Alemanya., Music listening in today’s digital spaces is highly characterized by the availability of huge music catalogues, accessible by people all over the world. In this scenario, recommender systems are designed to guide listeners in finding tracks and artists that best fit their requests, having therefore the power to influence the diversity of the music they listen to. Albeit several works have proposed new techniques for developing diversity-aware recommendations, little is known about how people perceive diversity while interacting with music recommendations. In this study, we interview several listeners about the role that diversity plays in their listening experience, trying to get a better understanding of how they interact with music recommendations. We recruit the listeners among the participants of a previous quantitative study, where they were confronted with the notion of diversity when asked to identify, from a series of electronic music lists, the most diverse ones according to their beliefs. As a follow-up, in this qualitative study we carry out semi-structured interviews to understand how listeners may assess the diversity of a music list and to investigate their experiences with music recommendation diversity. We report here our main findings on 1) what can influence the diversity assessment of tracks and artists’ music lists, and 2) which factors can characterize listeners’ interaction with music recommendation diversity., This work is partially supported by the European Commission under the TROMPA project (H2020 - grant agreement No. 770376). It is also part of the project Musical AI - PID2019-111403GB-I00/AEI/10.13039/501100011033 funded by the Spanish Ministerio de Ciencia, Innovación y Universidades (MCIU) and the Agencia Estatal de Investigación (AEI). This work is also partially supported by the HUMAINT programme (Human Behaviour and Machine Intelligence), Joint Research Centre, European Commission. The project leading to these results received funding from "la Caixa" Foundation (ID 100010434), under the agreement LCF/PR/PR16/51110009.
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Applications of biological and physiological signals in commercial video gaming and game research: a review

  • Hughes, Alayna
  • Jordà Puig, Sergi
Video gaming is now available as a fully immersive experience that creates responsive inputs and outputs concerning the user, and some experimental developers have integrated the use of the voice, brain, or muscles as input controls. The use of physiological signal equipment can provide valuable information regarding the emotion of a player or patient during gameplay. In this article, we discuss five of the most common biosignals that are used in gaming research, and their function and devices that may be used for measurement. We break down those individual signals and present examples of research studies that implement them. We also discuss the usage of biological signals within commercial gaming and conclude with some possible future directions for the use of biological signals in gaming and game research., This research has been partially supported by the project Musical AI - PID2019-111403GB-I00/AEI/10.13039/501100011033 funded by the Spanish Ministerio de Ciencia, Innovación y Universidades (MCIU) and the Agencia Estatal de Investigación (AEI).
Repositorio Digital de la UPF
Publicación Contribución de congreso (ConferenceOutput)

Personalized musically induced emotions of not-so-popular Colombian music

  • Gómez Cañón, Juan Sebastián
  • Herrera Boyer, Perfecto, 1964-
  • Cano, Estefanía
  • Gómez Gutiérrez, Emilia, 1975-
Comunicació presentada al workshop Human Centered AI inclòs a: 35th Conference on Neural Information Processing Systems (NeurIPS 2021) celebrat el 13 de desembre de manera virtual., This work presents an initial proof of concept of how Music Emotion Recognition (MER) systems could be intentionally biased with respect to annotations of musically-induced emotions in a political context. In specific, we analyze traditional Colombian music containing politically-charged lyrics of two types: (1) vallenatos and social songs from the “left-wing” guerrilla Fuerzas Armadas Revolucionarias de Colombia (FARC) and (2) corridos from the “right-wing” paramilitaries Autodefensas Unidas de Colombia (AUC). We train personalized machine learning models to predict induced emotions for three users with diverse political views – we aim at identifying the songs that may induce negative emotions for a particular user, such as anger and fear. To this extent, a user’s emotion judgements could be interpreted as problematizing data – subjective emotional judgments could in turn be used to influence the user in a human-centered machine learning environment. In short, highly desired “emotion regulation” applications could potentially deviate to “emotion manipulation” – the recent discredit of emotion recognition technologies might transcend ethical issues of diversity and inclusion., The research work conducted at the Universitat Pompeu Fabra is partially supported by the Eu- ropean Commission under the TROMPA project (H2020 770376) and the Project Musical AI - PID2019-111403GB-I00/AEI/10.13039/501100011033 funded by the Spanish Ministerio de Ciencia, Innovación y Universidades (MCIU) and the Agencia Estatal de Investigación (AEI).
Repositorio Digital de la UPF
Publicación Contribución de congreso (ConferenceOutput)

MusAV: a dataset of relative arousal-valence annotations for validation of audio models

  • Bogdanov, Dmitry
  • Lizarraga Seijas, Xavier
  • Alonso-Jiménez, Pablo
  • Serra, Xavier
Comunicació presentada a 23nd International Society for Music Information Retrieval Conference (ISMIR 2022), celebrat del 4 al 8 de desembre de 2022 a Bangalore, Índia., We present MusAV, a new public benchmark dataset for comparative validation of arousal and valence (AV) regression models for audio-based music emotion recognition. To gather the ground truth, we rely on relative judgments instead of absolute values to simplify the manual annotation process and improve its consistency. We build MusAV by gathering comparative annotations of arousal and valence on pairs of tracks, using track audio previews and metadata from the Spotify API. The resulting dataset contains 2,092 track previews covering 1,404 genres, with pairwise relative AV judgments by 20 annotators and various subsets of the ground truth based on different levels of annotation agreement. We demonstrate the use of the dataset in an example study evaluating nine models for AV regression that we train based on state-of-the-art audio embeddings and three existing datasets of absolute AV annotations. The results on MusAV offer a view of the performance of the models complementary to the metrics obtained during training and provide insights into the impact of the considered datasets and embeddings on the generalization abilities of the models., This research was carried out under the project Musical AI - PID2019-111403GB-I00/AEI/10.13039/501100011033, funded by the Spanish Ministerio de Ciencia e Innovación and the Agencia Estatal de Investigación. We also thank Juan Sebastián Gómez Cañón for his suggestions and all participating annotators.
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

The creative drummer: an eeg-based pilot study on the correlates of emotions and creative drum playing

  • Ramírez, Rafael, 1966-
  • Reija, Xavier
It is reasonable to assume that emotional processes are involved in creative tasks and the generation of creative ideas. In this pilot study, we investigate the emotional correlates in professional drummers during different degrees of creative music playing. Ten participants performed three tasks: repetitive rhythmic drum playing, pattern-based improvisation, and attention-intensive free improvisation, while their EEG activity was recorded. Arousal and valence levels were estimated from the EEG data at baseline and for the three tasks. Results show significantly increased levels of valence (i.e., increased prefrontal right alpha power compared to prefrontal left alpha power) during pattern-based and free improvisation relative to baseline, and significantly increased levels of valence during free improvisation relative to pattern-based improvisation. These results seem to indicate that positive emotion (characterized as increased valence) is associated with the creation of original ideas in drum playing and that the freer the creative process, the greater the positive effect. The implication of these results may be of particular relevance in the fields of music-based therapeutic interventions and music pedagogy., This work was partly sponsored by the European Union Horizon 2020 research and innovation program under grant agreement No. 688269 (TELMI project) and by the Musical AI project—(PID2019-111403GB-I00/AEI/10.13039/501100011033).
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Error monitoring and correction in violin performance: an EEG study

  • Blanco, Angel David
  • Costa-Faidella, Jordi
  • Pérez Carrillo, Alfonso Antonio, 1977-
  • Dalmazzo, David
  • Ramirez Melendez, Rafael
  • San Miguel Insúa, Iria
Previous research using electroencephalography (EEG) has investigated neural error-related music production processes in expert pianists reporting a frontal event-related negativity followed by a P300 event-related potential (ERP) after mistuned notes. However, piano playing does not rely on auditory feedback and corrective movements when dealing with accurate note tuning, thus offering an incomplete picture about error monitoring processes in players of many other musical instruments, including the human voice. We here designed a setup allowing us to collect the EEG of expert bow-string players while performing short melodies on the violin, (a fretless instrument) while also manipulating in real-time the produced pitch in a controlled manner. 15 expert bow-string players were asked to 1) listen to a reference melody of four notes; 2) play that melody with a violin (Active Condition); and 3) listen passively to the replay of their performance (Replayed Condition). Randomly, the auditory feedback of one of the four melody notes was manipulated in the Active Condition by lowering or lifting the pitch by half semitone. We found that mistuned notes, independently of whether they were manipulated externally or were produced by natural playing errors, elicited an f-ERN followed by a P300. We also found that the amplitude of the f-ERN was larger for faster corrective movements than for slower ones. In addition, between the f-ERN and the P300, we also identified two negative components-one in parietal areas and another in central-right regions-whose amplitudes were greater for slower corrective movements. We suggest the existence of two error-monitoring systems: a "fast" system mediated by the medial frontal cortex that quickly corrects errors when certainty is high and a "slow" system mediated by the posterior parietal cortex that accumulates evidence under uncertainty before triggering correction., This work was partly sponsored by the Musical AI project (PID2019-111403GB-I00) funded by MCIN/AEI/ 10.13039/501100011033 and by 'ERDF A way of making Europe'; by the Project PID2021-128790NB-I00, funded by MCIN/AEI/ 10.13039/501100011033 / FEDER UE; and by the European Union Horizon 2020 research and innovation program under grant agreement No. 688269 (TELMI project). It has been additionally supported by the Excellence Research Group 2017SGR-974 funded by the Secretaria d'Universitats i Recerca del Departament d'Empresa i Coneixement de la Generalitat de Catalunya and by the MDM-2017-0729-18-2M Maria de Maeztu Center of Excellence UBNeuro, funded by MCIN/AEI/ 10.13039/501100011033. ISM was supported by grant RYC-2013-12577, funded by MCIN/AEI/ 10.13039/501100011033 and by 'ESF Investing in your future'. Open access publication fees were partly covered by the University of Barcelona (Ajudes de la Universitat de Barcelona per a publicar en accés obert).
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

TROMPA-MER: an open dataset for personalized music emotion recognition

  • Gómez Cañón, Juan Sebastián
  • Gutiérrez Páez, Nicolás Felipe
  • Porcaro, Lorenzo
  • Porter, Alastair
  • Cano, Estefanía
  • Herrera Boyer, Perfecto, 1964-
  • Gkiokas, Aggelos
  • Santos Rodríguez, Patrícia
  • Hernández Leo, Davinia
  • Karreman, Casper
  • Gómez Gutiérrez, Emilia, 1975-
We present a platform and a dataset to help research on Music Emotion Recognition (MER). We developed the Music Enthusiasts platform aiming to improve the gathering and analysis of the so-called “ground truth” needed as input to MER systems. Firstly, our platform involves engaging participants using citizen science strategies and generate music emotion annotations – the platform presents didactic information and musical recommendations as incentivization, and collects data regarding demographics, mood, and language from each participant. Participants annotated each music excerpt with single free-text emotion words (in native language), distinct forced-choice emotion categories, preference, and familiarity. Additionally, participants stated the reasons for each annotation – including those distinctive of emotion perception and emotion induction. Secondly, our dataset was created for personalized MER and contains information from 181 participants, 4721 annotations, and 1161 music excerpts. To showcase the use of the dataset, we present a methodology for personalization of MER models based on active learning. The experiments show evidence that using the judgment of the crowd as prior knowledge for active learning allows for more effective personalization of MER systems for this particular dataset. Our dataset is publicly available and we invite researchers to use it for testing MER systems., This work was funded by the European Commission under the TROMPA project (H2020 770376) and the Project Musical AI - PID2019-111403GB-I00/AEI/10.13039/501100011033 funded by the Spanish Ministerio de Ciencia, Innovación y Universidades (MCIU) and the Agencia Estatal de Investigación (AEI).
Repositorio Digital de la UPF
Publicación Contribución de congreso (ConferenceOutput)

NeuralMidiFx: a wrapper template for deploying neural networks as VST3 plugins

  • Haki, Behzad
  • Lenz, Julian
  • Jordà Puig, Sergi
Comunicació presentada a AIMC 2023, celebrada del 30 d'agost al 1 de setembre de 2023 a Sussex, Anglaterra., Proper research, development and evaluation of AI-based generative systems of music that focus on performance or composition require active user-system interactions. To include a diverse group of users that can properly engage with a given system, researchers should provide easy access to their developed systems. Given that many users (i.e. musicians) are non-technical to the field of AI and the development frameworks involved, the researchers should aim to make their systems accessible within the environments commonly used in production/composition workflows (e.g. in the form of plugins hosted in digital audio workstations). Unfortunately, deploying generative systems in this manner is highly expensive. As such, researchers with limited resources are often unable to provide easy access to their works, and subsequently, are not able to properly evaluate and encourage active engagement with their systems. Facing these limitations, we have been working on a solution that allows for easy, effective and accessible deployment of generative systems. To this end, we propose a wrapper/template called NeuralMidiFx, which streamlines the deployment of neural network based symbolic music generation systems as VST3 plugins. The proposed wrapper is intended to allow researchers to develop plugins with ease while requiring minimal familiarity with plugin development., This research was partly funded by the Ministry of Science and Innovation of the Spanish Government. Agencia Estatal de Investigación (AEI). (Reference: PID2019-111403GB-I00)
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Music-enhanced emotion identification of facial emotions in autistic spectrum disorder children: a pilot EEG study

  • Ramírez, Rafael, 1966-
  • Matamoros, Elisabet
  • Hernández-Leo, Davinia
  • Mirabel, Júlia
  • Sánchez, Elisabet
  • Escude, Núria
The Autistic Spectrum Disorder (ASD) is characterized by a difficulty in expressing and interpreting others’ emotions. In particular, people with ASD have difficulties when interpreting emotions encoded in facial expressions. In the past, music interventions have been shown to improve autistic individuals’ emotional and social skills. The present study describes a pilot study to explore the usefulness of music as a tool for improving autistic children’s emotion recognition in facial expressions. Twenty-five children (mean age = 8.8 y, SD = 1.24) with high-functioning ASD and normal hearing participated in the study consisting of four weekly sessions of 15 min each. Twenty-five participants were randomly divided into an experimental group (N = 14) and a control group (N = 11). During each session, participants in the experimental group were exposed to images of facial expressions for four emotions (happy, sad, angry, and fear). Images were shown in three conditions, with the second condition consisting of music of congruent emotion with the shown images. Participants in the control group were shown only images in all three conditions. For six participants in each group, EEG data were acquired during the sessions, and instantaneous emotional responses (arousal and valence values) were extracted from the EEG data. Inter- and intra-session emotion identification improvement was measured in terms of verbal response accuracy, and EEG response differences were analyzed. A comparison of the verbal responses of the experimental group pre- and post-intervention showed a significant (p = 0.001) average improvement in emotion identification accuracy responses of 26% (SD = 3.4). Furthermore, emotional responses of the experimental group at the end of the study showed a higher correlation with the emotional stimuli being presented, compared with their emotional responses at the beginning of the study. No similar verbal responses improvement or EEG-stimuli correlation was found in the control group. These results seem to indicate that music can be used to improve both emotion identification in facial expressions and emotion induction through facial stimuli in children with high-functioning ASD., This work was partly sponsored by the European Union Horizon 2020 research and innovation program under grant agreement No. 688269 (TELMI project) and by the Spanish Ministerio de Ciencia, Innovación y Universidades (MCIU) and the Agencia Estatal de Investigación (AEI) project Musical AI—PID2019-111403GB-I00/AEI/10.13039/501100011033.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico antes de ser publicado, versión del autor (Preprint)

Bottlenecks and solutions for audio to score alignment research

  • Morsi, Alia
  • Serra, Xavier
This work has been accepted at the 23rd International Society for Music Information Retrieval Conference (ISMIR 2022), at Bengaluru, India. December 4-8, 2022., Although audio to score alignment is a classic Music Information Retrieval problem, it has not been defined uniquely with the scope of musical scenarios representing its core. The absence of a unified vision makes it difficult to pinpoint its state-of-the-art and determine directions for improvement. To get past this bottleneck, it is necessary to consolidate datasets and evaluation methodologies to allow comprehensive benchmarking. In our review of prior work,we demonstrate the extent of variation in problem scope, datasets, and evaluation practices across audio to score alignment research. To circumvent the high cost of creating large-scale datasets with various instruments, styles, performance conditions, and musician proficiency from scratch, the research community could generate ground truth approximations from non-audio to score alignment datasets which include a temporal mapping between a music score and its corresponding audio. We show a methodology for adapting the Aligned Scores and Performances dataset, created originally for beat tracking and music transcription. We filter the dataset semi-automatically by applying a set of Dynamic Time Warping based Audio to Score Alignment methods using out-of-the-box Chroma and Constant-Q Transform extraction algorithms, suitable for the characteristics of the piano performances of the dataset. We use the results to discuss the limitations of the generated ground truths and data adaptation method. While the adapted dataset does not provide the necessary diversity for solving the initial problem, we conclude with ideas for expansion, and identify future directions for curating more comprehensive datasets through data adaptation, or synthesis., This research was carried out under the project Musical AI - PID2019-111403GB-I00/AEI/10.13039/501100011033, funded by the Spanish Ministerio de Ciencia e Innovación and the Agencia Estatal de Investigación.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico antes de ser publicado, versión del autor (Preprint)

Score-Informed MIDI Velocity Estimation for Piano Performance by FiLM Conditioning

  • Kim, Hyon
  • Miron, Marius
  • Serra, Xavier
This work has been accepted at the SMC Sound and Music Computing Conference 2023, at Stockholm, Sweden. June 15-17, 2023., Piano is one of the most popular instruments among people that learn to play music. When playing the piano, the level of loudness is crucial for expressing emotions as well as manipulating tempo. These elements convey the expressiveness of music performance. Detecting the loudness of each note could provide more valuable feedback for music students, helping to improve their performance dynamics. This can be achieved by visualizing the loudness levels not only for self-learning purposes but also for effective communication between teachers and students. Also, given the polyphonic nature of piano music, which often involves parallel melodic streams, determining the loudness of each note is more informative than analyzing the cumulative loudness of a specific time frame. This research proposes a method using Deep Neural Network (DNN) with score information to estimate note-level MIDI velocity of piano performances from audio input. In addition, when score information is available, we condition the DNN with score information using a Feature-wise Linear Modulation (FiLM) layer. To the best of our knowledge, this is the first attempt to estimate the MIDI velocity using a neural network in an end to end fashion. The model proposed in this study achieved improved accuracy in both MIDI velocity estimation and estimation error deviation, as well as higher recall accuracy for note classification when compared to the DNN model that did not use score information, This research was carried out under the project Musical AI - PID2019- 111403GB-I00/AEI/10.13039/501100011033, funded by the Spanish Ministerio de Ciencia e Innovacion ́ and the Agencia Estatal de Investigacion.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Contribución de congreso (ConferenceOutput)

Bottlenecks and solutions for audio to score alignment research

  • Morsi, Alia
  • Serra, Xavier
Comunicació presentada a 23nd International Society for Music Information Retrieval Conference (ISMIR 2022), celebrat del 4 al 8 de desembre de 2022 a Bangalore, Índia., Although audio to score alignment is a classic Music Information Retrieval problem, it has not been defined uniquely with the scope of musical scenarios representing its core. The absence of a unified vision makes it difficult to pinpoint its state-of-the-art and determine directions for improvement. To get past this bottleneck, it is necessary to consolidate datasets and evaluation methodologies to allow comprehensive benchmarking. In our review of prior work, we demonstrate the extent of variation in problem scope, datasets, and evaluation practices across audio to score alignment research. To circumvent the high cost of creating large-scale datasets with various instruments, styles, performance conditions, and musician proficiency levels from scratch, the research community could generate ground truth approximations from non-audio to score alignment datasets which include a temporal mapping between a music score and its corresponding audio. We show a methodology for adapting the Aligned Scores and Performances dataset, created originally for beat tracking and music transcription. We filter the dataset semi-automatically by applying a set of Dynamic Time Warping based Audio to Score Alignment methods using out-of-the-box Chroma and Constant-Q Transform extraction algorithms, suitable for the characteristics of the piano performances of the dataset. We use the results to discuss the limitations of the generated ground truths and data adaptation method. While the adapted dataset does not provide the necessary diversity for solving the initial problem, we conclude with ideas for expansion, and identify future directions for curating more comprehensive datasets through data adaptation, or synthesis., This research was carried out under the project Musical AI - PID2019-111403GB-I00/AEI/10.13039/501100011033, funded by the Spanish Ministerio de Ciencia e Innovación and the Agencia Estatal de Investigación.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico antes de ser publicado, versión del autor (Preprint)

A diffusion-inspired training strategy for singing voice extraction in the waveform domain

  • Plaja-Roglans, Genís
  • Miron, Marius
  • Serra, Xavier
This work has been accepted at the 23rd International Society for Music Information Retrieval Conference (ISMIR 2022), at Bengaluru, India. December 4-8, 2022., Notable progress in music source separation has been achieved using multi-branch networks that operate on both temporal and spectral domains. However, such networks tend to be complex and heavy-weighted. In this work, we tackle the task of singing voice extraction from polyphonic music signals in an end-to-end manner using an approach inspired by the training procedure of denoising diffusion models. We perform unconditional signal modelling to gradually convert an input mixture signal to the corresponding singing voice or accompaniment. We use fewer parameters than the state-of-the-art models while operating on the waveform domain, bypassing phase-related problems. More concisely, we train a non-causal WaveNet using a diffusion-inspired strategy improving the said network for singing voice extraction and obtaining performance comparable to the end-to-end state-of-the-art on MUSDB18. We further report results on a non-MUSDB-overlapping version of MedleyDB and the multi-track audio of the Saraga Carnatic dataset showing good generalization, and run perceptual tests of our approach. Code, models, and audio examples are made available., This work was carried out under the projects Musical AI - PID2019-111403GB-I00/AEI/10.13039/501100011033 and NextCore - RTC2019-007248-7 funded by the Spanish Ministerio de Ciencia, Innovación y Universidades (MCIU) and the Agencia Estatal de Investigación (AEI).
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Audio and music analysis on the web using Essentia.js

  • Correya, Albin Andrew
  • Marcos Fernández, Jorge
  • Joglar-Ongay, Luis
  • Alonso Jiménez, Pablo
  • Serra, Xavier
  • Bogdanov, Dmitry
Open-source software libraries have a significant impact on the development of Audio Signal Processing and Music Information Retrieval (MIR) systems. Despite the abundance of such tools, there is a lack of an extensive and easy-to-use reference library for audio feature extraction on Web clients. In this article, we present Essentia.js, an open-source JavaScript (JS) library for audio and music analysis on both web clients and JS engines. Along with the Web Audio API, it can be used for both offline and real-time audio feature extraction on web browsers. Essentia.js is modular, lightweight, and easy-to-use, deploy, maintain, and integrate into the existing plethora of JS libraries and web technologies. It is powered by a WebAssembly back end cross-compiled from the Essentia C++ library, which facilitates a JS interface to a wide range of low-level and high-level audio features, including signal processing MIR algorithms as well as pre-trained TensorFlow.js machine learning models. It also provides a higher-level JS API and add-on MIR utility modules along with extensive documentation, usage examples, and tutorials. We benchmark the proposed library on two popular web browsers and the Node.js engine, and four devices, including mobile Android and iOS, comparing it to the native performance of Essentia and the Meyda JS library., The work on Essentia.js has been partially funded by the Ministry of Science and Innovation of the Spanish Government under the grant agreement PID2019-111403GB-I00 (Musical AI).
Recercat. Dipósit de la Recerca de Catalunya
Publicación Contribución de congreso (ConferenceOutput)

Analysis of musical dynamics in vocal performances

  • Narang, Jyoti
  • Miron, Marius
  • Lizarraga Seijas, Xavier
  • Serra, Xavier
Comunicació presentada a: The 15th International Symposium on Computer Music Multidisciplinary Research celebrat del 15 al 19 de novembre de 2021 de manera virtual., Dynamics are one of the fundamental tools of expressivity in a performance. While the usage of this tool is highly subjective, a systematic methodology to derive loudness markings based on a performance can be highly beneficial. With this goal in mind, this paper is a first step towards developing a methodology to automatically transcribe dynamic markings from vocal rock and pop performances. To this end, we make use of commercial recordings of some popular songs followed by source separation and compare them to the karaoke versions of the same songs. The dynamic variations in the original commercial recordings are found to be structurally very similar to the aligned karaoke/multi-track versions of the same tracks. We compare and show the differences between tracks using statistical analysis, with an eventual goal to use the transcribed markings as guiding tools, to help students adapt with a specific interpretation of a given piece of music. We perform a qualitative analysis of the proposed methodology with the teachers in terms of informativeness and accuracy., Part of this research is funded by the projects Musical AI (PID2019-111403GB-I00/AEI/10.13039/501100011033 funded by the Spanish Ministerio de Ciencia, Innovación y Universidades (MCIU) and the Agencia Estatal de Investigación (AEI)) and NextCore (RTC2019-007248-7 funded by the Spanish Ministerio de Ciencia, Innovación y Universidades (MCIU) and the Agencia Estatal de Investigación (AEI)).
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico antes de ser publicado, versión del autor (Preprint)

TapTamDrum: a dataset for dualized drum patterns

  • Haki, Behzad
  • Kotowski, Błażej
  • Lee, Cheuk Lun Isaac
  • Jordà Puig, Sergi
This work has been accepted at the 24th International Society for Music Information Retrieval Conference (ISMIR 2023), at Milan, Italy. October 5-9, 2023., Drummers spend extensive time practicing rudiments to develop technique, speed, coordination, and phrasing. These rudiments are often practiced on "silent" practice pads using only the hands. Additionally, many percussive instruments across cultures are played exclusively with the hands. Building on these concepts and inspired by Einstein's probably apocryphal quote, "Make everything as simple as possible, but not simpler," we hypothesize that a dual-voice reduction could serve as a natural and meaningful compressed representation of multi-voiced drum patterns. This representation would retain more information than its corresponding monotonic representation while maintaining relative simplicity for tasks such as rhythm analysis and generation. To validate this potential representation, we investigate whether experienced drummers can consistently represent and reproduce the rhythmic essence of a given drum pattern using only their two hands. We present TapTamDrum: a novel dataset of repeated dualizations from four experienced drummers, along with preliminary analysis and tools for further exploration of the data., This research was partly funded by the Ministry of Science and Innovation of the Spanish Government. Agencia Estatal de Investigación (AEI). (Reference: PID2019- 111403GB-I00).
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Repertoire-specific vocal pitch data generation for improved melodic analysis of carnatic music

  • Plaja-Roglans, Genís
  • Nutall, Thomas
  • Pearson, Lara
  • Serra, Xavier
  • Miron, Marius
Deep Learning methods achieve state-of-the-art in many tasks, including vocal pitch extraction. However, these methods rely on the availability of pitch track annotations without errors, which are scarce and expensive to obtain for Carnatic Music. Here we identify the tradition-related challenges and propose tailored solutions to generate a novel, large, and open dataset, the Saraga-Carnatic-Melody-Synth (SCMS), comprising audio mixtures and time-aligned vocal pitch annotations. Through a cross-cultural evaluation leveraging this novel dataset, we show improvements in the performance of Deep Learning vocal pitch extraction methods on Indian Art Music recordings. Additional experiments show that the trained models outperform the currently used heuristic-based pitch extraction solutions for the computational melodic analysis of Carnatic Music and that this improvement leads to better results in the musicologically relevant task of repeated melodic pattern discovery when evaluated using expert annotations. The code and annotations are made available for reproducibility. The novel dataset and trained models are also integrated into the Python package compIAM1 which allows them to be used out-of-the-box., This work is funded by the Spanish Ministerio de Ciencia, Innovación y Universidades (MCIU) and the Agencia Estatal de Investigación (AEI) within the Musical AI Project – PID2019-111403GB-I00/AEI/10.13039/501100011033.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Contribución de congreso (ConferenceOutput)

Personalized musically induced emotions of not-so-popular Colombian music

  • Gómez Cañón, Juan Sebastián
  • Herrera Boyer, Perfecto, 1964-
  • Cano, Estefanía
  • Gómez Gutiérrez, Emilia, 1975-
Comunicació presentada al workshop Human Centered AI inclòs a: 35th Conference on Neural Information Processing Systems (NeurIPS 2021) celebrat el 13 de desembre de manera virtual., This work presents an initial proof of concept of how Music Emotion Recognition (MER) systems could be intentionally biased with respect to annotations of musically-induced emotions in a political context. In specific, we analyze traditional Colombian music containing politically-charged lyrics of two types: (1) vallenatos and social songs from the “left-wing” guerrilla Fuerzas Armadas Revolucionarias de Colombia (FARC) and (2) corridos from the “right-wing” paramilitaries Autodefensas Unidas de Colombia (AUC). We train personalized machine learning models to predict induced emotions for three users with diverse political views – we aim at identifying the songs that may induce negative emotions for a particular user, such as anger and fear. To this extent, a user’s emotion judgements could be interpreted as problematizing data – subjective emotional judgments could in turn be used to influence the user in a human-centered machine learning environment. In short, highly desired “emotion regulation” applications could potentially deviate to “emotion manipulation” – the recent discredit of emotion recognition technologies might transcend ethical issues of diversity and inclusion., The research work conducted at the Universitat Pompeu Fabra is partially supported by the Eu- ropean Commission under the TROMPA project (H2020 770376) and the Project Musical AI - PID2019-111403GB-I00/AEI/10.13039/501100011033 funded by the Spanish Ministerio de Ciencia, Innovación y Universidades (MCIU) and the Agencia Estatal de Investigación (AEI).
Recercat. Dipósit de la Recerca de Catalunya
Publicación Contribución de congreso (ConferenceOutput)

MusAV: a dataset of relative arousal-valence annotations for validation of audio models

  • Bogdanov, Dmitry
  • Lizarraga Seijas, Xavier
  • Alonso-Jiménez, Pablo
  • Serra, Xavier
Comunicació presentada a 23nd International Society for Music Information Retrieval Conference (ISMIR 2022), celebrat del 4 al 8 de desembre de 2022 a Bangalore, Índia., We present MusAV, a new public benchmark dataset for comparative validation of arousal and valence (AV) regression models for audio-based music emotion recognition. To gather the ground truth, we rely on relative judgments instead of absolute values to simplify the manual annotation process and improve its consistency. We build MusAV by gathering comparative annotations of arousal and valence on pairs of tracks, using track audio previews and metadata from the Spotify API. The resulting dataset contains 2,092 track previews covering 1,404 genres, with pairwise relative AV judgments by 20 annotators and various subsets of the ground truth based on different levels of annotation agreement. We demonstrate the use of the dataset in an example study evaluating nine models for AV regression that we train based on state-of-the-art audio embeddings and three existing datasets of absolute AV annotations. The results on MusAV offer a view of the performance of the models complementary to the metrics obtained during training and provide insights into the impact of the considered datasets and embeddings on the generalization abilities of the models., This research was carried out under the project Musical AI - PID2019-111403GB-I00/AEI/10.13039/501100011033, funded by the Spanish Ministerio de Ciencia e Innovación and the Agencia Estatal de Investigación. We also thank Juan Sebastián Gómez Cañón for his suggestions and all participating annotators.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Contribución de congreso (ConferenceOutput)

Diversity in the music listening experience: insights from focus group interviews

  • Porcaro, Lorenzo
  • Gómez Gutiérrez, Emilia, 1975-
  • Castillo, Carlos
Comunicació presentada a: CHIIR '22: ACM SIGIR Conference on Human Information Interaction and Retrieval, celebrat del 14 al 18 de març de 2022 a Regensburg, Alemanya., Music listening in today’s digital spaces is highly characterized by the availability of huge music catalogues, accessible by people all over the world. In this scenario, recommender systems are designed to guide listeners in finding tracks and artists that best fit their requests, having therefore the power to influence the diversity of the music they listen to. Albeit several works have proposed new techniques for developing diversity-aware recommendations, little is known about how people perceive diversity while interacting with music recommendations. In this study, we interview several listeners about the role that diversity plays in their listening experience, trying to get a better understanding of how they interact with music recommendations. We recruit the listeners among the participants of a previous quantitative study, where they were confronted with the notion of diversity when asked to identify, from a series of electronic music lists, the most diverse ones according to their beliefs. As a follow-up, in this qualitative study we carry out semi-structured interviews to understand how listeners may assess the diversity of a music list and to investigate their experiences with music recommendation diversity. We report here our main findings on 1) what can influence the diversity assessment of tracks and artists’ music lists, and 2) which factors can characterize listeners’ interaction with music recommendation diversity., This work is partially supported by the European Commission under the TROMPA project (H2020 - grant agreement No. 770376). It is also part of the project Musical AI - PID2019-111403GB-I00/AEI/10.13039/501100011033 funded by the Spanish Ministerio de Ciencia, Innovación y Universidades (MCIU) and the Agencia Estatal de Investigación (AEI). This work is also partially supported by the HUMAINT programme (Human Behaviour and Machine Intelligence), Joint Research Centre, European Commission. The project leading to these results received funding from "la Caixa" Foundation (ID 100010434), under the agreement LCF/PR/PR16/51110009.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico antes de ser publicado, versión del autor (Preprint)

DiffVel: note-level MIDI velocity estimation for piano performance by a double conditioned diffusion model

  • Kim, Hyon
  • Serra, Xavier
This work has been accepted at the CMMR2023, the 16th International Symposium on Computer Music Multidisciplinary Research, at Tokyo, Japan. November 13-17, 2023., In any piano performance, expressiveness is paramount for effectively conveying the intent of the performer, and one of the most significant aspects of expressiveness is the loudness at the individual key or note level. However, accurately detecting note-level loudness poses a considerable technical challenge due to the polyphonic nature of piano performances, wherein multiple notes are played simultaneously, as well as the similarity of harmonic elements. MIDI velocity is crucial for indicating loudness in piano notes. This study conducted experiments for estimating a note-level MIDI velocity expanding the DiffRoll model: the Diffusion Model for piano performance transcription. By adopting double conditioning—audio and score information—and implementing noise removal as a post-processing, our findings highlight the model’s potential in estimating note level MIDI velocity., This research was carried out under the project Musical AI - PID2019- 111403GBI00/AEI/10.13039/501100011033, funded by the Spanish Ministerio de Ciencia e Innovación and the Agencia Estatal de Investigación.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Contribución de congreso (ConferenceOutput)

The Matrix profile for motif discovery in audio - an example application in Carnatic music

  • Nuttall, Thomas
  • Plaja-Roglans, Genís
  • Pearson, Lara
  • Serra, Xavier
Comunicació presentada a: The 15th International Symposium on Computer Music Multidisciplinary Research celebrat del 15 al 19 de novembre de 2021 de manera virtual., We present here a pipeline for the automated discovery of repeated motifs in audio. Our approach relies on state-of-the-art source separation, pre- dominant pitch extraction and time series motif detection via the matrix profile. Owing to the appropriateness of this approach for the task of motif recognition in the Carnatic musical style of South India, and with access to the recently re- leased Saraga Dataset of Indian Art Music, we provide an example application on a recording of a performance in the Carnatic r ̄aga, R ̄itigaul.a, finding 56 dis- tinct patterns of varying lengths that occur at least 3 times in the recording. The authors include a discussion of the potential musicological significance of this motif finding approach in relation to the particular tradition and beyond., This research was funded by the MUSICAL AI project (PID2019- 111403GB-I00) granted by the Ministry of Science and Innovation of the Spanish Gov- ernment. We also thank Rafael Caro Repetto for his continued guidance and input.
Recercat. Dipósit de la Recerca de Catalunya
Publicación

Simulating piano performance mistakes for music learning

  • Morsi, Alia
  • Zhang, Huan
  • Maezawa, Akira
  • Dixon, Simon
  • Serra, Xavier
This work has been accepted at 21st Sound and Music Computing Conference SMC 2024, at Porto, Portugal. July 4-6, 2024., The development of machine-learning based technologies to support music instrument learning needs large-scale datasets that capture the different stages of learning in a manner that is both realistic and computation-friendly. We are interested in modeling the mistakes of beginnerintermediate piano performances in practice or work-inprogress settings. In the absence of large-scale data representing our target case, our approach is to start by understanding such mistakes from real data and then provide a methodology for their simulation, thus creating synthetic data to support the training of performance assessment models. The main goals of this paper are: a) to propose a taxonomy of performance mistakes, specifically apt for simulating or reproducing/recreating them on mistake-free MIDI performances, and b) to provide a pipeline for creating synthetic datasets based on the former. We incorporate prior research in related contexts to facilitate the understanding of common mistake behaviours. Then, we design a hierarchical mistake taxonomy to categorize two real-world datasets capturing relevant piano performance contexts. Finally, we discuss our approach with 3 music teachers through a listening test and subsequent discussions., This research was partially supported by the project Musical AI-PID2019-111403GB-I00/AEI/10.13039/501100011033, funded by the Spanish Ministerio de Ciencia e Innovación and the Agencia Estatal de Investigación, and the UKRI Centre for Doctoral Training in Artificial Intelligence and Music (grant number EP/S022694/1).
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico antes de ser publicado, versión del autor (Preprint)

Note level midi velocity estimation for piano performance

  • Kim, Hyon
  • Miron, Marius
  • Serra, Xavier
Piano is one of the most popular music instruments. During the piano performance, loudness is an important factor for expressiveness, alongside tempo, changes in dynamics play with expectation, convey various emotions, and render expressiveness. Due to the polyphonic characteristics and with the goal of better analysing the expressiveness of performance of piano with multiple notes playing simultaneously, it is more useful to find loudness for each note than looking at accumulated loudness for a single time frame. Most of the research in this topic uses Non-negative Matrix Factorization (NMF) techniques to find note level loudness. In contrast, we propose to use Deep Neural Networks (DNNs) conditioned with score information to estimate the loudness based on MIDI velocity for each note performed by piano players. To our best knowledge, this is a novel research for note level MIDI velocity estimation by a DNN model in end to end fashion having FiLM conditioning., This research was carried out under the project Musical AI - PID2019- 111403GB-I00/AEI/10.13039/501100011033, funded by the Spanish Ministerio de Ciencia e Innovación and the Agencia Estatal de Investigación.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Contribución de congreso (ConferenceOutput)

PYWDF: an open source library for prototyping and simulating wave filter circuits in Python

  • Anthon, Gustav
  • Lizarraga Seijas, Xavier
  • Font Corbera, Frederic
Comunicació presentada a 26th International Conference on Digital Audio Effects (DAFx-23), celebrada del 4 al 7 desetembre de 2023 a Copenhaguen, Dinamarca., This paper introduces a new open-source Python library for the modeling and simulation of wave digital filter (WDF) circuits. The library, called pwydf, allows users to easily create and analyze WDF circuit models in a high-level, object-oriented manner. The library includes a variety of built-in components, such as voltage sources, capacitors, diodes etc., as well as the ability to create custom components and circuits. Additionally, pywdf includes a variety of analysis tools, such as frequency response and transient analysis, to aid in the design and optimization of WDF circuits. We demonstrate the library’s efficacy in replicating the nonlinear behavior of an analog diode clipper circuit, and in creating an allpass filter that cannot be realized in the analog world. The library is well-documented and includes several examples to help users get started. Overall, pywdf is a powerful tool for anyone working with WDF circuits, and we hope it can be of great use to researchers and engineers in the field., This research was carried out under the project Musical AI -PID2019- 111403GB-I00/AEI/10.13039/501100011033, funded by the Spanish Ministerio de Ciencia e Innovación and the Agencia Estatal de Investigación.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

TROMPA-MER: an open dataset for personalized music emotion recognition

  • Gómez Cañón, Juan Sebastián
  • Gutiérrez Páez, Nicolás Felipe
  • Porcaro, Lorenzo
  • Porter, Alastair
  • Cano, Estefanía
  • Herrera Boyer, Perfecto, 1964-
  • Gkiokas, Aggelos
  • Santos Rodríguez, Patrícia
  • Hernández Leo, Davinia
  • Karreman, Casper
  • Gómez Gutiérrez, Emilia, 1975-
We present a platform and a dataset to help research on Music Emotion Recognition (MER). We developed the Music Enthusiasts platform aiming to improve the gathering and analysis of the so-called “ground truth” needed as input to MER systems. Firstly, our platform involves engaging participants using citizen science strategies and generate music emotion annotations – the platform presents didactic information and musical recommendations as incentivization, and collects data regarding demographics, mood, and language from each participant. Participants annotated each music excerpt with single free-text emotion words (in native language), distinct forced-choice emotion categories, preference, and familiarity. Additionally, participants stated the reasons for each annotation – including those distinctive of emotion perception and emotion induction. Secondly, our dataset was created for personalized MER and contains information from 181 participants, 4721 annotations, and 1161 music excerpts. To showcase the use of the dataset, we present a methodology for personalization of MER models based on active learning. The experiments show evidence that using the judgment of the crowd as prior knowledge for active learning allows for more effective personalization of MER systems for this particular dataset. Our dataset is publicly available and we invite researchers to use it for testing MER systems., This work was funded by the European Commission under the TROMPA project (H2020 770376) and the Project Musical AI - PID2019-111403GB-I00/AEI/10.13039/501100011033 funded by the Spanish Ministerio de Ciencia, Innovación y Universidades (MCIU) and the Agencia Estatal de Investigación (AEI).
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico antes de ser publicado, versión del autor (Preprint)

High-resolution violin transcription using weak labels

  • Tamer, Nazif Can
  • Özer, Yigitcan
  • Müller, Meinard
  • Serra, Xavier
This work has been accepted at the 24th International Society for Music Information Retrieval Conference (ISMIR 2023), at Milan, Italy. October 5-9, 2023., A descriptive transcription of a violin performance requires detecting not only the notes but also the fine-grained pitch variations, such as vibrato. Most existing deep learning methods for music transcription do not capture these variations and often need frame-level annotations, which are scarce for the violin. In this paper, we propose a novel method for high-resolution violin transcription that can leverage piece-level weak labels for training. Our conformer-based model works on the raw audio waveform and transcribes violin notes and their corresponding pitch deviations with 5.8 ms frame resolution and 10-cent frequency resolution. We demonstrate that our method (1) outperforms generic systems in the proxy tasks of violin transcription and pitch estimation, and (2) can automatically generate new training labels by aligning its feature representations with unseen scores. We share our model along with 34 hours of score-aligned solo violin performance dataset, notably including the 24 Paganini Caprices., This research is funded by the project Musical AI- PID2019-111403GB-I00/AEI/10.13039/501100011033 funded by the Spanish Ministerio de Ciencia, Innovación y Universidades (MCIU) and the Agencia Estatal de Investigación (AEI), and by the German Research Foundation(DFG MU 2686/10-2).
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

The creative drummer: an eeg-based pilot study on the correlates of emotions and creative drum playing

  • Ramírez, Rafael, 1966-
  • Reija, Xavier
It is reasonable to assume that emotional processes are involved in creative tasks and the generation of creative ideas. In this pilot study, we investigate the emotional correlates in professional drummers during different degrees of creative music playing. Ten participants performed three tasks: repetitive rhythmic drum playing, pattern-based improvisation, and attention-intensive free improvisation, while their EEG activity was recorded. Arousal and valence levels were estimated from the EEG data at baseline and for the three tasks. Results show significantly increased levels of valence (i.e., increased prefrontal right alpha power compared to prefrontal left alpha power) during pattern-based and free improvisation relative to baseline, and significantly increased levels of valence during free improvisation relative to pattern-based improvisation. These results seem to indicate that positive emotion (characterized as increased valence) is associated with the creation of original ideas in drum playing and that the freer the creative process, the greater the positive effect. The implication of these results may be of particular relevance in the fields of music-based therapeutic interventions and music pedagogy., This work was partly sponsored by the European Union Horizon 2020 research and innovation program under grant agreement No. 688269 (TELMI project) and by the Musical AI project—(PID2019-111403GB-I00/AEI/10.13039/501100011033).
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Personalisation and profiling using algorithms and not-so-popular Colombian music: goal-directed mechanisms in music emotion recognition

  • Gómez-Cañon, Juan Sebastián
  • Lennie, Thomas Magnus
  • Eerola, Tuomas
  • Aragón, Pablo
  • Cano, Estefanía
  • Herrera, Perfecto
  • Gómez, Emilia
This work investigates how personalised Music Emotion Recognition (MER) systems may lead to sensitive profiling when applied to musically induced emotions in politically charged contexts. We focus on traditional Colombian music with explicit political content, including (1) vallenatos and social songs aligned with the left-wing guerrilla Fuerzas Armadas Revolucionarias de Colombia (FARC), and (2) corridos linked to sympathisers of the right-wing paramilitary group Autodefensas Unidas de Colombia (AUC). Using data from 49 participants with diverse political leanings, we train personalised machine learning models to predict induced emotional responses -particularly negative emotions. Our findings reveal that political identity plays a significant role in shaping emotional experiences of music with explicit political content, and that emotion recognition models can capture this variation to a certain extent. These results raise critical concerns about the potential misuse of emotion recognition technologies. What is often framed as a tool for wellbeing and emotional regulation could, in politically sensitive contexts, be repurposed for user profiling. This work highlights the ethical risks of deploying AI-driven emotion analysis without safeguards, particularly among populations that are politically or socially vulnerable. We argue that subjective emotional responses may constitute sensitive personal data, and that failing to account for their sociopolitical context could amplify harm and exclusion., JSGC's contribution was partially supported by the European Commission through the TROMPA project (H2020 770376) and the project Musical AI (PID2019-111403GB-I00/AEI/10.13039/501100011033), funded by the Spanish Ministerio de Ciencia, Innovación y Universidades (MCIU) and the Agencia Estatal de Investigación (AEI). TML was supported by the Arts and Humanities Research Council (AHRC) through a Northern Bridge Doctoral Training Partnership scholarship (grant code: AH/L503927/1).
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico antes de ser publicado, versión del autor (Preprint)

Carnatic singing voice separation using cold diffusion on training data with bleeding

  • Plaja-Roglans, Genís
  • Miron, Marius
  • Shankar, Adithi
  • Serra, Xavier
This work has been accepted at the 24th International Society for Music Information Retrieval Conference (ISMIR 2023), at Milan, Italy. October 5-9, 2023., Supervised music source separation systems using deep learning are trained by minimizing a loss function between pairs of predicted separations and ground-truth isolated sources. However, open datasets comprising isolated sources are few, small, and restricted to a few music styles. At the same time, multi-track datasets with source bleeding are usually found larger in size, and are easier to compile. In this work, we address the task of singing voice separation when the ground-truth signals have bleeding and only the target vocals and the corresponding mixture are available. We train a cold diffusion model on the frequency domain to iteratively transform a mixture into the corresponding vocals with bleeding. Next, we build the final separation masks by clustering spectrogram bins according to their evolution along the transformation steps. We test our approach on a Carnatic music scenario for which solely datasets with bleeding exist, while current research on this repertoire commonly uses source separation models trained solely with Western commercial music. Our evaluation on a Carnatic test set shows that our system improves Spleeter on interference removal and it is competitive in terms of signal distortion. Code is open sourced., This work was carried out under the projects Musical AI - PID2019-111403GB-I00/AEI/10.13039/501100011033 funded by the Spanish Ministerio de Ciencia, Innovación y Universidades (MCIU) and the Agencia Estatal de Investigación (AEI).
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico antes de ser publicado, versión del autor (Preprint)

TAPE: An End-to-End Timbre-Aware Pitch Estimator

  • Tamer, Nazif C
  • Özer, Yigitcan
  • Müller, Meinard
  • Serra, Xavier
This work has been accepted at the 2023 IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP 2023), at Rhodes, Greece. June 4-10, 2023., Pitch estimation of a target musical source within a multi-source polyphonic signal is of great interest for music performance analysis. One possible approach for extracting the pitch of a target source is to first perform source separation and then estimate the pitch of the separated track. However, as we will show, this typically leads to poor results. As an alternative to this approach, we introduce a timbre-aware pitch estimator (TAPE), which estimates the pitch of a target source in an end-to-end manner without the need for an explicit source separation step. Opposed to existing approaches that assume the predominance of a lead voice, our approach builds upon other cues that only rely on the timbral characteristics. Our results on real violin-piano duets show that, without any pre-processing step, TAPE outperforms the sequential procedure of source separation and pitch estimation under many settings, even if the target source is not predominant., This research is funded by the project Musical AI - PID2019-111403GB-I00/AEI/10.13039/501100011033 funded by the Spanish Ministerio de Ciencia, Innovacion y Universidades (MCIU) and the Agencia Estatal de Investigacion (AEI), and by the German Research Foundation (DFG MU 2686/10-2).
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico antes de ser publicado, versión del autor (Preprint)

MUSAV: a dataset of relative arousal-valence annotations for validation of audio models

  • Bogdanov, Dmitry
  • Lizarraga Seijas, Xavier
  • Alonso-Jiménez, Pablo
  • Serra, Xavier
We present MusAV, a new public benchmark dataset for comparative validation of arousal and valence (AV) regression models for audio-based music emotion recognition. To gather the ground truth, we rely on relative judgments instead of absolute values to simplify the manual annotation process and improve its consistency. We build MusAV by gathering comparative annotations of arousal and valence on pairs of tracks, using track audio previews and metadata from the Spotify API. The resulting dataset contains 2,092 track previews covering 1,404 genres, with pairwise relative AV judgments by 20 annotators and various subsets of the ground truth based on different levels of annotation agreement. We demonstrate the use of the dataset in an example study evaluating nine models for AV regression that we train based on state-of-the-art audio embeddings and three existing datasets of absolute AV annotations. The results on MusAV offer a view of the performance of the models complementary to the metrics obtained during training and provide insights into the impact of the considered datasets and embeddings on the generalization abilities of the models., This research was carried out under the project Musical AI - PID2019-111403GB-I00/AEI/10.13039/501100011033, funded by the Spanish Ministerio de Ciencia e Innovación and the Agencia Estatal de Investigación.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico antes de ser publicado, versión del autor (Preprint)

Predicting performance difficulty from piano sheet music images

  • Ramoneda, Pedro
  • Valero-Mas, Jose J.
  • Jeong, Dasaem
  • Serra, Xavier
This work has been accepted at the 24th International Society for Music Information Retrieval Conference (ISMIR 2023), at Milan, Italy. October 5-9, 2023., Estimating the performance difficulty of a musical score is crucial in music education for adequately designing the learning curriculum of the students. Although the Music Information Retrieval community has recently shown interest in this task, existing approaches mainly use machinereadable scores, leaving the broader case of sheet music images unaddressed. Based on previous works involving sheet music images, we use a mid-level representation, bootleg score, describing notehead positions relative to staff lines coupled with a transformer model. This architecture is adapted to our task by introducing an encoding scheme that reduces the encoded sequence length to oneeighth of the original size. In terms of evaluation, we consider five datasets—more than 7500 scores with up to 9 difficulty levels—, two of them particularly compiled for this work. The results obtained when pretraining the scheme on the IMSLP corpus and fine-tuning it on the considered datasets prove the proposal’s validity, achieving the bestperforming model with a balanced accuracy of 40.34% and a mean square error of 1.33. Finally, we provide access to our code, data, and models for transparency and reproducibility., This work is funded by the Spanish Ministerio de Ciencia, Innovación y Universidades (MCIU) and the Agencia Estatal de Investigación (AEI) within the Musical AI Project – PID2019-111403GBI00/AEI/10.13039/501100011033 and the Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Korea Government (MSIT) (NRF-2022R1F1A1074566).
Recercat. Dipósit de la Recerca de Catalunya
Publicación Contribución de congreso (ConferenceOutput)

Violin etudes: a comprehensive dataset for f0 estimation and performance analysis

  • Can Tamer, Nazif
  • Ramoneda, Pedro
  • Serra, Xavier
Comunicació presentada a 23nd International Society for Music Information Retrieval Conference (ISMIR 2022), celebrat del 4 al 8 de desembre de 2022 a Bangalore, Índia., Violin performance analysis requires accurate and robust f0 estimates to give feedback on the playing accuracy. Despite the recent advancements in data-driven f0 estimators, their application to performance analysis remains a challenge due to style-specific and dataset-induced biases. In this paper, we address this problem by introducing Violin Etudes, a 27.8-hours violin performance dataset constructed with domain knowledge in instrument pedagogy and a novel automatic f0-labeling paradigm. Experimental results on unseen datasets show that the CREPE f0 estimator trained on Violin Etudes outperforms the widely-used pre-trained version trained on multiple manually-labeled datasets. Further preliminary findings suggest that (i) existing data-driven f0 estimators may overfit to equal temperament, and (ii) iterative re-labeling regularized by our novel Constrained Harmonic Resynthesis method can simultaneously enhance datasets and f0 estimators. Our dataset curation methodology is easily scalable to other instruments owing to the quantity of pedagogical data online. It also supports a range of MIR research directions thanks to the performance difficulty labels from educational institutions., We would like to thank Esteban Maestre for his valuable insights on the violin resonance structure. This research was carried out under the project Musical AI - PID2019- 111403GB-I00/AEI/10.13039/501100011033, funded by the Spanish Ministerio de Ciencia e Innovación and the Agencia Estatal de Investigación.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Contribución de congreso (ConferenceOutput)

Music representation learning based on editorial metadata from Discogs

  • Alonso-Jiménez, Pablo
  • Serra, Xavier
  • Bogdanov, Dmitry
Comunicació presentada a 23nd International Society for Music Information Retrieval Conference (ISMIR 2022), celebrat del 4 al 8 de desembre de 2022 a Bangalore, Índia., This paper revisits the idea of music representation learning supervised by editorial metadata, contributing to the state of the art in two ways. First, we exploit the public editorial metadata available on Discogs, an extensive community-maintained music database containing information about artists, releases, and record labels. Second, we use a contrastive learning setup based on COLA, different from previous systems based on triplet loss. We train models targeting several associations derived from the metadata and experiment with stacked combinations of learned representations, evaluating them on standard music classification tasks. Additionally, we consider learning all the associations jointly in a multi-task setup. We show that it is possible to improve the performance of current self-supervised models by using inexpensive metadata commonly available in music collections, producing representations comparable to those learned on classification setups. We find that the resulting representations based on editorial metadata outperform a system trained with music style tags available in the same large-scale dataset, which motivates further research using this type of supervision. Additionally, we give insights on how to preprocess Discogs metadata to build training objectives and provide public pre-trained models., This research was carried out under the project Musical AI - PID2019-111403GB-I00/AEI/10.13039/501100011033, funded by the Spanish Ministerio de Ciencia e Innovación and the Agencia Estatal de Investigación.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico antes de ser publicado, versión del autor (Preprint)

Efficient supervised training of audio transformers for music representation learning

  • Alonso Jiménez, Pablo
  • Serra, Xavier
  • Bogdanov, Dmitry
This work has been accepted at the 24th International Society for Music Information Retrieval Conference (ISMIR 2023), at Milan, Italy. October 5-9, 2023., In this work, we address music representation learning using convolution-free transformers. We build on top of existing spectrogram-based audio transformers such as AST and train our models on a supervised task using patchout training similar to PaSST. In contrast to previous works, we study how specific design decisions affect downstream music tagging tasks instead of focusing on the training task. We assess the impact of initializing the models with different pre-trained weights, using various input audio segment lengths, using learned representations from different blocks and tokens of the transformer for downstream tasks, and applying patchout at inference to speed up feature extraction. We find that 1) initializing the model from ImageNet or AudioSet weights and using longer input segments are beneficial both for the training and downstream tasks, 2) the best representations for the considered downstream tasks are located in the middle blocks of the transformer, and 3) using patchout at inference allows faster processing than our convolutional baselines while maintaining superior performance. The resulting models, MAEST, 1 are publicly available and obtain the best performance among open models in music tagging tasks., This work has been supported by the Musical AI project - PID2019-111403GB-I00/AEI/10.13039/501100011033, funded by the Spanish Ministerio de Ciencia e Innovación and the Agencia Estatal de Investigación.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Music-enhanced emotion identification of facial emotions in autistic spectrum disorder children: a pilot EEG study

  • Ramírez, Rafael, 1966-
  • Matamoros, Elisabet
  • Hernández-Leo, Davinia
  • Mirabel, Júlia
  • Sánchez, Elisabet
  • Escude, Núria
The Autistic Spectrum Disorder (ASD) is characterized by a difficulty in expressing and interpreting others’ emotions. In particular, people with ASD have difficulties when interpreting emotions encoded in facial expressions. In the past, music interventions have been shown to improve autistic individuals’ emotional and social skills. The present study describes a pilot study to explore the usefulness of music as a tool for improving autistic children’s emotion recognition in facial expressions. Twenty-five children (mean age = 8.8 y, SD = 1.24) with high-functioning ASD and normal hearing participated in the study consisting of four weekly sessions of 15 min each. Twenty-five participants were randomly divided into an experimental group (N = 14) and a control group (N = 11). During each session, participants in the experimental group were exposed to images of facial expressions for four emotions (happy, sad, angry, and fear). Images were shown in three conditions, with the second condition consisting of music of congruent emotion with the shown images. Participants in the control group were shown only images in all three conditions. For six participants in each group, EEG data were acquired during the sessions, and instantaneous emotional responses (arousal and valence values) were extracted from the EEG data. Inter- and intra-session emotion identification improvement was measured in terms of verbal response accuracy, and EEG response differences were analyzed. A comparison of the verbal responses of the experimental group pre- and post-intervention showed a significant (p = 0.001) average improvement in emotion identification accuracy responses of 26% (SD = 3.4). Furthermore, emotional responses of the experimental group at the end of the study showed a higher correlation with the emotional stimuli being presented, compared with their emotional responses at the beginning of the study. No similar verbal responses improvement or EEG-stimuli correlation was found in the control group. These results seem to indicate that music can be used to improve both emotion identification in facial expressions and emotion induction through facial stimuli in children with high-functioning ASD., This work was partly sponsored by the European Union Horizon 2020 research and innovation program under grant agreement No. 688269 (TELMI project) and by the Spanish Ministerio de Ciencia, Innovación y Universidades (MCIU) and the Agencia Estatal de Investigación (AEI) project Musical AI—PID2019-111403GB-I00/AEI/10.13039/501100011033.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Contribución de congreso (ConferenceOutput)

An overview of automatic piano performance assessment within the music education context

  • Kim, Hyon
  • Ramoneda, Pedro
  • Miron, Marius
  • Serra, Xavier
Comunicació presentada a la 14th International Conference on Computer Supported Education, celebrada del 22 a 24 d'abril de 2021 de manera virtual., Piano is one of the most popular instruments among music learners. Technologies to evaluate piano performances have been researched and developed in recent years rapidly, including data driven methods using machine learning. Despite the demand from people and speed of the development, there are still gaps between the methods and the pedagogical setup for real use case scenarios due to lack of accuracy of methods, insufficient amount of training data or the biases in training machine learning models, ignoring actual use case of the technology and such. In this paper, we first propose a feedback approach in piano performance education and review methods for Automated Piano Performance Assessment (APPA). After that, we discuss about gaps between a feedback approach and current methods, emphasizing their music education application. As a future work we propose a potential approach to overcome the gaps., This project is supported by Project Musical AI funded by the Spanish Ministerio de Ciencia, Innovacion y Universidades (MCIU) and the Agencia Estatal de Investigacion (AEI).
Recercat. Dipósit de la Recerca de Catalunya
Publicación Contribución de congreso (ConferenceOutput)

A diffusion-inspired training strategy for singing voice extraction in the waveform domain

  • Plaja-Roglans, Genís
  • Miron, Marius
  • Serra, Xavier
Comunicació presentada a 23nd International Society for Music Information Retrieval Conference (ISMIR 2022), celebrat del 4 al 8 de desembre de 2022 a Bangalore, Índia., Notable progress in music source separation has been achieved using multi-branch networks that operate on both temporal and spectral domains. However, such networks tend to be complex and heavy-weighted. In this work, we tackle the task of singing voice extraction from polyphonic music signals in an end-to-end manner using an approach inspired by the training and sampling process of denoising diffusion models. We perform unconditional signal modelling to gradually convert an input mixture signal to the corresponding singing voice or accompaniment. We use fewer parameters than the state-of-the-art models while operating on the waveform domain, bypassing the phase estimation problem. More concisely, we train a non-causal WaveNet using a diffusion-inspired strategy while improving the said network for singing voice extraction and obtaining performance comparable to the end-to-end stateof-the-art on MUSDB18. We further report results on a non-MUSDB-overlapping version of MedleyDB and the multi-track audio of Saraga Carnatic showing good generalization, and run perceptual tests of our approach. Code, models, and audio examples are made available., This work was carried out under the projects Musical AI - PID2019-111403GB-I00/AEI/10.13039/501100011033 and NextCore - RTC2019-007248-7 funded by the Spanish Ministerio de Ciencia, Innovación y Universidades (MCIU) and the Agencia Estatal de Investigación (AEI).
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico antes de ser publicado, versión del autor (Preprint)

Violin etudes: a comprehensive dataset for f0 estimation and performance analysis

  • Tamer, Nazif C
  • Ramoneda, Pedro
  • Serra, Xavier
This work has been accepted at the 23rd International Society for Music Information Retrieval Conference (ISMIR 2022), at Bengaluru, India. December 4-8, 2022., Violin performance analysis requires accurate and robust f0 estimates to give feedback on the playing accuracy. Despite the recent advancements in data-driven f0 estimators, their application to performance analysis remains a challenge due to style-specific and dataset-induced biases. In this paper, we address this problem by introducing Violin Etudes, a 27.8-hours violin performance dataset constructed with domain knowledge in instrument pedagogy and a novel automatic f0-labeling paradigm. Experimental results on unseen datasets show that the CREPE f0 estimator trained on Violin Etudes outperforms the widely-used pre-trained version trained on multiple manually-labeled datasets. Further preliminary findings suggest that (i) existing data-driven f0 estimators may overfit to equal temperament, and (ii) iterative re-labeling regularized by our novel Constrained Harmonic Resynthesis method can simultaneously enhance datasets and f0 estimators. Our dataset curation methodology is easily scalable to other instruments owing to the quantity of pedagogical data online. It also supports a range of MIR research directions thanks to the performance difficulty labels from educational institutions., We would like to thank Esteban Maestre for his valuable insights on the violin resonance structure. This research was carried out under the project Musical AI - PID2019- 111403GB-I00/AEI/10.13039/501100011033, funded by the Spanish Ministerio de Ciencia e Innovación and the Agencia Estatal de Investigación.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Contribución de congreso (ConferenceOutput)

NeuralMidiFx: a wrapper template for deploying neural networks as VST3 plugins

  • Haki, Behzad
  • Lenz, Julian
  • Jordà Puig, Sergi
Comunicació presentada a AIMC 2023, celebrada del 30 d'agost al 1 de setembre de 2023 a Sussex, Anglaterra., Proper research, development and evaluation of AI-based generative systems of music that focus on performance or composition require active user-system interactions. To include a diverse group of users that can properly engage with a given system, researchers should provide easy access to their developed systems. Given that many users (i.e. musicians) are non-technical to the field of AI and the development frameworks involved, the researchers should aim to make their systems accessible within the environments commonly used in production/composition workflows (e.g. in the form of plugins hosted in digital audio workstations). Unfortunately, deploying generative systems in this manner is highly expensive. As such, researchers with limited resources are often unable to provide easy access to their works, and subsequently, are not able to properly evaluate and encourage active engagement with their systems. Facing these limitations, we have been working on a solution that allows for easy, effective and accessible deployment of generative systems. To this end, we propose a wrapper/template called NeuralMidiFx, which streamlines the deployment of neural network based symbolic music generation systems as VST3 plugins. The proposed wrapper is intended to allow researchers to develop plugins with ease while requiring minimal familiarity with plugin development., This research was partly funded by the Ministry of Science and Innovation of the Spanish Government. Agencia Estatal de Investigación (AEI). (Reference: PID2019-111403GB-I00)
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

A deep-learning based framework for source separation, analysis, and synthesis of choral ensembles

  • Chandna, Pritish
  • Cuesta, Helena
  • Petermann, Darius
  • Gómez Gutiérrez, Emilia, 1975-
Choral singing in the soprano, alto, tenor and bass (SATB) format is a widely practiced and studied art form with significant cultural importance. Despite the popularity of the choral setting, it has received little attention in the field of Music Information Retrieval. However, the recent publication of high-quality choral singing datasets as well as recent developments in deep learning based methodologies applied to the field of music and speech processing, have opened new avenues for research in this field. In this paper, we use some of the publicly available choral singing datasets to train and evaluate state-of-the-art source separation algorithms from the speech and music domains for the case of choral singing. Furthermore, we evaluate existing monophonic F0 estimators on the separated unison stems and propose an approximation of the perceived F0 of a unison signal. Additionally, we present a set of applications combining the proposed methodologies, including synthesizing a single singer voice from the unison, and transposing and remixing the separated stems into a synthetic multi-singer choral signal. We finally conduct a set of listening tests to perform a perceptual evaluation of the results we obtain with the proposed methodologies., This work is partially supported by the European Commission under the TROMPA project (H2020 770376), and the project Musical AI (PID 2019-111403GB-I00/AEI/10.13039/501100011033) funded by the Spanish Ministerio de Ciencia, Innovación y Universidades (MCIU) and the Agencia Estatal de Investigación (AEI).
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico antes de ser publicado, versión del autor (Preprint)

Music representation learning based on editorial metadata from discogs

  • Alonso-Jiménez, Pablo
  • Serra, Xavier
  • Bogdanov, Dmitry
This work has been accepted at the 23rd International Society for Music Information Retrieval Conference (ISMIR 2022), at Bengaluru, India. December 4-8, 2022., This paper revisits the idea of music representation learning supervised by editorial metadata, contributing to the state of the art in two ways. First, we exploit the public editorial metadata available on Discogs, an extensive community-maintained music database containing information about artists, releases, and record labels. Second, we use a contrastive learning setup based on COLA, different from previous systems based on triplet loss. We train models targeting several associations derived from the metadata and experiment with stacked combinations of learned representations, evaluating them on standard music classification tasks. Additionally, we consider learning all the associations jointly in a multi-task setup. We show that it is possible to improve the performance of current self-supervised models by using inexpensive metadata commonly available in music collections, producing representations comparable to those learned on classification setups. We find that the resulting representations based on editorial metadata outperform a system trained with music style tags available in the same large-scale dataset, which motivates further research using this type of supervision. Additionally, we give insights on how to preprocess Discogs metadata to build training objectives and provide public pre-trained models., This research was carried out under the project Musical AI - PID2019-111403GB-I00/AEI/10.13039/501100011033, funded by the Spanish Ministerio de Ciencia e Innovación and the Agencia Estatal de Investigación.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico antes de ser publicado, versión del autor (Preprint)

Essentia API: a web API for music audio analysis

  • Correya, Albin Andrew
  • Bogdanov, Dmitry
  • Alonso Jiménez, Pablo
  • Serra, Xavier
We present Essentia API, a web API to access a collection of state-of-the-art music audio analysis and description algorithms based on Essentia, an open-source library and machine learning (ML) models for audio and music analysis. We are developing it as part of a broader project in which we explore strategies for the commercial viability of technologies developed at Music Technology Group (MTG) following open science and open source practices, which involves finding licensing schemes and building custom solutions. Currently, the API supports music auto-tagging and classification algorithms (for genre, instrumentation, mood/emotion, danceability, approachability, and engagement), and algorithms for musical key, tempo, loudness, and many more. In the future, we envision expanding it with new machine learning models developed by the MTG and our collaborators to facilitate their access for a broader community of users.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Error monitoring and correction in violin performance: an EEG study

  • Blanco, Angel David
  • Costa-Faidella, Jordi
  • Pérez Carrillo, Alfonso Antonio, 1977-
  • Dalmazzo, David
  • Ramirez Melendez, Rafael
  • San Miguel Insúa, Iria
Previous research using electroencephalography (EEG) has investigated neural error-related music production processes in expert pianists reporting a frontal event-related negativity followed by a P300 event-related potential (ERP) after mistuned notes. However, piano playing does not rely on auditory feedback and corrective movements when dealing with accurate note tuning, thus offering an incomplete picture about error monitoring processes in players of many other musical instruments, including the human voice. We here designed a setup allowing us to collect the EEG of expert bow-string players while performing short melodies on the violin, (a fretless instrument) while also manipulating in real-time the produced pitch in a controlled manner. 15 expert bow-string players were asked to 1) listen to a reference melody of four notes; 2) play that melody with a violin (Active Condition); and 3) listen passively to the replay of their performance (Replayed Condition). Randomly, the auditory feedback of one of the four melody notes was manipulated in the Active Condition by lowering or lifting the pitch by half semitone. We found that mistuned notes, independently of whether they were manipulated externally or were produced by natural playing errors, elicited an f-ERN followed by a P300. We also found that the amplitude of the f-ERN was larger for faster corrective movements than for slower ones. In addition, between the f-ERN and the P300, we also identified two negative components-one in parietal areas and another in central-right regions-whose amplitudes were greater for slower corrective movements. We suggest the existence of two error-monitoring systems: a "fast" system mediated by the medial frontal cortex that quickly corrects errors when certainty is high and a "slow" system mediated by the posterior parietal cortex that accumulates evidence under uncertainty before triggering correction., This work was partly sponsored by the Musical AI project (PID2019-111403GB-I00) funded by MCIN/AEI/ 10.13039/501100011033 and by 'ERDF A way of making Europe'; by the Project PID2021-128790NB-I00, funded by MCIN/AEI/ 10.13039/501100011033 / FEDER UE; and by the European Union Horizon 2020 research and innovation program under grant agreement No. 688269 (TELMI project). It has been additionally supported by the Excellence Research Group 2017SGR-974 funded by the Secretaria d'Universitats i Recerca del Departament d'Empresa i Coneixement de la Generalitat de Catalunya and by the MDM-2017-0729-18-2M Maria de Maeztu Center of Excellence UBNeuro, funded by MCIN/AEI/ 10.13039/501100011033. ISM was supported by grant RYC-2013-12577, funded by MCIN/AEI/ 10.13039/501100011033 and by 'ESF Investing in your future'. Open access publication fees were partly covered by the University of Barcelona (Ajudes de la Universitat de Barcelona per a publicar en accés obert).
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Voice assignment in vocal quartets using deep learning models based on pitch salience

  • Cuesta, Helena
  • Gómez Gutiérrez, Emilia, 1975-
This paper deals with the automatic transcription of four-part, a cappella singing, audio performances. In particular, we exploit an existing, deep-learning based, multiple F0 estimation method and complement it with two neural network architectures for voice assignment (VA) in order to create a music transcription system that converts an input audio mixture into four pitch contours. To train our VA models, we create a novel synthetic dataset by collecting 5381 choral music scores from public-domain music archives, which we make publicly available for further research. We compare the performance of the proposed VA models on different types of input data, as well as to a hidden Markov model-based baseline system. In addition, we assess the generalization capabilities of these models on audio recordings with differing pitch distributions and vocal music styles. Our experiments show that the two proposed models, a CNN and a ConvLSTM, have very similar performance, and both of them outperform the baseline HMM-based system. We also observe a high confusion rate between the alto and tenor voice parts, which commonly have overlapping pitch ranges, while the bass voice has the highest scores in all evaluated scenarios., This work is partially supported by the European Commission under the TROMPA project (H2020 770376), the Spanish Ministry of Science and Innovation under the Musical AI project (PID2019-111403GB-I00), and by AGAUR (Generalitat de Catalunya) through an FI Predoctoral Grant (2018FI-B01015).
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Applications of biological and physiological signals in commercial video gaming and game research: a review

  • Hughes, Alayna
  • Jordà Puig, Sergi
Video gaming is now available as a fully immersive experience that creates responsive inputs and outputs concerning the user, and some experimental developers have integrated the use of the voice, brain, or muscles as input controls. The use of physiological signal equipment can provide valuable information regarding the emotion of a player or patient during gameplay. In this article, we discuss five of the most common biosignals that are used in gaming research, and their function and devices that may be used for measurement. We break down those individual signals and present examples of research studies that implement them. We also discuss the usage of biological signals within commercial gaming and conclude with some possible future directions for the use of biological signals in gaming and game research., This research has been partially supported by the project Musical AI - PID2019-111403GB-I00/AEI/10.13039/501100011033 funded by the Spanish Ministerio de Ciencia, Innovación y Universidades (MCIU) and the Agencia Estatal de Investigación (AEI).
Recercat. Dipósit de la Recerca de Catalunya
Publicación Contribución de congreso (ConferenceOutput)

Multi-type outer product-based fusion of respiratory sounds for detecting COVID-19

  • Mallol Ragolta, Adrià
  • Cuesta, Helena
  • Gómez Gutiérrez, Emilia, 1975-
  • Schuller, Björn
Comunicació presentada a Interspeech 2022, celebrat del 18 al 22 de setembre de 2022 a Inchon, Corea del Sud., This work presents an outer product-based approach to fuse the embedded representations learnt from the spectrograms of cough, breath, and speech samples for the automatic detection of COVID-19. To extract deep learnt representations from the spectrograms, we compare the performance of specific Convolutional Neural Networks (CNNs) trained from scratch and ResNet18- based CNNs fine-tuned for the task at hand. Furthermore, we investigate whether the patients’ sex and the use of contextual attention mechanisms are beneficial. Our experiments use the dataset released as part of the Second Diagnosing COVID-19 using Acoustics (DiCOVA) Challenge. The results suggest the suitability of fusing breath and speech information to detect COVID-19. An Area Under the Curve (AUC) of 84.06 % is obtained on the test partition when using specific CNNs trained from scratch with contextual attention mechanisms. When using ResNet18-based CNNs for feature extraction, the baseline model scores the highest performance with an AUC of 84.26 %., This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No. 826506 (sustAGE), and from the Spanish Ministry of Science and Innovation under the Musical AI project (PID2019-111403GB-I00).
Recercat. Dipósit de la Recerca de Catalunya
Publicación Contribución de congreso (ConferenceOutput)

In search of Sañcāras: tradition-informed repeated melodic pattern recognition in carnatic music

  • Nuttall, Thomas
  • Plaja-Roglans, Genís
  • Pearson, Lara
  • Serra, Xavier
Comunicació presentada a 23nd International Society for Music Information Retrieval Conference (ISMIR 2022), celebrat del 4 al 8 de desembre de 2022 a Bangalore, Índia., Carnatic Music is a South Indian art and devotional musical practice in which melodic patterns (motifs and phrases), known as sañcaras Å , play a crucial structural and expressive role. We demonstrate how the combination of transposition invariant features learnt by a Complex Autoencoder (CAE) and predominant pitch tracks extracted using a Frequency-Temporal Attention Network (FTANet) can be used to annotate and group regions of variable-length, repeated, melodic patterns in audio recordings of multiple Carnatic Music performances. These models are trained on novel, expert-curated datasets of hundreds of Carnatic audio recordings and the extraction process tailored to account for the unique characteristics of sañcaras Å in Carnatic Music. Experimental results show that the proposed method is able to identify 54% of all sañcaras Å annotated by a professional Carnatic vocalist. Code to reproduce and interact with these results is available online., This research work was carried out under the project Musical AI - PID2019- 111403GBI00/AEI/10.13039/501100011033, funded by the Spanish Ministerio de Ciencia e Innovación and the Agencia Estatal de Investigación.