Pasar al contenido principal

IA MULTIMODAL PARA EL PROCESAMIENTO DE AUDIO

PID2023-152250OB-I00

•
Nombre agencia financiadora Agencia Estatal de Investigación
Acrónimo agencia financiadora AEI
Programa Programa Estatal para Impulsar la Investigación Científico-Técnica y su Transferencia
Subprograma Subprograma Estatal de Generación de Conocimiento
Convocatoria Proyectos de I+D+i (Generación de Conocimiento y Retos Investigación)
Año convocatoria 2023
Plan Estatal 2021-2023 Plan Estatal de Investigación Científica y Técnica y de Innovación
Centro beneficiario UNIVERSITAT POMPEU FABRA CCT

Resultados relacionados

Resultados totales (Incluyendo duplicados): 20
Encontrada(s) 1 página(s)

Repositorio Digital de la UPF
Publicación Contribución de congreso (ConferenceOutput)

Leveraging Carnatic live recordings for singing voice separation using regression-guided latent diffusion

  • Plaja-Roglans, Genís
  • Serra, Xavier
  • Rocamora, Martín
Comunicació presentada a la 26th International Society for Music Information Retrieval Conference (ISMIR 2025), celebrada a Daejeon (Korea) del 21 a 25 de setembre de 2025., Diffusion models have demonstrated potential to separate individual sources from music mixtures in a generative fashion, enabling a new solution for this challenging problem. However, existing works require clean multi-stem data, which is scarce for several repertoires, consequently compromising generalization. We explore the potential of generative modeling to perform weakly-supervised singing voice separation for Carnatic Music, a music repertoire for which large quantities of multi-stem recordings with bleeding between sources have been collected from live performances. We pre-train a latent diffusion model to perform preliminary vocal separation conditioning on the corresponding mixture. Then, using a regressive model which is separately trained on a clean, smaller, and out-of-domain dataset, we estimate the level of bleeding in the preliminary separations and use that information to guide the diffusion model toward generating cleaner samples. The objective and perceptual evaluations show the potential of the proposed generative system for Carnatic vocal separation., This work is supported by IA y Música: Cátedra en Inteligencia Artificial y Música (TSI-100929-2023-1), funded by the Secretaría de Estado de Digitalización e Inteligencia Artificial, and the European Union-Next Generation EU, under the program Cátedras ENIA 2022 para la creación de cátedras universidad-empresa en IA, and IMPA: Multimodal AI for Audio Processing (PID2023-152250OB-I00), funded by the Ministry of Science, Innovation and Universities of the Spanish Government, the Agencia Estatal de Investigación (AEI) and co-financed by the European Union.
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Velocity2DMs: a contextual modeling approach to dynamics marking prediction in piano performance

  • Kim, Hyon
  • Benetos, Emmanouil
  • Serra, Xavier
Expressive dynamics in music performance are subjective and context-dependent, yet most symbolic models treat Dynamics Markings (DMs) as static with fixed MIDI velocities. This paper proposes a method for predicting DMs in piano performance by combining MusicXML score information with performance MIDI data through a novel tokenization scheme and an adapted RoBERTa-based Masked Language Model (MLM). Our approach focuses on contextual aggregated MIDI velocities and corresponding DMs, accounting for subjective interpretations of pianists. Note-level features are serialized and translated into a sequence of tokens to predict both constant (e.g., mp, ff) and non-constant DMs (e.g., crescendo, fp). Evaluation across three expert performance datasets shows that the model effectively learns dynamics transitions from contextual note blocks and generalizes beyond constant markings. This is the first study to model both constant and non-constant dynamics in a unified framework using contextual sequence learning. The results suggest promising applications for expressive music analysis, performance modeling, and computer-assisted music education., This work was supported by IMPA under Project PID2023-152250OB-I00 funded by MCIU/AEI/10.13039/501100011033/FEDER, UE.
Repositorio Digital de la UPF
Publicación Contribución de congreso (ConferenceOutput)

MusGO: a community-driven framework for assessing openness in music-generative AI

  • Batlle-Roca, Roser
  • Ibáñez-Martínez, Laura
  • Serra, Xavier
  • Gómez, Emilia
  • Rocamora, Martín
Comunicació presentada al 26th International Society for Music Information Retrieval Conference (ISMIR 2025), celebrada a Daejeon (Korea) del 21 al 25 de setembre del 2025, Since 2023, generative AI has rapidly advanced in the music domain. Despite significant technological advancements, music-generative models raise critical ethical challenges, including a lack of transparency and accountability, along with risks such as the replication of artists' works, which highlights the importance of fostering openness. With upcoming regulations such as the EU AI Act encouraging open models, many generative models are being released labelled as 'open'. However, the definition of an open model remains widely debated. In this article, we adapt a recently proposed evidence-based framework for assessing openness in LLMs to the music domain. Using feedback from a survey of 110 participants from the Music Information Retrieval (MIR) community, we refine the framework into MusGO (Music-Generative Open AI), which comprises 13 openness categories: 8 essential and 5 desirable. We evaluate 16 state-of-the-art generative models and provide an openness leaderboard that is fully open to public scrutiny and community contributions. Through this work, we aim to clarify the concept of openness in music-generative AI and promote its transparent and responsible development., This work has been supported by IA y Música: Cátedra en Inteligencia Artificial y Música (TSI-100929-2023-1), funded by the Secretaría de Estado de Digitalización e Inteligencia Artificial and the European Union-Next Generation EU, and IMPA: Multimodal AI for Audio Processing (PID2023-152250OB-I00), funded by the Ministry of Science, Innovation and Universities of the Spanish Government, the Agencia Estatal de Investigación (AEI) and cofinanced by the European Union. We thank our colleagues at the Music Technology Group at Universitat Pompeu Fabra for their thoughtful insights, constructive discussions and active engagement throughout the development of this work
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Tapping into rhythm: how polyphonic drum patterns map to monophonic contours and support pattern retrieval

  • Gómez-Marín, Daniel
  • Clark, Peter
  • Jordà Puig, Sergi
  • Herrera Boyer, Perfecto, 1964-
This study investigates how listeners reduce polyphonic drum patterns into simplified monophonic rhythm contours through tapping behavior. We examine how these contours relate to metrical structure, onset density, syncopation, and frequency-weighted onset density (FWOD), and whether these features can support rhythm retrieval tasks. Across three experiments, participants tapped rhythmic contours using a force-sensitive interface while listening to polyphonic drum patterns. The results show that tap force is shaped primarily by metrical structure, while FWOD captures an additional bottom-up component related to timbral and density-based salience. Among the analytical contour representations tested, FWOD also provided the best performance in rhythm retrieval tasks within a two-dimensional rhythm-space embedding. Together, these findings suggest that listeners simplify polyphonic rhythmic information through a combination of metrical expectations and spectrally weighted onset density. The study further supports FWOD as a compact and behaviorally relevant representation for rhythm analysis, retrieval, and interactive music systems, with potential applications in rhythm-based interaction, music production workflows, and generative musical interfaces., This work has been partially funded by the European UnionNextGenerationEU, Ministry of Universities and Recovery, Transformation and Resilience Plan, through a call from the Pompeu Fabra University (Barcelona). This work has been partially supported by IMPA project PID2023-152250OB-I00 funded by MCIU/AEI/10.13039/501100011033/FEDER, UE.
Repositorio Digital de la UPF
Publicación Contribución de congreso (ConferenceOutput)

Hierarchical and multimodal learning for heterogeneous sound classification

  • Anastasopoulou, Panagiota
  • Dal Rí, Francesco Ardan
  • Serra, Xavier
  • Font, Frederic
This paper investigates multimodal and hierarchical classification strategies to enhance performance in real-world sound classification tasks, centered on the two-level structure of the Broad Sound Taxonomy. We propose a framework that enables the system to consider high-level sound categories when refining its predictions at the subclass level, thereby aligning with the natural hierarchy of sound semantics. To improve accuracy, we integrate acoustic features with semantic cues extracted from crowdsourced textual metadata of the sounds such as titles, tags, and descriptions. During training, we utilize and compare pre-trained embeddings across these modalities, enabling better generalization across acoustically heterogeneous yet semantically related categories. Our experiments show that the use of text-audio embeddings improve classification. We also observe that, although hierarchical supervision does not significantly impact accuracy, it leads to more coherent and perceptually structured latent representations. These improvements in classification performance and representation quality make the system more suitable for downstream tasks such as sound retrieval, description, and similarity search., This research was partially funded by the Secretaria d’Universitats i Recerca del Departament de Recerca i Universitats de la Generalitat de Catalunya (2024FI-100252) under the Joan Oró program, the IA y Música: Cátedra en Inteligencia Artificial y Música (TSI-100929-2023-1) by the Secretaría de Estado de Digitalización e Inteligencia Artificial and the European Union-Next Generation EU under the program Cátedras ENIA 2022, and the IMPA project (PID2023-152250OBI00) by the Ministry of Science, Innovation and Universities of the Spanish Government, the Agencia Estatal de Investigación (AEI) and co-financed by the European Union
Repositorio Digital de la UPF
Publicación Contribución de congreso (ConferenceOutput)

Exploring situated stabilities of a rhythm generation system through variational cross-examination

  • Kotowski, Błażej
  • Evans, Nicholas
  • Haki, Behzad
  • Font Corbera, Frederic
  • Jordà Puig, Sergi
This paper investigates GrooveTransformer, a real-time rhythm generation system, through the postphenomenological framework of Variational Cross-Examination (VCE). By reflecting on its deployment across three distinct artistic contexts, we identify three stabilities: an autonomous drum accompaniment generator, a rhythmic control voltage sequencer in Eurorack format, and a rhythm driver for a harmonic accompaniment system. The versatility of its applications was not an explicit goal from the outset of the project. Thus, we ask: how did this multistability emerge? Through VCE, we identify three key contributors to its emergence: the affordances of system invariants, the interdisciplinary collaboration, and the situated nature of its development. We conclude by reflecting on the viability of VCE as a descriptive and analytical method for Digital Musical Instrument (DMI) design, emphasizing its value in uncovering how technologies mediate, co-shape, and are co-shaped by users and contexts., This research was funded by (1) the Secretaría de Estado de Digitalización e Inteligencia Artificial, and the European Union-Next Generation EU, under the program Cátedras ENIA 2022. "IA y Música: Cátedra en Inteligencia Artificial y Música" (Reference: TSI-100929-2023-1), (2) the Maria de Maeztu Strategic Research Program (CEX2021-001195-M) and (3) the IMPA Project PID2023-152250OB-I00 funded by MCIU/AEI/10.13039/501100011033/FEDER, UE.
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Difficulty-aware score generation for piano sight-reading

  • Ramoneda, Pedro
  • Suzuki, Masahiro
  • Maezawa, Akira
  • Serra, Xavier
Sight-reading is a core skill in music education. It refers to the ability to play a written piece of music correctly the first time it is seen. Developing this skill requires frequent practice with completely new musical excerpts that match the difficulty level of the student. However, creating new sight-reading exercises at a specific difficulty level requires significant time and expert knowledge. As a result, students and teachers often rely on pieces from the existing piano literature, even though sight-reading exams typically use compositions written specifically for the exam. Generative music systems provide a promising approach for creating new sight-reading material with explicit control over performance difficulty. In this work, we frame the creation of sight-reading exercises as a symbolic music generation task that produces piano scores with controllable difficulty. Existing approaches typically rely on control tokens to guide generation, but we show that this strategy does not result in piano scores with reliably controlled difficulty. To address this issue, we introduce an auxiliary difficulty prediction objective using synthetic difficulty labels produced by an expert-based system, enabling scalable training. Our method improves difficulty conditioning accuracy from 69.3% to 92.9% compared to a baseline that conditions generation solely on difficulty control tokens, and reduces mean squared error from 0.30 to 0.09. A user study with expert pianists shows that the generated scores are rated comparable or superior to human-written material in readability and naturalness, while maintaining appropriate playability across difficulty levels. These results represent a step toward the generation of music exercises for a variety of educational applications., We want to thank IMPA Project PID2023-152250OB-I00 funded by MCIU/AEI/10.13039/501100011033/FEDER, UE for partially funding this project.
Repositorio Digital de la UPF
Publicación Capítulo o parte de un libro (BookPart)

Evaluating disentangled representations for controllable music generation

  • Ibáñez-Martínez, Laura
  • Nkama, Chukwuemeka
  • Poltronieri, Andrea
  • Serra, Xavier
  • Rocamora, Martín
Comunicació presentada a la IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), celebrada a Barcelona del 3 al 8 de maig de 2026., Recent approaches in music generation rely on disentangled representations, often labeled as structure and timbre or local and global, to enable controllable synthesis. Yet the underlying properties of these embeddings remain underexplored. In this work, we evaluate such disentangled representations in a set of music audio models for controllable generation using a probing-based framework that goes beyond standard downstream tasks. The selected models reflect diverse un-supervised disentanglement strategies, including inductive biases, data augmentations, adversarial objectives, and staged training procedures. We further isolate specific strategies to analyze their effect. Our analysis spans four key axes: informativeness, equivariance, invariance, and disentanglement, which are assessed across datasets, tasks, and controlled transformations. Our findings reveal inconsistencies between intended and actual semantics of the embeddings, suggesting that current strategies fall short of producing truly disentangled representations, and prompting a re-examination of how controllability is approached in music generation., This work has been supported by IA y Musica: Cátedra en Inteligencia Artificial y Música (TSI-100929-2023-1), funded by the Secretaría de Estado de Digitalización e Inteligencia Artificial and the European Union-Next Generation EU, and IMPA: Multimodal AI for Audio Processing (PID2023-152250OB-I00), funded by the Ministry of Science, Innovation and Universities of the Spanish Government, the Agencia Estatal de Investigación (AEI) and co-financed by the European Union.
Repositorio Digital de la UPF
Publicación Contribución de congreso (ConferenceOutput)

The Freesound API: Advances in Audio Search and Retrieval

  • Anastasopoulou, Panagiota
  • Porter, Alastair
  • Font Corbera, Frederic
Comunicació presentada al The 9th Web Audio Conference WAC-2025, celebrada a París del 19 al 21 de novembre, This paper showcases the latest advances of the Freesound API, focusing on its updated features for improved access and exploration of community audio content shared on the web. Important advancements include restructured metadata retrieval, integration of audio analysis descriptors for richer, content-based search, and further reliance on Solr search technology for both efficient metadata-based querying and vector search. The API introduces a streamlined query system to combine metadata and content-based queries, new similarity spaces that enable more precise sound retrieval, and a unified, easy naming scheme for all audio analysis descriptors. This system is designed to support a range of creative applications, including live performance, browser-based composition tools, educational uses, and generally sound exploration. We overview the new features, demonstrating how users can leverage the API to build more sophisticated audio applications and workflows, and how online sound collections can be utilized for accessible and flexible creative work., This research was partially funded by the Generalitat de Catalunya (2025FI-100252, Joan Oró program), the IA y Música Cátedra (TSI-100929-2023-1, Cátedras ENIA 2022, SE Digitalización e IA, EU NGEU), and the IMPA project (PID2023-152250OB-I00, MCIU, AEI, co-funded by EU).
Repositorio Digital de la UPF
Publicación Contribución de congreso (ConferenceOutput)

From discord to harmony: decomposed consonance-based training for improved audio chord estimation

  • Poltronieri, Andrea
  • Serra, Xavier
  • Rocamora, Martín
Comunicació presentada al 26th International Society for Music Information Retrieval Conference (ISMIR 2025), celebrada a Daejeon (Korea) del 21 al 25 de setembre del 2025, Audio Chord Estimation (ACE) holds a pivotal role in music information research, having garnered attention for over two decades due to its relevance for music transcription and analysis. Despite notable advancements, challenges persist in the task, particularly concerning unique characteristics of harmonic content, which have resulted in existing systems' performances reaching a glass ceiling. These challenges include annotator subjectivity, where varying interpretations among annotators lead to inconsistencies, and class imbalance within chord datasets, where certain chord classes are over-represented compared to others, posing difficulties in model training and evaluation. As a first contribution, this paper presents an evaluation of inter-annotator agreement in chord annotations, using metrics that extend beyond traditional binary measures. In addition, we propose a consonance-informed distance metric that reflects the perceptual similarity between harmonic annotations. Our analysis suggests that consonance-based distance metrics more effectively capture musically meaningful agreement between annotations. Expanding on these findings, we introduce a novel ACE conformer-based model that integrates consonance concepts into the model through consonance-based label smoothing. The proposed model also addresses class imbalance by separately estimating root, bass, and all note activations, enabling the reconstruction of chord labels from decomposed outputs., This work is supported by IA y Música: Cátedra en Inteligencia Artificial y Música (TSI-100929-2023-1), funded by the Secretaría de Estado de Digitalización e Inteligencia Artificial, and the European Union-Next Generation EU, under the program Cátedras ENIA 2022 para la creación de cátedras universidad-empresa en IA, and IMPA: Multimodal AI for Audio Processing (PID2023- 152250OB-I00), funded by the Ministry of Science, Innovation and Universities of the Spanish Government, the Agencia Estatal de Investigación (AEI) and co-financed by the European Union.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Contribución de congreso (ConferenceOutput)

Leveraging Carnatic live recordings for singing voice separation using regression-guided latent diffusion

  • Plaja-Roglans, Genís
  • Serra, Xavier
  • Rocamora, Martín
Comunicació presentada a la 26th International Society for Music Information Retrieval Conference (ISMIR 2025), celebrada a Daejeon (Korea) del 21 a 25 de setembre de 2025., Diffusion models have demonstrated potential to separate individual sources from music mixtures in a generative fashion, enabling a new solution for this challenging problem. However, existing works require clean multi-stem data, which is scarce for several repertoires, consequently compromising generalization. We explore the potential of generative modeling to perform weakly-supervised singing voice separation for Carnatic Music, a music repertoire for which large quantities of multi-stem recordings with bleeding between sources have been collected from live performances. We pre-train a latent diffusion model to perform preliminary vocal separation conditioning on the corresponding mixture. Then, using a regressive model which is separately trained on a clean, smaller, and out-of-domain dataset, we estimate the level of bleeding in the preliminary separations and use that information to guide the diffusion model toward generating cleaner samples. The objective and perceptual evaluations show the potential of the proposed generative system for Carnatic vocal separation., This work is supported by IA y Música: Cátedra en Inteligencia Artificial y Música (TSI-100929-2023-1), funded by the Secretaría de Estado de Digitalización e Inteligencia Artificial, and the European Union-Next Generation EU, under the program Cátedras ENIA 2022 para la creación de cátedras universidad-empresa en IA, and IMPA: Multimodal AI for Audio Processing (PID2023-152250OB-I00), funded by the Ministry of Science, Innovation and Universities of the Spanish Government, the Agencia Estatal de Investigación (AEI) and co-financed by the European Union.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Velocity2DMs: a contextual modeling approach to dynamics marking prediction in piano performance

  • Kim, Hyon
  • Benetos, Emmanouil
  • Serra, Xavier
Expressive dynamics in music performance are subjective and context-dependent, yet most symbolic models treat Dynamics Markings (DMs) as static with fixed MIDI velocities. This paper proposes a method for predicting DMs in piano performance by combining MusicXML score information with performance MIDI data through a novel tokenization scheme and an adapted RoBERTa-based Masked Language Model (MLM). Our approach focuses on contextual aggregated MIDI velocities and corresponding DMs, accounting for subjective interpretations of pianists. Note-level features are serialized and translated into a sequence of tokens to predict both constant (e.g., mp, ff) and non-constant DMs (e.g., crescendo, fp). Evaluation across three expert performance datasets shows that the model effectively learns dynamics transitions from contextual note blocks and generalizes beyond constant markings. This is the first study to model both constant and non-constant dynamics in a unified framework using contextual sequence learning. The results suggest promising applications for expressive music analysis, performance modeling, and computer-assisted music education., This work was supported by IMPA under Project PID2023-152250OB-I00 funded by MCIU/AEI/10.13039/501100011033/FEDER, UE.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Contribución de congreso (ConferenceOutput)

From discord to harmony: decomposed consonance-based training for improved audio chord estimation

  • Poltronieri, Andrea
  • Serra, Xavier
  • Rocamora, Martín
Comunicació presentada al 26th International Society for Music Information Retrieval Conference (ISMIR 2025), celebrada a Daejeon (Korea) del 21 al 25 de setembre del 2025, Audio Chord Estimation (ACE) holds a pivotal role in music information research, having garnered attention for over two decades due to its relevance for music transcription and analysis. Despite notable advancements, challenges persist in the task, particularly concerning unique characteristics of harmonic content, which have resulted in existing systems' performances reaching a glass ceiling. These challenges include annotator subjectivity, where varying interpretations among annotators lead to inconsistencies, and class imbalance within chord datasets, where certain chord classes are over-represented compared to others, posing difficulties in model training and evaluation. As a first contribution, this paper presents an evaluation of inter-annotator agreement in chord annotations, using metrics that extend beyond traditional binary measures. In addition, we propose a consonance-informed distance metric that reflects the perceptual similarity between harmonic annotations. Our analysis suggests that consonance-based distance metrics more effectively capture musically meaningful agreement between annotations. Expanding on these findings, we introduce a novel ACE conformer-based model that integrates consonance concepts into the model through consonance-based label smoothing. The proposed model also addresses class imbalance by separately estimating root, bass, and all note activations, enabling the reconstruction of chord labels from decomposed outputs., This work is supported by IA y Música: Cátedra en Inteligencia Artificial y Música (TSI-100929-2023-1), funded by the Secretaría de Estado de Digitalización e Inteligencia Artificial, and the European Union-Next Generation EU, under the program Cátedras ENIA 2022 para la creación de cátedras universidad-empresa en IA, and IMPA: Multimodal AI for Audio Processing (PID2023- 152250OB-I00), funded by the Ministry of Science, Innovation and Universities of the Spanish Government, the Agencia Estatal de Investigación (AEI) and co-financed by the European Union.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Contribución de congreso (ConferenceOutput)

Hierarchical and multimodal learning for heterogeneous sound classification

  • Anastasopoulou, Panagiota
  • Dal Rí, Francesco Ardan
  • Serra, Xavier
  • Font, Frederic
This paper investigates multimodal and hierarchical classification strategies to enhance performance in real-world sound classification tasks, centered on the two-level structure of the Broad Sound Taxonomy. We propose a framework that enables the system to consider high-level sound categories when refining its predictions at the subclass level, thereby aligning with the natural hierarchy of sound semantics. To improve accuracy, we integrate acoustic features with semantic cues extracted from crowdsourced textual metadata of the sounds such as titles, tags, and descriptions. During training, we utilize and compare pre-trained embeddings across these modalities, enabling better generalization across acoustically heterogeneous yet semantically related categories. Our experiments show that the use of text-audio embeddings improve classification. We also observe that, although hierarchical supervision does not significantly impact accuracy, it leads to more coherent and perceptually structured latent representations. These improvements in classification performance and representation quality make the system more suitable for downstream tasks such as sound retrieval, description, and similarity search., This research was partially funded by the Secretaria d’Universitats i Recerca del Departament de Recerca i Universitats de la Generalitat de Catalunya (2024FI-100252) under the Joan Oró program, the IA y Música: Cátedra en Inteligencia Artificial y Música (TSI-100929-2023-1) by the Secretaría de Estado de Digitalización e Inteligencia Artificial and the European Union-Next Generation EU under the program Cátedras ENIA 2022, and the IMPA project (PID2023-152250OBI00) by the Ministry of Science, Innovation and Universities of the Spanish Government, the Agencia Estatal de Investigación (AEI) and co-financed by the European Union
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Tapping into rhythm: how polyphonic drum patterns map to monophonic contours and support pattern retrieval

  • Gómez-Marín, Daniel
  • Clark, Peter
  • Jordà Puig, Sergi
  • Herrera Boyer, Perfecto, 1964-
This study investigates how listeners reduce polyphonic drum patterns into simplified monophonic rhythm contours through tapping behavior. We examine how these contours relate to metrical structure, onset density, syncopation, and frequency-weighted onset density (FWOD), and whether these features can support rhythm retrieval tasks. Across three experiments, participants tapped rhythmic contours using a force-sensitive interface while listening to polyphonic drum patterns. The results show that tap force is shaped primarily by metrical structure, while FWOD captures an additional bottom-up component related to timbral and density-based salience. Among the analytical contour representations tested, FWOD also provided the best performance in rhythm retrieval tasks within a two-dimensional rhythm-space embedding. Together, these findings suggest that listeners simplify polyphonic rhythmic information through a combination of metrical expectations and spectrally weighted onset density. The study further supports FWOD as a compact and behaviorally relevant representation for rhythm analysis, retrieval, and interactive music systems, with potential applications in rhythm-based interaction, music production workflows, and generative musical interfaces., This work has been partially funded by the European UnionNextGenerationEU, Ministry of Universities and Recovery, Transformation and Resilience Plan, through a call from the Pompeu Fabra University (Barcelona). This work has been partially supported by IMPA project PID2023-152250OB-I00 funded by MCIU/AEI/10.13039/501100011033/FEDER, UE.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Difficulty-aware score generation for piano sight-reading

  • Ramoneda, Pedro
  • Suzuki, Masahiro
  • Maezawa, Akira
  • Serra, Xavier
Sight-reading is a core skill in music education. It refers to the ability to play a written piece of music correctly the first time it is seen. Developing this skill requires frequent practice with completely new musical excerpts that match the difficulty level of the student. However, creating new sight-reading exercises at a specific difficulty level requires significant time and expert knowledge. As a result, students and teachers often rely on pieces from the existing piano literature, even though sight-reading exams typically use compositions written specifically for the exam. Generative music systems provide a promising approach for creating new sight-reading material with explicit control over performance difficulty. In this work, we frame the creation of sight-reading exercises as a symbolic music generation task that produces piano scores with controllable difficulty. Existing approaches typically rely on control tokens to guide generation, but we show that this strategy does not result in piano scores with reliably controlled difficulty. To address this issue, we introduce an auxiliary difficulty prediction objective using synthetic difficulty labels produced by an expert-based system, enabling scalable training. Our method improves difficulty conditioning accuracy from 69.3% to 92.9% compared to a baseline that conditions generation solely on difficulty control tokens, and reduces mean squared error from 0.30 to 0.09. A user study with expert pianists shows that the generated scores are rated comparable or superior to human-written material in readability and naturalness, while maintaining appropriate playability across difficulty levels. These results represent a step toward the generation of music exercises for a variety of educational applications., We want to thank IMPA Project PID2023-152250OB-I00 funded by MCIU/AEI/10.13039/501100011033/FEDER, UE for partially funding this project.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Contribución de congreso (ConferenceOutput)

MusGO: a community-driven framework for assessing openness in music-generative AI

  • Batlle-Roca, Roser
  • Ibáñez-Martínez, Laura
  • Serra, Xavier
  • Gómez, Emilia
  • Rocamora, Martín
Comunicació presentada al 26th International Society for Music Information Retrieval Conference (ISMIR 2025), celebrada a Daejeon (Korea) del 21 al 25 de setembre del 2025, Since 2023, generative AI has rapidly advanced in the music domain. Despite significant technological advancements, music-generative models raise critical ethical challenges, including a lack of transparency and accountability, along with risks such as the replication of artists' works, which highlights the importance of fostering openness. With upcoming regulations such as the EU AI Act encouraging open models, many generative models are being released labelled as 'open'. However, the definition of an open model remains widely debated. In this article, we adapt a recently proposed evidence-based framework for assessing openness in LLMs to the music domain. Using feedback from a survey of 110 participants from the Music Information Retrieval (MIR) community, we refine the framework into MusGO (Music-Generative Open AI), which comprises 13 openness categories: 8 essential and 5 desirable. We evaluate 16 state-of-the-art generative models and provide an openness leaderboard that is fully open to public scrutiny and community contributions. Through this work, we aim to clarify the concept of openness in music-generative AI and promote its transparent and responsible development., This work has been supported by IA y Música: Cátedra en Inteligencia Artificial y Música (TSI-100929-2023-1), funded by the Secretaría de Estado de Digitalización e Inteligencia Artificial and the European Union-Next Generation EU, and IMPA: Multimodal AI for Audio Processing (PID2023-152250OB-I00), funded by the Ministry of Science, Innovation and Universities of the Spanish Government, the Agencia Estatal de Investigación (AEI) and cofinanced by the European Union. We thank our colleagues at the Music Technology Group at Universitat Pompeu Fabra for their thoughtful insights, constructive discussions and active engagement throughout the development of this work
Recercat. Dipósit de la Recerca de Catalunya
Publicación Contribución de congreso (ConferenceOutput)

The Freesound API: Advances in Audio Search and Retrieval

  • Anastasopoulou, Panagiota
  • Porter, Alastair
  • Font Corbera, Frederic
Comunicació presentada al The 9th Web Audio Conference WAC-2025, celebrada a París del 19 al 21 de novembre, This paper showcases the latest advances of the Freesound API, focusing on its updated features for improved access and exploration of community audio content shared on the web. Important advancements include restructured metadata retrieval, integration of audio analysis descriptors for richer, content-based search, and further reliance on Solr search technology for both efficient metadata-based querying and vector search. The API introduces a streamlined query system to combine metadata and content-based queries, new similarity spaces that enable more precise sound retrieval, and a unified, easy naming scheme for all audio analysis descriptors. This system is designed to support a range of creative applications, including live performance, browser-based composition tools, educational uses, and generally sound exploration. We overview the new features, demonstrating how users can leverage the API to build more sophisticated audio applications and workflows, and how online sound collections can be utilized for accessible and flexible creative work., This research was partially funded by the Generalitat de Catalunya (2025FI-100252, Joan Oró program), the IA y Música Cátedra (TSI-100929-2023-1, Cátedras ENIA 2022, SE Digitalización e IA, EU NGEU), and the IMPA project (PID2023-152250OB-I00, MCIU, AEI, co-funded by EU).
Recercat. Dipósit de la Recerca de Catalunya
Publicación Contribución de congreso (ConferenceOutput)

Exploring situated stabilities of a rhythm generation system through variational cross-examination

  • Kotowski, Błażej
  • Evans, Nicholas
  • Haki, Behzad
  • Font Corbera, Frederic
  • Jordà Puig, Sergi
This paper investigates GrooveTransformer, a real-time rhythm generation system, through the postphenomenological framework of Variational Cross-Examination (VCE). By reflecting on its deployment across three distinct artistic contexts, we identify three stabilities: an autonomous drum accompaniment generator, a rhythmic control voltage sequencer in Eurorack format, and a rhythm driver for a harmonic accompaniment system. The versatility of its applications was not an explicit goal from the outset of the project. Thus, we ask: how did this multistability emerge? Through VCE, we identify three key contributors to its emergence: the affordances of system invariants, the interdisciplinary collaboration, and the situated nature of its development. We conclude by reflecting on the viability of VCE as a descriptive and analytical method for Digital Musical Instrument (DMI) design, emphasizing its value in uncovering how technologies mediate, co-shape, and are co-shaped by users and contexts., This research was funded by (1) the Secretaría de Estado de Digitalización e Inteligencia Artificial, and the European Union-Next Generation EU, under the program Cátedras ENIA 2022. "IA y Música: Cátedra en Inteligencia Artificial y Música" (Reference: TSI-100929-2023-1), (2) the Maria de Maeztu Strategic Research Program (CEX2021-001195-M) and (3) the IMPA Project PID2023-152250OB-I00 funded by MCIU/AEI/10.13039/501100011033/FEDER, UE.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Capítulo o parte de un libro (BookPart)

Evaluating disentangled representations for controllable music generation

  • Ibáñez-Martínez, Laura
  • Nkama, Chukwuemeka
  • Poltronieri, Andrea
  • Serra, Xavier
  • Rocamora, Martín
Comunicació presentada a la IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), celebrada a Barcelona del 3 al 8 de maig de 2026., Recent approaches in music generation rely on disentangled representations, often labeled as structure and timbre or local and global, to enable controllable synthesis. Yet the underlying properties of these embeddings remain underexplored. In this work, we evaluate such disentangled representations in a set of music audio models for controllable generation using a probing-based framework that goes beyond standard downstream tasks. The selected models reflect diverse un-supervised disentanglement strategies, including inductive biases, data augmentations, adversarial objectives, and staged training procedures. We further isolate specific strategies to analyze their effect. Our analysis spans four key axes: informativeness, equivariance, invariance, and disentanglement, which are assessed across datasets, tasks, and controlled transformations. Our findings reveal inconsistencies between intended and actual semantics of the embeddings, suggesting that current strategies fall short of producing truly disentangled representations, and prompting a re-examination of how controllability is approached in music generation., This work has been supported by IA y Musica: Cátedra en Inteligencia Artificial y Música (TSI-100929-2023-1), funded by the Secretaría de Estado de Digitalización e Inteligencia Artificial and the European Union-Next Generation EU, and IMPA: Multimodal AI for Audio Processing (PID2023-152250OB-I00), funded by the Ministry of Science, Innovation and Universities of the Spanish Government, the Agencia Estatal de Investigación (AEI) and co-financed by the European Union.