Pasar al contenido principal

IA MULTIMODAL PARA EL PROCESAMIENTO DE AUDIO

PID2023-152250OB-I00

•
Nombre agencia financiadora Agencia Estatal de Investigación
Acrónimo agencia financiadora AEI
Programa Programa Estatal para Impulsar la Investigación Científico-Técnica y su Transferencia
Subprograma Subprograma Estatal de Generación de Conocimiento
Convocatoria Proyectos de I+D+i (Generación de Conocimiento y Retos Investigación)
Año convocatoria 2023
Plan Estatal 2021-2023 Plan Estatal de Investigación Científica y Técnica y de Innovación
Centro beneficiario UNIVERSITAT POMPEU FABRA CCT

Resultados relacionados

Resultados totales (Incluyendo duplicados): 6
Encontrada(s) 1 página(s)

Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Velocity2DMs: a contextual modeling approach to dynamics marking prediction in piano performance

  • Kim, Hyon
  • Benetos, Emmanouil
  • Serra, Xavier
Expressive dynamics in music performance are subjective and context-dependent, yet most symbolic models treat Dynamics Markings (DMs) as static with fixed MIDI velocities. This paper proposes a method for predicting DMs in piano performance by combining MusicXML score information with performance MIDI data through a novel tokenization scheme and an adapted RoBERTa-based Masked Language Model (MLM). Our approach focuses on contextual aggregated MIDI velocities and corresponding DMs, accounting for subjective interpretations of pianists. Note-level features are serialized and translated into a sequence of tokens to predict both constant (e.g., mp, ff) and non-constant DMs (e.g., crescendo, fp). Evaluation across three expert performance datasets shows that the model effectively learns dynamics transitions from contextual note blocks and generalizes beyond constant markings. This is the first study to model both constant and non-constant dynamics in a unified framework using contextual sequence learning. The results suggest promising applications for expressive music analysis, performance modeling, and computer-assisted music education., This work was supported by IMPA under Project PID2023-152250OB-I00 funded by MCIU/AEI/10.13039/501100011033/FEDER, UE.
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Tapping into rhythm: how polyphonic drum patterns map to monophonic contours and support pattern retrieval

  • Gómez-Marín, Daniel
  • Clark, Peter
  • Jordà Puig, Sergi
  • Herrera Boyer, Perfecto, 1964-
This study investigates how listeners reduce polyphonic drum patterns into simplified monophonic rhythm contours through tapping behavior. We examine how these contours relate to metrical structure, onset density, syncopation, and frequency-weighted onset density (FWOD), and whether these features can support rhythm retrieval tasks. Across three experiments, participants tapped rhythmic contours using a force-sensitive interface while listening to polyphonic drum patterns. The results show that tap force is shaped primarily by metrical structure, while FWOD captures an additional bottom-up component related to timbral and density-based salience. Among the analytical contour representations tested, FWOD also provided the best performance in rhythm retrieval tasks within a two-dimensional rhythm-space embedding. Together, these findings suggest that listeners simplify polyphonic rhythmic information through a combination of metrical expectations and spectrally weighted onset density. The study further supports FWOD as a compact and behaviorally relevant representation for rhythm analysis, retrieval, and interactive music systems, with potential applications in rhythm-based interaction, music production workflows, and generative musical interfaces., This work has been partially funded by the European UnionNextGenerationEU, Ministry of Universities and Recovery, Transformation and Resilience Plan, through a call from the Pompeu Fabra University (Barcelona). This work has been partially supported by IMPA project PID2023-152250OB-I00 funded by MCIU/AEI/10.13039/501100011033/FEDER, UE.
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Difficulty-aware score generation for piano sight-reading

  • Ramoneda, Pedro
  • Suzuki, Masahiro
  • Maezawa, Akira
  • Serra, Xavier
Sight-reading is a core skill in music education. It refers to the ability to play a written piece of music correctly the first time it is seen. Developing this skill requires frequent practice with completely new musical excerpts that match the difficulty level of the student. However, creating new sight-reading exercises at a specific difficulty level requires significant time and expert knowledge. As a result, students and teachers often rely on pieces from the existing piano literature, even though sight-reading exams typically use compositions written specifically for the exam. Generative music systems provide a promising approach for creating new sight-reading material with explicit control over performance difficulty. In this work, we frame the creation of sight-reading exercises as a symbolic music generation task that produces piano scores with controllable difficulty. Existing approaches typically rely on control tokens to guide generation, but we show that this strategy does not result in piano scores with reliably controlled difficulty. To address this issue, we introduce an auxiliary difficulty prediction objective using synthetic difficulty labels produced by an expert-based system, enabling scalable training. Our method improves difficulty conditioning accuracy from 69.3% to 92.9% compared to a baseline that conditions generation solely on difficulty control tokens, and reduces mean squared error from 0.30 to 0.09. A user study with expert pianists shows that the generated scores are rated comparable or superior to human-written material in readability and naturalness, while maintaining appropriate playability across difficulty levels. These results represent a step toward the generation of music exercises for a variety of educational applications., We want to thank IMPA Project PID2023-152250OB-I00 funded by MCIU/AEI/10.13039/501100011033/FEDER, UE for partially funding this project.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Velocity2DMs: a contextual modeling approach to dynamics marking prediction in piano performance

  • Kim, Hyon
  • Benetos, Emmanouil
  • Serra, Xavier
Expressive dynamics in music performance are subjective and context-dependent, yet most symbolic models treat Dynamics Markings (DMs) as static with fixed MIDI velocities. This paper proposes a method for predicting DMs in piano performance by combining MusicXML score information with performance MIDI data through a novel tokenization scheme and an adapted RoBERTa-based Masked Language Model (MLM). Our approach focuses on contextual aggregated MIDI velocities and corresponding DMs, accounting for subjective interpretations of pianists. Note-level features are serialized and translated into a sequence of tokens to predict both constant (e.g., mp, ff) and non-constant DMs (e.g., crescendo, fp). Evaluation across three expert performance datasets shows that the model effectively learns dynamics transitions from contextual note blocks and generalizes beyond constant markings. This is the first study to model both constant and non-constant dynamics in a unified framework using contextual sequence learning. The results suggest promising applications for expressive music analysis, performance modeling, and computer-assisted music education., This work was supported by IMPA under Project PID2023-152250OB-I00 funded by MCIU/AEI/10.13039/501100011033/FEDER, UE.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Tapping into rhythm: how polyphonic drum patterns map to monophonic contours and support pattern retrieval

  • Gómez-Marín, Daniel
  • Clark, Peter
  • Jordà Puig, Sergi
  • Herrera Boyer, Perfecto, 1964-
This study investigates how listeners reduce polyphonic drum patterns into simplified monophonic rhythm contours through tapping behavior. We examine how these contours relate to metrical structure, onset density, syncopation, and frequency-weighted onset density (FWOD), and whether these features can support rhythm retrieval tasks. Across three experiments, participants tapped rhythmic contours using a force-sensitive interface while listening to polyphonic drum patterns. The results show that tap force is shaped primarily by metrical structure, while FWOD captures an additional bottom-up component related to timbral and density-based salience. Among the analytical contour representations tested, FWOD also provided the best performance in rhythm retrieval tasks within a two-dimensional rhythm-space embedding. Together, these findings suggest that listeners simplify polyphonic rhythmic information through a combination of metrical expectations and spectrally weighted onset density. The study further supports FWOD as a compact and behaviorally relevant representation for rhythm analysis, retrieval, and interactive music systems, with potential applications in rhythm-based interaction, music production workflows, and generative musical interfaces., This work has been partially funded by the European UnionNextGenerationEU, Ministry of Universities and Recovery, Transformation and Resilience Plan, through a call from the Pompeu Fabra University (Barcelona). This work has been partially supported by IMPA project PID2023-152250OB-I00 funded by MCIU/AEI/10.13039/501100011033/FEDER, UE.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Difficulty-aware score generation for piano sight-reading

  • Ramoneda, Pedro
  • Suzuki, Masahiro
  • Maezawa, Akira
  • Serra, Xavier
Sight-reading is a core skill in music education. It refers to the ability to play a written piece of music correctly the first time it is seen. Developing this skill requires frequent practice with completely new musical excerpts that match the difficulty level of the student. However, creating new sight-reading exercises at a specific difficulty level requires significant time and expert knowledge. As a result, students and teachers often rely on pieces from the existing piano literature, even though sight-reading exams typically use compositions written specifically for the exam. Generative music systems provide a promising approach for creating new sight-reading material with explicit control over performance difficulty. In this work, we frame the creation of sight-reading exercises as a symbolic music generation task that produces piano scores with controllable difficulty. Existing approaches typically rely on control tokens to guide generation, but we show that this strategy does not result in piano scores with reliably controlled difficulty. To address this issue, we introduce an auxiliary difficulty prediction objective using synthetic difficulty labels produced by an expert-based system, enabling scalable training. Our method improves difficulty conditioning accuracy from 69.3% to 92.9% compared to a baseline that conditions generation solely on difficulty control tokens, and reduces mean squared error from 0.30 to 0.09. A user study with expert pianists shows that the generated scores are rated comparable or superior to human-written material in readability and naturalness, while maintaining appropriate playability across difficulty levels. These results represent a step toward the generation of music exercises for a variety of educational applications., We want to thank IMPA Project PID2023-152250OB-I00 funded by MCIU/AEI/10.13039/501100011033/FEDER, UE for partially funding this project.