Pasar al contenido principal

TECNOLOGIAS INTERACTIVAS PARA EL APRENDIZAJE DE MUSICA

TIN2013-48152-C2-2-R

•
Nombre agencia financiadora Ministerio de Economía y Competitividad
Acrónimo agencia financiadora MINECO
Programa Programa Estatal de I+D+i Orientado a los Retos de la Sociedad
Convocatoria Proyectos I+D+i Retos de Investigación
Año convocatoria 2013
Plan Estatal 2013-2016 Plan Estatal de Investigación Científica y Técnica y de Innovación
Unidad de gestión Secretaría de Estado de Educación, Formación Profesional y Universidades
Centro beneficiario UNIVERSITAT POMPEU FABRA CCT
Identificador persistente http://dx.doi.org/10.13039/501100003329

Resultados relacionados

Resultados totales (Incluyendo duplicados): 28
Encontrada(s) 1 página(s)

Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Phrase-Level modeling of expression in violin performances

  • Ortega, Fabio J. M.
  • Giraldo, Sergio
  • Pérez Carrillo, Alfonso Antonio, 1977-
  • Ramírez, Rafael, 1966-
Background: Expression is a key skill in music performance, and one that is difficult to address in music lessons. Computational models that learn from expert performances can help providing suggestions and feedback to students. Aim: We propose and analyze an approach to modeling variations in dynamics and note onset timing for solo violin pieces with the purpose of facilitating expressive performance learning in new pieces, for which no reference performance is available. Method: The method generates phrase–level predictions based on musical score information on the assumption that expressiveness is idiomatic, and thus influenced by similar–sounding melodies. Predictions were evaluated numerically using three different datasets and against note–level machine–learning models, and also perceptually by listeners, who were presented to synthesized versions of musical excerpts, and asked to choose the most human–sounding one. Some of the presented excerpts were synthesized to reflect the variations in dynamics and timing predicted by the model, whereas others were shaped to reflect the dynamics and timing of an actual expert performance, and a third group was presented with no expressive variations. Results: surprisingly, none of the three synthesized versions was consistently selected as human–like nor preferred with statistical significance by listeners. Possible interpretations of these results include the fact that the melodies might have been impossible to interpret outside their musical context, or that expressive features that were left out of the modeling such as note articulation and vibrato are, in fact, essential to the perception of expression in violin performance. Positive feedback by some listeners toward the modeled melodies in a blind setting indicate that the modeling approach was capable of generating appropriate renditions at least for a subset of the data. Numerically, performance in phrase–level suffers a small degradation if compared to note–level, but produces predictions easier to interpret visually, thus more useful in a pedagogical setting., This work has been partly sponsored by the Spanish TIN project TIMUL (TIN 2013-48152-C2-2-R), the European Union Horizon 2020 research and innovation programme under grant agreement No. 688269 (TELMI project), and the Spanish Ministry of Economy and Competitiveness under the Maria de Maeztu Units of Excellence Programme (MDM-2015-0502).
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Musical neurofeedback for treating depression in elderly people

  • Ramírez, Rafael, 1966-
  • Palencia-Lefler Ors, Manuel
  • Giraldo, Sergio
  • Vamvakousis, Zacharias
We introduce a new neurofeedback approach, which allows users to manipulate expressive parameters in music performances using their emotional state, and we present the results of a pilot clinical experiment applying the approach to alleviate depression in elderly people. Ten adults (9 female and 1 male, mean = 84, SD = 5.8) with normal hearing participated in the neurofeedback study consisting of 10 sessions (2 sessions per week) of 15 min each. EEG data was acquired using the Emotiv EPOC EEG device. In all sessions, subjects were asked to sit in a comfortable chair facing two loudspeakers, to close their eyes, and to avoid moving during the experiment. Participants listened to music pieces preselected according to their music preferences, and were encouraged to increase the loudness and tempo of the pieces, based on their arousal and valence levels. The neurofeedback system was tuned so that increased arousal, computed as beta to alpha activity ratio in the frontal cortex corresponded to increased loudness, and increased valence, computed as relative frontal alpha activity in the right lobe compared to the left lobe, corresponded to increased tempo. Pre and post evaluation of six participants was performed using the BDI depression test, showing an average improvement of 17.2% (1.3) in their BDI scores at the end of the study. In addition, an analysis of the collected EEG data of the participants showed a significant decrease of relative alpha activity in their left frontal lobe (p = 0.00008), which may be interpreted as an improvement of their depression condition., This work has been partly sponsored by the Ministerio de Economia y Competitividad under Grant TIN2013-48152-C2-2-R (TIMuL Project).
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

A Machine learning approach to ornamentation modeling and synthesis in jazz guitar

  • Ramírez, Rafael, 1966-
  • Giraldo, Sergio
We present a machine learning approach to automatically generate expressive (ornamented) jazz performances from un-expressive music scores. Features extracted from the scores and the corresponding audio recordings performed by a professional guitarist were used to train computational models for predicting melody ornamentation. As a first step, several machine learning techniques were explored to induce regression models for timing, onset, and dynamics (i.e. note duration and energy) transformations, and an ornamentation model for classifying notes as ornamented or non-ornamented. In a second step, the most suitable ornament for predicted ornamented notes was selected based on note context similarity. Finally, concatenative synthesis was used to automatically synthesize expressive performances of new pieces using the induced models. Supplemental online material for this article containing musical examples of the automatically generated ornamented pieces can be accessed at doi: 10.1080/17459737.2016.1207814 and https://soundcloud.com/machine-learning-and-jazz. In the Online Supplement we present an example of the musical piece Yesterdays by Jerome Kern, which was modeled using our methodology for expressive music performance in jazz guitar., This project has received funding from: the European Union Horizon 2020 research and innovation programme [grant agreement No 688269]; the Spanish TIN project TIMUL [grant agreement TIN2013-48152-C2-2-R].
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Evaluation of a sound quality visual feedback system for bow learning technique in violin beginners: an EEG study

  • Blanco, Angel David
  • Ramírez, Rafael, 1966-
Current music technologies can assist in the process of learning to play a musical instrument and provide objective measures for evaluating the improvement of music students in concrete music tasks. In this paper, we investigated the effects of a sound quality visual feedback system (SQVFS) in violin learning. In particular, we studied the EEG activity of a group of participants with no previous violin playing experience while they learned to produce a stable sound (regarding pitch, dynamics, and timbre) in order to find motor learning biomarkers in a music task. Eighteen subjects with no prior experience in violin playing were divided into two groups: participants in the first group (experimental group, N = 9) practiced with instructional videos and offline feedback from the SQVFS provided in alternation with their performance, while participants in a second group (control group, N = 9) practiced with the instructional videos only. A third group of violin experts (players with more than 6 years of experience) performed the same task for comparative purposes (N = 7). All participants were asked to perform 20 trials (4 blocks of 5 trials) consisting of a violin bowing exercise while their EEG activity and their produced sound was recorded. Significant sound quality improvements along the session were found in all participants with the exception of participants in the expert group. In addition, participants in the experimental group showed increased interest in the learning process and significant improvement after the second block not present in the control group. A significant correlation between the levels of frontal gamma band power and the sound improvement along the task was found in both the experimental and control group. This result is consistent with the temporal binding model which associates gamma band power with the role of integrating (binding) information processed in distributed cortical areas. Task complexity demands more cognitive resources, more binding and thus, gamma band power enhancement, which may be reduced as the demanded task begins to be automated as it is likely to be the case in both beginners groups., This work was partly sponsored by Fundación Memora, the Spanish TIN project TIMUL (TIN2013-48152-C2-2-R), and the European Union Horizon 2020 research and innovation program under grant agreement No. 688269 (TELMI project).
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Applying deep learning techniques to estimate patterns of musical gesture

  • Dalmazzo, David
  • Waddell, George
  • Ramírez, Rafael, 1966-
Repetitive practice is one of the most important factors in improving the performance of motor skills. This paper focuses on the analysis and classification of forearm gestures in the context of violin playing. We recorded five experts and three students performing eight traditional classical violin bow-strokes: martelé, staccato, detaché, ricochet, legato, trémolo, collé, and col legno. To record inertial motion information, we utilized the Myo sensor, which reports a multidimensional time-series signal. We synchronized inertial motion recordings with audio data to extract the spatiotemporal dynamics of each gesture. Applying state-of-the-art deep neural networks, we implemented and compared different architectures where convolutional neural networks (CNN) models demonstrated recognition rates of 97.147%, 3DMultiHeaded_CNN models showed rates of 98.553%, and rates of 99.234% were demonstrated by CNN_LSTM models. The collected data (quaternion of the bowing arm of a violinist) contained sufficient information to distinguish the bowing techniques studied, and deep learning methods were capable of learning the movement patterns that distinguish these techniques. Each of the learning algorithms investigated (CNN, 3DMultiHeaded_CNN, and CNN_LSTM) produced high classification accuracies which supported the feasibility of training classifiers. The resulting classifiers may provide the foundation of a digital assistant to enhance musicians' time spent practicing alone, providing real-time feedback on the accuracy and consistency of their musical gestures in performance., This work has been partly sponsored by the Spanish TIN project TIMUL (TIN 2013-48152-C2-2-R), the European Union Horizon 2020 research and innovation program under grant agreement No. 688269 (TELMI project), and the Spanish Ministry of Economy and Competitiveness under the Maria de Maeztu Units of Excellence Programme (MDM-2015-0502).
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

EEG-Based Analysis of the Emotional Effect of Music Therapy on Palliative Care Cancer Patients

  • Ramírez, Rafael, 1966-
  • Planas Domingo, Josep
  • Escudé, Nuria
  • Mercadé, Jordi
  • Farriols, Cristina
Music is known to have the power to induce strong emotions. The present study assessed, based on Electroencephalography (EEG) data, the emotional response of terminally ill cancer patients to a music therapy intervention in a randomized controlled trial. A sample of 40 participants from the palliative care unit in the Hospital del Mar in Barcelona was randomly assigned to two groups of 20. The first group [experimental group (EG)] participated in a session of music therapy (MT), and the second group [control group (CG)] was provided with company. Based on our previous work on EEG-based emotion detection, instantaneous emotional indicators in the form of a coordinate in the arousal-valence plane were extracted from the participants' EEG data. The emotional indicators were analyzed in order to quantify (1) the overall emotional effect of MT on the patients compared to controls, and (2) the relative effect of the different MT techniques applied during each session. During each MT session, five conditions were considered: I (initial patient's state before MT starts), C1 (passive listening), C2 (active listening), R (relaxation), and F (final patient's state). EEG data analysis showed a significant increase in valence (p = 0.0004) and arousal (p = 0.003) between I and F in the EG. No significant changes were found in the CG. This results can be interpreted as a positive emotional effect of MT in advanced cancer patients. In addition, according to pre- and post-intervention questionnaire responses, participants in the EG also showed a significant decrease in tiredness, anxiety and breathing difficulties, as well as an increase in levels of well-being. No equivalent changes were observed in the CG., This work was partly sponsored by Fundación Memora, the Spanish TIN project TIMUL (TIN2013-48152-C2-2-R), and the European Union Horizon 2020 research and innovation program under grant agreement No. 688269 (TELMI project).
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Machine learning and music generation

  • Iñesta, José M.
  • Conklin, Darrell
  • Ramírez, Rafael, 1966-
We guest editors acknowledge the financial support of the following funding sources: the Ministerio de Economía y Competitividad project TIMuL [No. TIN2013–48152–C2–1–R]; [No. TIN2013–48152–C2–2–R, supported by UE FEDER funds]; the project Lrn2Cre8, which is funded by the Future and Emerging Technologies (FET) programme within the Seventh Framework Programme for Research of the European Commission [FET grant number 610859]; the European Union Horizon 2020 research and innovation programme [grant agreement number 688269].
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

The EyeHarp: A gaze-controlled digital musical instrument

  • Vamvakousis, Zacharias
  • Ramírez, Rafael, 1966-
We present and evaluate the EyeHarp, a new gaze-controlled Digital Musical Instrument, which aims to enable people with severe motor disabilities to learn, perform, and compose music using only their gaze as control mechanism. It consists of (1) a step-sequencer layer, which serves for constructing chords/arpeggios, and (2) a melody layer, for playing melodies and changing the chords/arpeggios. We have conducted a pilot evaluation of the EyeHarp involving 39 participants with no disabilities from both a performer and an audience perspective. In the first case, eight people with normal vision and no motor disability participated in a music-playing session in which both quantitative and qualitative data were collected. In the second case 31 people qualitatively evaluated the EyeHarp in a concert setting consisting of two parts: a solo performance part, and an ensemble (EyeHarp, two guitars, and flute) performance part. The obtained results indicate that, similarly to traditional music instruments, the proposed digital musical instrument has a steep learning curve, and allows to produce expressive performances both from the performer and audience perspective., This project has received funding from the European Unions Horizon 2020 research and innovation program under grant agreement No. 688269, as well as from the Spanish TIN project TIMUL under grant agreement TIN2013-48152-C2-2-R.
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Mapping between dynamic markings and performed loudness: a machine learning approach

  • Kosta, Katerina
  • Ramírez, Rafael, 1966-
  • Bandtlow, Oscar F.
  • Chew, Elaine
Loudness variation is one of the foremost tools for expressivity in music performance. Loudness is frequently notated as dynamic markings such as p (piano, meaning soft) or f (forte, meaning loud). While dynamic markings in music scores are important indicators of how music pieces should be interpreted, their meaning is less straightforward than it may seem, and depends highly on the context in which they appear. In this article, we investigate the relationship between dynamic markings in the score and performed loudness by applying machine learning techniques – decision trees, support vector machines, artificial neural networks, and a k-nearest neighbor method – to the prediction of loudness levels corresponding to dynamic markings, and to the classification of dynamic markings given loudness values. The methods are applied to 44 recordings of performances of Chopin's Mazurkas, each by 8 pianists. The results show that loudness values and markings can be predicted relatively well when trained across recordings of the same piece, but fail dismally when trained across the pianist's recordings of other pieces, demonstrating that score features may trump individual style when modeling loudness choices. Evidence suggests that all the features chosen for the task are relevant, and analysis of the results reveals the forms (such as the return of the theme) and structures (such as dynamic-marking repetitions) that influence the predictability of loudness and markings. Modeling of loudness trends in expressive performance appears to be a delicate matter, and sometimes loudness expression can be a matter of the performer's idiosyncracy., This work was supported in part by a UK EPSRC Platform Grant for Digital Music [EP/K009559/1]; the Spanish TIN project TIMUL [TIN2013-48152-C2-2-R]; and the European Union's Horizon 2020 research and innovation programme [grant agreement No. 688269].
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Bowing gestures classification in violin performance: a machine learning approach

  • Dalmazzo, David
  • Ramírez, Rafael, 1966-
Gestures in music are of paramount importance partly because they are directly linked to musicians' sound and expressiveness. At the same time, current motion capture technologies are capable of detecting body motion/gestures details very accurately. We present a machine learning approach to automatic violin bow gesture classification based on Hierarchical Hidden Markov Models (HHMM) and motion data. We recorded motion and audio data corresponding to seven representative bow techniques (Détaché, Martelé, Spiccato, Ricochet, Sautillé, Staccato, and Bariolage) performed by a professional violin player. We used the commercial Myo device for recording inertial motion information from the right forearm and synchronized it with audio recordings. Data was uploaded into an online public repository. After extracting features from both the motion and audio data, we trained an HHMM to identify the different bowing techniques automatically. Our model can determine the studied bowing techniques with over 94% accuracy. The results make feasible the application of this work in a practical learning scenario, where violin students can benefit from the real-time feedback provided by the system., This work has been partly sponsored by the Spanish TIN project TIMUL (TIN 2013-48152-C2-2-R), the European Union Horizon 2020 research and innovation programme under grant agreement No. 688269 (TELMI project), and the Spanish Ministry of Economy and Competitiveness under the Maria de Maeztu Units of Excellence Programme (MDM-2015-0502).
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

A Machine learning approach to discover rules for expressive performance actions in jazz guitar music

  • Giraldo, Sergio
  • Ramírez, Rafael, 1966-
Expert musicians introduce expression in their performances by manipulating sound properties such as timing, energy, pitch, and timbre. Here, we present a data driven computational approach to induce expressive performance rule models for note duration, onset, energy, and ornamentation transformations in jazz guitar music. We extract high-level features from a set of 16 commercial audio recordings (and corresponding music scores) of jazz guitarist Grant Green in order to characterize the expression in the pieces. We apply machine learning techniques to the resulting features to learn expressive performance rule models. We (1) quantitatively evaluate the accuracy of the induced models, (2) analyse the relative importance of the considered musical features, (3) discuss some of the learnt expressive performance rules in the context of previous work, and (4) assess their generailty. The accuracies of the induced predictive models is significantly above base-line levels indicating that the audio performances and the musical features extracted contain sufficient information to automatically learn informative expressive performance patterns. Feature analysis shows that the most important musical features for predicting expressive transformations are note duration, pitch, metrical strength, phrase position, Narmour structure, and tempo and key of the piece. Similarities and differences between the induced expressive rules and the rules reported in the literature were found. Differences may be due to the fact that most previously studied performance data has consisted of classical music recordings. Finally, the rules' performer specificity/generality is assessed by applying the induced rules to performances of the same pieces performed by two other professional jazz guitar players. Results show a consistency in the ornamentation patterns between Grant Green and the other two musicians, which may be interpreted as a good indicator for generality of the ornamentation rules., This work has been partly sponsored by the Spanish TIN project TIMUL (TIN2013-48152-C2-2-R), and the European Union Horizon 2020 research and innovation programme under grant agreement No. 688269 (TELMI project).
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Air violin: a machine learning approach to fingering gesture recognition

  • Dalmazzo, David
  • Ramírez, Rafael, 1966-
Comunicació presentada a: 1st ACM SIGCHI International Workshop on Multimodal Interaction for Education, celebrat el 13 de novembre de 2017 a Glasgow, Regne Unit., We train and evaluate two machine learning models for predicting fingering in violin performances using motion and EMG sensors integrated in the Myo device. Our aim is twofold: first, provide a fingering recognition model in the context of a gamification virtual violin application where we measure both right hand (i.e. bow) and left hand (i.e. fingering) gestures, and second, implement a tracking system for a computer assisted pedagogical tool for self-regulated learners in high-level music education. Our approach is based on the principle of mapping-by-demonstration in which the model is trained by the performer. We evaluated a model based on Decision Trees and compared it with a Hidden Markovian Model., This work has been partly sponsored by the Spanish TIN project TIMUL (TIN 2013-48152-C2-2-R), the European Union Horizon 2020 research and innovation programme under grant agreement No. 688269 (TELMI project), and the Spanish Ministry of Economy and Competitiveness under the Maria de Maeztu Units of Excellence Programme (MDM-2015-0502).
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Automatic assessment of tone quality in violin music performance

  • Giraldo, Sergio
  • Waddell, George
  • Nou Plana, Ignasi
  • Ortega, Ariadna
  • Mayor, Oscar
  • Perez, Alfonso
  • Williamon, Aaron
  • Ramírez, Rafael, 1966-
The automatic assessment of music performance has become an area of increasing interest due to the growing number of technology-enhanced music learning systems. In most of these systems, the assessment of musical performance is based on pitch and onset accuracy, but very few pay attention to other important aspects of performance, such as sound quality or timbre. This is particularly true in violin education, where the quality of timbre plays a significant role in the assessment of musical performances. However, obtaining quantifiable criteria for the assessment of timbre quality is challenging, as it relies on consensus among the subjective interpretations of experts. We present an approach to assess the quality of timbre in violin performances using machine learning techniques. We collected audio recordings of several tone qualities and performed perceptual tests to find correlations among different timbre dimensions. We processed the audio recordings to extract acoustic features for training tone-quality models. Correlations among the extracted features were analyzed and feature information for discriminating different timbre qualities were investigated. A real-time feedback system designed for pedagogical use was implemented in which users can train their own timbre models to assess and receive feedback on their performances., This work has been partly sponsored by the Spanish TIN project TIMUL (TIN2013-48152-C2-2-R), the European Union Horizon 2020 research and innovation programme under grant agreement No. 688269 (TELMI project), and the Spanish Ministry of Economy and Competitiveness under the Maria de Maeztu Units of Excellence Programme (MDM-2015-0502).
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Real-time sound and motion feedback for violin bow technique learning: a controlled, randomized trial

  • Blanco, Angel David
  • Tassani, Simone
  • Ramírez, Rafael, 1966-
The production of good sound generation in the violin is a complex task that requires coordination and spatiotemporal control of bowing gestures. The use of motion-capture technologies to improve performance or reduce injury risks in the area of kinesiology is becoming widespread. The combination of motion accuracy and sound quality feedback has the potential of becoming an important aid in violin learning. In this study, we evaluate motion-capture and sound-quality analysis technologies developed inside the context of the TELMI, a technology-enhanced music learning project. We analyzed the sound and bow motion of 50 participants with no prior violin experience while learning to produce a stable sound in the violin. Participants were divided into two groups: the experimental group (N = 24) received real-time visual feedback both on kinematics and sound quality, while participants in the control group (N = 26) practiced without any type of external help. An additional third group of violin experts performed the same task for comparative purposes (N = 15). After the practice session, all groups were evaluated in a transfer phase without feedback. At the practice phase, the experimental group improved their bowing kinematics in comparison to the control group, but this was at the expense of impairing the sound quality of their performance. At the retention phase, the experimental group showed better results in sound quality, especially concerning control of sound dynamics. Besides, we found that the expert group improved the stability of their sound while using the technology. All in all, these results emphasize the importance of feedback technologies in learning complex tasks, such as musical instrument learning., This work was partly sponsored by Fundación Memora, the Spanish TIN project TIMUL (TIN2013-48152-C2-2-R), and the European Union Horizon 2020 research and innovation program under grant agreement No. 688269 (TELMI project).
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Automatic assessment of tone quality in violin music performance

  • Giraldo, Sergio
  • Waddell, George
  • Nou Plana, Ignasi
  • Ortega, Ariadna
  • Mayor, Oscar
  • Perez, Alfonso
  • Williamon, Aaron
  • Ramírez, Rafael, 1966-
The automatic assessment of music performance has become an area of increasing interest due to the growing number of technology-enhanced music learning systems. In most of these systems, the assessment of musical performance is based on pitch and onset accuracy, but very few pay attention to other important aspects of performance, such as sound quality or timbre. This is particularly true in violin education, where the quality of timbre plays a significant role in the assessment of musical performances. However, obtaining quantifiable criteria for the assessment of timbre quality is challenging, as it relies on consensus among the subjective interpretations of experts. We present an approach to assess the quality of timbre in violin performances using machine learning techniques. We collected audio recordings of several tone qualities and performed perceptual tests to find correlations among different timbre dimensions. We processed the audio recordings to extract acoustic features for training tone-quality models. Correlations among the extracted features were analyzed and feature information for discriminating different timbre qualities were investigated. A real-time feedback system designed for pedagogical use was implemented in which users can train their own timbre models to assess and receive feedback on their performances., This work has been partly sponsored by the Spanish TIN project TIMUL (TIN2013-48152-C2-2-R), the European Union Horizon 2020 research and innovation programme under grant agreement No. 688269 (TELMI project), and the Spanish Ministry of Economy and Competitiveness under the Maria de Maeztu Units of Excellence Programme (MDM-2015-0502).
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Phrase-Level modeling of expression in violin performances

  • Ortega, Fabio J. M.
  • Giraldo, Sergio
  • Pérez Carrillo, Alfonso Antonio, 1977-
  • Ramírez, Rafael, 1966-
Background: Expression is a key skill in music performance, and one that is difficult to address in music lessons. Computational models that learn from expert performances can help providing suggestions and feedback to students. Aim: We propose and analyze an approach to modeling variations in dynamics and note onset timing for solo violin pieces with the purpose of facilitating expressive performance learning in new pieces, for which no reference performance is available. Method: The method generates phrase–level predictions based on musical score information on the assumption that expressiveness is idiomatic, and thus influenced by similar–sounding melodies. Predictions were evaluated numerically using three different datasets and against note–level machine–learning models, and also perceptually by listeners, who were presented to synthesized versions of musical excerpts, and asked to choose the most human–sounding one. Some of the presented excerpts were synthesized to reflect the variations in dynamics and timing predicted by the model, whereas others were shaped to reflect the dynamics and timing of an actual expert performance, and a third group was presented with no expressive variations. Results: surprisingly, none of the three synthesized versions was consistently selected as human–like nor preferred with statistical significance by listeners. Possible interpretations of these results include the fact that the melodies might have been impossible to interpret outside their musical context, or that expressive features that were left out of the modeling such as note articulation and vibrato are, in fact, essential to the perception of expression in violin performance. Positive feedback by some listeners toward the modeled melodies in a blind setting indicate that the modeling approach was capable of generating appropriate renditions at least for a subset of the data. Numerically, performance in phrase–level suffers a small degradation if compared to note–level, but produces predictions easier to interpret visually, thus more useful in a pedagogical setting., This work has been partly sponsored by the Spanish TIN project TIMUL (TIN 2013-48152-C2-2-R), the European Union Horizon 2020 research and innovation programme under grant agreement No. 688269 (TELMI project), and the Spanish Ministry of Economy and Competitiveness under the Maria de Maeztu Units of Excellence Programme (MDM-2015-0502).
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

A Machine learning approach to ornamentation modeling and synthesis in jazz guitar

  • Ramírez, Rafael, 1966-
  • Giraldo, Sergio
We present a machine learning approach to automatically generate expressive (ornamented) jazz performances from un-expressive music scores. Features extracted from the scores and the corresponding audio recordings performed by a professional guitarist were used to train computational models for predicting melody ornamentation. As a first step, several machine learning techniques were explored to induce regression models for timing, onset, and dynamics (i.e. note duration and energy) transformations, and an ornamentation model for classifying notes as ornamented or non-ornamented. In a second step, the most suitable ornament for predicted ornamented notes was selected based on note context similarity. Finally, concatenative synthesis was used to automatically synthesize expressive performances of new pieces using the induced models. Supplemental online material for this article containing musical examples of the automatically generated ornamented pieces can be accessed at doi: 10.1080/17459737.2016.1207814 and https://soundcloud.com/machine-learning-and-jazz. In the Online Supplement we present an example of the musical piece Yesterdays by Jerome Kern, which was modeled using our methodology for expressive music performance in jazz guitar., This project has received funding from: the European Union Horizon 2020 research and innovation programme [grant agreement No 688269]; the Spanish TIN project TIMUL [grant agreement TIN2013-48152-C2-2-R].
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Air violin: a machine learning approach to fingering gesture recognition

  • Dalmazzo, David
  • Ramírez, Rafael, 1966-
Comunicació presentada a: 1st ACM SIGCHI International Workshop on Multimodal Interaction for Education, celebrat el 13 de novembre de 2017 a Glasgow, Regne Unit., We train and evaluate two machine learning models for predicting fingering in violin performances using motion and EMG sensors integrated in the Myo device. Our aim is twofold: first, provide a fingering recognition model in the context of a gamification virtual violin application where we measure both right hand (i.e. bow) and left hand (i.e. fingering) gestures, and second, implement a tracking system for a computer assisted pedagogical tool for self-regulated learners in high-level music education. Our approach is based on the principle of mapping-by-demonstration in which the model is trained by the performer. We evaluated a model based on Decision Trees and compared it with a Hidden Markovian Model., This work has been partly sponsored by the Spanish TIN project TIMUL (TIN 2013-48152-C2-2-R), the European Union Horizon 2020 research and innovation programme under grant agreement No. 688269 (TELMI project), and the Spanish Ministry of Economy and Competitiveness under the Maria de Maeztu Units of Excellence Programme (MDM-2015-0502).
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Evaluation of a sound quality visual feedback system for bow learning technique in violin beginners: an EEG study

  • Blanco, Angel David
  • Ramírez, Rafael, 1966-
Current music technologies can assist in the process of learning to play a musical instrument and provide objective measures for evaluating the improvement of music students in concrete music tasks. In this paper, we investigated the effects of a sound quality visual feedback system (SQVFS) in violin learning. In particular, we studied the EEG activity of a group of participants with no previous violin playing experience while they learned to produce a stable sound (regarding pitch, dynamics, and timbre) in order to find motor learning biomarkers in a music task. Eighteen subjects with no prior experience in violin playing were divided into two groups: participants in the first group (experimental group, N = 9) practiced with instructional videos and offline feedback from the SQVFS provided in alternation with their performance, while participants in a second group (control group, N = 9) practiced with the instructional videos only. A third group of violin experts (players with more than 6 years of experience) performed the same task for comparative purposes (N = 7). All participants were asked to perform 20 trials (4 blocks of 5 trials) consisting of a violin bowing exercise while their EEG activity and their produced sound was recorded. Significant sound quality improvements along the session were found in all participants with the exception of participants in the expert group. In addition, participants in the experimental group showed increased interest in the learning process and significant improvement after the second block not present in the control group. A significant correlation between the levels of frontal gamma band power and the sound improvement along the task was found in both the experimental and control group. This result is consistent with the temporal binding model which associates gamma band power with the role of integrating (binding) information processed in distributed cortical areas. Task complexity demands more cognitive resources, more binding and thus, gamma band power enhancement, which may be reduced as the demanded task begins to be automated as it is likely to be the case in both beginners groups., This work was partly sponsored by Fundación Memora, the Spanish TIN project TIMUL (TIN2013-48152-C2-2-R), and the European Union Horizon 2020 research and innovation program under grant agreement No. 688269 (TELMI project).
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Applying deep learning techniques to estimate patterns of musical gesture

  • Dalmazzo, David
  • Waddell, George
  • Ramírez, Rafael, 1966-
Repetitive practice is one of the most important factors in improving the performance of motor skills. This paper focuses on the analysis and classification of forearm gestures in the context of violin playing. We recorded five experts and three students performing eight traditional classical violin bow-strokes: martelé, staccato, detaché, ricochet, legato, trémolo, collé, and col legno. To record inertial motion information, we utilized the Myo sensor, which reports a multidimensional time-series signal. We synchronized inertial motion recordings with audio data to extract the spatiotemporal dynamics of each gesture. Applying state-of-the-art deep neural networks, we implemented and compared different architectures where convolutional neural networks (CNN) models demonstrated recognition rates of 97.147%, 3DMultiHeaded_CNN models showed rates of 98.553%, and rates of 99.234% were demonstrated by CNN_LSTM models. The collected data (quaternion of the bowing arm of a violinist) contained sufficient information to distinguish the bowing techniques studied, and deep learning methods were capable of learning the movement patterns that distinguish these techniques. Each of the learning algorithms investigated (CNN, 3DMultiHeaded_CNN, and CNN_LSTM) produced high classification accuracies which supported the feasibility of training classifiers. The resulting classifiers may provide the foundation of a digital assistant to enhance musicians' time spent practicing alone, providing real-time feedback on the accuracy and consistency of their musical gestures in performance., This work has been partly sponsored by the Spanish TIN project TIMUL (TIN 2013-48152-C2-2-R), the European Union Horizon 2020 research and innovation program under grant agreement No. 688269 (TELMI project), and the Spanish Ministry of Economy and Competitiveness under the Maria de Maeztu Units of Excellence Programme (MDM-2015-0502).
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Machine learning and music generation

  • Iñesta, José M.
  • Conklin, Darrell
  • Ramírez, Rafael, 1966-
We guest editors acknowledge the financial support of the following funding sources: the Ministerio de Economía y Competitividad project TIMuL [No. TIN2013–48152–C2–1–R]; [No. TIN2013–48152–C2–2–R, supported by UE FEDER funds]; the project Lrn2Cre8, which is funded by the Future and Emerging Technologies (FET) programme within the Seventh Framework Programme for Research of the European Commission [FET grant number 610859]; the European Union Horizon 2020 research and innovation programme [grant agreement number 688269].
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Mapping between dynamic markings and performed loudness: a machine learning approach

  • Kosta, Katerina
  • Ramírez, Rafael, 1966-
  • Bandtlow, Oscar F.
  • Chew, Elaine
Loudness variation is one of the foremost tools for expressivity in music performance. Loudness is frequently notated as dynamic markings such as p (piano, meaning soft) or f (forte, meaning loud). While dynamic markings in music scores are important indicators of how music pieces should be interpreted, their meaning is less straightforward than it may seem, and depends highly on the context in which they appear. In this article, we investigate the relationship between dynamic markings in the score and performed loudness by applying machine learning techniques – decision trees, support vector machines, artificial neural networks, and a k-nearest neighbor method – to the prediction of loudness levels corresponding to dynamic markings, and to the classification of dynamic markings given loudness values. The methods are applied to 44 recordings of performances of Chopin's Mazurkas, each by 8 pianists. The results show that loudness values and markings can be predicted relatively well when trained across recordings of the same piece, but fail dismally when trained across the pianist's recordings of other pieces, demonstrating that score features may trump individual style when modeling loudness choices. Evidence suggests that all the features chosen for the task are relevant, and analysis of the results reveals the forms (such as the return of the theme) and structures (such as dynamic-marking repetitions) that influence the predictability of loudness and markings. Modeling of loudness trends in expressive performance appears to be a delicate matter, and sometimes loudness expression can be a matter of the performer's idiosyncracy., This work was supported in part by a UK EPSRC Platform Grant for Digital Music [EP/K009559/1]; the Spanish TIN project TIMUL [TIN2013-48152-C2-2-R]; and the European Union's Horizon 2020 research and innovation programme [grant agreement No. 688269].
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Bowing gestures classification in violin performance: a machine learning approach

  • Dalmazzo, David
  • Ramírez, Rafael, 1966-
Gestures in music are of paramount importance partly because they are directly linked to musicians' sound and expressiveness. At the same time, current motion capture technologies are capable of detecting body motion/gestures details very accurately. We present a machine learning approach to automatic violin bow gesture classification based on Hierarchical Hidden Markov Models (HHMM) and motion data. We recorded motion and audio data corresponding to seven representative bow techniques (Détaché, Martelé, Spiccato, Ricochet, Sautillé, Staccato, and Bariolage) performed by a professional violin player. We used the commercial Myo device for recording inertial motion information from the right forearm and synchronized it with audio recordings. Data was uploaded into an online public repository. After extracting features from both the motion and audio data, we trained an HHMM to identify the different bowing techniques automatically. Our model can determine the studied bowing techniques with over 94% accuracy. The results make feasible the application of this work in a practical learning scenario, where violin students can benefit from the real-time feedback provided by the system., This work has been partly sponsored by the Spanish TIN project TIMUL (TIN 2013-48152-C2-2-R), the European Union Horizon 2020 research and innovation programme under grant agreement No. 688269 (TELMI project), and the Spanish Ministry of Economy and Competitiveness under the Maria de Maeztu Units of Excellence Programme (MDM-2015-0502).
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Musical neurofeedback for treating depression in elderly people

  • Ramírez, Rafael, 1966-
  • Palencia-Lefler Ors, Manuel
  • Giraldo, Sergio
  • Vamvakousis, Zacharias
We introduce a new neurofeedback approach, which allows users to manipulate expressive parameters in music performances using their emotional state, and we present the results of a pilot clinical experiment applying the approach to alleviate depression in elderly people. Ten adults (9 female and 1 male, mean = 84, SD = 5.8) with normal hearing participated in the neurofeedback study consisting of 10 sessions (2 sessions per week) of 15 min each. EEG data was acquired using the Emotiv EPOC EEG device. In all sessions, subjects were asked to sit in a comfortable chair facing two loudspeakers, to close their eyes, and to avoid moving during the experiment. Participants listened to music pieces preselected according to their music preferences, and were encouraged to increase the loudness and tempo of the pieces, based on their arousal and valence levels. The neurofeedback system was tuned so that increased arousal, computed as beta to alpha activity ratio in the frontal cortex corresponded to increased loudness, and increased valence, computed as relative frontal alpha activity in the right lobe compared to the left lobe, corresponded to increased tempo. Pre and post evaluation of six participants was performed using the BDI depression test, showing an average improvement of 17.2% (1.3) in their BDI scores at the end of the study. In addition, an analysis of the collected EEG data of the participants showed a significant decrease of relative alpha activity in their left frontal lobe (p = 0.00008), which may be interpreted as an improvement of their depression condition., This work has been partly sponsored by the Ministerio de Economia y Competitividad under Grant TIN2013-48152-C2-2-R (TIMuL Project).
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

A Machine learning approach to discover rules for expressive performance actions in jazz guitar music

  • Giraldo, Sergio
  • Ramírez, Rafael, 1966-
Expert musicians introduce expression in their performances by manipulating sound properties such as timing, energy, pitch, and timbre. Here, we present a data driven computational approach to induce expressive performance rule models for note duration, onset, energy, and ornamentation transformations in jazz guitar music. We extract high-level features from a set of 16 commercial audio recordings (and corresponding music scores) of jazz guitarist Grant Green in order to characterize the expression in the pieces. We apply machine learning techniques to the resulting features to learn expressive performance rule models. We (1) quantitatively evaluate the accuracy of the induced models, (2) analyse the relative importance of the considered musical features, (3) discuss some of the learnt expressive performance rules in the context of previous work, and (4) assess their generailty. The accuracies of the induced predictive models is significantly above base-line levels indicating that the audio performances and the musical features extracted contain sufficient information to automatically learn informative expressive performance patterns. Feature analysis shows that the most important musical features for predicting expressive transformations are note duration, pitch, metrical strength, phrase position, Narmour structure, and tempo and key of the piece. Similarities and differences between the induced expressive rules and the rules reported in the literature were found. Differences may be due to the fact that most previously studied performance data has consisted of classical music recordings. Finally, the rules' performer specificity/generality is assessed by applying the induced rules to performances of the same pieces performed by two other professional jazz guitar players. Results show a consistency in the ornamentation patterns between Grant Green and the other two musicians, which may be interpreted as a good indicator for generality of the ornamentation rules., This work has been partly sponsored by the Spanish TIN project TIMUL (TIN2013-48152-C2-2-R), and the European Union Horizon 2020 research and innovation programme under grant agreement No. 688269 (TELMI project).
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Real-time sound and motion feedback for violin bow technique learning: a controlled, randomized trial

  • Blanco, Angel David
  • Tassani, Simone
  • Ramírez, Rafael, 1966-
The production of good sound generation in the violin is a complex task that requires coordination and spatiotemporal control of bowing gestures. The use of motion-capture technologies to improve performance or reduce injury risks in the area of kinesiology is becoming widespread. The combination of motion accuracy and sound quality feedback has the potential of becoming an important aid in violin learning. In this study, we evaluate motion-capture and sound-quality analysis technologies developed inside the context of the TELMI, a technology-enhanced music learning project. We analyzed the sound and bow motion of 50 participants with no prior violin experience while learning to produce a stable sound in the violin. Participants were divided into two groups: the experimental group (N = 24) received real-time visual feedback both on kinematics and sound quality, while participants in the control group (N = 26) practiced without any type of external help. An additional third group of violin experts performed the same task for comparative purposes (N = 15). After the practice session, all groups were evaluated in a transfer phase without feedback. At the practice phase, the experimental group improved their bowing kinematics in comparison to the control group, but this was at the expense of impairing the sound quality of their performance. At the retention phase, the experimental group showed better results in sound quality, especially concerning control of sound dynamics. Besides, we found that the expert group improved the stability of their sound while using the technology. All in all, these results emphasize the importance of feedback technologies in learning complex tasks, such as musical instrument learning., This work was partly sponsored by Fundación Memora, the Spanish TIN project TIMUL (TIN2013-48152-C2-2-R), and the European Union Horizon 2020 research and innovation program under grant agreement No. 688269 (TELMI project).
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

The EyeHarp: A gaze-controlled digital musical instrument

  • Vamvakousis, Zacharias
  • Ramírez, Rafael, 1966-
We present and evaluate the EyeHarp, a new gaze-controlled Digital Musical Instrument, which aims to enable people with severe motor disabilities to learn, perform, and compose music using only their gaze as control mechanism. It consists of (1) a step-sequencer layer, which serves for constructing chords/arpeggios, and (2) a melody layer, for playing melodies and changing the chords/arpeggios. We have conducted a pilot evaluation of the EyeHarp involving 39 participants with no disabilities from both a performer and an audience perspective. In the first case, eight people with normal vision and no motor disability participated in a music-playing session in which both quantitative and qualitative data were collected. In the second case 31 people qualitatively evaluated the EyeHarp in a concert setting consisting of two parts: a solo performance part, and an ensemble (EyeHarp, two guitars, and flute) performance part. The obtained results indicate that, similarly to traditional music instruments, the proposed digital musical instrument has a steep learning curve, and allows to produce expressive performances both from the performer and audience perspective., This project has received funding from the European Unions Horizon 2020 research and innovation program under grant agreement No. 688269, as well as from the Spanish TIN project TIMUL under grant agreement TIN2013-48152-C2-2-R.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

EEG-Based Analysis of the Emotional Effect of Music Therapy on Palliative Care Cancer Patients

  • Ramírez, Rafael, 1966-
  • Planas Domingo, Josep
  • Escudé, Nuria
  • Mercadé, Jordi
  • Farriols, Cristina
Music is known to have the power to induce strong emotions. The present study assessed, based on Electroencephalography (EEG) data, the emotional response of terminally ill cancer patients to a music therapy intervention in a randomized controlled trial. A sample of 40 participants from the palliative care unit in the Hospital del Mar in Barcelona was randomly assigned to two groups of 20. The first group [experimental group (EG)] participated in a session of music therapy (MT), and the second group [control group (CG)] was provided with company. Based on our previous work on EEG-based emotion detection, instantaneous emotional indicators in the form of a coordinate in the arousal-valence plane were extracted from the participants' EEG data. The emotional indicators were analyzed in order to quantify (1) the overall emotional effect of MT on the patients compared to controls, and (2) the relative effect of the different MT techniques applied during each session. During each MT session, five conditions were considered: I (initial patient's state before MT starts), C1 (passive listening), C2 (active listening), R (relaxation), and F (final patient's state). EEG data analysis showed a significant increase in valence (p = 0.0004) and arousal (p = 0.003) between I and F in the EG. No significant changes were found in the CG. This results can be interpreted as a positive emotional effect of MT in advanced cancer patients. In addition, according to pre- and post-intervention questionnaire responses, participants in the EG also showed a significant decrease in tiredness, anxiety and breathing difficulties, as well as an increase in levels of well-being. No equivalent changes were observed in the CG., This work was partly sponsored by Fundación Memora, the Spanish TIN project TIMUL (TIN2013-48152-C2-2-R), and the European Union Horizon 2020 research and innovation program under grant agreement No. 688269 (TELMI project).