Skip to main content

Resultados relacionados

Resultados totales (Incluyendo duplicados): 64
Encontrada(s) 1 página(s)

Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

Enhancing music learning with smart technologies

  • Ramírez, Rafael, 1966-
  • Canepa, Corrado
  • Ghisio, Simone
  • Kolykhalova, Ksenia
  • Mancini, Maurizio
  • Volta, Erica
  • Volpe, Gualtiero
  • Giraldo, Sergio
  • Mayor, Oscar
  • Perez, Alfonso
  • Waddell, George
  • Williamon, Aaron
Comunicació presentada a: 5th International Conference on Movement and Computing, celebrat del 28 al 30 de juny de 2018 a Gènova, Itàlia., Learning to play a musical instrument is a difficult task, requiring the development of sophisticated skills. Nowadays, such a learning process is mostly based on the master-apprentice model. Technologies are rarely employed and are usually restricted to audio and video recording and playback. The TELMI (Technology Enhanced Learning of Musical Instrument Performance) Project seeks to design and implement new interaction paradigms for music learning and training based on state-of-the-art multimodal (audio, image, video, and motion) technologies. Figure 1: Analysis of coordination, applying RecurrenceQuantification Analysis to the kinetic energy of the right wrist. Figure 2: A sample screen-shot of the intonation feedback on Piano Roll Mode. Figure 3: A sample screen-shot of the intonation feedback on Score View Mode. The project focuses on the violin as a case study. This practice work is intended as demo, showing to MOCO attendants the results the project obtained along two years of work. The demo simulates a setup at a higher education music institution, where attendants with any level of previous violin experience (and even with no experience at all) are invited to try the technologies themselves, performing basic tests of violin skill and pre-defined exercises under the guidance of the researchers involved in the project.
Repositorio Digital de la UPF
Publication Artículo científico (JournalArticle)

Phrase-Level modeling of expression in violin performances

  • Ortega, Fabio J. M.
  • Giraldo, Sergio
  • Pérez Carrillo, Alfonso Antonio, 1977-
  • Ramírez, Rafael, 1966-
Background: Expression is a key skill in music performance, and one that is difficult to address in music lessons. Computational models that learn from expert performances can help providing suggestions and feedback to students. Aim: We propose and analyze an approach to modeling variations in dynamics and note onset timing for solo violin pieces with the purpose of facilitating expressive performance learning in new pieces, for which no reference performance is available. Method: The method generates phrase–level predictions based on musical score information on the assumption that expressiveness is idiomatic, and thus influenced by similar–sounding melodies. Predictions were evaluated numerically using three different datasets and against note–level machine–learning models, and also perceptually by listeners, who were presented to synthesized versions of musical excerpts, and asked to choose the most human–sounding one. Some of the presented excerpts were synthesized to reflect the variations in dynamics and timing predicted by the model, whereas others were shaped to reflect the dynamics and timing of an actual expert performance, and a third group was presented with no expressive variations. Results: surprisingly, none of the three synthesized versions was consistently selected as human–like nor preferred with statistical significance by listeners. Possible interpretations of these results include the fact that the melodies might have been impossible to interpret outside their musical context, or that expressive features that were left out of the modeling such as note articulation and vibrato are, in fact, essential to the perception of expression in violin performance. Positive feedback by some listeners toward the modeled melodies in a blind setting indicate that the modeling approach was capable of generating appropriate renditions at least for a subset of the data. Numerically, performance in phrase–level suffers a small degradation if compared to note–level, but produces predictions easier to interpret visually, thus more useful in a pedagogical setting., This work has been partly sponsored by the Spanish TIN project TIMUL (TIN 2013-48152-C2-2-R), the European Union Horizon 2020 research and innovation programme under grant agreement No. 688269 (TELMI project), and the Spanish Ministry of Economy and Competitiveness under the Maria de Maeztu Units of Excellence Programme (MDM-2015-0502).
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

A Real-time feedback learning tool to visualize sound quality in violin performances

  • Giraldo, Sergio
  • Ramírez, Rafael, 1966-
  • Waddell, George
  • Williamon, Aaron
Comunicació presentada a: 10th International Workshop on Machine Learning and Music (MML), celebrat a Barcelona (Espanya), el 6 d'octubre de 2017., The assessment of the sound properties of a performed mu- sical note has been widely studied in the past. Although a consensus exist on what is a good or a bad musical performance, there is not a formal de nition of performance tone quality due to its subjectivity. In this study we present a computational approach for the automatic assess- ment of violin sound production. We investigate the correlations among extracted features from audio performances and the perceptual quality of violin sounds rated by listeners using machine learning techniques. The obtained models are used for implementing a real-time feedback learning system., This work has been partly sponsored by the Spanish TIN project TIMUL (TIN2013-48152-C2-2-R), the European Union Horizon 2020 research and inno- vation programme under grant agreement No. 688269 (TELMI project), and the Spanish Ministry of Economy and Competitiveness under the Maria de Maeztu Units of Excellence Programme (MDM-2015-0502).
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

Acquisition of violin instrumental gestures using an infrared depth camera

  • Vamvakousis, Zacharias
  • Perez, Alfonso
  • Ramírez, Rafael, 1966-
Comunicació presentada a: 15th Sound and Music Computing Conference (SMC2018), celebrat del 4 al 7 de juliol a Limassol, Xipre., We present a low-cost six-degrees of freedom violin and bow pose-tracking system. The infrared and depth streams of an infrared-based Microsoft Kinect Mbox One depth camera are utilized to track the 3-dimensional position of 6 infrared markers attached on the violin and the bow. After computing the bow pose on the violin coordinate system, a number of bowing features are extracted. In order to evaluate the system’s performance, we recorded 4 bowing exercises using simultaneously our system along with a commercial two-sensor 3D tracking system based on electromagnetic field (EMF) sensing. The mean pearson coefficient values were 0.996 for the bow position, 0.889 for the bow velocity, 0.966 for the bow tilt, 0.692 for the bowbridge distance and 0.998 for the bow inclination. Compared to existing bowing-tracking solutions, the proposed solution might be appealing because of its low cost, easy setup and high performance., This work was partly sponsored by the Spanish TIN project TIMUL (TIN2013-48152-C2-2-R), and the European Union Horizon 2020 research and innovation program under grant agreement No. 688269 (TELMI project).
Repositorio Digital de la UPF
Publication Artículo científico (JournalArticle)

A Machine learning approach to ornamentation modeling and synthesis in jazz guitar

  • Ramírez, Rafael, 1966-
  • Giraldo, Sergio
We present a machine learning approach to automatically generate expressive (ornamented) jazz performances from un-expressive music scores. Features extracted from the scores and the corresponding audio recordings performed by a professional guitarist were used to train computational models for predicting melody ornamentation. As a first step, several machine learning techniques were explored to induce regression models for timing, onset, and dynamics (i.e. note duration and energy) transformations, and an ornamentation model for classifying notes as ornamented or non-ornamented. In a second step, the most suitable ornament for predicted ornamented notes was selected based on note context similarity. Finally, concatenative synthesis was used to automatically synthesize expressive performances of new pieces using the induced models. Supplemental online material for this article containing musical examples of the automatically generated ornamented pieces can be accessed at doi: 10.1080/17459737.2016.1207814 and https://soundcloud.com/machine-learning-and-jazz. In the Online Supplement we present an example of the musical piece Yesterdays by Jerome Kern, which was modeled using our methodology for expressive music performance in jazz guitar., This project has received funding from: the European Union Horizon 2020 research and innovation programme [grant agreement No 688269]; the Spanish TIN project TIMUL [grant agreement TIN2013-48152-C2-2-R].
Repositorio Digital de la UPF
Publication Artículo científico (JournalArticle)

Evaluation of a sound quality visual feedback system for bow learning technique in violin beginners: an EEG study

  • Blanco, Angel David
  • Ramírez, Rafael, 1966-
Current music technologies can assist in the process of learning to play a musical instrument and provide objective measures for evaluating the improvement of music students in concrete music tasks. In this paper, we investigated the effects of a sound quality visual feedback system (SQVFS) in violin learning. In particular, we studied the EEG activity of a group of participants with no previous violin playing experience while they learned to produce a stable sound (regarding pitch, dynamics, and timbre) in order to find motor learning biomarkers in a music task. Eighteen subjects with no prior experience in violin playing were divided into two groups: participants in the first group (experimental group, N = 9) practiced with instructional videos and offline feedback from the SQVFS provided in alternation with their performance, while participants in a second group (control group, N = 9) practiced with the instructional videos only. A third group of violin experts (players with more than 6 years of experience) performed the same task for comparative purposes (N = 7). All participants were asked to perform 20 trials (4 blocks of 5 trials) consisting of a violin bowing exercise while their EEG activity and their produced sound was recorded. Significant sound quality improvements along the session were found in all participants with the exception of participants in the expert group. In addition, participants in the experimental group showed increased interest in the learning process and significant improvement after the second block not present in the control group. A significant correlation between the levels of frontal gamma band power and the sound improvement along the task was found in both the experimental and control group. This result is consistent with the temporal binding model which associates gamma band power with the role of integrating (binding) information processed in distributed cortical areas. Task complexity demands more cognitive resources, more binding and thus, gamma band power enhancement, which may be reduced as the demanded task begins to be automated as it is likely to be the case in both beginners groups., This work was partly sponsored by Fundación Memora, the Spanish TIN project TIMUL (TIN2013-48152-C2-2-R), and the European Union Horizon 2020 research and innovation program under grant agreement No. 688269 (TELMI project).
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

JamSketch: improvisation support system with GA-based melody creation from user’s drawing

  • Kitahara, Tetsuro
  • Giraldo, Sergio
  • Ramírez, Rafael, 1966-
Comunicació presentada al 13th International Symposium (CMMR), celebrat del 25 al 28 de setembre de 2017 a Matosinhos, Portugal., Improvisation is an enjoyable form of music performance but requires advanced skills and knowledge of music because the player has to create melodies immediately during the performance. To support improvisations by people without skills or knowledge of music, we have to develop (1) a human interface that can be used without skills or knowledge of music and (2) automatic melody generation from the user’s input that may be musically abstract or incomplete. In this paper, we develop an improvisation support system based on melodic outlines, which represent the overall contour of melodies, with a function of melody generation using a genetic algorithm (GA). Once the user draws a melodic outline on the piano-roll display with the mouse or touch screen, the system immediately generates a melody using a GA with a fitness function based on the similarity to the outline, an N-gram probability, and entropy. The generated melody is performed expressively based on expression parameters calculated with an machine learning approach. The results of listening tests for comparing human performances and the system’s performances suggest that generated melodies have quality similar to performances by non-expert human performers., This work was supported by JSPS KAKENHI Grant Numbers 26240025, 26280089, 16K16180, 16H01744, 16KT0136, and 17H00749 as well as by the Spanish TIMuL Project (TIN2013-48152-C2-2-R), and the TELMI Project of the Horizon 2020 Research and Innovation Programme (grant agreement No. 688269).
Repositorio Digital de la UPF
Publication Artículo científico (JournalArticle)

Applying deep learning techniques to estimate patterns of musical gesture

  • Dalmazzo, David
  • Waddell, George
  • Ramírez, Rafael, 1966-
Repetitive practice is one of the most important factors in improving the performance of motor skills. This paper focuses on the analysis and classification of forearm gestures in the context of violin playing. We recorded five experts and three students performing eight traditional classical violin bow-strokes: martelé, staccato, detaché, ricochet, legato, trémolo, collé, and col legno. To record inertial motion information, we utilized the Myo sensor, which reports a multidimensional time-series signal. We synchronized inertial motion recordings with audio data to extract the spatiotemporal dynamics of each gesture. Applying state-of-the-art deep neural networks, we implemented and compared different architectures where convolutional neural networks (CNN) models demonstrated recognition rates of 97.147%, 3DMultiHeaded_CNN models showed rates of 98.553%, and rates of 99.234% were demonstrated by CNN_LSTM models. The collected data (quaternion of the bowing arm of a violinist) contained sufficient information to distinguish the bowing techniques studied, and deep learning methods were capable of learning the movement patterns that distinguish these techniques. Each of the learning algorithms investigated (CNN, 3DMultiHeaded_CNN, and CNN_LSTM) produced high classification accuracies which supported the feasibility of training classifiers. The resulting classifiers may provide the foundation of a digital assistant to enhance musicians' time spent practicing alone, providing real-time feedback on the accuracy and consistency of their musical gestures in performance., This work has been partly sponsored by the Spanish TIN project TIMUL (TIN 2013-48152-C2-2-R), the European Union Horizon 2020 research and innovation program under grant agreement No. 688269 (TELMI project), and the Spanish Ministry of Economy and Competitiveness under the Maria de Maeztu Units of Excellence Programme (MDM-2015-0502).
Repositorio Digital de la UPF
Publication Artículo científico (JournalArticle)

EEG-Based Analysis of the Emotional Effect of Music Therapy on Palliative Care Cancer Patients

  • Ramírez, Rafael, 1966-
  • Planas Domingo, Josep
  • Escudé, Nuria
  • Mercadé, Jordi
  • Farriols, Cristina
Music is known to have the power to induce strong emotions. The present study assessed, based on Electroencephalography (EEG) data, the emotional response of terminally ill cancer patients to a music therapy intervention in a randomized controlled trial. A sample of 40 participants from the palliative care unit in the Hospital del Mar in Barcelona was randomly assigned to two groups of 20. The first group [experimental group (EG)] participated in a session of music therapy (MT), and the second group [control group (CG)] was provided with company. Based on our previous work on EEG-based emotion detection, instantaneous emotional indicators in the form of a coordinate in the arousal-valence plane were extracted from the participants' EEG data. The emotional indicators were analyzed in order to quantify (1) the overall emotional effect of MT on the patients compared to controls, and (2) the relative effect of the different MT techniques applied during each session. During each MT session, five conditions were considered: I (initial patient's state before MT starts), C1 (passive listening), C2 (active listening), R (relaxation), and F (final patient's state). EEG data analysis showed a significant increase in valence (p = 0.0004) and arousal (p = 0.003) between I and F in the EG. No significant changes were found in the CG. This results can be interpreted as a positive emotional effect of MT in advanced cancer patients. In addition, according to pre- and post-intervention questionnaire responses, participants in the EG also showed a significant decrease in tiredness, anxiety and breathing difficulties, as well as an increase in levels of well-being. No equivalent changes were observed in the CG., This work was partly sponsored by Fundación Memora, the Spanish TIN project TIMUL (TIN2013-48152-C2-2-R), and the European Union Horizon 2020 research and innovation program under grant agreement No. 688269 (TELMI project).
Repositorio Digital de la UPF
Publication Artículo científico (JournalArticle)

Machine learning and music generation

  • Iñesta, José M.
  • Conklin, Darrell
  • Ramírez, Rafael, 1966-
We guest editors acknowledge the financial support of the following funding sources: the Ministerio de Economía y Competitividad project TIMuL [No. TIN2013–48152–C2–1–R]; [No. TIN2013–48152–C2–2–R, supported by UE FEDER funds]; the project Lrn2Cre8, which is funded by the Future and Emerging Technologies (FET) programme within the Seventh Framework Programme for Research of the European Commission [FET grant number 610859]; the European Union Horizon 2020 research and innovation programme [grant agreement number 688269].
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

Computational modelling of timbre dimensions for automatic violin tone quality assessment

  • Giraldo, Sergio
  • Ramírez, Rafael, 1966-
  • Waddell, George
  • Williamon, Aaron
Automatic assessment of music performance is an open research area widely studied in the past. A vast amount of systems aiming to enhance the learning process of a musical instrument are being developed in the recent years. However, most of the systems focus on the assessment of pitch and onset accuracy, and very few pay attention to tone quality. This is particularly true in violin music education, where although a consensus exist on what is a good or a bad tone quality, there is not a formal definition due to its subjectivity. We present a machine learning approach for the automatic assessment of violin tone quality. We depart from our previous work on the preliminary modelling of several dimensions involving tone quality. Based on recorded examples of tones with different qualities defined and recorded by a professional violinist, we applied machine learning techniques to learn computational models able to evaluate tone quality from extracted audio features. The tone quality models were implemented into a real-time-visual-feedback system., This work has been partly sponsored by the Spanish TIN project TIMUL (TIN2013-48152-C2-2-R), the European Union Horizon 2020 research and innovation programme under grant agreement No. 688269 (TELMI project), and the Spanish Ministry of Economy and Competitiveness under the Maria de Maeztu Units of Excellence Programme (MDM-2015-0502).
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

JamSketch: a drawing-based real-time evolutionary improvisation support system

  • Kitahara, Tetsuro
  • Giraldo, Sergio
  • Ramírez, Rafael, 1966-
Comunicació presentada a: International Conference on New Interfaces for Musical Expression celebrat del 15 al 19 de maig de 2017 a Copenhagen, Dinamarca., In this paper, we present JamSketch, a real-time improvisation support system which automatically generates melodies according to melodic outlines drawn by the users. The system generates the improvised melodies based on (1) an outline sketched by the user using a mouse or a touch screen, (2) a genetic algorithm based on a dataset of existing music pieces as well as musical knowledge, and (3) an expressive performance model for timing and dynamic transformations. The aim of the system is to allow people with no prior musical knowledge to be able to enjoy playing music by improvising melodies in real time., This work was supported by JSPS KAKENHI Grant Numbers 26240025, 26280089, 16K16180, 16H01744, and 16KT0136, as well as by the Spanish TIMuL Project (TIN2013-48152-C2-2-R), and the TELMI Project of the Horizon 2020 Research and Innovation Programme (grant agreement No. 688269).
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

A Machine learning approach to violin bow technique classification: a comparison between IMU and MOCAP systems

  • Dalmazzo, David
  • Tassani, Simone
  • Ramírez, Rafael, 1966-
Comunicació presentada a: 5th international Workshop on Sensor-based Activity Recognition and Interaction celebrat el 20 i 21 de setembre de 2018 a Berlin, Alemanya., Motion Capture (MOCAP) Systems have been used to analyze body motion and postures in biomedicine, sports, rehabilitation, and music. With the aim to compare the precision of low-cost devices for motion tracking (e.g. Myo) with the precision of MOCAP systems in the context of music performance, we recorded MOCAP and Myo data of a top professional violinist executing four fundamental bowing techniques (i.e. Détaché, Martelé, Spiccato and Ricochet). Using the recorded data we applied machine learning techniques to train models to classify the four bowing techniques. Despite intrinsic differences between the MOCAP and low-cost data, the Myo-based classifier resulted in slightly higher accuracy than the MOCAP-based classifier. This result shows that it is possible to develop music-gesture learning applications based on low-cost technology which can be used in home environments for self-learning practitioners., This work has been partly sponsored by the Spanish TIN project TIMUL (TIN 2013-48152-C2-2-R), the European Union Horizon 2020 research and innovation programme under grant agreement No. 688269 (TELMI project), and the Spanish Ministry of Economy and Competitiveness under the Maria de Maeztu Units of Excellence Programme (MDM-2015-0502).
Repositorio Digital de la UPF
Publication Artículo científico (JournalArticle)

The EyeHarp: A gaze-controlled digital musical instrument

  • Vamvakousis, Zacharias
  • Ramírez, Rafael, 1966-
We present and evaluate the EyeHarp, a new gaze-controlled Digital Musical Instrument, which aims to enable people with severe motor disabilities to learn, perform, and compose music using only their gaze as control mechanism. It consists of (1) a step-sequencer layer, which serves for constructing chords/arpeggios, and (2) a melody layer, for playing melodies and changing the chords/arpeggios. We have conducted a pilot evaluation of the EyeHarp involving 39 participants with no disabilities from both a performer and an audience perspective. In the first case, eight people with normal vision and no motor disability participated in a music-playing session in which both quantitative and qualitative data were collected. In the second case 31 people qualitatively evaluated the EyeHarp in a concert setting consisting of two parts: a solo performance part, and an ensemble (EyeHarp, two guitars, and flute) performance part. The obtained results indicate that, similarly to traditional music instruments, the proposed digital musical instrument has a steep learning curve, and allows to produce expressive performances both from the performer and audience perspective., This project has received funding from the European Unions Horizon 2020 research and innovation program under grant agreement No. 688269, as well as from the Spanish TIN project TIMUL under grant agreement TIN2013-48152-C2-2-R.
Repositorio Digital de la UPF
Publication Artículo científico (JournalArticle)

Mapping between dynamic markings and performed loudness: a machine learning approach

  • Kosta, Katerina
  • Ramírez, Rafael, 1966-
  • Bandtlow, Oscar F.
  • Chew, Elaine
Loudness variation is one of the foremost tools for expressivity in music performance. Loudness is frequently notated as dynamic markings such as p (piano, meaning soft) or f (forte, meaning loud). While dynamic markings in music scores are important indicators of how music pieces should be interpreted, their meaning is less straightforward than it may seem, and depends highly on the context in which they appear. In this article, we investigate the relationship between dynamic markings in the score and performed loudness by applying machine learning techniques – decision trees, support vector machines, artificial neural networks, and a k-nearest neighbor method – to the prediction of loudness levels corresponding to dynamic markings, and to the classification of dynamic markings given loudness values. The methods are applied to 44 recordings of performances of Chopin's Mazurkas, each by 8 pianists. The results show that loudness values and markings can be predicted relatively well when trained across recordings of the same piece, but fail dismally when trained across the pianist's recordings of other pieces, demonstrating that score features may trump individual style when modeling loudness choices. Evidence suggests that all the features chosen for the task are relevant, and analysis of the results reveals the forms (such as the return of the theme) and structures (such as dynamic-marking repetitions) that influence the predictability of loudness and markings. Modeling of loudness trends in expressive performance appears to be a delicate matter, and sometimes loudness expression can be a matter of the performer's idiosyncracy., This work was supported in part by a UK EPSRC Platform Grant for Digital Music [EP/K009559/1]; the Spanish TIN project TIMUL [TIN2013-48152-C2-2-R]; and the European Union's Horizon 2020 research and innovation programme [grant agreement No. 688269].
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

Computational modelling of expressive music performance in hexaphonic guitar

  • Siquier, Marc
  • Giraldo, Sergio
  • Ramírez, Rafael, 1966-
Comunicació presentada a: 10th International Workshop on Machine Learning and Music (MML), celebrat a Barcelona (Espanya), el 6 d'octubre de 2017., Computational modelling of expressive music performance has been widely studied in the past. While previous work in this area has been mainly focused on classical piano music, there has been very little work on guitar music, and such work has focused on monophonic guitar playing. In this work, we present a machine learning approach to automatically generate expressive performances from non expressive music scores for polyphonic guitar. We treated guitar as an hexaphonic instrument, obtaining a polyphonic transcription of performed musical pieces. Features were extracted from the scores and performance actions were calculated from the deviations of the score and the performance. Machine learning techniques were used to train computational models to predict the aforementioned performance actions. Qualitative and quantitative evaluations of the models and the predicted pieces were performed., This work has been partly sponsored by the Spanish TIN project TIMUL (TIN2013-48152-C2-2-R), the European Union Horizon 2020 research and inno- vation programme under grant agreement No. 688269 (TELMI project), and the Spanish Ministry of Economy and Competitiveness under the Maria de Maeztu Units of Excellence Programme (MDM-2015-0502).
Repositorio Digital de la UPF
Publication Artículo científico (JournalArticle)

The creative drummer: an eeg-based pilot study on the correlates of emotions and creative drum playing

  • Ramírez, Rafael, 1966-
  • Reija, Xavier
It is reasonable to assume that emotional processes are involved in creative tasks and the generation of creative ideas. In this pilot study, we investigate the emotional correlates in professional drummers during different degrees of creative music playing. Ten participants performed three tasks: repetitive rhythmic drum playing, pattern-based improvisation, and attention-intensive free improvisation, while their EEG activity was recorded. Arousal and valence levels were estimated from the EEG data at baseline and for the three tasks. Results show significantly increased levels of valence (i.e., increased prefrontal right alpha power compared to prefrontal left alpha power) during pattern-based and free improvisation relative to baseline, and significantly increased levels of valence during free improvisation relative to pattern-based improvisation. These results seem to indicate that positive emotion (characterized as increased valence) is associated with the creation of original ideas in drum playing and that the freer the creative process, the greater the positive effect. The implication of these results may be of particular relevance in the fields of music-based therapeutic interventions and music pedagogy., This work was partly sponsored by the European Union Horizon 2020 research and innovation program under grant agreement No. 688269 (TELMI project) and by the Musical AI project—(PID2019-111403GB-I00/AEI/10.13039/501100011033).
Repositorio Digital de la UPF
Publication Artículo científico (JournalArticle)

Bowing gestures classification in violin performance: a machine learning approach

  • Dalmazzo, David
  • Ramírez, Rafael, 1966-
Gestures in music are of paramount importance partly because they are directly linked to musicians' sound and expressiveness. At the same time, current motion capture technologies are capable of detecting body motion/gestures details very accurately. We present a machine learning approach to automatic violin bow gesture classification based on Hierarchical Hidden Markov Models (HHMM) and motion data. We recorded motion and audio data corresponding to seven representative bow techniques (Détaché, Martelé, Spiccato, Ricochet, Sautillé, Staccato, and Bariolage) performed by a professional violin player. We used the commercial Myo device for recording inertial motion information from the right forearm and synchronized it with audio recordings. Data was uploaded into an online public repository. After extracting features from both the motion and audio data, we trained an HHMM to identify the different bowing techniques automatically. Our model can determine the studied bowing techniques with over 94% accuracy. The results make feasible the application of this work in a practical learning scenario, where violin students can benefit from the real-time feedback provided by the system., This work has been partly sponsored by the Spanish TIN project TIMUL (TIN 2013-48152-C2-2-R), the European Union Horizon 2020 research and innovation programme under grant agreement No. 688269 (TELMI project), and the Spanish Ministry of Economy and Competitiveness under the Maria de Maeztu Units of Excellence Programme (MDM-2015-0502).
Repositorio Digital de la UPF
Publication Artículo científico (JournalArticle)

A Machine learning approach to discover rules for expressive performance actions in jazz guitar music

  • Giraldo, Sergio
  • Ramírez, Rafael, 1966-
Expert musicians introduce expression in their performances by manipulating sound properties such as timing, energy, pitch, and timbre. Here, we present a data driven computational approach to induce expressive performance rule models for note duration, onset, energy, and ornamentation transformations in jazz guitar music. We extract high-level features from a set of 16 commercial audio recordings (and corresponding music scores) of jazz guitarist Grant Green in order to characterize the expression in the pieces. We apply machine learning techniques to the resulting features to learn expressive performance rule models. We (1) quantitatively evaluate the accuracy of the induced models, (2) analyse the relative importance of the considered musical features, (3) discuss some of the learnt expressive performance rules in the context of previous work, and (4) assess their generailty. The accuracies of the induced predictive models is significantly above base-line levels indicating that the audio performances and the musical features extracted contain sufficient information to automatically learn informative expressive performance patterns. Feature analysis shows that the most important musical features for predicting expressive transformations are note duration, pitch, metrical strength, phrase position, Narmour structure, and tempo and key of the piece. Similarities and differences between the induced expressive rules and the rules reported in the literature were found. Differences may be due to the fact that most previously studied performance data has consisted of classical music recordings. Finally, the rules' performer specificity/generality is assessed by applying the induced rules to performances of the same pieces performed by two other professional jazz guitar players. Results show a consistency in the ornamentation patterns between Grant Green and the other two musicians, which may be interpreted as a good indicator for generality of the ornamentation rules., This work has been partly sponsored by the Spanish TIN project TIMUL (TIN2013-48152-C2-2-R), and the European Union Horizon 2020 research and innovation programme under grant agreement No. 688269 (TELMI project).
Repositorio Digital de la UPF
Publication Artículo científico (JournalArticle)

Finger-string interaction analysis in guitar playing with optical motion capture

  • Pérez Carrillo, Alfonso Antonio, 1977-
We present a method for the analysis of the finger-string interaction in guitar performances and the computation of fine actions during the plucking gesture. The method is based on Motion Capture using high-speed cameras that can track the position of reflective markers placed on the guitar and fingers, in combination with audio analysis. A major problem inherent in optical motion capture is that of marker occlusion and, in guitar playing, it is the right hand of the guitarist that is extremely difficult to capture, especially during the plucking process, where the track of the markers at the fingertips is lost very frequently. This work presents two models that allow the reconstruction of the position of occluded markers: a rigid-body model to track the motion of the guitar strings and a flexible-body model to track the motion of the hands. In combination with audio analysis (onset and pitch detection), the method can estimate a comprehensive set of sound control features that include the plucked string, the plucking finger, and the characteristics of the plucking gesture in the phases of contact, pressure and release (e.g., position, timing, velocity, direction, or string displacement)., This work has been sponsored by the European Union Horizon 2020 research and innovation program under grant agreement No. 688269 (TELMI project) and Beatriu de Pinos grant 2010 BP-A 00209 by the Catalan Research Agency (AGAUR).
Repositorio Digital de la UPF
Publication Capítulo o parte de un libro (BookPart)

Technology enhanced learning of performance

  • Ramírez, Rafael, 1966-
  • Waddell, George
Technology pervades musical practice. It has driven the development of the instruments musicians play. It has ushered in an explosion of online music making, sharing, and collaboration (see Lisboa et al., this part, for a discussion of innovations in distance learning and performance). It dictates the ways musicians make their living, how they reach their audiences, and how their performances are consumed. It has fostered entire genres of musical expression and new multimodal experiences. However, despite this technological revolution, the way in which musicians learn to perform continues largely the same as it did centuries ago. Students still turn to expert teachers in one-to-one lessons for personalized assessment and guidance in a masterapprentice model that has proved highly resilient to change, interspersed with countless hours spent alone in practice (Creech & Gaunt, 2012; Gaunt, 2017). Over the decades, some digital tools have slowly made their way into common use in the practice room and teaching studio—metronomes, tuners, audio and video recording devices, digital scores—although the pace of innovation has not necessarily been characteristic of the technological explosion that has been seen in other areas of musical practice. This may be about to change. This chapter provides a window into the recent work that musicians, scientists, designers, and engineers are doing to build the next generation of technologies for music learning. While it does not attempt to summarize the full array of technologies available or in development (this would go well beyond the scope of one chapter and would quickly become outdated), it highlights the tools and techniques, from state-of-the-art measurement sensors to the latest in artificial intelligence, that are being brought to bear on the challenges of music learning and what they mean for musicians. Finally, it gives an optimistic glimpse at what musicians might expect to see in the coming decades, and how they can prepare themselves and their students to be technologically literate and adventurous in pushing the boundaries of their practice., The research reported in this chapter was supported by TELMI (Technology Enhanced Learning of Musical Instrument Performance), a project funded by the European Commission’s Horizon 2020 research and innovation program (grant no. 688269)
Repositorio Digital de la UPF
Publication Artículo científico (JournalArticle)

Error monitoring and correction in violin performance: an EEG study

  • Blanco, Angel David
  • Costa-Faidella, Jordi
  • Pérez Carrillo, Alfonso Antonio, 1977-
  • Dalmazzo, David
  • Ramirez Melendez, Rafael
  • San Miguel Insúa, Iria
Previous research using electroencephalography (EEG) has investigated neural error-related music production processes in expert pianists reporting a frontal event-related negativity followed by a P300 event-related potential (ERP) after mistuned notes. However, piano playing does not rely on auditory feedback and corrective movements when dealing with accurate note tuning, thus offering an incomplete picture about error monitoring processes in players of many other musical instruments, including the human voice. We here designed a setup allowing us to collect the EEG of expert bow-string players while performing short melodies on the violin, (a fretless instrument) while also manipulating in real-time the produced pitch in a controlled manner. 15 expert bow-string players were asked to 1) listen to a reference melody of four notes; 2) play that melody with a violin (Active Condition); and 3) listen passively to the replay of their performance (Replayed Condition). Randomly, the auditory feedback of one of the four melody notes was manipulated in the Active Condition by lowering or lifting the pitch by half semitone. We found that mistuned notes, independently of whether they were manipulated externally or were produced by natural playing errors, elicited an f-ERN followed by a P300. We also found that the amplitude of the f-ERN was larger for faster corrective movements than for slower ones. In addition, between the f-ERN and the P300, we also identified two negative components-one in parietal areas and another in central-right regions-whose amplitudes were greater for slower corrective movements. We suggest the existence of two error-monitoring systems: a "fast" system mediated by the medial frontal cortex that quickly corrects errors when certainty is high and a "slow" system mediated by the posterior parietal cortex that accumulates evidence under uncertainty before triggering correction., This work was partly sponsored by the Musical AI project (PID2019-111403GB-I00) funded by MCIN/AEI/ 10.13039/501100011033 and by 'ERDF A way of making Europe'; by the Project PID2021-128790NB-I00, funded by MCIN/AEI/ 10.13039/501100011033 / FEDER UE; and by the European Union Horizon 2020 research and innovation program under grant agreement No. 688269 (TELMI project). It has been additionally supported by the Excellence Research Group 2017SGR-974 funded by the Secretaria d'Universitats i Recerca del Departament d'Empresa i Coneixement de la Generalitat de Catalunya and by the MDM-2017-0729-18-2M Maria de Maeztu Center of Excellence UBNeuro, funded by MCIN/AEI/ 10.13039/501100011033. ISM was supported by grant RYC-2013-12577, funded by MCIN/AEI/ 10.13039/501100011033 and by 'ESF Investing in your future'. Open access publication fees were partly covered by the University of Barcelona (Ajudes de la Universitat de Barcelona per a publicar en accés obert).
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

Automatic assessment of violin performance using dynamic time warping classification

  • Giraldo, Sergio
  • Ortega, Ariadna
  • Pérez Carrillo, Alfonso Antonio, 1977-
  • Ramírez, Rafael, 1966-
  • Waddell, George
  • Williamon, Aaron
Comunicació presentada a: 26th Signal Processing and Communications Applications Conference, celebrada a Izmir, Turkey, del 2 al 5 de maig de 2018., The automatic assessment of music performance has become an area of special interest due to the increasing amount of technology-enhanced music learning systems. However, in most of these systems the assessment of the musical performance is based on the accuracy of onsets and pitch, paying little attention to other relevant aspects of performance. In this paper we present a preliminary study to assess the quality of violin performance using machine learning techniques. We collect recording examples of selected violin exercises varying from expert to amateur performances. We process the audio signal to extract features to train models using clustering based on Dynamic Time Warping distance. The quality of new performances is evaluated based on the level of match/miss-match to each of the recorded training examples., This work has been partly sponsored by the Spanish TIN project TIMUL (TIN 2013-48152-C2-2-R), the European Union Horizon 2020 research and innovation programme under grant agreement No. 688269 (TELMI project), and the Spanish Ministry of Economy and Competitiveness under the Maria de Maeztu Units of Excellence Programme (MDM-2015-0502).
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

Evaluation of audio-based feedback technologies for bow learning technique in violin beginners

  • Blanco, Angel David
  • Ramírez, Rafael, 1966-
Comunicació presentada a: 1st ACM SIGCHI International Workshop on Multimodal Interaction for Education (MIE'17), celebrat en Glasgow (Regne Unit) el 13 de novembre de 2017., We present a study of the effects of feedback technologies on the learning process of novice violin students. Twenty-one subjects participated in our experiment, divided into two groups: Beginners (participants with no prior violin playing experience, N=14), and experts (participants with more than 6 years of violin playing experience, N=7). The beginners group was further divided into two: a group of beginners learning with Youtube videos (N=7), and a group of beginners with additional feedback related to the quality of their performance (N=7). Participants were asked to perform a violin exercise during 21 trials while their audio was recorded and analyzed. Three different audio descriptors were extracted from each audio in order to evaluate the quality of the performance: Dynamic stability, pitch stability and aperiodicity. Beginners showed a significant improvement during the session(i.e. by comparing the beginning and the end of the session)in the quality of the sound recorded, while experts maintained their results. However, only the beginner group with feedback showed significant improvement between the middle and late part of the session, while the group without feedback remained stable., This work has been partly sponsored by the Spanish TIN project TIMUL (TIN 2013-48152-C2-2-R), the European Union Horizon 2020 research and innovation programme under grant agreement No. 688269 (TELMI project), and the Spanish Ministry of Economy and Competitiveness under the Maria de Maeztu Units of Excellence Programme (MDM-2015-0502).
Repositorio Digital de la UPF
Publication Artículo científico (JournalArticle)

Air violin: a machine learning approach to fingering gesture recognition

  • Dalmazzo, David
  • Ramírez, Rafael, 1966-
Comunicació presentada a: 1st ACM SIGCHI International Workshop on Multimodal Interaction for Education, celebrat el 13 de novembre de 2017 a Glasgow, Regne Unit., We train and evaluate two machine learning models for predicting fingering in violin performances using motion and EMG sensors integrated in the Myo device. Our aim is twofold: first, provide a fingering recognition model in the context of a gamification virtual violin application where we measure both right hand (i.e. bow) and left hand (i.e. fingering) gestures, and second, implement a tracking system for a computer assisted pedagogical tool for self-regulated learners in high-level music education. Our approach is based on the principle of mapping-by-demonstration in which the model is trained by the performer. We evaluated a model based on Decision Trees and compared it with a Hidden Markovian Model., This work has been partly sponsored by the Spanish TIN project TIMUL (TIN 2013-48152-C2-2-R), the European Union Horizon 2020 research and innovation programme under grant agreement No. 688269 (TELMI project), and the Spanish Ministry of Economy and Competitiveness under the Maria de Maeztu Units of Excellence Programme (MDM-2015-0502).
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

Jazz ensemble expressive performance modeling

  • Bantula, Helena
  • Giraldo, Sergio
  • Ramírez, Rafael, 1966-
Comunicació presentada a la 17th International Society for Music Information Retrieval Conference (ISMIR 2016), celebrada els dies 7 a 11 d'agost de 2016 a Nova York, EUA., Computational expressive music performance studies the analysis and characterisation of the deviations that a musician introduces when performing a musical piece. It has been studied in a classical context where timing and dynamic deviations are modeled using machine learning techniques. In jazz music, work has been done previously on the study of ornament prediction in guitar performance, as well as in saxophone expressive modeling. However, little work has been done on expressive ensemble performance. In this work, we analysed the musical expressivity of jazz guitar and piano from two different perspectives: solo and ensemble performance. The aim of this paper is to study the influence of piano accompaniment into the performance of a guitar melody and vice versa. Based on a set of recordings made by professional musicians, we extracted descriptors from the score, we transcribed the guitar and the piano performances and calculated performance actions for both instruments. We applied machine learning techniques to train models for each performance action, taking into account both solo and ensemble descriptors. Finally, we compared the accuracy of the induced models. The accuracy of most models increased when ensemble information was considered, which can be explained by the interaction between musicians., This work has been partly sponsored by the Spanish TIN project TIMUL (TIN2013-48152-C2-2-R) and the H2020-ICT-688269 TELMI project.
Repositorio Digital de la UPF
Publication Artículo científico (JournalArticle)

Effects of visual and auditory feedback in violin and singing voice pitch matching tasks

  • Blanco, Angel David
  • Tassani, Simone
  • Ramírez, Rafael, 1966-
Auditory-guided vocal learning is a mechanism that operates both in humans and other animal species making us capable to imitate arbitrary sounds. Both auditory memories and auditory feedback interact to guide vocal learning. This may explain why it is easier for humans to imitate the pitch of a human voice than the pitch of a synthesized sound. In this study, we compared the effects of two different feedback modalities in learning pitch-matching abilities using a synthesized pure tone in 47 participants with no prior music experience. Participants were divided into three groups: a feedback group (N = 15) receiving real-time visual feedback of their pitch as well as knowledge of results; an equal-timbre group (N = 17) receiving additional auditory feedback of the target note with a similar timbre to the instrument being used (i.e., violin or human voice); and a control group (N = 15) practicing without any feedback or knowledge of results. An additional fourth group of violin experts performed the same task for comparative purposes (N = 15). All groups were posteriorly evaluated in a transfer phase. Both experimental groups (i.e., the feedback and equal-timbre groups) improved their intonation abilities with the synthesized sound after receiving feedback. Participants from the equal-timber group seemed as capable as the feedback group of producing the required pitch with the voice after listening to the human voice, but not with the violin (although they also showed improvement). In addition, only participants receiving real-time visual feedback learned and retained in the transfer phase the mapping between the synthesized pitch and its correspondence with the produced vocal or violin pitch. It is suggested that both the effect of an objective external reward, together with the experience of exploring the pitch space with their instrument in an explicit manner, helped participants to understand how to control their pitch production, strengthening their schemas, and favoring retention., This work was partly sponsored by the Musical AI project—(PID2019-111403GB-I00/AEI/10.13039/501100011033) and the European Union Horizon 2020 research and innovation program under grant agreement No. 688269 (TELMI project).
Repositorio Digital de la UPF
Publication Artículo científico (JournalArticle)

Automatic assessment of tone quality in violin music performance

  • Giraldo, Sergio
  • Waddell, George
  • Nou Plana, Ignasi
  • Ortega, Ariadna
  • Mayor, Oscar
  • Perez, Alfonso
  • Williamon, Aaron
  • Ramírez, Rafael, 1966-
The automatic assessment of music performance has become an area of increasing interest due to the growing number of technology-enhanced music learning systems. In most of these systems, the assessment of musical performance is based on pitch and onset accuracy, but very few pay attention to other important aspects of performance, such as sound quality or timbre. This is particularly true in violin education, where the quality of timbre plays a significant role in the assessment of musical performances. However, obtaining quantifiable criteria for the assessment of timbre quality is challenging, as it relies on consensus among the subjective interpretations of experts. We present an approach to assess the quality of timbre in violin performances using machine learning techniques. We collected audio recordings of several tone qualities and performed perceptual tests to find correlations among different timbre dimensions. We processed the audio recordings to extract acoustic features for training tone-quality models. Correlations among the extracted features were analyzed and feature information for discriminating different timbre qualities were investigated. A real-time feedback system designed for pedagogical use was implemented in which users can train their own timbre models to assess and receive feedback on their performances., This work has been partly sponsored by the Spanish TIN project TIMUL (TIN2013-48152-C2-2-R), the European Union Horizon 2020 research and innovation programme under grant agreement No. 688269 (TELMI project), and the Spanish Ministry of Economy and Competitiveness under the Maria de Maeztu Units of Excellence Programme (MDM-2015-0502).
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

Bowing modeling for violin students assistance

  • Ortega, Fabio J. M.
  • Giraldo, Sergio
  • Ramírez, Rafael, 1966-
Comunicació presentada a: 1st ACM SIGCHI International Workshop on Multimodal Interaction for Education, que va tenir lloc el 13 de novembre de 2017 a Glasgow, UK., Though musicians tend to agree on the importance of practicing expressivity in performance, not many tools and techniques are available for the task. A machine learning model is proposed for predicting bowing velocity during performances of violin pieces. Our aim is to provide feedback to violin students in a technology–enhanced learning setting. Predictions are generated for musical phrases in a score by matching them to melodically and rhythmically similar phrases in performances by experts and adapting the bow velocity curve measured in the experts’ performance. Results show that mean error in velocity predictions and bowing direction classification accuracy outperform our baseline when reference phrases similar to the predicted ones are available., This work has been partly sponsored by the Spanish TIN project TIMUL (TIN 2013-48152-C2-2-R), the European Union Horizon 2020 research and innovation programme under grant agreement No. 688269 (TELMI project), and the Spanish Ministry of Economy and Competitiveness under the Maria de Maeztu Units of Excellence Programme (MDM-2015-0502).
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

Phrase-level modeling of expression in violin performances

  • Ortega, Fabio J. M.
  • Giraldo, Sergio
  • Ramírez, Rafael, 1966-
Comunicació presentada a: 10th International Workshop on Machine Learning and Music (MML), celebrat a Barcelona (Espanya), el 6 d'octubre de 2017., A model is proposed for predicting expressive variations in dynamics for violin performances with the purpose of facilitating expressive performance learning by students. The model uses phrases rather than single notes as units of analysis in a lazy learning approach: each phrase in a new score is matched to phrases from expressive performances by experts, adapting the experts’ transformations to render an expressive performance of the new score. In preliminary tests, the model approximates the dynamics of actual performances better than an unexpressive baseline model whenever the reference dataset contains melodies similar to those being predicted., This work has been partly sponsored by the Spanish TIN project TIMUL (TIN 2013-48152-C2-2-R), the European Union Horizon 2020 research and innovation programme under grant agreement No. 688269 (TELMI project), and the Spanish Ministry of Economy and Competitiveness under the Maria de Maeztu Units of Excellence Programme (MDM-2015-0502).
Repositorio Digital de la UPF
Publication Artículo científico (JournalArticle)

Music-enhanced emotion identification of facial emotions in autistic spectrum disorder children: a pilot EEG study

  • Ramírez, Rafael, 1966-
  • Matamoros, Elisabet
  • Hernández-Leo, Davinia
  • Mirabel, Júlia
  • Sánchez, Elisabet
  • Escude, Núria
The Autistic Spectrum Disorder (ASD) is characterized by a difficulty in expressing and interpreting others’ emotions. In particular, people with ASD have difficulties when interpreting emotions encoded in facial expressions. In the past, music interventions have been shown to improve autistic individuals’ emotional and social skills. The present study describes a pilot study to explore the usefulness of music as a tool for improving autistic children’s emotion recognition in facial expressions. Twenty-five children (mean age = 8.8 y, SD = 1.24) with high-functioning ASD and normal hearing participated in the study consisting of four weekly sessions of 15 min each. Twenty-five participants were randomly divided into an experimental group (N = 14) and a control group (N = 11). During each session, participants in the experimental group were exposed to images of facial expressions for four emotions (happy, sad, angry, and fear). Images were shown in three conditions, with the second condition consisting of music of congruent emotion with the shown images. Participants in the control group were shown only images in all three conditions. For six participants in each group, EEG data were acquired during the sessions, and instantaneous emotional responses (arousal and valence values) were extracted from the EEG data. Inter- and intra-session emotion identification improvement was measured in terms of verbal response accuracy, and EEG response differences were analyzed. A comparison of the verbal responses of the experimental group pre- and post-intervention showed a significant (p = 0.001) average improvement in emotion identification accuracy responses of 26% (SD = 3.4). Furthermore, emotional responses of the experimental group at the end of the study showed a higher correlation with the emotional stimuli being presented, compared with their emotional responses at the beginning of the study. No similar verbal responses improvement or EEG-stimuli correlation was found in the control group. These results seem to indicate that music can be used to improve both emotion identification in facial expressions and emotion induction through facial stimuli in children with high-functioning ASD., This work was partly sponsored by the European Union Horizon 2020 research and innovation program under grant agreement No. 688269 (TELMI project) and by the Spanish Ministerio de Ciencia, Innovación y Universidades (MCIU) and the Agencia Estatal de Investigación (AEI) project Musical AI—PID2019-111403GB-I00/AEI/10.13039/501100011033.
Repositorio Digital de la UPF
Publication Artículo científico (JournalArticle)

Real-time sound and motion feedback for violin bow technique learning: a controlled, randomized trial

  • Blanco, Angel David
  • Tassani, Simone
  • Ramírez, Rafael, 1966-
The production of good sound generation in the violin is a complex task that requires coordination and spatiotemporal control of bowing gestures. The use of motion-capture technologies to improve performance or reduce injury risks in the area of kinesiology is becoming widespread. The combination of motion accuracy and sound quality feedback has the potential of becoming an important aid in violin learning. In this study, we evaluate motion-capture and sound-quality analysis technologies developed inside the context of the TELMI, a technology-enhanced music learning project. We analyzed the sound and bow motion of 50 participants with no prior violin experience while learning to produce a stable sound in the violin. Participants were divided into two groups: the experimental group (N = 24) received real-time visual feedback both on kinematics and sound quality, while participants in the control group (N = 26) practiced without any type of external help. An additional third group of violin experts performed the same task for comparative purposes (N = 15). After the practice session, all groups were evaluated in a transfer phase without feedback. At the practice phase, the experimental group improved their bowing kinematics in comparison to the control group, but this was at the expense of impairing the sound quality of their performance. At the retention phase, the experimental group showed better results in sound quality, especially concerning control of sound dynamics. Besides, we found that the expert group improved the stability of their sound while using the technology. All in all, these results emphasize the importance of feedback technologies in learning complex tasks, such as musical instrument learning., This work was partly sponsored by Fundación Memora, the Spanish TIN project TIMUL (TIN2013-48152-C2-2-R), and the European Union Horizon 2020 research and innovation program under grant agreement No. 688269 (TELMI project).
Recercat. Dipósit de la Recerca de Catalunya
Publication Capítulo o parte de un libro (BookPart)

Technology enhanced learning of performance

  • Ramírez, Rafael, 1966-
  • Waddell, George
Technology pervades musical practice. It has driven the development of the instruments musicians play. It has ushered in an explosion of online music making, sharing, and collaboration (see Lisboa et al., this part, for a discussion of innovations in distance learning and performance). It dictates the ways musicians make their living, how they reach their audiences, and how their performances are consumed. It has fostered entire genres of musical expression and new multimodal experiences. However, despite this technological revolution, the way in which musicians learn to perform continues largely the same as it did centuries ago. Students still turn to expert teachers in one-to-one lessons for personalized assessment and guidance in a masterapprentice model that has proved highly resilient to change, interspersed with countless hours spent alone in practice (Creech & Gaunt, 2012; Gaunt, 2017). Over the decades, some digital tools have slowly made their way into common use in the practice room and teaching studio—metronomes, tuners, audio and video recording devices, digital scores—although the pace of innovation has not necessarily been characteristic of the technological explosion that has been seen in other areas of musical practice. This may be about to change. This chapter provides a window into the recent work that musicians, scientists, designers, and engineers are doing to build the next generation of technologies for music learning. While it does not attempt to summarize the full array of technologies available or in development (this would go well beyond the scope of one chapter and would quickly become outdated), it highlights the tools and techniques, from state-of-the-art measurement sensors to the latest in artificial intelligence, that are being brought to bear on the challenges of music learning and what they mean for musicians. Finally, it gives an optimistic glimpse at what musicians might expect to see in the coming decades, and how they can prepare themselves and their students to be technologically literate and adventurous in pushing the boundaries of their practice., The research reported in this chapter was supported by TELMI (Technology Enhanced Learning of Musical Instrument Performance), a project funded by the European Commission’s Horizon 2020 research and innovation program (grant no. 688269)
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

Automatic assessment of tone quality in violin music performance

  • Giraldo, Sergio
  • Waddell, George
  • Nou Plana, Ignasi
  • Ortega, Ariadna
  • Mayor, Oscar
  • Perez, Alfonso
  • Williamon, Aaron
  • Ramírez, Rafael, 1966-
The automatic assessment of music performance has become an area of increasing interest due to the growing number of technology-enhanced music learning systems. In most of these systems, the assessment of musical performance is based on pitch and onset accuracy, but very few pay attention to other important aspects of performance, such as sound quality or timbre. This is particularly true in violin education, where the quality of timbre plays a significant role in the assessment of musical performances. However, obtaining quantifiable criteria for the assessment of timbre quality is challenging, as it relies on consensus among the subjective interpretations of experts. We present an approach to assess the quality of timbre in violin performances using machine learning techniques. We collected audio recordings of several tone qualities and performed perceptual tests to find correlations among different timbre dimensions. We processed the audio recordings to extract acoustic features for training tone-quality models. Correlations among the extracted features were analyzed and feature information for discriminating different timbre qualities were investigated. A real-time feedback system designed for pedagogical use was implemented in which users can train their own timbre models to assess and receive feedback on their performances., This work has been partly sponsored by the Spanish TIN project TIMUL (TIN2013-48152-C2-2-R), the European Union Horizon 2020 research and innovation programme under grant agreement No. 688269 (TELMI project), and the Spanish Ministry of Economy and Competitiveness under the Maria de Maeztu Units of Excellence Programme (MDM-2015-0502).
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

Phrase-Level modeling of expression in violin performances

  • Ortega, Fabio J. M.
  • Giraldo, Sergio
  • Pérez Carrillo, Alfonso Antonio, 1977-
  • Ramírez, Rafael, 1966-
Background: Expression is a key skill in music performance, and one that is difficult to address in music lessons. Computational models that learn from expert performances can help providing suggestions and feedback to students. Aim: We propose and analyze an approach to modeling variations in dynamics and note onset timing for solo violin pieces with the purpose of facilitating expressive performance learning in new pieces, for which no reference performance is available. Method: The method generates phrase–level predictions based on musical score information on the assumption that expressiveness is idiomatic, and thus influenced by similar–sounding melodies. Predictions were evaluated numerically using three different datasets and against note–level machine–learning models, and also perceptually by listeners, who were presented to synthesized versions of musical excerpts, and asked to choose the most human–sounding one. Some of the presented excerpts were synthesized to reflect the variations in dynamics and timing predicted by the model, whereas others were shaped to reflect the dynamics and timing of an actual expert performance, and a third group was presented with no expressive variations. Results: surprisingly, none of the three synthesized versions was consistently selected as human–like nor preferred with statistical significance by listeners. Possible interpretations of these results include the fact that the melodies might have been impossible to interpret outside their musical context, or that expressive features that were left out of the modeling such as note articulation and vibrato are, in fact, essential to the perception of expression in violin performance. Positive feedback by some listeners toward the modeled melodies in a blind setting indicate that the modeling approach was capable of generating appropriate renditions at least for a subset of the data. Numerically, performance in phrase–level suffers a small degradation if compared to note–level, but produces predictions easier to interpret visually, thus more useful in a pedagogical setting., This work has been partly sponsored by the Spanish TIN project TIMUL (TIN 2013-48152-C2-2-R), the European Union Horizon 2020 research and innovation programme under grant agreement No. 688269 (TELMI project), and the Spanish Ministry of Economy and Competitiveness under the Maria de Maeztu Units of Excellence Programme (MDM-2015-0502).
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Phrase-level modeling of expression in violin performances

  • Ortega, Fabio J. M.
  • Giraldo, Sergio
  • Ramírez, Rafael, 1966-
Comunicació presentada a: 10th International Workshop on Machine Learning and Music (MML), celebrat a Barcelona (Espanya), el 6 d'octubre de 2017., A model is proposed for predicting expressive variations in dynamics for violin performances with the purpose of facilitating expressive performance learning by students. The model uses phrases rather than single notes as units of analysis in a lazy learning approach: each phrase in a new score is matched to phrases from expressive performances by experts, adapting the experts’ transformations to render an expressive performance of the new score. In preliminary tests, the model approximates the dynamics of actual performances better than an unexpressive baseline model whenever the reference dataset contains melodies similar to those being predicted., This work has been partly sponsored by the Spanish TIN project TIMUL (TIN 2013-48152-C2-2-R), the European Union Horizon 2020 research and innovation programme under grant agreement No. 688269 (TELMI project), and the Spanish Ministry of Economy and Competitiveness under the Maria de Maeztu Units of Excellence Programme (MDM-2015-0502).
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Evaluation of audio-based feedback technologies for bow learning technique in violin beginners

  • Blanco, Angel David
  • Ramírez, Rafael, 1966-
Comunicació presentada a: 1st ACM SIGCHI International Workshop on Multimodal Interaction for Education (MIE'17), celebrat en Glasgow (Regne Unit) el 13 de novembre de 2017., We present a study of the effects of feedback technologies on the learning process of novice violin students. Twenty-one subjects participated in our experiment, divided into two groups: Beginners (participants with no prior violin playing experience, N=14), and experts (participants with more than 6 years of violin playing experience, N=7). The beginners group was further divided into two: a group of beginners learning with Youtube videos (N=7), and a group of beginners with additional feedback related to the quality of their performance (N=7). Participants were asked to perform a violin exercise during 21 trials while their audio was recorded and analyzed. Three different audio descriptors were extracted from each audio in order to evaluate the quality of the performance: Dynamic stability, pitch stability and aperiodicity. Beginners showed a significant improvement during the session(i.e. by comparing the beginning and the end of the session)in the quality of the sound recorded, while experts maintained their results. However, only the beginner group with feedback showed significant improvement between the middle and late part of the session, while the group without feedback remained stable., This work has been partly sponsored by the Spanish TIN project TIMUL (TIN 2013-48152-C2-2-R), the European Union Horizon 2020 research and innovation programme under grant agreement No. 688269 (TELMI project), and the Spanish Ministry of Economy and Competitiveness under the Maria de Maeztu Units of Excellence Programme (MDM-2015-0502).
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

A Machine learning approach to ornamentation modeling and synthesis in jazz guitar

  • Ramírez, Rafael, 1966-
  • Giraldo, Sergio
We present a machine learning approach to automatically generate expressive (ornamented) jazz performances from un-expressive music scores. Features extracted from the scores and the corresponding audio recordings performed by a professional guitarist were used to train computational models for predicting melody ornamentation. As a first step, several machine learning techniques were explored to induce regression models for timing, onset, and dynamics (i.e. note duration and energy) transformations, and an ornamentation model for classifying notes as ornamented or non-ornamented. In a second step, the most suitable ornament for predicted ornamented notes was selected based on note context similarity. Finally, concatenative synthesis was used to automatically synthesize expressive performances of new pieces using the induced models. Supplemental online material for this article containing musical examples of the automatically generated ornamented pieces can be accessed at doi: 10.1080/17459737.2016.1207814 and https://soundcloud.com/machine-learning-and-jazz. In the Online Supplement we present an example of the musical piece Yesterdays by Jerome Kern, which was modeled using our methodology for expressive music performance in jazz guitar., This project has received funding from: the European Union Horizon 2020 research and innovation programme [grant agreement No 688269]; the Spanish TIN project TIMUL [grant agreement TIN2013-48152-C2-2-R].
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Computational modelling of timbre dimensions for automatic violin tone quality assessment

  • Giraldo, Sergio
  • Ramírez, Rafael, 1966-
  • Waddell, George
  • Williamon, Aaron
Automatic assessment of music performance is an open research area widely studied in the past. A vast amount of systems aiming to enhance the learning process of a musical instrument are being developed in the recent years. However, most of the systems focus on the assessment of pitch and onset accuracy, and very few pay attention to tone quality. This is particularly true in violin music education, where although a consensus exist on what is a good or a bad tone quality, there is not a formal definition due to its subjectivity. We present a machine learning approach for the automatic assessment of violin tone quality. We depart from our previous work on the preliminary modelling of several dimensions involving tone quality. Based on recorded examples of tones with different qualities defined and recorded by a professional violinist, we applied machine learning techniques to learn computational models able to evaluate tone quality from extracted audio features. The tone quality models were implemented into a real-time-visual-feedback system., This work has been partly sponsored by the Spanish TIN project TIMUL (TIN2013-48152-C2-2-R), the European Union Horizon 2020 research and innovation programme under grant agreement No. 688269 (TELMI project), and the Spanish Ministry of Economy and Competitiveness under the Maria de Maeztu Units of Excellence Programme (MDM-2015-0502).
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Automatic assessment of violin performance using dynamic time warping classification

  • Giraldo, Sergio
  • Ortega, Ariadna
  • Pérez Carrillo, Alfonso Antonio, 1977-
  • Ramírez, Rafael, 1966-
  • Waddell, George
  • Williamon, Aaron
Comunicació presentada a: 26th Signal Processing and Communications Applications Conference, celebrada a Izmir, Turkey, del 2 al 5 de maig de 2018., The automatic assessment of music performance has become an area of special interest due to the increasing amount of technology-enhanced music learning systems. However, in most of these systems the assessment of the musical performance is based on the accuracy of onsets and pitch, paying little attention to other relevant aspects of performance. In this paper we present a preliminary study to assess the quality of violin performance using machine learning techniques. We collect recording examples of selected violin exercises varying from expert to amateur performances. We process the audio signal to extract features to train models using clustering based on Dynamic Time Warping distance. The quality of new performances is evaluated based on the level of match/miss-match to each of the recorded training examples., This work has been partly sponsored by the Spanish TIN project TIMUL (TIN 2013-48152-C2-2-R), the European Union Horizon 2020 research and innovation programme under grant agreement No. 688269 (TELMI project), and the Spanish Ministry of Economy and Competitiveness under the Maria de Maeztu Units of Excellence Programme (MDM-2015-0502).
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

Effects of visual and auditory feedback in violin and singing voice pitch matching tasks

  • Blanco, Angel David
  • Tassani, Simone
  • Ramírez, Rafael, 1966-
Auditory-guided vocal learning is a mechanism that operates both in humans and other animal species making us capable to imitate arbitrary sounds. Both auditory memories and auditory feedback interact to guide vocal learning. This may explain why it is easier for humans to imitate the pitch of a human voice than the pitch of a synthesized sound. In this study, we compared the effects of two different feedback modalities in learning pitch-matching abilities using a synthesized pure tone in 47 participants with no prior music experience. Participants were divided into three groups: a feedback group (N = 15) receiving real-time visual feedback of their pitch as well as knowledge of results; an equal-timbre group (N = 17) receiving additional auditory feedback of the target note with a similar timbre to the instrument being used (i.e., violin or human voice); and a control group (N = 15) practicing without any feedback or knowledge of results. An additional fourth group of violin experts performed the same task for comparative purposes (N = 15). All groups were posteriorly evaluated in a transfer phase. Both experimental groups (i.e., the feedback and equal-timbre groups) improved their intonation abilities with the synthesized sound after receiving feedback. Participants from the equal-timber group seemed as capable as the feedback group of producing the required pitch with the voice after listening to the human voice, but not with the violin (although they also showed improvement). In addition, only participants receiving real-time visual feedback learned and retained in the transfer phase the mapping between the synthesized pitch and its correspondence with the produced vocal or violin pitch. It is suggested that both the effect of an objective external reward, together with the experience of exploring the pitch space with their instrument in an explicit manner, helped participants to understand how to control their pitch production, strengthening their schemas, and favoring retention., This work was partly sponsored by the Musical AI project—(PID2019-111403GB-I00/AEI/10.13039/501100011033) and the European Union Horizon 2020 research and innovation program under grant agreement No. 688269 (TELMI project).
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

Air violin: a machine learning approach to fingering gesture recognition

  • Dalmazzo, David
  • Ramírez, Rafael, 1966-
Comunicació presentada a: 1st ACM SIGCHI International Workshop on Multimodal Interaction for Education, celebrat el 13 de novembre de 2017 a Glasgow, Regne Unit., We train and evaluate two machine learning models for predicting fingering in violin performances using motion and EMG sensors integrated in the Myo device. Our aim is twofold: first, provide a fingering recognition model in the context of a gamification virtual violin application where we measure both right hand (i.e. bow) and left hand (i.e. fingering) gestures, and second, implement a tracking system for a computer assisted pedagogical tool for self-regulated learners in high-level music education. Our approach is based on the principle of mapping-by-demonstration in which the model is trained by the performer. We evaluated a model based on Decision Trees and compared it with a Hidden Markovian Model., This work has been partly sponsored by the Spanish TIN project TIMUL (TIN 2013-48152-C2-2-R), the European Union Horizon 2020 research and innovation programme under grant agreement No. 688269 (TELMI project), and the Spanish Ministry of Economy and Competitiveness under the Maria de Maeztu Units of Excellence Programme (MDM-2015-0502).
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

JamSketch: a drawing-based real-time evolutionary improvisation support system

  • Kitahara, Tetsuro
  • Giraldo, Sergio
  • Ramírez, Rafael, 1966-
Comunicació presentada a: International Conference on New Interfaces for Musical Expression celebrat del 15 al 19 de maig de 2017 a Copenhagen, Dinamarca., In this paper, we present JamSketch, a real-time improvisation support system which automatically generates melodies according to melodic outlines drawn by the users. The system generates the improvised melodies based on (1) an outline sketched by the user using a mouse or a touch screen, (2) a genetic algorithm based on a dataset of existing music pieces as well as musical knowledge, and (3) an expressive performance model for timing and dynamic transformations. The aim of the system is to allow people with no prior musical knowledge to be able to enjoy playing music by improvising melodies in real time., This work was supported by JSPS KAKENHI Grant Numbers 26240025, 26280089, 16K16180, 16H01744, and 16KT0136, as well as by the Spanish TIMuL Project (TIN2013-48152-C2-2-R), and the TELMI Project of the Horizon 2020 Research and Innovation Programme (grant agreement No. 688269).
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Computational modelling of expressive music performance in hexaphonic guitar

  • Siquier, Marc
  • Giraldo, Sergio
  • Ramírez, Rafael, 1966-
Comunicació presentada a: 10th International Workshop on Machine Learning and Music (MML), celebrat a Barcelona (Espanya), el 6 d'octubre de 2017., Computational modelling of expressive music performance has been widely studied in the past. While previous work in this area has been mainly focused on classical piano music, there has been very little work on guitar music, and such work has focused on monophonic guitar playing. In this work, we present a machine learning approach to automatically generate expressive performances from non expressive music scores for polyphonic guitar. We treated guitar as an hexaphonic instrument, obtaining a polyphonic transcription of performed musical pieces. Features were extracted from the scores and performance actions were calculated from the deviations of the score and the performance. Machine learning techniques were used to train computational models to predict the aforementioned performance actions. Qualitative and quantitative evaluations of the models and the predicted pieces were performed., This work has been partly sponsored by the Spanish TIN project TIMUL (TIN2013-48152-C2-2-R), the European Union Horizon 2020 research and inno- vation programme under grant agreement No. 688269 (TELMI project), and the Spanish Ministry of Economy and Competitiveness under the Maria de Maeztu Units of Excellence Programme (MDM-2015-0502).
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Enhancing music learning with smart technologies

  • Ramírez, Rafael, 1966-
  • Canepa, Corrado
  • Ghisio, Simone
  • Kolykhalova, Ksenia
  • Mancini, Maurizio
  • Volta, Erica
  • Volpe, Gualtiero
  • Giraldo, Sergio
  • Mayor, Oscar
  • Perez, Alfonso
  • Waddell, George
  • Williamon, Aaron
Comunicació presentada a: 5th International Conference on Movement and Computing, celebrat del 28 al 30 de juny de 2018 a Gènova, Itàlia., Learning to play a musical instrument is a difficult task, requiring the development of sophisticated skills. Nowadays, such a learning process is mostly based on the master-apprentice model. Technologies are rarely employed and are usually restricted to audio and video recording and playback. The TELMI (Technology Enhanced Learning of Musical Instrument Performance) Project seeks to design and implement new interaction paradigms for music learning and training based on state-of-the-art multimodal (audio, image, video, and motion) technologies. Figure 1: Analysis of coordination, applying RecurrenceQuantification Analysis to the kinetic energy of the right wrist. Figure 2: A sample screen-shot of the intonation feedback on Piano Roll Mode. Figure 3: A sample screen-shot of the intonation feedback on Score View Mode. The project focuses on the violin as a case study. This practice work is intended as demo, showing to MOCO attendants the results the project obtained along two years of work. The demo simulates a setup at a higher education music institution, where attendants with any level of previous violin experience (and even with no experience at all) are invited to try the technologies themselves, performing basic tests of violin skill and pre-defined exercises under the guidance of the researchers involved in the project.
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

Evaluation of a sound quality visual feedback system for bow learning technique in violin beginners: an EEG study

  • Blanco, Angel David
  • Ramírez, Rafael, 1966-
Current music technologies can assist in the process of learning to play a musical instrument and provide objective measures for evaluating the improvement of music students in concrete music tasks. In this paper, we investigated the effects of a sound quality visual feedback system (SQVFS) in violin learning. In particular, we studied the EEG activity of a group of participants with no previous violin playing experience while they learned to produce a stable sound (regarding pitch, dynamics, and timbre) in order to find motor learning biomarkers in a music task. Eighteen subjects with no prior experience in violin playing were divided into two groups: participants in the first group (experimental group, N = 9) practiced with instructional videos and offline feedback from the SQVFS provided in alternation with their performance, while participants in a second group (control group, N = 9) practiced with the instructional videos only. A third group of violin experts (players with more than 6 years of experience) performed the same task for comparative purposes (N = 7). All participants were asked to perform 20 trials (4 blocks of 5 trials) consisting of a violin bowing exercise while their EEG activity and their produced sound was recorded. Significant sound quality improvements along the session were found in all participants with the exception of participants in the expert group. In addition, participants in the experimental group showed increased interest in the learning process and significant improvement after the second block not present in the control group. A significant correlation between the levels of frontal gamma band power and the sound improvement along the task was found in both the experimental and control group. This result is consistent with the temporal binding model which associates gamma band power with the role of integrating (binding) information processed in distributed cortical areas. Task complexity demands more cognitive resources, more binding and thus, gamma band power enhancement, which may be reduced as the demanded task begins to be automated as it is likely to be the case in both beginners groups., This work was partly sponsored by Fundación Memora, the Spanish TIN project TIMUL (TIN2013-48152-C2-2-R), and the European Union Horizon 2020 research and innovation program under grant agreement No. 688269 (TELMI project).
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Jazz ensemble expressive performance modeling

  • Bantula, Helena
  • Giraldo, Sergio
  • Ramírez, Rafael, 1966-
Comunicació presentada a la 17th International Society for Music Information Retrieval Conference (ISMIR 2016), celebrada els dies 7 a 11 d'agost de 2016 a Nova York, EUA., Computational expressive music performance studies the analysis and characterisation of the deviations that a musician introduces when performing a musical piece. It has been studied in a classical context where timing and dynamic deviations are modeled using machine learning techniques. In jazz music, work has been done previously on the study of ornament prediction in guitar performance, as well as in saxophone expressive modeling. However, little work has been done on expressive ensemble performance. In this work, we analysed the musical expressivity of jazz guitar and piano from two different perspectives: solo and ensemble performance. The aim of this paper is to study the influence of piano accompaniment into the performance of a guitar melody and vice versa. Based on a set of recordings made by professional musicians, we extracted descriptors from the score, we transcribed the guitar and the piano performances and calculated performance actions for both instruments. We applied machine learning techniques to train models for each performance action, taking into account both solo and ensemble descriptors. Finally, we compared the accuracy of the induced models. The accuracy of most models increased when ensemble information was considered, which can be explained by the interaction between musicians., This work has been partly sponsored by the Spanish TIN project TIMUL (TIN2013-48152-C2-2-R) and the H2020-ICT-688269 TELMI project.
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

Applying deep learning techniques to estimate patterns of musical gesture

  • Dalmazzo, David
  • Waddell, George
  • Ramírez, Rafael, 1966-
Repetitive practice is one of the most important factors in improving the performance of motor skills. This paper focuses on the analysis and classification of forearm gestures in the context of violin playing. We recorded five experts and three students performing eight traditional classical violin bow-strokes: martelé, staccato, detaché, ricochet, legato, trémolo, collé, and col legno. To record inertial motion information, we utilized the Myo sensor, which reports a multidimensional time-series signal. We synchronized inertial motion recordings with audio data to extract the spatiotemporal dynamics of each gesture. Applying state-of-the-art deep neural networks, we implemented and compared different architectures where convolutional neural networks (CNN) models demonstrated recognition rates of 97.147%, 3DMultiHeaded_CNN models showed rates of 98.553%, and rates of 99.234% were demonstrated by CNN_LSTM models. The collected data (quaternion of the bowing arm of a violinist) contained sufficient information to distinguish the bowing techniques studied, and deep learning methods were capable of learning the movement patterns that distinguish these techniques. Each of the learning algorithms investigated (CNN, 3DMultiHeaded_CNN, and CNN_LSTM) produced high classification accuracies which supported the feasibility of training classifiers. The resulting classifiers may provide the foundation of a digital assistant to enhance musicians' time spent practicing alone, providing real-time feedback on the accuracy and consistency of their musical gestures in performance., This work has been partly sponsored by the Spanish TIN project TIMUL (TIN 2013-48152-C2-2-R), the European Union Horizon 2020 research and innovation program under grant agreement No. 688269 (TELMI project), and the Spanish Ministry of Economy and Competitiveness under the Maria de Maeztu Units of Excellence Programme (MDM-2015-0502).
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

The creative drummer: an eeg-based pilot study on the correlates of emotions and creative drum playing

  • Ramírez, Rafael, 1966-
  • Reija, Xavier
It is reasonable to assume that emotional processes are involved in creative tasks and the generation of creative ideas. In this pilot study, we investigate the emotional correlates in professional drummers during different degrees of creative music playing. Ten participants performed three tasks: repetitive rhythmic drum playing, pattern-based improvisation, and attention-intensive free improvisation, while their EEG activity was recorded. Arousal and valence levels were estimated from the EEG data at baseline and for the three tasks. Results show significantly increased levels of valence (i.e., increased prefrontal right alpha power compared to prefrontal left alpha power) during pattern-based and free improvisation relative to baseline, and significantly increased levels of valence during free improvisation relative to pattern-based improvisation. These results seem to indicate that positive emotion (characterized as increased valence) is associated with the creation of original ideas in drum playing and that the freer the creative process, the greater the positive effect. The implication of these results may be of particular relevance in the fields of music-based therapeutic interventions and music pedagogy., This work was partly sponsored by the European Union Horizon 2020 research and innovation program under grant agreement No. 688269 (TELMI project) and by the Musical AI project—(PID2019-111403GB-I00/AEI/10.13039/501100011033).
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Bowing modeling for violin students assistance

  • Ortega, Fabio J. M.
  • Giraldo, Sergio
  • Ramírez, Rafael, 1966-
Comunicació presentada a: 1st ACM SIGCHI International Workshop on Multimodal Interaction for Education, que va tenir lloc el 13 de novembre de 2017 a Glasgow, UK., Though musicians tend to agree on the importance of practicing expressivity in performance, not many tools and techniques are available for the task. A machine learning model is proposed for predicting bowing velocity during performances of violin pieces. Our aim is to provide feedback to violin students in a technology–enhanced learning setting. Predictions are generated for musical phrases in a score by matching them to melodically and rhythmically similar phrases in performances by experts and adapting the bow velocity curve measured in the experts’ performance. Results show that mean error in velocity predictions and bowing direction classification accuracy outperform our baseline when reference phrases similar to the predicted ones are available., This work has been partly sponsored by the Spanish TIN project TIMUL (TIN 2013-48152-C2-2-R), the European Union Horizon 2020 research and innovation programme under grant agreement No. 688269 (TELMI project), and the Spanish Ministry of Economy and Competitiveness under the Maria de Maeztu Units of Excellence Programme (MDM-2015-0502).
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

Machine learning and music generation

  • Iñesta, José M.
  • Conklin, Darrell
  • Ramírez, Rafael, 1966-
We guest editors acknowledge the financial support of the following funding sources: the Ministerio de Economía y Competitividad project TIMuL [No. TIN2013–48152–C2–1–R]; [No. TIN2013–48152–C2–2–R, supported by UE FEDER funds]; the project Lrn2Cre8, which is funded by the Future and Emerging Technologies (FET) programme within the Seventh Framework Programme for Research of the European Commission [FET grant number 610859]; the European Union Horizon 2020 research and innovation programme [grant agreement number 688269].
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

Mapping between dynamic markings and performed loudness: a machine learning approach

  • Kosta, Katerina
  • Ramírez, Rafael, 1966-
  • Bandtlow, Oscar F.
  • Chew, Elaine
Loudness variation is one of the foremost tools for expressivity in music performance. Loudness is frequently notated as dynamic markings such as p (piano, meaning soft) or f (forte, meaning loud). While dynamic markings in music scores are important indicators of how music pieces should be interpreted, their meaning is less straightforward than it may seem, and depends highly on the context in which they appear. In this article, we investigate the relationship between dynamic markings in the score and performed loudness by applying machine learning techniques – decision trees, support vector machines, artificial neural networks, and a k-nearest neighbor method – to the prediction of loudness levels corresponding to dynamic markings, and to the classification of dynamic markings given loudness values. The methods are applied to 44 recordings of performances of Chopin's Mazurkas, each by 8 pianists. The results show that loudness values and markings can be predicted relatively well when trained across recordings of the same piece, but fail dismally when trained across the pianist's recordings of other pieces, demonstrating that score features may trump individual style when modeling loudness choices. Evidence suggests that all the features chosen for the task are relevant, and analysis of the results reveals the forms (such as the return of the theme) and structures (such as dynamic-marking repetitions) that influence the predictability of loudness and markings. Modeling of loudness trends in expressive performance appears to be a delicate matter, and sometimes loudness expression can be a matter of the performer's idiosyncracy., This work was supported in part by a UK EPSRC Platform Grant for Digital Music [EP/K009559/1]; the Spanish TIN project TIMUL [TIN2013-48152-C2-2-R]; and the European Union's Horizon 2020 research and innovation programme [grant agreement No. 688269].
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

Bowing gestures classification in violin performance: a machine learning approach

  • Dalmazzo, David
  • Ramírez, Rafael, 1966-
Gestures in music are of paramount importance partly because they are directly linked to musicians' sound and expressiveness. At the same time, current motion capture technologies are capable of detecting body motion/gestures details very accurately. We present a machine learning approach to automatic violin bow gesture classification based on Hierarchical Hidden Markov Models (HHMM) and motion data. We recorded motion and audio data corresponding to seven representative bow techniques (Détaché, Martelé, Spiccato, Ricochet, Sautillé, Staccato, and Bariolage) performed by a professional violin player. We used the commercial Myo device for recording inertial motion information from the right forearm and synchronized it with audio recordings. Data was uploaded into an online public repository. After extracting features from both the motion and audio data, we trained an HHMM to identify the different bowing techniques automatically. Our model can determine the studied bowing techniques with over 94% accuracy. The results make feasible the application of this work in a practical learning scenario, where violin students can benefit from the real-time feedback provided by the system., This work has been partly sponsored by the Spanish TIN project TIMUL (TIN 2013-48152-C2-2-R), the European Union Horizon 2020 research and innovation programme under grant agreement No. 688269 (TELMI project), and the Spanish Ministry of Economy and Competitiveness under the Maria de Maeztu Units of Excellence Programme (MDM-2015-0502).
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Acquisition of violin instrumental gestures using an infrared depth camera

  • Vamvakousis, Zacharias
  • Perez, Alfonso
  • Ramírez, Rafael, 1966-
Comunicació presentada a: 15th Sound and Music Computing Conference (SMC2018), celebrat del 4 al 7 de juliol a Limassol, Xipre., We present a low-cost six-degrees of freedom violin and bow pose-tracking system. The infrared and depth streams of an infrared-based Microsoft Kinect Mbox One depth camera are utilized to track the 3-dimensional position of 6 infrared markers attached on the violin and the bow. After computing the bow pose on the violin coordinate system, a number of bowing features are extracted. In order to evaluate the system’s performance, we recorded 4 bowing exercises using simultaneously our system along with a commercial two-sensor 3D tracking system based on electromagnetic field (EMF) sensing. The mean pearson coefficient values were 0.996 for the bow position, 0.889 for the bow velocity, 0.966 for the bow tilt, 0.692 for the bowbridge distance and 0.998 for the bow inclination. Compared to existing bowing-tracking solutions, the proposed solution might be appealing because of its low cost, easy setup and high performance., This work was partly sponsored by the Spanish TIN project TIMUL (TIN2013-48152-C2-2-R), and the European Union Horizon 2020 research and innovation program under grant agreement No. 688269 (TELMI project).
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

A Machine learning approach to discover rules for expressive performance actions in jazz guitar music

  • Giraldo, Sergio
  • Ramírez, Rafael, 1966-
Expert musicians introduce expression in their performances by manipulating sound properties such as timing, energy, pitch, and timbre. Here, we present a data driven computational approach to induce expressive performance rule models for note duration, onset, energy, and ornamentation transformations in jazz guitar music. We extract high-level features from a set of 16 commercial audio recordings (and corresponding music scores) of jazz guitarist Grant Green in order to characterize the expression in the pieces. We apply machine learning techniques to the resulting features to learn expressive performance rule models. We (1) quantitatively evaluate the accuracy of the induced models, (2) analyse the relative importance of the considered musical features, (3) discuss some of the learnt expressive performance rules in the context of previous work, and (4) assess their generailty. The accuracies of the induced predictive models is significantly above base-line levels indicating that the audio performances and the musical features extracted contain sufficient information to automatically learn informative expressive performance patterns. Feature analysis shows that the most important musical features for predicting expressive transformations are note duration, pitch, metrical strength, phrase position, Narmour structure, and tempo and key of the piece. Similarities and differences between the induced expressive rules and the rules reported in the literature were found. Differences may be due to the fact that most previously studied performance data has consisted of classical music recordings. Finally, the rules' performer specificity/generality is assessed by applying the induced rules to performances of the same pieces performed by two other professional jazz guitar players. Results show a consistency in the ornamentation patterns between Grant Green and the other two musicians, which may be interpreted as a good indicator for generality of the ornamentation rules., This work has been partly sponsored by the Spanish TIN project TIMUL (TIN2013-48152-C2-2-R), and the European Union Horizon 2020 research and innovation programme under grant agreement No. 688269 (TELMI project).
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

Music-enhanced emotion identification of facial emotions in autistic spectrum disorder children: a pilot EEG study

  • Ramírez, Rafael, 1966-
  • Matamoros, Elisabet
  • Hernández-Leo, Davinia
  • Mirabel, Júlia
  • Sánchez, Elisabet
  • Escude, Núria
The Autistic Spectrum Disorder (ASD) is characterized by a difficulty in expressing and interpreting others’ emotions. In particular, people with ASD have difficulties when interpreting emotions encoded in facial expressions. In the past, music interventions have been shown to improve autistic individuals’ emotional and social skills. The present study describes a pilot study to explore the usefulness of music as a tool for improving autistic children’s emotion recognition in facial expressions. Twenty-five children (mean age = 8.8 y, SD = 1.24) with high-functioning ASD and normal hearing participated in the study consisting of four weekly sessions of 15 min each. Twenty-five participants were randomly divided into an experimental group (N = 14) and a control group (N = 11). During each session, participants in the experimental group were exposed to images of facial expressions for four emotions (happy, sad, angry, and fear). Images were shown in three conditions, with the second condition consisting of music of congruent emotion with the shown images. Participants in the control group were shown only images in all three conditions. For six participants in each group, EEG data were acquired during the sessions, and instantaneous emotional responses (arousal and valence values) were extracted from the EEG data. Inter- and intra-session emotion identification improvement was measured in terms of verbal response accuracy, and EEG response differences were analyzed. A comparison of the verbal responses of the experimental group pre- and post-intervention showed a significant (p = 0.001) average improvement in emotion identification accuracy responses of 26% (SD = 3.4). Furthermore, emotional responses of the experimental group at the end of the study showed a higher correlation with the emotional stimuli being presented, compared with their emotional responses at the beginning of the study. No similar verbal responses improvement or EEG-stimuli correlation was found in the control group. These results seem to indicate that music can be used to improve both emotion identification in facial expressions and emotion induction through facial stimuli in children with high-functioning ASD., This work was partly sponsored by the European Union Horizon 2020 research and innovation program under grant agreement No. 688269 (TELMI project) and by the Spanish Ministerio de Ciencia, Innovación y Universidades (MCIU) and the Agencia Estatal de Investigación (AEI) project Musical AI—PID2019-111403GB-I00/AEI/10.13039/501100011033.
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

Real-time sound and motion feedback for violin bow technique learning: a controlled, randomized trial

  • Blanco, Angel David
  • Tassani, Simone
  • Ramírez, Rafael, 1966-
The production of good sound generation in the violin is a complex task that requires coordination and spatiotemporal control of bowing gestures. The use of motion-capture technologies to improve performance or reduce injury risks in the area of kinesiology is becoming widespread. The combination of motion accuracy and sound quality feedback has the potential of becoming an important aid in violin learning. In this study, we evaluate motion-capture and sound-quality analysis technologies developed inside the context of the TELMI, a technology-enhanced music learning project. We analyzed the sound and bow motion of 50 participants with no prior violin experience while learning to produce a stable sound in the violin. Participants were divided into two groups: the experimental group (N = 24) received real-time visual feedback both on kinematics and sound quality, while participants in the control group (N = 26) practiced without any type of external help. An additional third group of violin experts performed the same task for comparative purposes (N = 15). After the practice session, all groups were evaluated in a transfer phase without feedback. At the practice phase, the experimental group improved their bowing kinematics in comparison to the control group, but this was at the expense of impairing the sound quality of their performance. At the retention phase, the experimental group showed better results in sound quality, especially concerning control of sound dynamics. Besides, we found that the expert group improved the stability of their sound while using the technology. All in all, these results emphasize the importance of feedback technologies in learning complex tasks, such as musical instrument learning., This work was partly sponsored by Fundación Memora, the Spanish TIN project TIMUL (TIN2013-48152-C2-2-R), and the European Union Horizon 2020 research and innovation program under grant agreement No. 688269 (TELMI project).
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

The EyeHarp: A gaze-controlled digital musical instrument

  • Vamvakousis, Zacharias
  • Ramírez, Rafael, 1966-
We present and evaluate the EyeHarp, a new gaze-controlled Digital Musical Instrument, which aims to enable people with severe motor disabilities to learn, perform, and compose music using only their gaze as control mechanism. It consists of (1) a step-sequencer layer, which serves for constructing chords/arpeggios, and (2) a melody layer, for playing melodies and changing the chords/arpeggios. We have conducted a pilot evaluation of the EyeHarp involving 39 participants with no disabilities from both a performer and an audience perspective. In the first case, eight people with normal vision and no motor disability participated in a music-playing session in which both quantitative and qualitative data were collected. In the second case 31 people qualitatively evaluated the EyeHarp in a concert setting consisting of two parts: a solo performance part, and an ensemble (EyeHarp, two guitars, and flute) performance part. The obtained results indicate that, similarly to traditional music instruments, the proposed digital musical instrument has a steep learning curve, and allows to produce expressive performances both from the performer and audience perspective., This project has received funding from the European Unions Horizon 2020 research and innovation program under grant agreement No. 688269, as well as from the Spanish TIN project TIMUL under grant agreement TIN2013-48152-C2-2-R.
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

EEG-Based Analysis of the Emotional Effect of Music Therapy on Palliative Care Cancer Patients

  • Ramírez, Rafael, 1966-
  • Planas Domingo, Josep
  • Escudé, Nuria
  • Mercadé, Jordi
  • Farriols, Cristina
Music is known to have the power to induce strong emotions. The present study assessed, based on Electroencephalography (EEG) data, the emotional response of terminally ill cancer patients to a music therapy intervention in a randomized controlled trial. A sample of 40 participants from the palliative care unit in the Hospital del Mar in Barcelona was randomly assigned to two groups of 20. The first group [experimental group (EG)] participated in a session of music therapy (MT), and the second group [control group (CG)] was provided with company. Based on our previous work on EEG-based emotion detection, instantaneous emotional indicators in the form of a coordinate in the arousal-valence plane were extracted from the participants' EEG data. The emotional indicators were analyzed in order to quantify (1) the overall emotional effect of MT on the patients compared to controls, and (2) the relative effect of the different MT techniques applied during each session. During each MT session, five conditions were considered: I (initial patient's state before MT starts), C1 (passive listening), C2 (active listening), R (relaxation), and F (final patient's state). EEG data analysis showed a significant increase in valence (p = 0.0004) and arousal (p = 0.003) between I and F in the EG. No significant changes were found in the CG. This results can be interpreted as a positive emotional effect of MT in advanced cancer patients. In addition, according to pre- and post-intervention questionnaire responses, participants in the EG also showed a significant decrease in tiredness, anxiety and breathing difficulties, as well as an increase in levels of well-being. No equivalent changes were observed in the CG., This work was partly sponsored by Fundación Memora, the Spanish TIN project TIMUL (TIN2013-48152-C2-2-R), and the European Union Horizon 2020 research and innovation program under grant agreement No. 688269 (TELMI project).
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

A Real-time feedback learning tool to visualize sound quality in violin performances

  • Giraldo, Sergio
  • Ramírez, Rafael, 1966-
  • Waddell, George
  • Williamon, Aaron
Comunicació presentada a: 10th International Workshop on Machine Learning and Music (MML), celebrat a Barcelona (Espanya), el 6 d'octubre de 2017., The assessment of the sound properties of a performed mu- sical note has been widely studied in the past. Although a consensus exist on what is a good or a bad musical performance, there is not a formal de nition of performance tone quality due to its subjectivity. In this study we present a computational approach for the automatic assess- ment of violin sound production. We investigate the correlations among extracted features from audio performances and the perceptual quality of violin sounds rated by listeners using machine learning techniques. The obtained models are used for implementing a real-time feedback learning system., This work has been partly sponsored by the Spanish TIN project TIMUL (TIN2013-48152-C2-2-R), the European Union Horizon 2020 research and inno- vation programme under grant agreement No. 688269 (TELMI project), and the Spanish Ministry of Economy and Competitiveness under the Maria de Maeztu Units of Excellence Programme (MDM-2015-0502).
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

Finger-string interaction analysis in guitar playing with optical motion capture

  • Pérez Carrillo, Alfonso Antonio, 1977-
We present a method for the analysis of the finger-string interaction in guitar performances and the computation of fine actions during the plucking gesture. The method is based on Motion Capture using high-speed cameras that can track the position of reflective markers placed on the guitar and fingers, in combination with audio analysis. A major problem inherent in optical motion capture is that of marker occlusion and, in guitar playing, it is the right hand of the guitarist that is extremely difficult to capture, especially during the plucking process, where the track of the markers at the fingertips is lost very frequently. This work presents two models that allow the reconstruction of the position of occluded markers: a rigid-body model to track the motion of the guitar strings and a flexible-body model to track the motion of the hands. In combination with audio analysis (onset and pitch detection), the method can estimate a comprehensive set of sound control features that include the plucked string, the plucking finger, and the characteristics of the plucking gesture in the phases of contact, pressure and release (e.g., position, timing, velocity, direction, or string displacement)., This work has been sponsored by the European Union Horizon 2020 research and innovation program under grant agreement No. 688269 (TELMI project) and Beatriu de Pinos grant 2010 BP-A 00209 by the Catalan Research Agency (AGAUR).
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

Error monitoring and correction in violin performance: an EEG study

  • Blanco, Angel David
  • Costa-Faidella, Jordi
  • Pérez Carrillo, Alfonso Antonio, 1977-
  • Dalmazzo, David
  • Ramirez Melendez, Rafael
  • San Miguel Insúa, Iria
Previous research using electroencephalography (EEG) has investigated neural error-related music production processes in expert pianists reporting a frontal event-related negativity followed by a P300 event-related potential (ERP) after mistuned notes. However, piano playing does not rely on auditory feedback and corrective movements when dealing with accurate note tuning, thus offering an incomplete picture about error monitoring processes in players of many other musical instruments, including the human voice. We here designed a setup allowing us to collect the EEG of expert bow-string players while performing short melodies on the violin, (a fretless instrument) while also manipulating in real-time the produced pitch in a controlled manner. 15 expert bow-string players were asked to 1) listen to a reference melody of four notes; 2) play that melody with a violin (Active Condition); and 3) listen passively to the replay of their performance (Replayed Condition). Randomly, the auditory feedback of one of the four melody notes was manipulated in the Active Condition by lowering or lifting the pitch by half semitone. We found that mistuned notes, independently of whether they were manipulated externally or were produced by natural playing errors, elicited an f-ERN followed by a P300. We also found that the amplitude of the f-ERN was larger for faster corrective movements than for slower ones. In addition, between the f-ERN and the P300, we also identified two negative components-one in parietal areas and another in central-right regions-whose amplitudes were greater for slower corrective movements. We suggest the existence of two error-monitoring systems: a "fast" system mediated by the medial frontal cortex that quickly corrects errors when certainty is high and a "slow" system mediated by the posterior parietal cortex that accumulates evidence under uncertainty before triggering correction., This work was partly sponsored by the Musical AI project (PID2019-111403GB-I00) funded by MCIN/AEI/ 10.13039/501100011033 and by 'ERDF A way of making Europe'; by the Project PID2021-128790NB-I00, funded by MCIN/AEI/ 10.13039/501100011033 / FEDER UE; and by the European Union Horizon 2020 research and innovation program under grant agreement No. 688269 (TELMI project). It has been additionally supported by the Excellence Research Group 2017SGR-974 funded by the Secretaria d'Universitats i Recerca del Departament d'Empresa i Coneixement de la Generalitat de Catalunya and by the MDM-2017-0729-18-2M Maria de Maeztu Center of Excellence UBNeuro, funded by MCIN/AEI/ 10.13039/501100011033. ISM was supported by grant RYC-2013-12577, funded by MCIN/AEI/ 10.13039/501100011033 and by 'ESF Investing in your future'. Open access publication fees were partly covered by the University of Barcelona (Ajudes de la Universitat de Barcelona per a publicar en accés obert).
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

A Machine learning approach to violin bow technique classification: a comparison between IMU and MOCAP systems

  • Dalmazzo, David
  • Tassani, Simone
  • Ramírez, Rafael, 1966-
Comunicació presentada a: 5th international Workshop on Sensor-based Activity Recognition and Interaction celebrat el 20 i 21 de setembre de 2018 a Berlin, Alemanya., Motion Capture (MOCAP) Systems have been used to analyze body motion and postures in biomedicine, sports, rehabilitation, and music. With the aim to compare the precision of low-cost devices for motion tracking (e.g. Myo) with the precision of MOCAP systems in the context of music performance, we recorded MOCAP and Myo data of a top professional violinist executing four fundamental bowing techniques (i.e. Détaché, Martelé, Spiccato and Ricochet). Using the recorded data we applied machine learning techniques to train models to classify the four bowing techniques. Despite intrinsic differences between the MOCAP and low-cost data, the Myo-based classifier resulted in slightly higher accuracy than the MOCAP-based classifier. This result shows that it is possible to develop music-gesture learning applications based on low-cost technology which can be used in home environments for self-learning practitioners., This work has been partly sponsored by the Spanish TIN project TIMUL (TIN 2013-48152-C2-2-R), the European Union Horizon 2020 research and innovation programme under grant agreement No. 688269 (TELMI project), and the Spanish Ministry of Economy and Competitiveness under the Maria de Maeztu Units of Excellence Programme (MDM-2015-0502).
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

JamSketch: improvisation support system with GA-based melody creation from user’s drawing

  • Kitahara, Tetsuro
  • Giraldo, Sergio
  • Ramírez, Rafael, 1966-
Comunicació presentada al 13th International Symposium (CMMR), celebrat del 25 al 28 de setembre de 2017 a Matosinhos, Portugal., Improvisation is an enjoyable form of music performance but requires advanced skills and knowledge of music because the player has to create melodies immediately during the performance. To support improvisations by people without skills or knowledge of music, we have to develop (1) a human interface that can be used without skills or knowledge of music and (2) automatic melody generation from the user’s input that may be musically abstract or incomplete. In this paper, we develop an improvisation support system based on melodic outlines, which represent the overall contour of melodies, with a function of melody generation using a genetic algorithm (GA). Once the user draws a melodic outline on the piano-roll display with the mouse or touch screen, the system immediately generates a melody using a GA with a fitness function based on the similarity to the outline, an N-gram probability, and entropy. The generated melody is performed expressively based on expression parameters calculated with an machine learning approach. The results of listening tests for comparing human performances and the system’s performances suggest that generated melodies have quality similar to performances by non-expert human performers., This work was supported by JSPS KAKENHI Grant Numbers 26240025, 26280089, 16K16180, 16H01744, 16KT0136, and 17H00749 as well as by the Spanish TIMuL Project (TIN2013-48152-C2-2-R), and the TELMI Project of the Horizon 2020 Research and Innovation Programme (grant agreement No. 688269).