Skip to main content

Resultados relacionados

Resultados totales (Incluyendo duplicados): 38
Encontrada(s) 1 página(s)

Repositorio Digital de la UPF
Publication Artículo científico (JournalArticle)

On the interrelation between listener characteristics and the perception of emotion in classical orchestra music

  • Schedl, Markus
  • Gómez Gutiérrez, Emilia, 1975-
  • Trent, Erika S.
  • Tkalčič, Marko
  • Eghbal-Zadeh, Hamid
  • Martorell Domínguez, Agustín
This study deals with the strong relationship between emotions and music, investigating three main research questions: (RQ1) Are there differences in human music perception (e.g., emotions, tempo, instrumentation, and complexity), according to musical education, experience, demographics, and personality traits?; (RQ2) Do certain perceived music characteristics correlate (e.g., tension and sadness), irrespective of a particular listener's background or personality?; (RQ3) Does human perception of music characteristics, such as emotions and tempo, correlate with descriptors extracted from music audio signals? To investigate our research questions, we conducted two user studies focusing on different groups of subjects. We used web-based surveys to collect information about demographics, listening habits, musical education, personality, and perceptual ratings with respect to perceived emotions, tempo, complexity, and instrumentation for 15 segments of Beethoven's 3rd symphony, “Eroica”. Our experiments showed that all three research questions can be affirmed, at least partly. We found strong support for RQ2 and RQ3, while RQ1 could be confirmed only for some perceptual aspects and user groups., This research is supported by the EU-FP7 project no. 601166 (“PHENICX”), the Austrian Science Fund (FWF): P25655 and the Spanish Government (CASAS project TIN2015-70816-R).
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

Musical instrument recognition in user-generated videos using a multimodal convolutional neural network architecture

  • Slizovskaia, Olga
  • Gómez Gutiérrez, Emilia, 1975-
  • Haro Ortega, Gloria
Comunicació presentada a la International Conference on Multimedia Retrieval celebrada del 6 al 9 de juny de 2017 a Bucarest, Romania., This paper presents a method for recognizing musical instruments in user-generated videos. Musical instrument recognition from music signals is a well-known task in the music information retrieval (MIR) field, where current approaches rely on the analysis of the good-quality audio material. This work addresses a real-world scenario with several research challenges, i.e. the analysis of user-generated videos that are varied in terms of recording conditions and quality and may contain multiple instruments sounding simultaneously and background noise. Our approach does not only focus on the analysis of audio information, but we exploit the multimodal information embedded in the audio and visual domains. In order to do so, we develop a Convolutional Neural Network (CNN) architecture which combines learned representations from both modalities at a late fusion stage. Our approach is trained and evaluated on two large-scale video datasets: YouTube-8M and FCVID. The proposed architectures demonstrate state-of-the-art results in audio and video object recognition, provide additional robustness to missing modalities, and remains computationally cheap to train., is work is partly supported by the Spanish Ministry of Economy and Competitiveness under the Maria de Maeztu Units of Excellence Programme (MDM-2015-0502), the CASAS Spanish research project (TIN2015-70816-R), and project TIN2015-70410-C2- 1-R (MINECO/FEDER, UE). We gratefully acknowledge the support of NVIDIA Corporation with the donation of the Titan X GPU used for this research.
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

A plan for sustainable MIR evaluation

  • McFee, Brian
  • Humphrey, Eric J.
  • Urbano, Julián
The Music Information Retrieval Evaluation eXchange (MIREX) is a valuable community service, having established standard datasets, metrics, baselines, methodologies, and infrastructure for comparing MIR methods. While MIREX has managed to successfully maintain operations for over a decade, its long-term sustainability is at risk. The imposed constraint that input data cannot be made freely available to participants necessitates that all algorithms run on centralized computational resources, which are administered by a limited number of people. This incurs an approximately linear cost with the number of submissions, exacting significant tolls on both human and financial resources, such that the current paradigm becomes less tenable as participation increases. To alleviate the recurring costs of future evaluation campaigns, we propose a distributed, community-centric paradigm for system evaluation, built upon the principles of openness, transparency, reproducibility, and incremental evaluation. We argue that this proposal has the potential to reduce operating costs to sustainable levels. Moreover, the proposed paradigm would improve scalability, and eventually result in the release of large, open datasets for improving both MIR techniques and evaluation methods., B.M. is supported by the Moore-Sloan Data Science Environment at NYU. J.U. is supported by the Spanish Government: JdC postdoctoral fellowship, and projects TIN2015-70816-R and MDM-2015-0502.
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

Data augmentation for deep learning source separation of HipHop songs

  • Martel Baro, Héctor
  • Miron, Marius
Comunicació presentada a: 10th International Workshop on Machine Learning and Music, celebrat el 6 d'octbure de 2017 a Barcelona., Training deep learning source separation methods involves computationally intensive procedures relying on large multi-track datasets. In this paper we use data augmentation to improve hip hop source sepa- ration using small training datasets. We analyze different training strate- gies and data augmentation techniques with respect to their generaliza- tion capabilities. Moreover, we propose a hip hop multi-track dataset and we implemented a web demo to demonstrate our use scenario. The evaluation is done on a part of the dataset and hip-hop songs from an external dataset.
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

A comparison of melody extraction methods based on source-filter modelling

  • Bosch, Juan J.
  • Bittner, Rachel M.
  • Salamon, Justin
  • Gómez Gutiérrez, Emilia, 1975-
This work explores the use of source-filter models for pitch/nsalience estimation and their combination with different/npitch tracking and voicing estimation methods for automatic/nmelody extraction. Source-filter models are used/nto create a mid-level representation of pitch that implicitly/nincorporates timbre information. The spectrogram of/na musical audio signal is modelled as the sum of the leading/nvoice (produced by human voice or pitched musical instruments)/nand accompaniment. The leading voice is then/nmodelled with a Smoothed Instantaneous Mixture Model/n(SIMM) based on a source-filter model. The main advantage/nof such a pitch salience function is that it enhances/nthe leading voice even without explicitly separating it from/nthe rest of the signal. We show that this is beneficial/nfor melody extraction, increasing pitch estimation accuracy/nand reducing octave errors in comparison with simpler/npitch salience functions. The adequate combination with/nvoicing detection techniques based on pitch contour characterisation/nleads to significant improvements over stateof-the-art/nmethods, for both vocal and instrumental music., This work is partially supported by the European Union/nunder the PHENICX project (FP7-ICT-601166) and the/nSpanish Ministry of Economy and Competitiveness under/nCASAS project (TIN2015-70816-R) and Maria de Maeztu/nUnits of Excellence Programme (MDM-2015-0502)
Repositorio Digital de la UPF
Publication Artículo científico (JournalArticle)

Score-informed source separation for multichannel orchestral recordings

  • Miron, Marius
  • Carabias Orti, Julio J.
  • Bosch, Juan J.
  • Gómez Gutiérrez, Emilia, 1975-
  • Janer Mestres, Jordi
This paper proposes a system for score-informed audio source separation for multichannel orchestral recordings. The orchestral music repertoire relies on the existence of scores. Thus, a reliable separation requires a good alignment of the score with the audio of the performance. To that extent, automatic score alignment methods are reliable when allowing a tolerance window around the actual onset and offset. Moreover, several factors increase the difficulty of our task: a high reverberant image, large ensembles having rich polyphony, and a large variety of instruments recorded within a distant-microphone setup. To solve these problems, we design context-specific methods such as the refinement of score-following output in order to obtain a more precise alignment. Moreover, we extend a close-microphone separation framework to deal with the distant-microphone orchestral recordings. Then, we propose the first open evaluation dataset in this musical context, including annotations of the notes played by multiple instruments from an orchestral ensemble. The evaluation aims at analyzing the interactions of important parts of the separation framework on the quality of separation. Results show that we are able to align the original score with the audio of the performance and separate the sources corresponding to the instrument sections., This work is partially supported by the Spanish Ministry of Economy and Competitiveness under CASAS Project (TIN2015-70816-R).
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

Automatic transcription of flamenco guitar falsetas

  • Rodríguez, Sonia
  • Gómez Gutiérrez, Emilia, 1975-
  • Cuesta, Helena
Comunicació presentada al 8th International Workshop on Folk Music Analysis celebrat del 26 al 29 de juny de 2018 a Tessalònica, Grècia., This work deals with the automatic transcription and characterization of flamenco guitar, with a focus on short melodic interludes improvised between sung verses. These are called falsetas in the flamenco argot and are very challenging for manual and automatic transcription due to their fast and highly ornamented nature. However, they are a key resource for guitar players to practice. We adapted a state of the art singing transcription algorithm to process an audio signal containing one or several guitar falsetas and extract their symbolic representation. The algorithms first perform a segmentation to locate the guitar fragments and then a symbolic transcription of these segments into symbolic representation. In order to evaluate it, we collected the first (to our knowledge) annotated falseta datasets. Our results confirm the difficulty of the task, and a detailed study of two transcriptions revealed that combining the algorithm with specific musical knowledge about the scale used by the song, improves the performance of the system. Our approach follows the principles of research reproducibility, and the system is integrated in a computer-assisted paradigm, where the user complements the automatic annotation with a priory knowledge to generate a final transcription., This work is partially supported by the Spanish Ministry of Economy and Competitiveness under the CASAS project (TIN2015-70816-R).
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

Monaural score-informed source separation for classical music using convolutional neural networks

  • Miron, Marius
  • Janer Mestres, Jordi
  • Gómez Gutiérrez, Emilia, 1975-
Comunicació presentada a la 18th International Society for Music Information Retrieval Conference (ISMIR 2017), celebrada els dies 23 a 27 d'octubre de 2017 a Suzhou, Xina., Score information has been shown to improve music source separation when included into non-negative matrix factorization (NMF) frameworks. Recently, deep learning approaches have outperformed NMF methods in terms of separation quality and processing time, and there is scope to extend them with score information. In this paper, we propose a score-informed separation system for classical music that is based on deep learning. We propose a method to derive training features from audio files and the corresponding coarsely aligned scores for a set of classical music pieces. Additionally, we introduce a convolutional neural network architecture (CNN) with the goal of estimating time-frequency masks for source separation. Our system is trained with synthetic renditions derived from the original scores and can be used to separate real-life performances based on the same scores, provided a coarse audio-to-score alignment. The proposed system achieves better performance (SDR and SIR) and is less computationally intensive than a score-informed NMF system on a dataset comprising Bach chorales., This work is partially supported by the Spanish Ministry of Economy and Competitiveness under CASAS project (TIN2015-70816-R) and by the Spanish Ministry of Economy and Competitiveness under the Maria de Maeztu Units of Excellence Programme (MDM2015-0502).
Repositorio Digital de la UPF
Publication Artículo científico (JournalArticle)

Mapping by observation: building a user-tailored conducting system from spontaneous movements

  • Sarasúa Berodia, Álvaro
  • Urbano, Julián
  • Gómez Gutiérrez, Emilia, 1975-
Metaphors are commonly used in interface design within Human-Computer Interaction (HCI). Interface metaphors provide users with a way to interact with the computer that resembles a known activity, giving instantaneous knowledge or intuition about how the interaction works. A widely used one in Digital Musical Instruments (DMIs) is the conductor-orchestra metaphor, where the orchestra is considered as an instrument controlled by the movements of the conductor. We propose a DMI based on the conductor metaphor that allows to control tempo and dynamics and adapts its mapping specifically for each user by observing spontaneous conducting movements (i.e., movements performed on top of fixed music without any instructions). We refer to this as mapping by observation given that, even though the system is trained specifically for each user, this training is not done explicitly and consciously by the user. More specifically, the system adapts its mapping based on the tendency of the user to anticipate or fall behind the beat and observing the Motion Capture descriptors that best correlate to loudness during spontaneous conducting. We evaluate the proposed system in an experiment with twenty four (24) participants where we compare it with a baseline that does not perform this user-specific adaptation. The comparison is done in a context where the user does not receive instructions and, instead, is allowed to discover by playing. We evaluate objective and subjective measures from tasks where participants have to make the orchestra play at different loudness levels or in synchrony with a metronome. Results of the experiment prove that the usability of the system that automatically learns its mapping from spontaneous movements is better both in terms of providing a more intuitive control over loudness and a more precise control over beat timing. Interestingly, the results also show a strong correlation between measures taken from the data used for training and the improvement introduced by the adapting system. This indicates that it is possible to estimate in advance how useful the observation of spontaneous movements is to build user-specific adaptations. This opens interesting directions for creating more intuitive and expressive DMIs, particularly in public installations., This work was supported by the European Union Seventh Framework Programme FP7 / 2007-2013 through the PHENICX project (grant agreement no. 601166) and by the CASAS project (TIN2015-70816-R).
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

Monoaural audio source separation using deep convolutional neural networks

  • Chandna, Pritish
  • Miron, Marius
  • Janer Mestres, Jordi
  • Gómez Gutiérrez, Emilia, 1975-
Comunicació presentada a 13th International Conference on Latent Variable Analysis and Signal Separation, celebrada a Grenoble (França) els dies 21 a 23 de febrer de 2017., In this paper we introduce a low-latency monaural source separation framework using a Convolutional Neural Network (CNN). We use a CNN to estimate time-frequency soft masks which are applied for source separation. We evaluate the performance of the neural network on a database comprising of musical mixtures of three instruments: voice, drums, bass as well as other instruments which vary from song to song. The proposed architecture is compared to a Multilayer Perceptron (MLP), achieving on-par results and a significant improvement in processing time. The algorithm was submitted to source separation evaluation campaigns to test efficiency, and achieved competitive results., This work is partially supported by the Spanish Ministry of Economy and Competitiveness under CASAS project (TIN2015-70816-R).
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

Melody extraction based on a source-filter model using pitch contour selection

  • Bosch, Juan J.
  • Gómez Gutiérrez, Emilia, 1975-
This work proposes a melody extraction method which combines a pitch salience function based on source-filter modelling with melody tracking based on pitch contour selection. We model the spectrogram of a musical audio signal as the sum of the leading voice and accompaniment. The leading voice is modelled with a Smoothed Instantaneous Mixture Model (SIMM), and the accompaniment is modelled with a Non-negative Matrix Factorization (NMF). The main benefit of this representation is that it incorporates timbre information, and that the leading voice is enhanced, even without an explicit separation from the rest of the signal. Two different salience functions based on SIMM are proposed, in order to adapt the output of such model to the pitch contour based tracking. Candidate melody pitch contours are then created by grouping pitch sequences, using auditory streaming cues. Finally, melody pitch contours are selected using a set of heuristic rules based on contour characteristics and smoothness constraints. The evaluation on a large set of challenging polyphonic music material, shows that the proposed salience functions help increasing the salience of melody pitches in comparison to similar methods. The complete melody extraction methods also achieve a higher overall accuracy than state-of-the-art approaches when evaluated on both vocal and instrumental music., This work is partially supported by the European Union under the PHENICX project (FP7-ICT-601166) and the Spanish Ministry of Economy and Competitiveness under CASAS project (TIN2015-70816-R) and Maria de Maeztu Units of Excellence Programme (MDM-2015-0502).
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

A Framework for multi-f0 modeling in SATB choir recordings

  • Cuesta, Helena
  • Gómez Gutiérrez, Emilia, 1975-
  • Chandna, Pritish
Comunicació presentada a: Sound & Music Computing Conference celebrada del 28 al 31 de maig de 2019 a Màlaga, Espanya., Fundamental frequency (f0) modeling is an important but relatively unexplored aspect of choir singing. Performance evaluation as well as auditory analysis of singing, whether individually or in a choir, often depend on extracting f0 contours for the singing voice. However, due to the large number of singers, singing at a similar frequency range, extracting the exact individual pitch contours from choir recordings is a challenging task. In this paper, we address this task and develop a methodology for modeling pitch contours of SATB choir recordings. A typical SATB choir consists of four parts, each covering a distinct range of pitches and often with multiple singers each. We first evaluate some state-of-the-art multi-f0 estimation systems for the particular case of choirs with a single singer per part, and observe that the pitch of individual singers can be estimated to a relatively high degree of accuracy. We observe, however, that the scenario of multiple singers for each choir part (i.e. unison singing) is far more challenging. In this work we propose a methodology based on combining a multi-f0 estimation methodology based on deep learning followed by a set of traditional DSP techniques to model f0 and its dispersion instead of a single f0 trajectory for each choir part. We present and discuss our observations and test our framework with different singer configurations., This work is partially supported by the European Commission under the TROMPA project (H2020 770376) and by Spanish Ministry of Economy and Competitiveness under the CASAS project (TIN2015-70816-R). First author is supported by FI Predoctoral Grant from AGAUR (Generalitat de Catalunya).
Repositorio Digital de la UPF
Publication Artículo científico (JournalArticle)

A neural parametric singing synthesizer modeling timbre and expression from natural songs

  • Blaauw, Merlijn
  • Bonada, Jordi, 1973-
We recently presented a new model for singing synthesis based on a modified version of the WaveNet architecture. Instead of modeling raw waveform, we model features produced by a parametric vocoder that separates the influence of pitch and timbre. This allows conveniently modifying pitch to match any target melody, facilitates training on more modest dataset sizes, and significantly reduces training and generation times. Nonetheless, compared to modeling waveform directly, ways of effectively handling higher-dimensional outputs, multiple feature streams and regularization become more important with our approach. In this work, we extend our proposed system to include additional components for predicting F0 and phonetic timings from a musical score with lyrics. These expression-related features are learned together with timbrical features from a single set of natural songs. We compare our method to existing statistical parametric, concatenative, and neural network-based approaches using quantitative metrics as well as listening tests., This work is partially supported by the Spanish Ministry of Economy and Competitiveness under the CASAS project (TIN2015-70816-R).
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

Timbre analysis of music audio signals with convolutional neural networks

  • Pons Puig, Jordi
  • Slizovskaia, Olga
  • Gómez Gutiérrez, Emilia, 1975-
  • Serra, Xavier
Comunicació presentada a la EUSIPCO 2017: 25th European Signal Processing Conference, celebrada els dies 28 d'agost a 2 de setembre de 2017 a Kos, Grècia., The focus of this work is to study how to efficiently tailor Convolutional Neural Networks (CNNs) towards learning timbre representations from log-mel magnitude spectrograms. We first review the trends when designing CNN architectures. Through this literature overview we discuss which are the crucial points to consider for efficiently learning timbre representations using CNNs. From this discussion we propose a design strategy meant to capture the relevant time-frequency contexts for learning timbre, which permits using domain knowledge for designing architectures. In addition, one of our main goals is to design efficient CNN architectures – what reduces the risk of these models to over-fit, since CNNs’ number of parameters is minimized. Several architectures based on the design principles we propose are successfully assessed for different research tasks related to timbre: singing voice phoneme classification, musical instrument recognition and music auto-tagging., This work is partially supported by: the Maria de Maeztu Units of Excellence Programme (MDM-2015-0502), the CompMusic project (ERC grant agreement 267583) and the CASAS Spanish research project (TIN2015-70816-R).
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

Cante Flamenco Tech: aprendizaje del cante con la ayuda de la tecnología

  • Gómez Gutiérrez, Emilia, 1975-
  • Guerrero, Alba
  • Rodríguez, Sonia
Comunicació presentada al V Congreso Interdisciplinar Investigación y Flamenco (INFLA) celebrat el 7 i 8 de juny de 2018 a Sevilla., Cante Flamenco Tech es un proyecto pedagógico cuyo objetivo es el de generar materiales docentes online para el aprendizaje del cante flamenco, siguiendo un énfasis tecnolégico. Para ello, el proyecto integra las tecnologías actuales de análisis de sonido y música en su aplicación a uno de los aspectos más característicos del cante flamenco como son sus recursos vocales. Como aspecto innovador en la docencia, se propone un enfoque interdisciplinar que integra los ámbitos de la acústica, el procesado de sonido y el canto. En éste artículo se describen en primer lugar las caracaterísticas vocales que se han seleccionado en el curso. Además de los recursos audiovisuales y de evaluación propios de éste tipo de cursos (víıdeos, cuestionarios y fórums de discusión), nuestra propuesta es la de integrar tecnologías automáticas de análisis de voz cantada para visualizar los aspectos más característicos de la técnica vocal de los cantaores. Después de introducir las estrategias de visualización seleccionada, presentamos los resultados del proyecto y un ejemplo de la utilidad de las visualizaciones para la caracterización expresiva de diferentes interpretaciones de un mismo palo flamenco. Este tipo de curso es novedoso en cuanto al ámbito, orientación tecnológica e impacto potencial en la comunidad de amantes del flamenco y de la tecnología., Cante Flamenco Tech is a pedagogical project intended to create online teaching materials for flamenco singing following a music technology perspective. The project integrates current technologies of audio analysis for describing one of the most important features of flamenco singing: a set of characteristic vocal resources. Our project is interdisciplinary, as it integrates concepts from different disciplines such as acoustics, sound processing and singing practice. The paper first describes the set of vocal resources presented in the course. In addition to audiovisual resources and evaluation strategies commonly found in online teaching material (videos, questionnaires and discussion forums), our project integrates automatic singing analysis tools for visualizing the most important aspects of vocal techniques in singers. After presenting the selected visualization strategies, we provide some results of our analysis and an illustration of the usefulness of visualizations for the expressive characterization of different performances from the same flamenco style. Our project is novel with respect to context, technological orientation and potential impact in the community of flamenco lovers and technologists., Nuestro proyecto cuenta con el apoyo económico del plan de innovación docente PlaCLIK 2017-2018 de la Universidad Pompeu Fabra de Barcelona y del proyecto CASAS, Ministerio de Economía, Industria y Competitividad de España, TIN2015-70816-R.
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

Generating data to train convolutional neural networks for classical music source separation

  • Miron, Marius
  • Janer Mestres, Jordi
  • Gómez Gutiérrez, Emilia, 1975-
Comunicació presentada a la conferència 14th Sound and Music Computing Conference, celebrada a Finlàndia del 5 al 8 de juliol de 2017., Deep learning approaches have become increasingly popular in estimating time-frequency masks for audio source separation. However, training neural networks usually requires a considerable amount of data. Music data is scarce, particularly for the task of classical music source separation, where we need multi-track recordings with isolated instruments. In this work, we depart from the assumption that all the renditions of a piece are based on the same musical score, and we can generate multiple renditions of the score by synthesizing it with different performance properties, e.g. tempo, dynamics, timbre and local timing variations. We then use this data to train a convolutional neural network (CNN) which can separate with low latency all the renditions of a score or a set of scores. The trained model is tested on real life recordings and is able to effectively separate the corresponding sources. This work follows the principle of research reproducibility, providing related data and code, and can be extended to separate other pieces., The TITANX used for this research was donated by the NVIDIA Corporation. This work is partially supported by the Spanish Ministry of Economy and Competitiveness under CASAS project (TIN2015-70816-R) and by the Spanish Ministry of Economy and Competitiveness under the Maria de Maeztu Units of Excellence Programme (MDM- 2015-0502). We thank Agustin Martorell for his help with Sibelius and Pritish Chandna for his useful feedback.
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

Visual music transcription of clarinet video recordings trained with audio-based labelled data

  • Gómez Gutiérrez, Emilia, 1975-
  • Arias Martínez, Pablo
  • Zinemanas, Pablo
  • Haro Ortega, Gloria
Comunicació presentada a la 2017 IEEE International Conference on Computer Vision Workshops (ICCVW), celebrada els dies 22 a 29 d'octubre de 2017 a Venècia, Itàlia., Automatic transcription is a well-known task in the music information retrieval (MIR) domain, and consists on the computation of a symbolic music representation (e.g. MIDI) from an audio recording. In this work, we address the automatic transcription of video recordings when the audio modality is missing or it does not have enough quality, and thus analyze the visual information. We focus on the clarinet which is played by opening/closing a set of holes and keys. We propose a method for automatic visual note estimation by detecting the fingertips of the player and measuring their displacement with respect to the holes and keys of the clarinet. To this aim, we track the clarinet and determine its position on every frame. The relative positions of the fingertips are used as features of a machine learning algorithm trained for note pitch classification. For that purpose, a dataset is built in a semiautomatic way by estimating pitch information from audio signals in an existing collection of 4.5 hours of video recordings from six different songs performed by nine different players. Our results confirm the difficulty of performing visual vs audio automatic transcription mainly due to motion blur and occlusions that cannot be solved with a single view., This work is partly supported by the Spanish Ministry of Economy and Competitiveness under the Maria de Maeztu Units of Excellence Programme (MDM-2015-0502); the Spanish research projects CASAS (TIN2015-70816-R, MINECO/FEDER, UE) and MOTECAVI (TIN2015-70410-C2-1-R, MINECO/FEDER, UE); BPIFrance and Région Île de France in the framework of the FUI 18 Plein Phare project; the Office of Naval research by grant N00014-17-1-2552, ANR-DGA project ANR-12-ASTR-0035; and ANR-14-CE27-001 (MIRIAM).
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

A neural parametric singing synthesizer

  • Blaauw, Merlijn
  • Bonada, Jordi, 1973-
Comunicació i pòster presentats a l'Interspeech 2017 celebrat del 20 al 24 d'agost a Estocolm, Suècia., We present a new model for singing synthesis based on a modified version of the WaveNet architecture. Instead of modeling raw waveform, we model features produced by a parametric vocoder that separates the influence of pitch and timbre. This allows conveniently modifying pitch to match any target melody, facilitates training on more modest dataset sizes, and significantly reduces training and generation times. Our model makes frame-wise predictions using mixture density outputs rather than categorical outputs in order to reduce the required parameter count. As we found overfitting to be an issue with the relatively small datasets used in our experiments, we propose a method to regularize the model and make the autoregressive generation process more robust to prediction errors. Using a simple multi-stream architecture, harmonic, aperiodic and voiced/unvoiced components can all be predicted in a coherent manner. We compare our method to existing parametric statistical and state-of-the-art concatenative methods using quantitative metrics and a listening test. While naive implementations of the autoregressive generation algorithm tend to be inefficient, using a smart algorithm we can greatly speed up the process and obtain a system that’s competitive in both speed and quality., We gratefully acknowledge the support of NVIDIA Corporation with the donation of the Titan X Pascal GPU used for this research. We also thank Zya for providing the English datasets. Voctro Labs provided the Spanish dataset and the implementation of the fast generation algorithm. This work is partially supported by the Spanish Ministry of Economy and Competitiveness under the CASAS project (TIN2015-70816-R).
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

Analysis of intonation in unison choir singing

  • Cuesta, Helena
  • Gómez Gutiérrez, Emilia, 1975-
  • Martorell Domínguez, Agustín
  • Loáiciga, Felipe
Comunicació presentada a la 15th International Conference on Music Perception and Cognition celebrada del 23 al 28 de juliol a Graz, Àustria., Choir singers synchronize their fundamental frequency (f0) and timing when they perform together. In this work, we investigate several expressive characteristics of choir singing, with special emphasis on unison performances, to study how singers blend together and interact with each other by analyzing their f0 dispersion, intonation, and vibrato. We use the spectral autocorrelation algorithm to obtain the f0 trajectories of each individual singer and compute the f0 dispersion as the standard deviation of the f0 distribution. Using the Pearson correlation, we study the synchronization of singers in terms of their intonation and vibrato. Together with this study, we present a new choir singing dataset of voices singing together but recorded individually using directional close microphones. This dataset contains the individual recordings of 16 singers from a semiprofessional 4-part choir (SATB) performing 3 different songs. We found the f0 dispersion in unison sections to be in the range of 16 to 30 cents, and the largest average f0 dispersion was found in the basses. We observed that the intonation correlation varies a lot and is around 0.32 on average in all cases. The slowest piece was the one with the highest percentage of notes with vibrato, which had a frequency around 5 Hz for the whole dataset., This work is partially supported by the Spanish Ministry of Economy and Competitiveness under the CASAS project (TIN2015-70816-R) and the European Commission under the TROMPA project (H2020 770376).
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

A neural parametric singing synthesizer modeling timbre and expression from natural songs

  • Blaauw, Merlijn
  • Bonada, Jordi, 1973-
We recently presented a new model for singing synthesis based on a modified version of the WaveNet architecture. Instead of modeling raw waveform, we model features produced by a parametric vocoder that separates the influence of pitch and timbre. This allows conveniently modifying pitch to match any target melody, facilitates training on more modest dataset sizes, and significantly reduces training and generation times. Nonetheless, compared to modeling waveform directly, ways of effectively handling higher-dimensional outputs, multiple feature streams and regularization become more important with our approach. In this work, we extend our proposed system to include additional components for predicting F0 and phonetic timings from a musical score with lyrics. These expression-related features are learned together with timbrical features from a single set of natural songs. We compare our method to existing statistical parametric, concatenative, and neural network-based approaches using quantitative metrics as well as listening tests., This work is partially supported by the Spanish Ministry of Economy and Competitiveness under the CASAS project (TIN2015-70816-R).
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

A comparison of melody extraction methods based on source-filter modelling

  • Bosch, Juan J.
  • Bittner, Rachel M.
  • Salamon, Justin
  • Gómez Gutiérrez, Emilia, 1975-
This work explores the use of source-filter models for pitch/nsalience estimation and their combination with different/npitch tracking and voicing estimation methods for automatic/nmelody extraction. Source-filter models are used/nto create a mid-level representation of pitch that implicitly/nincorporates timbre information. The spectrogram of/na musical audio signal is modelled as the sum of the leading/nvoice (produced by human voice or pitched musical instruments)/nand accompaniment. The leading voice is then/nmodelled with a Smoothed Instantaneous Mixture Model/n(SIMM) based on a source-filter model. The main advantage/nof such a pitch salience function is that it enhances/nthe leading voice even without explicitly separating it from/nthe rest of the signal. We show that this is beneficial/nfor melody extraction, increasing pitch estimation accuracy/nand reducing octave errors in comparison with simpler/npitch salience functions. The adequate combination with/nvoicing detection techniques based on pitch contour characterisation/nleads to significant improvements over stateof-the-art/nmethods, for both vocal and instrumental music., This work is partially supported by the European Union/nunder the PHENICX project (FP7-ICT-601166) and the/nSpanish Ministry of Economy and Competitiveness under/nCASAS project (TIN2015-70816-R) and Maria de Maeztu/nUnits of Excellence Programme (MDM-2015-0502)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Monaural score-informed source separation for classical music using convolutional neural networks

  • Miron, Marius
  • Janer Mestres, Jordi
  • Gómez Gutiérrez, Emilia, 1975-
Comunicació presentada a la 18th International Society for Music Information Retrieval Conference (ISMIR 2017), celebrada els dies 23 a 27 d'octubre de 2017 a Suzhou, Xina., Score information has been shown to improve music source separation when included into non-negative matrix factorization (NMF) frameworks. Recently, deep learning approaches have outperformed NMF methods in terms of separation quality and processing time, and there is scope to extend them with score information. In this paper, we propose a score-informed separation system for classical music that is based on deep learning. We propose a method to derive training features from audio files and the corresponding coarsely aligned scores for a set of classical music pieces. Additionally, we introduce a convolutional neural network architecture (CNN) with the goal of estimating time-frequency masks for source separation. Our system is trained with synthetic renditions derived from the original scores and can be used to separate real-life performances based on the same scores, provided a coarse audio-to-score alignment. The proposed system achieves better performance (SDR and SIR) and is less computationally intensive than a score-informed NMF system on a dataset comprising Bach chorales., This work is partially supported by the Spanish Ministry of Economy and Competitiveness under CASAS project (TIN2015-70816-R) and by the Spanish Ministry of Economy and Competitiveness under the Maria de Maeztu Units of Excellence Programme (MDM2015-0502).
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Melody extraction based on a source-filter model using pitch contour selection

  • Bosch, Juan J.
  • Gómez Gutiérrez, Emilia, 1975-
This work proposes a melody extraction method which combines a pitch salience function based on source-filter modelling with melody tracking based on pitch contour selection. We model the spectrogram of a musical audio signal as the sum of the leading voice and accompaniment. The leading voice is modelled with a Smoothed Instantaneous Mixture Model (SIMM), and the accompaniment is modelled with a Non-negative Matrix Factorization (NMF). The main benefit of this representation is that it incorporates timbre information, and that the leading voice is enhanced, even without an explicit separation from the rest of the signal. Two different salience functions based on SIMM are proposed, in order to adapt the output of such model to the pitch contour based tracking. Candidate melody pitch contours are then created by grouping pitch sequences, using auditory streaming cues. Finally, melody pitch contours are selected using a set of heuristic rules based on contour characteristics and smoothness constraints. The evaluation on a large set of challenging polyphonic music material, shows that the proposed salience functions help increasing the salience of melody pitches in comparison to similar methods. The complete melody extraction methods also achieve a higher overall accuracy than state-of-the-art approaches when evaluated on both vocal and instrumental music., This work is partially supported by the European Union under the PHENICX project (FP7-ICT-601166) and the Spanish Ministry of Economy and Competitiveness under CASAS project (TIN2015-70816-R) and Maria de Maeztu Units of Excellence Programme (MDM-2015-0502).
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

A neural parametric singing synthesizer

  • Blaauw, Merlijn
  • Bonada, Jordi, 1973-
Comunicació i pòster presentats a l'Interspeech 2017 celebrat del 20 al 24 d'agost a Estocolm, Suècia., We present a new model for singing synthesis based on a modified version of the WaveNet architecture. Instead of modeling raw waveform, we model features produced by a parametric vocoder that separates the influence of pitch and timbre. This allows conveniently modifying pitch to match any target melody, facilitates training on more modest dataset sizes, and significantly reduces training and generation times. Our model makes frame-wise predictions using mixture density outputs rather than categorical outputs in order to reduce the required parameter count. As we found overfitting to be an issue with the relatively small datasets used in our experiments, we propose a method to regularize the model and make the autoregressive generation process more robust to prediction errors. Using a simple multi-stream architecture, harmonic, aperiodic and voiced/unvoiced components can all be predicted in a coherent manner. We compare our method to existing parametric statistical and state-of-the-art concatenative methods using quantitative metrics and a listening test. While naive implementations of the autoregressive generation algorithm tend to be inefficient, using a smart algorithm we can greatly speed up the process and obtain a system that’s competitive in both speed and quality., We gratefully acknowledge the support of NVIDIA Corporation with the donation of the Titan X Pascal GPU used for this research. We also thank Zya for providing the English datasets. Voctro Labs provided the Spanish dataset and the implementation of the fast generation algorithm. This work is partially supported by the Spanish Ministry of Economy and Competitiveness under the CASAS project (TIN2015-70816-R).
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

A plan for sustainable MIR evaluation

  • McFee, Brian
  • Humphrey, Eric J.
  • Urbano, Julián
The Music Information Retrieval Evaluation eXchange (MIREX) is a valuable community service, having established standard datasets, metrics, baselines, methodologies, and infrastructure for comparing MIR methods. While MIREX has managed to successfully maintain operations for over a decade, its long-term sustainability is at risk. The imposed constraint that input data cannot be made freely available to participants necessitates that all algorithms run on centralized computational resources, which are administered by a limited number of people. This incurs an approximately linear cost with the number of submissions, exacting significant tolls on both human and financial resources, such that the current paradigm becomes less tenable as participation increases. To alleviate the recurring costs of future evaluation campaigns, we propose a distributed, community-centric paradigm for system evaluation, built upon the principles of openness, transparency, reproducibility, and incremental evaluation. We argue that this proposal has the potential to reduce operating costs to sustainable levels. Moreover, the proposed paradigm would improve scalability, and eventually result in the release of large, open datasets for improving both MIR techniques and evaluation methods., B.M. is supported by the Moore-Sloan Data Science Environment at NYU. J.U. is supported by the Spanish Government: JdC postdoctoral fellowship, and projects TIN2015-70816-R and MDM-2015-0502.
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Visual music transcription of clarinet video recordings trained with audio-based labelled data

  • Gómez Gutiérrez, Emilia, 1975-
  • Arias Martínez, Pablo
  • Zinemanas, Pablo
  • Haro Ortega, Gloria
Comunicació presentada a la 2017 IEEE International Conference on Computer Vision Workshops (ICCVW), celebrada els dies 22 a 29 d'octubre de 2017 a Venècia, Itàlia., Automatic transcription is a well-known task in the music information retrieval (MIR) domain, and consists on the computation of a symbolic music representation (e.g. MIDI) from an audio recording. In this work, we address the automatic transcription of video recordings when the audio modality is missing or it does not have enough quality, and thus analyze the visual information. We focus on the clarinet which is played by opening/closing a set of holes and keys. We propose a method for automatic visual note estimation by detecting the fingertips of the player and measuring their displacement with respect to the holes and keys of the clarinet. To this aim, we track the clarinet and determine its position on every frame. The relative positions of the fingertips are used as features of a machine learning algorithm trained for note pitch classification. For that purpose, a dataset is built in a semiautomatic way by estimating pitch information from audio signals in an existing collection of 4.5 hours of video recordings from six different songs performed by nine different players. Our results confirm the difficulty of performing visual vs audio automatic transcription mainly due to motion blur and occlusions that cannot be solved with a single view., This work is partly supported by the Spanish Ministry of Economy and Competitiveness under the Maria de Maeztu Units of Excellence Programme (MDM-2015-0502); the Spanish research projects CASAS (TIN2015-70816-R, MINECO/FEDER, UE) and MOTECAVI (TIN2015-70410-C2-1-R, MINECO/FEDER, UE); BPIFrance and Région Île de France in the framework of the FUI 18 Plein Phare project; the Office of Naval research by grant N00014-17-1-2552, ANR-DGA project ANR-12-ASTR-0035; and ANR-14-CE27-001 (MIRIAM).
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

Mapping by observation: building a user-tailored conducting system from spontaneous movements

  • Sarasúa Berodia, Álvaro
  • Urbano, Julián
  • Gómez Gutiérrez, Emilia, 1975-
Metaphors are commonly used in interface design within Human-Computer Interaction (HCI). Interface metaphors provide users with a way to interact with the computer that resembles a known activity, giving instantaneous knowledge or intuition about how the interaction works. A widely used one in Digital Musical Instruments (DMIs) is the conductor-orchestra metaphor, where the orchestra is considered as an instrument controlled by the movements of the conductor. We propose a DMI based on the conductor metaphor that allows to control tempo and dynamics and adapts its mapping specifically for each user by observing spontaneous conducting movements (i.e., movements performed on top of fixed music without any instructions). We refer to this as mapping by observation given that, even though the system is trained specifically for each user, this training is not done explicitly and consciously by the user. More specifically, the system adapts its mapping based on the tendency of the user to anticipate or fall behind the beat and observing the Motion Capture descriptors that best correlate to loudness during spontaneous conducting. We evaluate the proposed system in an experiment with twenty four (24) participants where we compare it with a baseline that does not perform this user-specific adaptation. The comparison is done in a context where the user does not receive instructions and, instead, is allowed to discover by playing. We evaluate objective and subjective measures from tasks where participants have to make the orchestra play at different loudness levels or in synchrony with a metronome. Results of the experiment prove that the usability of the system that automatically learns its mapping from spontaneous movements is better both in terms of providing a more intuitive control over loudness and a more precise control over beat timing. Interestingly, the results also show a strong correlation between measures taken from the data used for training and the improvement introduced by the adapting system. This indicates that it is possible to estimate in advance how useful the observation of spontaneous movements is to build user-specific adaptations. This opens interesting directions for creating more intuitive and expressive DMIs, particularly in public installations., This work was supported by the European Union Seventh Framework Programme FP7 / 2007-2013 through the PHENICX project (grant agreement no. 601166) and by the CASAS project (TIN2015-70816-R).
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Monoaural audio source separation using deep convolutional neural networks

  • Chandna, Pritish
  • Miron, Marius
  • Janer Mestres, Jordi
  • Gómez Gutiérrez, Emilia, 1975-
Comunicació presentada a 13th International Conference on Latent Variable Analysis and Signal Separation, celebrada a Grenoble (França) els dies 21 a 23 de febrer de 2017., In this paper we introduce a low-latency monaural source separation framework using a Convolutional Neural Network (CNN). We use a CNN to estimate time-frequency soft masks which are applied for source separation. We evaluate the performance of the neural network on a database comprising of musical mixtures of three instruments: voice, drums, bass as well as other instruments which vary from song to song. The proposed architecture is compared to a Multilayer Perceptron (MLP), achieving on-par results and a significant improvement in processing time. The algorithm was submitted to source separation evaluation campaigns to test efficiency, and achieved competitive results., This work is partially supported by the Spanish Ministry of Economy and Competitiveness under CASAS project (TIN2015-70816-R).
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Analysis of intonation in unison choir singing

  • Cuesta, Helena
  • Gómez Gutiérrez, Emilia, 1975-
  • Martorell Domínguez, Agustín
  • Loáiciga, Felipe
Comunicació presentada a la 15th International Conference on Music Perception and Cognition celebrada del 23 al 28 de juliol a Graz, Àustria., Choir singers synchronize their fundamental frequency (f0) and timing when they perform together. In this work, we investigate several expressive characteristics of choir singing, with special emphasis on unison performances, to study how singers blend together and interact with each other by analyzing their f0 dispersion, intonation, and vibrato. We use the spectral autocorrelation algorithm to obtain the f0 trajectories of each individual singer and compute the f0 dispersion as the standard deviation of the f0 distribution. Using the Pearson correlation, we study the synchronization of singers in terms of their intonation and vibrato. Together with this study, we present a new choir singing dataset of voices singing together but recorded individually using directional close microphones. This dataset contains the individual recordings of 16 singers from a semiprofessional 4-part choir (SATB) performing 3 different songs. We found the f0 dispersion in unison sections to be in the range of 16 to 30 cents, and the largest average f0 dispersion was found in the basses. We observed that the intonation correlation varies a lot and is around 0.32 on average in all cases. The slowest piece was the one with the highest percentage of notes with vibrato, which had a frequency around 5 Hz for the whole dataset., This work is partially supported by the Spanish Ministry of Economy and Competitiveness under the CASAS project (TIN2015-70816-R) and the European Commission under the TROMPA project (H2020 770376).
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Data augmentation for deep learning source separation of HipHop songs

  • Martel Baro, Héctor
  • Miron, Marius
Comunicació presentada a: 10th International Workshop on Machine Learning and Music, celebrat el 6 d'octbure de 2017 a Barcelona., Training deep learning source separation methods involves computationally intensive procedures relying on large multi-track datasets. In this paper we use data augmentation to improve hip hop source sepa- ration using small training datasets. We analyze different training strate- gies and data augmentation techniques with respect to their generaliza- tion capabilities. Moreover, we propose a hip hop multi-track dataset and we implemented a web demo to demonstrate our use scenario. The evaluation is done on a part of the dataset and hip-hop songs from an external dataset.
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

On the interrelation between listener characteristics and the perception of emotion in classical orchestra music

  • Schedl, Markus
  • Gómez Gutiérrez, Emilia, 1975-
  • Trent, Erika S.
  • Tkalčič, Marko
  • Eghbal-Zadeh, Hamid
  • Martorell Domínguez, Agustín
This study deals with the strong relationship between emotions and music, investigating three main research questions: (RQ1) Are there differences in human music perception (e.g., emotions, tempo, instrumentation, and complexity), according to musical education, experience, demographics, and personality traits?; (RQ2) Do certain perceived music characteristics correlate (e.g., tension and sadness), irrespective of a particular listener's background or personality?; (RQ3) Does human perception of music characteristics, such as emotions and tempo, correlate with descriptors extracted from music audio signals? To investigate our research questions, we conducted two user studies focusing on different groups of subjects. We used web-based surveys to collect information about demographics, listening habits, musical education, personality, and perceptual ratings with respect to perceived emotions, tempo, complexity, and instrumentation for 15 segments of Beethoven's 3rd symphony, “Eroica”. Our experiments showed that all three research questions can be affirmed, at least partly. We found strong support for RQ2 and RQ3, while RQ1 could be confirmed only for some perceptual aspects and user groups., This research is supported by the EU-FP7 project no. 601166 (“PHENICX”), the Austrian Science Fund (FWF): P25655 and the Spanish Government (CASAS project TIN2015-70816-R).
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Cante Flamenco Tech: aprendizaje del cante con la ayuda de la tecnología

  • Gómez Gutiérrez, Emilia, 1975-
  • Guerrero, Alba
  • Rodríguez, Sonia
Comunicació presentada al V Congreso Interdisciplinar Investigación y Flamenco (INFLA) celebrat el 7 i 8 de juny de 2018 a Sevilla., Cante Flamenco Tech es un proyecto pedagógico cuyo objetivo es el de generar materiales docentes online para el aprendizaje del cante flamenco, siguiendo un énfasis tecnolégico. Para ello, el proyecto integra las tecnologías actuales de análisis de sonido y música en su aplicación a uno de los aspectos más característicos del cante flamenco como son sus recursos vocales. Como aspecto innovador en la docencia, se propone un enfoque interdisciplinar que integra los ámbitos de la acústica, el procesado de sonido y el canto. En éste artículo se describen en primer lugar las caracaterísticas vocales que se han seleccionado en el curso. Además de los recursos audiovisuales y de evaluación propios de éste tipo de cursos (víıdeos, cuestionarios y fórums de discusión), nuestra propuesta es la de integrar tecnologías automáticas de análisis de voz cantada para visualizar los aspectos más característicos de la técnica vocal de los cantaores. Después de introducir las estrategias de visualización seleccionada, presentamos los resultados del proyecto y un ejemplo de la utilidad de las visualizaciones para la caracterización expresiva de diferentes interpretaciones de un mismo palo flamenco. Este tipo de curso es novedoso en cuanto al ámbito, orientación tecnológica e impacto potencial en la comunidad de amantes del flamenco y de la tecnología., Cante Flamenco Tech is a pedagogical project intended to create online teaching materials for flamenco singing following a music technology perspective. The project integrates current technologies of audio analysis for describing one of the most important features of flamenco singing: a set of characteristic vocal resources. Our project is interdisciplinary, as it integrates concepts from different disciplines such as acoustics, sound processing and singing practice. The paper first describes the set of vocal resources presented in the course. In addition to audiovisual resources and evaluation strategies commonly found in online teaching material (videos, questionnaires and discussion forums), our project integrates automatic singing analysis tools for visualizing the most important aspects of vocal techniques in singers. After presenting the selected visualization strategies, we provide some results of our analysis and an illustration of the usefulness of visualizations for the expressive characterization of different performances from the same flamenco style. Our project is novel with respect to context, technological orientation and potential impact in the community of flamenco lovers and technologists., Nuestro proyecto cuenta con el apoyo económico del plan de innovación docente PlaCLIK 2017-2018 de la Universidad Pompeu Fabra de Barcelona y del proyecto CASAS, Ministerio de Economía, Industria y Competitividad de España, TIN2015-70816-R.
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

Score-informed source separation for multichannel orchestral recordings

  • Miron, Marius
  • Carabias Orti, Julio J.
  • Bosch, Juan J.
  • Gómez Gutiérrez, Emilia, 1975-
  • Janer Mestres, Jordi
This paper proposes a system for score-informed audio source separation for multichannel orchestral recordings. The orchestral music repertoire relies on the existence of scores. Thus, a reliable separation requires a good alignment of the score with the audio of the performance. To that extent, automatic score alignment methods are reliable when allowing a tolerance window around the actual onset and offset. Moreover, several factors increase the difficulty of our task: a high reverberant image, large ensembles having rich polyphony, and a large variety of instruments recorded within a distant-microphone setup. To solve these problems, we design context-specific methods such as the refinement of score-following output in order to obtain a more precise alignment. Moreover, we extend a close-microphone separation framework to deal with the distant-microphone orchestral recordings. Then, we propose the first open evaluation dataset in this musical context, including annotations of the notes played by multiple instruments from an orchestral ensemble. The evaluation aims at analyzing the interactions of important parts of the separation framework on the quality of separation. Results show that we are able to align the original score with the audio of the performance and separate the sources corresponding to the instrument sections., This work is partially supported by the Spanish Ministry of Economy and Competitiveness under CASAS Project (TIN2015-70816-R).
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Timbre analysis of music audio signals with convolutional neural networks

  • Pons Puig, Jordi
  • Slizovskaia, Olga
  • Gómez Gutiérrez, Emilia, 1975-
  • Serra, Xavier
Comunicació presentada a la EUSIPCO 2017: 25th European Signal Processing Conference, celebrada els dies 28 d'agost a 2 de setembre de 2017 a Kos, Grècia., The focus of this work is to study how to efficiently tailor Convolutional Neural Networks (CNNs) towards learning timbre representations from log-mel magnitude spectrograms. We first review the trends when designing CNN architectures. Through this literature overview we discuss which are the crucial points to consider for efficiently learning timbre representations using CNNs. From this discussion we propose a design strategy meant to capture the relevant time-frequency contexts for learning timbre, which permits using domain knowledge for designing architectures. In addition, one of our main goals is to design efficient CNN architectures – what reduces the risk of these models to over-fit, since CNNs’ number of parameters is minimized. Several architectures based on the design principles we propose are successfully assessed for different research tasks related to timbre: singing voice phoneme classification, musical instrument recognition and music auto-tagging., This work is partially supported by: the Maria de Maeztu Units of Excellence Programme (MDM-2015-0502), the CompMusic project (ERC grant agreement 267583) and the CASAS Spanish research project (TIN2015-70816-R).
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

A Framework for multi-f0 modeling in SATB choir recordings

  • Cuesta, Helena
  • Gómez Gutiérrez, Emilia, 1975-
  • Chandna, Pritish
Comunicació presentada a: Sound & Music Computing Conference celebrada del 28 al 31 de maig de 2019 a Màlaga, Espanya., Fundamental frequency (f0) modeling is an important but relatively unexplored aspect of choir singing. Performance evaluation as well as auditory analysis of singing, whether individually or in a choir, often depend on extracting f0 contours for the singing voice. However, due to the large number of singers, singing at a similar frequency range, extracting the exact individual pitch contours from choir recordings is a challenging task. In this paper, we address this task and develop a methodology for modeling pitch contours of SATB choir recordings. A typical SATB choir consists of four parts, each covering a distinct range of pitches and often with multiple singers each. We first evaluate some state-of-the-art multi-f0 estimation systems for the particular case of choirs with a single singer per part, and observe that the pitch of individual singers can be estimated to a relatively high degree of accuracy. We observe, however, that the scenario of multiple singers for each choir part (i.e. unison singing) is far more challenging. In this work we propose a methodology based on combining a multi-f0 estimation methodology based on deep learning followed by a set of traditional DSP techniques to model f0 and its dispersion instead of a single f0 trajectory for each choir part. We present and discuss our observations and test our framework with different singer configurations., This work is partially supported by the European Commission under the TROMPA project (H2020 770376) and by Spanish Ministry of Economy and Competitiveness under the CASAS project (TIN2015-70816-R). First author is supported by FI Predoctoral Grant from AGAUR (Generalitat de Catalunya).
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Generating data to train convolutional neural networks for classical music source separation

  • Miron, Marius
  • Janer Mestres, Jordi
  • Gómez Gutiérrez, Emilia, 1975-
Comunicació presentada a la conferència 14th Sound and Music Computing Conference, celebrada a Finlàndia del 5 al 8 de juliol de 2017., Deep learning approaches have become increasingly popular in estimating time-frequency masks for audio source separation. However, training neural networks usually requires a considerable amount of data. Music data is scarce, particularly for the task of classical music source separation, where we need multi-track recordings with isolated instruments. In this work, we depart from the assumption that all the renditions of a piece are based on the same musical score, and we can generate multiple renditions of the score by synthesizing it with different performance properties, e.g. tempo, dynamics, timbre and local timing variations. We then use this data to train a convolutional neural network (CNN) which can separate with low latency all the renditions of a score or a set of scores. The trained model is tested on real life recordings and is able to effectively separate the corresponding sources. This work follows the principle of research reproducibility, providing related data and code, and can be extended to separate other pieces., The TITANX used for this research was donated by the NVIDIA Corporation. This work is partially supported by the Spanish Ministry of Economy and Competitiveness under CASAS project (TIN2015-70816-R) and by the Spanish Ministry of Economy and Competitiveness under the Maria de Maeztu Units of Excellence Programme (MDM- 2015-0502). We thank Agustin Martorell for his help with Sibelius and Pritish Chandna for his useful feedback.
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Automatic transcription of flamenco guitar falsetas

  • Rodríguez, Sonia
  • Gómez Gutiérrez, Emilia, 1975-
  • Cuesta, Helena
Comunicació presentada al 8th International Workshop on Folk Music Analysis celebrat del 26 al 29 de juny de 2018 a Tessalònica, Grècia., This work deals with the automatic transcription and characterization of flamenco guitar, with a focus on short melodic interludes improvised between sung verses. These are called falsetas in the flamenco argot and are very challenging for manual and automatic transcription due to their fast and highly ornamented nature. However, they are a key resource for guitar players to practice. We adapted a state of the art singing transcription algorithm to process an audio signal containing one or several guitar falsetas and extract their symbolic representation. The algorithms first perform a segmentation to locate the guitar fragments and then a symbolic transcription of these segments into symbolic representation. In order to evaluate it, we collected the first (to our knowledge) annotated falseta datasets. Our results confirm the difficulty of the task, and a detailed study of two transcriptions revealed that combining the algorithm with specific musical knowledge about the scale used by the song, improves the performance of the system. Our approach follows the principles of research reproducibility, and the system is integrated in a computer-assisted paradigm, where the user complements the automatic annotation with a priory knowledge to generate a final transcription., This work is partially supported by the Spanish Ministry of Economy and Competitiveness under the CASAS project (TIN2015-70816-R).
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Musical instrument recognition in user-generated videos using a multimodal convolutional neural network architecture

  • Slizovskaia, Olga
  • Gómez Gutiérrez, Emilia, 1975-
  • Haro Ortega, Gloria
Comunicació presentada a la International Conference on Multimedia Retrieval celebrada del 6 al 9 de juny de 2017 a Bucarest, Romania., This paper presents a method for recognizing musical instruments in user-generated videos. Musical instrument recognition from music signals is a well-known task in the music information retrieval (MIR) field, where current approaches rely on the analysis of the good-quality audio material. This work addresses a real-world scenario with several research challenges, i.e. the analysis of user-generated videos that are varied in terms of recording conditions and quality and may contain multiple instruments sounding simultaneously and background noise. Our approach does not only focus on the analysis of audio information, but we exploit the multimodal information embedded in the audio and visual domains. In order to do so, we develop a Convolutional Neural Network (CNN) architecture which combines learned representations from both modalities at a late fusion stage. Our approach is trained and evaluated on two large-scale video datasets: YouTube-8M and FCVID. The proposed architectures demonstrate state-of-the-art results in audio and video object recognition, provide additional robustness to missing modalities, and remains computationally cheap to train., is work is partly supported by the Spanish Ministry of Economy and Competitiveness under the Maria de Maeztu Units of Excellence Programme (MDM-2015-0502), the CASAS Spanish research project (TIN2015-70816-R), and project TIN2015-70410-C2- 1-R (MINECO/FEDER, UE). We gratefully acknowledge the support of NVIDIA Corporation with the donation of the Titan X GPU used for this research.