Pasar al contenido principal

COMUNICACION MULTIMODAL (MCOM): LA INTEGRACION DE LA PROSODIA Y EL GESTO EN LA COMUNICACION HUMANA Y EN EL DESARROLLO Y APRENDIZAJE DEL LENGUAJE.

PID2021-123823NB-I00

•
Nombre agencia financiadora Agencia Estatal de Investigación
Acrónimo agencia financiadora AEI
Programa Programa Estatal para Impulsar la Investigación Científico-Técnica y su Transferencia
Subprograma Subprograma Estatal de Generación de Conocimiento
Convocatoria Proyectos de I+D+i (Generación de Conocimiento y Retos Investigación)
Año convocatoria 2021
Plan Estatal 2021-2023 Plan Estatal de Investigación Científica y Técnica y de Innovación
Centro beneficiario UNIVERSITAT POMPEU FABRA CCT

Resultados relacionados

Resultados totales (Incluyendo duplicados): 40
Encontrada(s) 1 página(s)

Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Encouraging participant embodiment during VR-assisted public speaking training improves persuasiveness and charisma and reduces anxiety in secondary school students

  • Valls-Ratés, Ïo
  • Niebuhr, Oliver
  • Prieto Vives, Pilar, 1965-
Practicing public speaking to simulated audiences created in virtual reality environments is reported to be effective for reducing public speaking anxiety. However, little is known about whether this effect can be enhanced by encouraging the use of gestures during VR-assisted public speaking training. In the present study two groups of secondary schools underwent a three-session public speaking training program in which they delivered short speeches to VR-simulated audiences. One group was encouraged to “embody” their speeches through gesture while the other was given no instructions regarding the use of gesture. Before and after the training sessions participants underwent respectively a pre- and a post-training session, which consisted of delivering a similar short speech to a small live audience. At pre- and post-training sessions, participants’ levels of anxiety were self-assessed, their speech performances were rated for persuasiveness and charisma by independent raters, and their verbal output was analyzed for prosodic features and gesture rate. Results showed that both groups significantly reduced their self-assessed anxiety between the pre- and post-training sessions. Persuasiveness and charisma ratings increased for both groups, but to a significantly greater extent in the gesture-using group. However, the prosodic and gestural features analyzed showed no significant differences across groups or from pre-to post-training speeches. Thus, our results seem to indicate that encouraging the use of gesture in VR-assisted public speaking practice can help students be more charismatic and their delivery more persuasive before presenting in front of a live audience., This work benefited from funding awarded by the Spanish Ministry of Economy and Competitiveness (PGC2018-097007-B-I00 and PID2021-123823NB-I00) and the Generalitat de Catalunya (2017 SGR_971). We also acknowledge support from the Recercaixa Project (RecerCaixa 2017ACUP 00249) and the Department of Translation at Universitat Pompeu Fabra through a 1-year doctoral grant to the first author.
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Towards a novel conceptualization of prosody that accounts for spoken and visual signals: the modality-neutral prosodic framework hypothesis

  • Prieto Vives, Pilar, 1965-
  • Esteve-Gibert, Núria
  • Shattuck-Hufnagel, Stefanie
Prosody influences speech organization by signaling phrasal prominence, grouping patterns, and speakers' pragmatic intentions. While traditionally viewed as restricted to speech, research shows prosody is also conveyed visually. This article reviews research showing strong parallels between spoken prosody and co-speech gestures in prominence marking, grouping phrasal structures, and signaling pragmatic intent. We extend this discussion and propose a modality-neutral prosodic framework hypothesis comprising three propositions: (a) prosody should be viewed as a modality-neutral grammar component that operates as an abstract level of representation while adapting to different sensory channels and language modalities; (b) in spoken languages, prosody is implemented flexibly through two distinct channels, spoken and gestural, which enable to mark prominence, grouping and meaning in a multimodal way; (c) parallel implementations are found in the way prosody is manifested in spoken and sign languages. A modality-neutral view of prosody will enrich current formal and developmental theories of language., We extend our sincere gratitude to our colleagues Marianne Gullberg, David House, Frank Kügler and Patrick Rohrer, as well as our reviewers Gilbert Ambrazaitis, Marisa Cruz and Sónia Frota for their invaluable feedback and fruitful discussions on an early version of the proposal. The authors would like to acknowledge the following financial support received to conduct this research: The Keith North Fund at the Speech Communication Group, Research Laboratory of Electronics, Department of Electrical Engineering and Computer Science, MIT (Ref: 3161400); the Agency for Management of University and Research Grants (Generalitat de Catalunya) (2021 SGR_922, 2021 SGR 01102); and the Spanish Ministry of Science and Innovation, Agencia Estatal de Investigación and Fondo Europeo de Desarrollo Regional (PID2021-123823NB-I00: 'Multimodal Communication: The Integration of Prosody and Gesture in Human Communication and in Language Learning'; PID2020-115385GA-I00: 'The Role of Prosody and Gestures in the Acquisition of Pragmatics: Typical and Atypical Populations'). The first author would like to acknowledge the support from the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) - [project ID 502013510, KU 2323/5-1 & PR 1780/1-1] - SPP 2392 `Visual Communication (ViCom).
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Unguided virtual-reality training can enhance the oral presentation skills of high-school students

  • Valls-Ratés, Ïo
  • Niebuhr, Oliver
  • Prieto Vives, Pilar, 1965-
Public speaking is fundamental in our daily life, and it happens to be challenging for many people. Like all aspects of language, these skills should be encouraged early on in educational settings. However, the high number of students per class and the extensive curriculum both limit the possibilities of the training and, moreover, entail that students give short in-class presentations under great time pressure. Virtual Reality (VR) environments can help speakers and teachers meet these challenges and foster oral skills. This experimental study employs a between-subjects pre- and post-training design with four Catalan high-school student groups, a VR group (N = 30) and a Non-VR group (N = 20). Both groups gave a 2-min speech in front of a live audience before (pre-training) and after (post-training) 3 training sessions (one session per week) in which they practiced public speaking either in front of a VR audience or alone in a classroom (Non-VR). Students assessed their anxiety measures right before performing every speech and filled out a satisfaction questionnaire at the end. Pre- and post-training speeches were assessed by 15 raters, who analyzed the persuasiveness of the message and the charisma of the presenter. Speeches were also analyzed for prosodic features and gesture rate. First, results showed that self-assessed anxiety got significantly reduced at post-training for both conditions. Second, acoustic analyses of both groups' speeches show that the VR group has, unlike the Non-VR group, developed a more clear and resonant voice quality in the post-training speeches, in terms of higher cepstral-peak prominence (CPP) (although no significant differences in f0- related parameters as a function of training were obtained), as well as significantly less erosion effects than the Non-VR group. However, these differences across groups did not trigger a direct improvement on the participants' gesture rate, persuasiveness and charisma at post-training. Furthermore, students perceived the training in the VR environment to be more useful and beneficial for their future oral presentations than the Non-VR group. All in all, short unguided VR training sessions can help students feel less anxious, promote a more clear and resonant voice style, and can prevent them from experiencing an erosion effect while practicing speeches in front of a real audience., This work benefited from funding awarded by the Spanish Ministry of Economy and Competitiveness (PGC2018-097007-B-I00, PID2021-123823NB-I00) and the Generalitat de Catalunya (2017 SGR_971). We also acknowledge the support from the Recercaixa Project (RecerCaixa 2017ACUP 00249) and the Department of Translation at Universitat Pompeu Fabra through a 1-year doctoral grant to ÏV-R.
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Preschoolers mark focus types through multimodal prominence: further evidence for the precursor role of gestures

  • Coego, Sara
  • Esteve Gibert, Núria
  • Prieto Vives, Pilar, 1965-
The present cross-sectional study assessed the role of multimodal cues in marking focus types during early childhood, focusing on prosodic prominence, gesture presence, and gestural prominence. A total of 116 Catalan-speaking three-, four- and five-year-olds participated in a semi-controlled interactive task eliciting words in three focus conditions: information, contrastive, and corrective. The data were coded manually using holistic assessments for all three measures. The results indicated, first, that children’s prosodic and gestural behavior was key in marking corrective focus. A significant tendency to use more gestures and increase both prosodic and gestural prominence was found in the corrective focus condition across the three age groups. Second, a developmental difference emerged in the acquisition of contrastive focus. Three-year-olds relied solely on gesture presence to encode contrastive focus, being unable to differentiate it prosodically from information focus. In turn, four- and five-year-olds used both gestures and prosody, with contrastive focus not only receiving more gestures than information focus but also increased prosodic prominence. This finding shows that gesture presence is a precursor to prosodic prominence in marking contrastive focus in Catalan, thus supporting the idea that gesture production can bootstrap the expression of focus type distinctions., This research was funded by grant PID2020-115385GA-I00, financed by MCIN/AEI/10.13039/501100011033 and by the European Regional Development Fund (ERDF), by the European Union. It was also supported by the grant AGAUR-FI Joan Oró 2025FI_B 00342, funded by the Agency for Management of University and Research Grants (Generalitat de Catalunya) together with the European Social Fund (ESF), by the European Union. The APC was funded by grants PID2021-123823NB-I00 and PID2020-115385GA-I00, financed by MCIN/AEI/10.13039/501100011033 and by the European Regional Development Fund (ERDF), by the European Union.
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Are nonwords really superior to real words to train l2 sounds? Insights from an HVPT study on L2 English front vowels

  • Gorba, Celia
  • Llompart, Miquel
  • Prieto Vives, Pilar, 1965-
This study investigated the effects of a high-variability phonetic training (HVPT) regime using nonwords (NW), compared to a comparable regime using real words (RW), on the perception of English front vowels by L1-Catalan/Spanish learners. Participants completed pre-and posttest tasks targeting both pre-lexical (vowel identification of NW and RW) and phono-lexical processing (word-picture matching, WPM, and lexical decision, LD). Training consisted of six sessions of phoneme identification with either NW or RW stimuli. Results revealed that the RW group consistently outperformed the NW group, with significant differences in RW identification and WPM. Importantly, learning generalized to new speakers and items, particularly in the RW condition. Lexical tasks proved more challenging than pre-lexical ones, with lower overall scores and no significant gains in LD. These findings indicate that RW training produces stronger perceptual and phono-lexical gains than NW training, challenging findings from prior studies that favored NW-based regimes., The authors would like to acknowledge the financial support awarded by the Generalitat de Catalunya (2021 SGR 00922) and the Spanish Ministry of Science and Innovation (PID2021-123823NB-I00). The first author's contribution was additionally supported by the Spanish Ministry of Universities (Margarita Salas Fellowship, ID-706211), as was the second author¿s (Beatriz Galindo Junior Research Fellowship, BG22/00161).
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Improving second language vowel production with hand gestures encoding visible articulation: evidence from picture‐naming and paragraph‐reading tasks

  • Xi, Xiaotong
  • Li, Peng
  • Prieto Vives, Pilar, 1965-
Includes supplementary materials for the online appendix., This study investigates whether audiovisual phonetic training with hand gestures encoding visible or nonvisible articulation features has a differential impact on learning second language sounds. Ninety-nine Catalan–Spanish bilingual students were trained to differentiate English /æ/ and /ʌ/, which differ in the visible lip aperture and nonvisible tongue position, with training involving no gestures, gestures representing the lip aperture, or gestures representing the tongue position. Before, immediately after, and 1 week after the training, participants’ perception of the targets was assessed through a word-identification task, and their production was tested through paragraph-reading, picture-naming, and word-imitation tasks. Although all participants improved in perception and production, the lip hand gesture was more effective in adjusting lip aperture than the other two conditions in the paragraph-reading and picture-naming tasks. These results suggest that hand gestures encoding visible rather than nonvisible articulation features are more effective for improving second language pronunciation., The study was supported by the Ministerio de Ciencia, Innovación y Universidades, Agencia Estatal de Investigación, and Fondo Europeo de Desarrollo Regional [PGC2018-097007-B-l00], and by the Ministerio de Ciencia e Innovación [PID2021-123823NB-I00]. The first author is supported by the Department d'Empresa i Coneixement de la Generalitat de Catalunya and the European Social Fund under a grant for the recruitment of early-stage research staff [2021FI_B 00137]. The second author is supported by the Research Council of Norway through its Centres of Excellence funding scheme [223265].
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Evaluating the feasibility and preliminary effectiveness of a multi-tiered multimodal narrative intervention program for preschool children

  • Florit-Pons, Júlia
  • Igualada Pérez, Alfonso
  • Prieto Vives, Pilar, 1965-
Purpose. The present study aims to evaluate the feasibility and preliminary effectiveness of a novel multi-tiered narrative intervention program -the multimodal narrative (MMN) program- in Catalan that was co-created to boost preschool children's narrative and pragmatic skills. Method. First, we describe here in detail the novel program, which consisted of a set of interventions oriented around the retelling of a narrative in a multimodal fashion, that is, with an abundant use of appropriate gesture and facial expression and careful attention to the pragmatic aspects of communication. We then describe the results of a self-reported feasibility study (Study 1) after this program was trial-implemented by 31 preschool teachers and speech-language therapists in their respective professional contexts. A pre- and post-intervention pilot study (Study 2) was conducted in which the researchers measured the effect of the MMN intervention on the 51 children who participated in the trial implementation. Results. Results from Study 1 revealed that most professionals adhered to the intervention protocol, that they found it enjoyable and easy to implement, and that it fostered active participation on the part of children. Results from Study 2 revealed that after the intervention, the narrative and pragmatic skills of all the children had improved. Conclusion. These results suggest that a full-fledged implementation of the MMN intervention program is feasible and has the potential to improve children's narrative and pragmatic skills in both clinical and educational contexts., The authors would like to acknowledge the financial support awarded by the Spanish Ministry of Science and Innovation, Agencia Estatal de Investigación and Fondo Europeo de Desarrollo Regional (PID2021-123823NB-I00: 'Multimodal Communication: The Integration of Prosody and Gesture in Human Communication and in Language Learning'); the Generalitat de Catalunya (2021 SGR_922); and the GEstures and Head Movements in language research network, funded by the Independent Research Fund Denmark (9055-00004B). The first author would like to acknowledge a PhD grant, awarded by the Generalitat de Catalunya and European Social Fund (2021 FI_B 00778). The authors of this study would like to express their gratitude to Imma Canal, Anna Marco, and Pere Mayans from the Supports and Linguistic Resources Service from the Regional Ministry of Education of the Generalitat de Catalunya, for their continuous support to the project, ranging from help in the organization of the oral narrative training for preschool teachers to continuous professional feedback on the training sessions. The authors are also grateful to Prudens Martínez, Glòria Medina, Dolors Blasco, and Ma. Isabel Fernández from the Inclusive School Subdivision from the Regional Ministry of Education (Generalitat de Catalunya) as well as the CDIAPs, CREDAs, and private speech therapy centers for their support in the recruitment of the clinical population. We are very grateful to Ïo Valls-Ratés for acting as storyteller for the intervention materials and to Mireia Gómez Martínez for her help in the data collection. Importantly, this work could not have been possible without the participation of the preschool teachers, speech-language therapists, and the schools and centers where they usually work, as well as the children themselves and their families.
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Visualizing prosodic structure: manual gestures as highlighters of prosodic heads and edges in English academic discourses

  • Rohrer, Patrick Louis
  • Delais-Roussarie, Elisabeth
  • Prieto Vives, Pilar, 1965-
Research has shown a close temporal relationship between prominence-lending tonal movements in speech and prominence in manual gesture. However, prosodic structure consists of not only prosodic heads (i.e., pitch accentuation) but also prosodic edges. To our knowledge, no previous studies have assessed the value of prosodic edges (nuclear vs. phrase-initial prenuclear pitch accents) as anchoring sites for different types of gestures (i.e., referential vs. non-referential) while independently controlling for the relative degree of prominence associated with the pitch accent. The English M3D-TED corpus, which contains over 23 minutes of multimodal speech, was analyzed in terms of prosody and gesture. Results showed that while the majority of manual gesture strokes overlapped a pitch accented syllable (85.99%), apex alignment occurred at a relatively low rate (50.4%) and alignment rates did not significantly differ between referential and non-referential gestures. At the phrasal level, crucially our results also showed that strokes align with phrase-initial prenuclear pitch accents over nuclear accents, and this relationship is not driven by relative prominence. These findings show that both prosodic heads and prosodic edges (i.e., phrase initial and final positions) are key sites for both referential and non-referential gesture production., The authors would like to acknowledge the financial support awarded by the Spanish Ministry of Science, Innovation and Universities (MCIU), Agencia Estatal de Investigación (AEI), and Fondo Europeo de Desarrollo Regional (FEDER) [grant numbers PGC2018-097007-B-I00 'Multimodal Language Learning (MLL): Prosodic and Gestural Integration in Pragmatic and Phonological Development'; PID2021-123823NB-I00 'Multimodal Communication (MCOM): The integration of prosody and gesture in human communication and in language learning'], by the Generalitat de Catalunya [grant number 2017 SGR_971], and by the GEHM (Gesture and Head Movements in Language) Research Network, funded by the Independent Research Fund Denmark [grant number 9055-00004B]. The first author would like to acknowledge a joint Ph.D. grant, awarded by the Department of Translation and Language Sciences, Universitat Pompeu Fabra, and SGR Grant, Generalitat de Catalunya, [grant number 2017 SGR_971]. We would like to express our gratitude to Ulya Tütücünbasi for her help with the gestural annotation of the corpus. Many thanks are also extended to Dr. Stefanie Shattuck-Hufnagel and Ada Ren (Massachusetts Institute of Technology), as well as the members of the GrEP research lab for their feedback provided on early versions of this study. We would also like to thank Steve Dunne for his help with proofreading the final manuscript.
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Multimodal mitigation: how facial and body cues index politeness in Catalan requests

  • Hübscher, Iris
  • Sánchez-Conde, Cristina
  • Borràs-Comes, Joan
  • Vincze, Laura
  • Prieto Vives, Pilar, 1965-
Recent cross-linguistic research has demonstrated that speakers use a prosodic mitigation strategy when addressing higher status interlocutors by talking more slowly, reducing the intensity and lowering the overall fundamental frequency (F0). Much less is known, however, about how politeness-related meaning is expressed multimodally (i.e., combining verbal and multimodal channels). The present study investigates how Catalan native speakers encode politeness-related meanings through facial and body cues. We test whether speakers apply a gestural mitigation strategy and use specific hedging devices in socially distant situations (e.g., when asking an older person of higher status for a favor). Twenty Catalan speakers were video-recorded while participating in a discourse elicitation task where they were required to produce requests in polite and non-polite contexts. In the resulting recordings, a set of 21 facial and body cues associated with speech were coded and analyzed. The results show that politeness-related meanings are expressed through gestural mitigation strategies that go hand-in-hand with previously reported prosodic mitigation strategies., This work was financed by the Spanish Ministry of Science, Innovation and Universities, Agencia Estatal de Investigacion, and Fondo Europeo ´ de Desarrollo Regional, grant numbers PGC2018-097007-B-100 and PID2021- 123823NB-I00; and by a grant awarded by the Generalitat de Catalunya, grant number 2017 SGR_971 to the Prosodic Studies Group.
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

An individualized multimodal narrative intervention for young children with neurodevelopmental disorders

  • Florit-Pons, Júlia
  • Prieto Vives, Pilar, 1965-
  • Igualada Pérez, Alfonso
Data de publicació electrònica: 11-04-2026, This study examines whether an individualized narrative intervention can boost the narrative (macrostructure and perspec-tive-taking) abilities of children with neurodevelopmental disorders (NDD). An experimental group of 16 children with NDD from Catalonia, Spain, received 9 MultiModal Narrative intervention sessions. Children's narrative skills were mea-sured pre- and post-intervention with a retelling task. Their performance was compared to two control groups not receiving the intervention (17 children with NDD and 17 with typical development). After each session, children in the experimental group underwent two Dynamic Assessment measures (retelling task and graduated prompting comprehension questions about the story), tracking their learning. Results showed an improvement in narrative macrostructure but not perspective-taking. The Dynamic Assessment revealed faster learning of macrostructure, individual differences in the prompts needed for successful responses, and that fewer prompts predicted better macrostructure outcomes post-intervention. These find-ings underscore the relevance of diverse measures in evaluating continuous learning in interventions., Open Access funding provided thanks to the CRUE-CSIC agreement with Springer Nature. This research received financial support awarded by the Spanish Ministry of Science and Innovation, Agencia Estatal de Investigación and Fondo Europeo de Desarrollo Regional (PID2021-123823NB-I00); by the Generalitat de Catalunya (2021 SGR_922). Also, the first author would like to acknowledge a Ph.D. grant, awarded by the Generalitat de Catalunya (2021 FI_B00778) and by the Department of Translation and Language Sciences (Universitat Pompeu Fabra).
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Musical perception skills predict speech imitation skills: differences between speakers of tone and intonation languages

  • Li, Peng
  • Zhang, Yuan
  • Baills, Florence
  • Prieto Vives, Pilar, 1965-
The ability to imitate speech is linked to individual cognitive abilities such as working memory and the auditory processing of music. However, little research has focused on the role of specific components of musical perception aptitude in relation to an individual’s native language from a crosslinguistic perspective. This study explores the predictive role of four components of musical perception skills and working memory on phonetic language abilities for speakers of two typologically different languages, Catalan (an intonation language) and Chinese (a tone language). Sixty-one Catalan and 144 Chinese participants completed four subtests (accent, melody, pitch and rhythm) of the Profile of Music Perception Skills, a forward digit span task and a speech imitation task. The results showed that for both groups of participants, musical perception skills predicted speech imitation accuracy but working memory did not. Importantly, among the components of musical perception skills, accent was the only predictive factor for Chinese speakers, whereas melody was the only predictive factor for Catalan speakers. These findings suggest that speech imitation ability is predicted by musical perception skills rather than working memory and that the predictive role of specific musical components may depend on the phonological properties of the native language., This study is funded by ‘Multimodal Communication: The integration of prosody and gesture in human communication and in language learning’ (PID2021-123823NB-I00) awarded by the Ministerio de Ciencia e Innovación and ‘Multimodal language learning: Prosodic and Gestural Integration in Pragmatic and Phonological Development’ (PGC2018-097007-B-l00), awarded by the Ministerio de Ciencia, Innovación y Universidades, Agencia Estatal de Investigación, and Fondo Europeo de Desarrollo Regional. P.L. is supported by the Research Council of Norway through its Centres of Excellence funding scheme (223265). F.B. acknowledges a Margarita Salas grant funded by the European Union-NextGenerationEU, Ministry of Universities and Recovery, Transformation and Resilience Plan, through a call from Pompeu Fabra University.
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Singing songs facilitates L2 pronunciation and vocabulary learning: a study with Chinese adolescent ESL learners

  • Zhang, Yuan
  • Baills, Florence
  • Prieto Vives, Pilar, 1965-
The present study assesses the effect of a three-session classroom-based training program involving singing songs with familiar melodies on second-language pronunciation and vocabulary learning. Ninety-five adolescent Chinese ESL learners (M = 14.04 years) were assigned to one of two groups. Participants learned the lyrics in English of three songs whose melodies were familiar to them either by singing or reciting the lyrics, following a native English singer/instructor. Before and after training, participants performed two vocabulary tasks (picture-naming and word meaning recall tasks) and two pronunciation tasks (word and sentence oral-reading tasks). The results revealed that although both groups showed gains in vocabulary and pronunciation after training, the singing group outperformed the speech group. These findings support the value of using songs with familiar melodies to teach second languages at the early stages of learning in an ESL classroom context., This research was supported by the Generalitat de Catalunya [2017 SGR_971]; Ministerio de Ciencia, Innovación y Universidades [PGC2018-097007-B-I00]; Ministerio de Ciencia e Innovación [PID2021-123823NB-I00]. The second author acknowledges funding by the European Union-NextGenerationEU, the Spanish Ministry of Universities and Recovery, Transformation and Resilience Plan, through a call from Pompeu Fabra University (Barcelona).
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Multidimensional labeling of gesture in communication: the M3D proposal

  • Rohrer, Patrick Louis
  • Tütüncübasi, Ulya
  • Florit-Pons, Júlia
  • Vilà-Giménez, Ingrid
  • Esteve-Gibert, Núria
  • Ren-Mitchell, Ada
  • Shattuck-Hufnagel, Stefanie
  • Prieto Vives, Pilar, 1965-
Communication is multimodal in that speakers use not only their voices, but also co-speech gestures to communicate. Recent insights suggest that gestural behavior has a strong association with prosodic structure and that a single gesture can communicate various semantic and pragmatic meanings. This highlights the importance of developing a comprehensive, flexible, and transparent approach to gesture annotation that accounts for multiple dimensions of gesture, including a gesture's form, prosodic properties, and semantic and pragmatic contributions. To address this need for an increasingly dimensionalized approach to multimodal data annotation, the main goal of this paper is to present and describe a novel labeling system for manual gestures. The MultiModal MultiDimensional (M3D) system consists of an open access package that has been developed in coordination with five different labs working on gesture and its interaction with speech. The package includes a set of reliable annotation guidelines, a validated training program, and two annotated audiovisual corpora that represent over 60 minutes of lecture-style speech., This work was partially funded by the Spanish Ministry of Science, Innovation and Universities (Grants numbers PID2021-123823NB-I00, PID2020-115385GA-I00, PGC2018-097007-B-I00), the Pla de Recerca de Catalunya from the Generalitat de Catalunya (Grant numbers 2017 SGR_971 and 2021 SGR 00922) and the GEHM (Gesture and Head Movements in Language) Research Network, funded by the Independent Research Fund Denmark (grant number 9055-00004B).
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Multimodal skills, but not motor skills, predict narrative and expressive pragmatic skills in children with typical development and neurodevelopmental disorders

  • Florit Pons, Júlia
  • Pronina, Mariia
  • Igualada, Alfonso
  • Prieto Vives, Pilar, 1965-
  • Norbury, Courtenay
To see whether communicative-based multimodal skills (compared to non-communicative motor skills) predicted complex language skills, this study examined the predictive power of multimodal and motor skills on narrative and expressive pragmatic abilities across two groups. Participants were children with typical development (N=88, Mage=5.34, 48% female) and with neurodevelopmental disorders (N = 51, Mage = 5.01, 25% female) mostly of white ethnicity (86.3%). We evaluated children's multimodal accuracy, motor skills, core language skills, and narrative and pragmatic skills. Results revealed that, in all groups, both multimodal skills and core language significantly predicted narrative (R2 = 0.569) and pragmatic skills (R2 = 0.621), while motor skills did not. These findings highlight the relevance of multimodality in the assessment of children's complex language skills., The authors would like to acknowledge the financial support awarded by the Spanish Ministry of Science, Innovation and Universities (PID2021-123823NB-I00: "Multimodal Communication: The Integration of Prosody and Gesture in Human Communication and in Language Learning"; MICIU/AEI/10.13039/501100011033) and awarded by the Generalitat de Catalunya (2021 SGR_922). Also, the first author would like to acknowledge a Ph.D. grant, awarded by the Generalitat de Catalunya (2021 FI_B 00778) and the Department of Translation and Language Sciences at Universitat Pompeu Fabra.
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Revisiting the link between second-language sound identification and word recognition with an eye on methodological similarity

  • Llompart, Miquel
  • Gorba, Celia
  • Prieto Vives, Pilar, 1965-
Data de publicació electrònica: 12-09-2025, This study revisits the relationship between second-language (L2) learners' ability to distinguish sounds in non-native phonological contrasts and to recognize spoken words when recognition depends on these sounds, while addressing the role of methodological similarity. Bilingual Catalan/Spanish learners of English were tested on the identification of two vowel contrasts (VI) of diverging difficulty, /i/-// (difficult) and //-/æ/ (easy), in monosyllabic minimal pairs, and on their recognition of the same pairs in a word-picture matching task (WPM). Learners performed substantially better with /i/-// in VI than in WPM, and individual scores were only weakly correlated. By replicating previous findings through a more symmetrical design, we show that an account of prior work rooted in methodological dissimilarity is improbable and provide additional support for the claim that accuracy in sound identification does not guarantee improvements in word recognition. This has implications for our understanding of L2-speech acquisition and L2 pronunciation training., The authors would like to acknowledge the financial support awarded by the Generalitat de Catalunya (2021 SGR 00922) and the Spanish Ministry of Science and Innovation (PID2021-123823NB-I00). The first author's contribution was additionally supported by the Spanish Ministry of Universities (Beatriz Galindo Junior Research Fellowship, BG22/00161), as was the second author's (Margarita Salas Fellowship, ID-706211). We would also like to thank the editor and three anonymous reviewers for their helpful suggestions.
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Multimodal development in children’s narrative speech: evidence for tight gesture–speech temporal alignment patterns as early as 5 years old

  • Florit-Pons, Júlia
  • Vilà-Giménez, Ingrid
  • Rohrer, Patrick Louis
  • Prieto Vives, Pilar, 1965-
The authors would like to acknowledge the financial support awarded by the Spanish Ministry of Science, Innovation and Universities, Agencia Estatal de Investigación, and Fondo Europeo de Desarrollo Regional (PGC2018-097007-B-100: “Multimodal Language Learning: Prosodic and Gestural Integration in Pragmatic and Phonological Development,” and PID2021-123823NB-I00: “Multimodal Communication: The Integration of Prosody and Gesture in Human Communication and in Language Learning”); by the Generalitat de Catalunya (2017 SGR_971 and 2021 SGR_922); and by the GEstures and Head Movements in language research network, funded by the Independent Research Fund Denmark (9055-00004B). The first author would also like to acknowledge a PhD grant awarded by the Agency for Management of University and Research Grants from the Generalitat de Catalunya, (Award no. 2021 FI_B 00778), and the third author would like to acknowledge a joint PhD grant awarded by the Department of Translation and Language Sciences, Universitat Pompeu Fabra, and SGR AGAUR Grant, Generalitat de Catalunya (Award no. 2017 SGR_971). Also, the second author would like to acknowledge a postdoctoral fellowship funded by European Union-NextGenerationEU, Ministry of Universities and Recovery, Transformation and Resilience Plan, through a call from Universitat Pompeu Fabra (Barcelona).
Repositori Institucional UIB
Publicación Artículo científico (JournalArticle)

The MultiModal Imitation Task: A Validation Study

  • Pronina, Mariia
  • Florit-Pons, Júlia
  • Coego, Sara
  • Prieto, Pilar
[eng] The MultiModal Imitation Task (MMIT) has been developed to jointly assess the imitation of sentence, prosody, and gestures in children. The MMIT focuses on multimodal language abilities and includes four scores (sentence imitation, prosody imitation, gesture imitation, MMIT total). Psychometric properties of the tool were analyzed on a sample of 235 typically developing children 3 to 6 years of age. The results showed good validity and reliability of the tool. The MMIT had excellent internal consistency and interrater reliability, suggesting that the items are coherent and intercorrelated and the scoring system is reliable. The MMIT showed adequate convergent validity with other previously validated and standardized tests to assess sentence and prosody imitation. The relevance and adequacy of the MMIT items were confirmed by expert evaluations. Overall, the MMIT is a valuable, reliable, appropriate, and practical tool for the assessment of multimodal imitation skills in children of preschool and primary school ages.
Repositori Obert UdL
Publicación Artículo científico (JournalArticle)

Singing Songs Facilitates L2 Pronunciation and Vocabulary Learning: A Study with Chinese Adolescent ESL Learners

  • Zhang, Yuan
  • Baills, Florence
  • Prieto, Pilar
The present study assesses the effect of a three-session classroom-based training program involving singing songs with familiar melodies on second-language pronunciation and vocabulary learning. Ninety-five adolescent Chinese ESL learners (M = 14.04 years) were assigned to one of two groups. Participants learned the lyrics in English of three songs whose melodies were familiar to them either by singing or reciting the lyrics, following a native English singer/instructor. Before and after training, participants performed two vocabulary tasks (picture-naming and word meaning recall tasks) and two pronunciation tasks (word and sentence oral-reading tasks). The results revealed that although both groups showed gains in vocabulary and pronunciation after training, the singing group outperformed the speech group. These findings support the value of using songs with familiar melodies to teach second languages at the early stages of learning in an ESL classroom context., This research was supported by the Generalitat de Catalunya [2017 SGR_971]; Ministerio de Ciencia, Innovación y Universidades [PGC2018-097007-B-I00]; Ministerio de Ciencia e Innovación [PID2021-123823NB-I00].
Repositori Obert UdL
Publicación Capítulo o parte de un libro (BookPart)

Gesture shape and gesture–speech alignment predict simultaneous L2 sound production accuracy

  • Li, Peng
  • Baills, Florence
  • Xi, Xiaotong
  • Prieto, Pilar
Previous research suggests hand gestures aid L2 pronunciation learning, but the underlying mechanisms remain unclear. This study examines if accurate gesture imitation during phonetic training predicts the accuracy of non-native sounds produced while doing a pedagogical hand gesture. Twenty-nine Catalan speakers, unfamiliar with Chinese, learned to pronounce Chinese aspirated plosives while mimicking gestures representing the aspirated sounds. Results revealed that more precise gesture shape and better gesture–speech alignment predicted more accurate speech imitation, emphasizing the significance of the spatiotemporal gesture–speech coupling in embodied L2 phonetic training., This study is funded by “Multimodal Communication: The integration of prosody and gesture in human communication and in language learning” [PID2021-123823NB-I00] awarded by the Ministerio de Ciencia e Innovación and “Multimodal language learning: Prosodic and Gestural Integration in Pragmatic and Phonological Development” [PGC2018- 097007-B-l00], awarded by the Ministerio de Ciencia, Innovación y Universidades, Agencia Estatal de Investigación, and Fondo Europeo de Desarrollo Regional. PL is supported by the Research Council of Norway through its Centres of Excellence funding scheme [223265]. FB acknowledges a Margarita Salas grant funded by European Union– NextGenerationEU, Ministry of Universities and Recovery, Transformation, and Resilience Plan, through a call from Pompeu Fabra University. XX is supported by the Secretaria d’Universitats i Recerca de la Generalitat de Catalunya and the European Social Fund under the Grant for the recruitment of early-stage research staff [2021FI_B 00137].
Repositori Obert UdL
Publicación Artículo científico (JournalArticle)

Musical perception skills predict speech imitation skills: differences between speakers of tone and intonation languages

  • Li, Peng
  • Zhang, Yuan
  • Baills, Florence
  • Prieto, Pilar
The ability to imitate speech is linked to individual cognitive abilities such as working memory and the auditory processing of music. However, little research has focused on the role of specific components of musical perception aptitude in relation to an individual’s native language from a crosslinguistic perspective. This study explores the predictive role of four components of musical perception skills and working memory on phonetic language abilities for speakers of two typologically different languages, Catalan (an intonation language) and Chinese (a tone language). Sixty-one Catalan and 144 Chinese participants completed four subtests (accent, melody, pitch and rhythm) of the Profile of Music Perception Skills, a forward digit span task and a speech imitation task. The results showed that for both groups of participants, musical perception skills predicted speech imitation accuracy but working memory did not. Importantly, among the components of musical perception skills, accent was the only predictive factor for Chinese speakers, whereas melody was the only predictive factor for Catalan speakers. These findings suggest that speech imitation ability is predicted by musical perception skills rather than working memory and that the predictive role of specific musical components may depend on the phonological properties of the native language., This study is funded by ‘Multimodal Communication: The integration of prosody and gesture in human communication and in language learning’ (PID2021-123823NB-I00) awarded by the Ministerio de Ciencia e Innovación and ‘Multimodal language learning: Prosodic and Gestural Integration in Pragmatic and Phonological Development’ (PGC2018-097007-B-l00), awarded by the Ministerio de Ciencia, Innovación y Universidades, Agencia Estatal de Investigación, and Fondo Europeo de Desarrollo Regional. P.L. is supported by the Research Council of Norway through its Centres of Excellence funding scheme (223265). F.B. acknowledges a Margarita Salas grant funded by the European Union-NextGenerationEU, Ministry of Universities and Recovery, Transformation and Resilience Plan, through a call from Pompeu Fabra University.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Singing Songs Facilitates L2 Pronunciation and Vocabulary Learning: A Study with Chinese Adolescent ESL Learners

  • Zhang, Yuan
  • Baills, Florence
  • Prieto, Pilar
The present study assesses the effect of a three-session classroom-based training program involving singing songs with familiar melodies on second-language pronunciation and vocabulary learning. Ninety-five adolescent Chinese ESL learners (M = 14.04 years) were assigned to one of two groups. Participants learned the lyrics in English of three songs whose melodies were familiar to them either by singing or reciting the lyrics, following a native English singer/instructor. Before and after training, participants performed two vocabulary tasks (picture-naming and word meaning recall tasks) and two pronunciation tasks (word and sentence oral-reading tasks). The results revealed that although both groups showed gains in vocabulary and pronunciation after training, the singing group outperformed the speech group. These findings support the value of using songs with familiar melodies to teach second languages at the early stages of learning in an ESL classroom context., This research was supported by the Generalitat de Catalunya [2017 SGR_971]; Ministerio de Ciencia, Innovación y Universidades [PGC2018-097007-B-I00]; Ministerio de Ciencia e Innovación [PID2021-123823NB-I00].
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Multimodal development in children’s narrative speech: evidence for tight gesture–speech temporal alignment patterns as early as 5 years old

  • Florit-Pons, Júlia
  • Vilà-Giménez, Ingrid
  • Rohrer, Patrick Louis
  • Prieto Vives, Pilar, 1965-
The authors would like to acknowledge the financial support awarded by the Spanish Ministry of Science, Innovation and Universities, Agencia Estatal de Investigación, and Fondo Europeo de Desarrollo Regional (PGC2018-097007-B-100: “Multimodal Language Learning: Prosodic and Gestural Integration in Pragmatic and Phonological Development,” and PID2021-123823NB-I00: “Multimodal Communication: The Integration of Prosody and Gesture in Human Communication and in Language Learning”); by the Generalitat de Catalunya (2017 SGR_971 and 2021 SGR_922); and by the GEstures and Head Movements in language research network, funded by the Independent Research Fund Denmark (9055-00004B). The first author would also like to acknowledge a PhD grant awarded by the Agency for Management of University and Research Grants from the Generalitat de Catalunya, (Award no. 2021 FI_B 00778), and the third author would like to acknowledge a joint PhD grant awarded by the Department of Translation and Language Sciences, Universitat Pompeu Fabra, and SGR AGAUR Grant, Generalitat de Catalunya (Award no. 2017 SGR_971). Also, the second author would like to acknowledge a postdoctoral fellowship funded by European Union-NextGenerationEU, Ministry of Universities and Recovery, Transformation and Resilience Plan, through a call from Universitat Pompeu Fabra (Barcelona).
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Improving second language vowel production with hand gestures encoding visible articulation: evidence from picture‐naming and paragraph‐reading tasks

  • Xi, Xiaotong
  • Li, Peng
  • Prieto Vives, Pilar, 1965-
Includes supplementary materials for the online appendix., This study investigates whether audiovisual phonetic training with hand gestures encoding visible or nonvisible articulation features has a differential impact on learning second language sounds. Ninety-nine Catalan–Spanish bilingual students were trained to differentiate English /æ/ and /ʌ/, which differ in the visible lip aperture and nonvisible tongue position, with training involving no gestures, gestures representing the lip aperture, or gestures representing the tongue position. Before, immediately after, and 1 week after the training, participants’ perception of the targets was assessed through a word-identification task, and their production was tested through paragraph-reading, picture-naming, and word-imitation tasks. Although all participants improved in perception and production, the lip hand gesture was more effective in adjusting lip aperture than the other two conditions in the paragraph-reading and picture-naming tasks. These results suggest that hand gestures encoding visible rather than nonvisible articulation features are more effective for improving second language pronunciation., The study was supported by the Ministerio de Ciencia, Innovación y Universidades, Agencia Estatal de Investigación, and Fondo Europeo de Desarrollo Regional [PGC2018-097007-B-l00], and by the Ministerio de Ciencia e Innovación [PID2021-123823NB-I00]. The first author is supported by the Department d'Empresa i Coneixement de la Generalitat de Catalunya and the European Social Fund under a grant for the recruitment of early-stage research staff [2021FI_B 00137]. The second author is supported by the Research Council of Norway through its Centres of Excellence funding scheme [223265].
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Evaluating the feasibility and preliminary effectiveness of a multi-tiered multimodal narrative intervention program for preschool children

  • Florit-Pons, Júlia
  • Igualada Pérez, Alfonso
  • Prieto Vives, Pilar, 1965-
Purpose. The present study aims to evaluate the feasibility and preliminary effectiveness of a novel multi-tiered narrative intervention program -the multimodal narrative (MMN) program- in Catalan that was co-created to boost preschool children's narrative and pragmatic skills. Method. First, we describe here in detail the novel program, which consisted of a set of interventions oriented around the retelling of a narrative in a multimodal fashion, that is, with an abundant use of appropriate gesture and facial expression and careful attention to the pragmatic aspects of communication. We then describe the results of a self-reported feasibility study (Study 1) after this program was trial-implemented by 31 preschool teachers and speech-language therapists in their respective professional contexts. A pre- and post-intervention pilot study (Study 2) was conducted in which the researchers measured the effect of the MMN intervention on the 51 children who participated in the trial implementation. Results. Results from Study 1 revealed that most professionals adhered to the intervention protocol, that they found it enjoyable and easy to implement, and that it fostered active participation on the part of children. Results from Study 2 revealed that after the intervention, the narrative and pragmatic skills of all the children had improved. Conclusion. These results suggest that a full-fledged implementation of the MMN intervention program is feasible and has the potential to improve children's narrative and pragmatic skills in both clinical and educational contexts., The authors would like to acknowledge the financial support awarded by the Spanish Ministry of Science and Innovation, Agencia Estatal de Investigación and Fondo Europeo de Desarrollo Regional (PID2021-123823NB-I00: 'Multimodal Communication: The Integration of Prosody and Gesture in Human Communication and in Language Learning'); the Generalitat de Catalunya (2021 SGR_922); and the GEstures and Head Movements in language research network, funded by the Independent Research Fund Denmark (9055-00004B). The first author would like to acknowledge a PhD grant, awarded by the Generalitat de Catalunya and European Social Fund (2021 FI_B 00778). The authors of this study would like to express their gratitude to Imma Canal, Anna Marco, and Pere Mayans from the Supports and Linguistic Resources Service from the Regional Ministry of Education of the Generalitat de Catalunya, for their continuous support to the project, ranging from help in the organization of the oral narrative training for preschool teachers to continuous professional feedback on the training sessions. The authors are also grateful to Prudens Martínez, Glòria Medina, Dolors Blasco, and Ma. Isabel Fernández from the Inclusive School Subdivision from the Regional Ministry of Education (Generalitat de Catalunya) as well as the CDIAPs, CREDAs, and private speech therapy centers for their support in the recruitment of the clinical population. We are very grateful to Ïo Valls-Ratés for acting as storyteller for the intervention materials and to Mireia Gómez Martínez for her help in the data collection. Importantly, this work could not have been possible without the participation of the preschool teachers, speech-language therapists, and the schools and centers where they usually work, as well as the children themselves and their families.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Musical perception skills predict speech imitation skills: differences between speakers of tone and intonation languages

  • Li, Peng
  • Zhang, Yuan
  • Baills, Florence
  • Prieto, Pilar
The ability to imitate speech is linked to individual cognitive abilities such as working memory and the auditory processing of music. However, little research has focused on the role of specific components of musical perception aptitude in relation to an individual’s native language from a crosslinguistic perspective. This study explores the predictive role of four components of musical perception skills and working memory on phonetic language abilities for speakers of two typologically different languages, Catalan (an intonation language) and Chinese (a tone language). Sixty-one Catalan and 144 Chinese participants completed four subtests (accent, melody, pitch and rhythm) of the Profile of Music Perception Skills, a forward digit span task and a speech imitation task. The results showed that for both groups of participants, musical perception skills predicted speech imitation accuracy but working memory did not. Importantly, among the components of musical perception skills, accent was the only predictive factor for Chinese speakers, whereas melody was the only predictive factor for Catalan speakers. These findings suggest that speech imitation ability is predicted by musical perception skills rather than working memory and that the predictive role of specific musical components may depend on the phonological properties of the native language., This study is funded by ‘Multimodal Communication: The integration of prosody and gesture in human communication and in language learning’ (PID2021-123823NB-I00) awarded by the Ministerio de Ciencia e Innovación and ‘Multimodal language learning: Prosodic and Gestural Integration in Pragmatic and Phonological Development’ (PGC2018-097007-B-l00), awarded by the Ministerio de Ciencia, Innovación y Universidades, Agencia Estatal de Investigación, and Fondo Europeo de Desarrollo Regional. P.L. is supported by the Research Council of Norway through its Centres of Excellence funding scheme (223265). F.B. acknowledges a Margarita Salas grant funded by the European Union-NextGenerationEU, Ministry of Universities and Recovery, Transformation and Resilience Plan, through a call from Pompeu Fabra University.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Singing songs facilitates L2 pronunciation and vocabulary learning: a study with Chinese adolescent ESL learners

  • Zhang, Yuan
  • Baills, Florence
  • Prieto Vives, Pilar, 1965-
The present study assesses the effect of a three-session classroom-based training program involving singing songs with familiar melodies on second-language pronunciation and vocabulary learning. Ninety-five adolescent Chinese ESL learners (M = 14.04 years) were assigned to one of two groups. Participants learned the lyrics in English of three songs whose melodies were familiar to them either by singing or reciting the lyrics, following a native English singer/instructor. Before and after training, participants performed two vocabulary tasks (picture-naming and word meaning recall tasks) and two pronunciation tasks (word and sentence oral-reading tasks). The results revealed that although both groups showed gains in vocabulary and pronunciation after training, the singing group outperformed the speech group. These findings support the value of using songs with familiar melodies to teach second languages at the early stages of learning in an ESL classroom context., This research was supported by the Generalitat de Catalunya [2017 SGR_971]; Ministerio de Ciencia, Innovación y Universidades [PGC2018-097007-B-I00]; Ministerio de Ciencia e Innovación [PID2021-123823NB-I00]. The second author acknowledges funding by the European Union-NextGenerationEU, the Spanish Ministry of Universities and Recovery, Transformation and Resilience Plan, through a call from Pompeu Fabra University (Barcelona).
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Unguided virtual-reality training can enhance the oral presentation skills of high-school students

  • Valls-Ratés, Ïo
  • Niebuhr, Oliver
  • Prieto Vives, Pilar, 1965-
Public speaking is fundamental in our daily life, and it happens to be challenging for many people. Like all aspects of language, these skills should be encouraged early on in educational settings. However, the high number of students per class and the extensive curriculum both limit the possibilities of the training and, moreover, entail that students give short in-class presentations under great time pressure. Virtual Reality (VR) environments can help speakers and teachers meet these challenges and foster oral skills. This experimental study employs a between-subjects pre- and post-training design with four Catalan high-school student groups, a VR group (N = 30) and a Non-VR group (N = 20). Both groups gave a 2-min speech in front of a live audience before (pre-training) and after (post-training) 3 training sessions (one session per week) in which they practiced public speaking either in front of a VR audience or alone in a classroom (Non-VR). Students assessed their anxiety measures right before performing every speech and filled out a satisfaction questionnaire at the end. Pre- and post-training speeches were assessed by 15 raters, who analyzed the persuasiveness of the message and the charisma of the presenter. Speeches were also analyzed for prosodic features and gesture rate. First, results showed that self-assessed anxiety got significantly reduced at post-training for both conditions. Second, acoustic analyses of both groups' speeches show that the VR group has, unlike the Non-VR group, developed a more clear and resonant voice quality in the post-training speeches, in terms of higher cepstral-peak prominence (CPP) (although no significant differences in f0- related parameters as a function of training were obtained), as well as significantly less erosion effects than the Non-VR group. However, these differences across groups did not trigger a direct improvement on the participants' gesture rate, persuasiveness and charisma at post-training. Furthermore, students perceived the training in the VR environment to be more useful and beneficial for their future oral presentations than the Non-VR group. All in all, short unguided VR training sessions can help students feel less anxious, promote a more clear and resonant voice style, and can prevent them from experiencing an erosion effect while practicing speeches in front of a real audience., This work benefited from funding awarded by the Spanish Ministry of Economy and Competitiveness (PGC2018-097007-B-I00, PID2021-123823NB-I00) and the Generalitat de Catalunya (2017 SGR_971). We also acknowledge the support from the Recercaixa Project (RecerCaixa 2017ACUP 00249) and the Department of Translation at Universitat Pompeu Fabra through a 1-year doctoral grant to ÏV-R.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Visualizing prosodic structure: manual gestures as highlighters of prosodic heads and edges in English academic discourses

  • Rohrer, Patrick Louis
  • Delais-Roussarie, Elisabeth
  • Prieto Vives, Pilar, 1965-
Research has shown a close temporal relationship between prominence-lending tonal movements in speech and prominence in manual gesture. However, prosodic structure consists of not only prosodic heads (i.e., pitch accentuation) but also prosodic edges. To our knowledge, no previous studies have assessed the value of prosodic edges (nuclear vs. phrase-initial prenuclear pitch accents) as anchoring sites for different types of gestures (i.e., referential vs. non-referential) while independently controlling for the relative degree of prominence associated with the pitch accent. The English M3D-TED corpus, which contains over 23 minutes of multimodal speech, was analyzed in terms of prosody and gesture. Results showed that while the majority of manual gesture strokes overlapped a pitch accented syllable (85.99%), apex alignment occurred at a relatively low rate (50.4%) and alignment rates did not significantly differ between referential and non-referential gestures. At the phrasal level, crucially our results also showed that strokes align with phrase-initial prenuclear pitch accents over nuclear accents, and this relationship is not driven by relative prominence. These findings show that both prosodic heads and prosodic edges (i.e., phrase initial and final positions) are key sites for both referential and non-referential gesture production., The authors would like to acknowledge the financial support awarded by the Spanish Ministry of Science, Innovation and Universities (MCIU), Agencia Estatal de Investigación (AEI), and Fondo Europeo de Desarrollo Regional (FEDER) [grant numbers PGC2018-097007-B-I00 'Multimodal Language Learning (MLL): Prosodic and Gestural Integration in Pragmatic and Phonological Development'; PID2021-123823NB-I00 'Multimodal Communication (MCOM): The integration of prosody and gesture in human communication and in language learning'], by the Generalitat de Catalunya [grant number 2017 SGR_971], and by the GEHM (Gesture and Head Movements in Language) Research Network, funded by the Independent Research Fund Denmark [grant number 9055-00004B]. The first author would like to acknowledge a joint Ph.D. grant, awarded by the Department of Translation and Language Sciences, Universitat Pompeu Fabra, and SGR Grant, Generalitat de Catalunya, [grant number 2017 SGR_971]. We would like to express our gratitude to Ulya Tütücünbasi for her help with the gestural annotation of the corpus. Many thanks are also extended to Dr. Stefanie Shattuck-Hufnagel and Ada Ren (Massachusetts Institute of Technology), as well as the members of the GrEP research lab for their feedback provided on early versions of this study. We would also like to thank Steve Dunne for his help with proofreading the final manuscript.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Towards a novel conceptualization of prosody that accounts for spoken and visual signals: the modality-neutral prosodic framework hypothesis

  • Prieto Vives, Pilar, 1965-
  • Esteve-Gibert, Núria
  • Shattuck-Hufnagel, Stefanie
Prosody influences speech organization by signaling phrasal prominence, grouping patterns, and speakers' pragmatic intentions. While traditionally viewed as restricted to speech, research shows prosody is also conveyed visually. This article reviews research showing strong parallels between spoken prosody and co-speech gestures in prominence marking, grouping phrasal structures, and signaling pragmatic intent. We extend this discussion and propose a modality-neutral prosodic framework hypothesis comprising three propositions: (a) prosody should be viewed as a modality-neutral grammar component that operates as an abstract level of representation while adapting to different sensory channels and language modalities; (b) in spoken languages, prosody is implemented flexibly through two distinct channels, spoken and gestural, which enable to mark prominence, grouping and meaning in a multimodal way; (c) parallel implementations are found in the way prosody is manifested in spoken and sign languages. A modality-neutral view of prosody will enrich current formal and developmental theories of language., We extend our sincere gratitude to our colleagues Marianne Gullberg, David House, Frank Kügler and Patrick Rohrer, as well as our reviewers Gilbert Ambrazaitis, Marisa Cruz and Sónia Frota for their invaluable feedback and fruitful discussions on an early version of the proposal. The authors would like to acknowledge the following financial support received to conduct this research: The Keith North Fund at the Speech Communication Group, Research Laboratory of Electronics, Department of Electrical Engineering and Computer Science, MIT (Ref: 3161400); the Agency for Management of University and Research Grants (Generalitat de Catalunya) (2021 SGR_922, 2021 SGR 01102); and the Spanish Ministry of Science and Innovation, Agencia Estatal de Investigación and Fondo Europeo de Desarrollo Regional (PID2021-123823NB-I00: 'Multimodal Communication: The Integration of Prosody and Gesture in Human Communication and in Language Learning'; PID2020-115385GA-I00: 'The Role of Prosody and Gestures in the Acquisition of Pragmatics: Typical and Atypical Populations'). The first author would like to acknowledge the support from the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) - [project ID 502013510, KU 2323/5-1 & PR 1780/1-1] - SPP 2392 `Visual Communication (ViCom).
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Revisiting the link between second-language sound identification and word recognition with an eye on methodological similarity

  • Llompart, Miquel
  • Gorba, Celia
  • Prieto Vives, Pilar, 1965-
Data de publicació electrònica: 12-09-2025, This study revisits the relationship between second-language (L2) learners' ability to distinguish sounds in non-native phonological contrasts and to recognize spoken words when recognition depends on these sounds, while addressing the role of methodological similarity. Bilingual Catalan/Spanish learners of English were tested on the identification of two vowel contrasts (VI) of diverging difficulty, /i/-// (difficult) and //-/æ/ (easy), in monosyllabic minimal pairs, and on their recognition of the same pairs in a word-picture matching task (WPM). Learners performed substantially better with /i/-// in VI than in WPM, and individual scores were only weakly correlated. By replicating previous findings through a more symmetrical design, we show that an account of prior work rooted in methodological dissimilarity is improbable and provide additional support for the claim that accuracy in sound identification does not guarantee improvements in word recognition. This has implications for our understanding of L2-speech acquisition and L2 pronunciation training., The authors would like to acknowledge the financial support awarded by the Generalitat de Catalunya (2021 SGR 00922) and the Spanish Ministry of Science and Innovation (PID2021-123823NB-I00). The first author's contribution was additionally supported by the Spanish Ministry of Universities (Beatriz Galindo Junior Research Fellowship, BG22/00161), as was the second author's (Margarita Salas Fellowship, ID-706211). We would also like to thank the editor and three anonymous reviewers for their helpful suggestions.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Multidimensional labeling of gesture in communication: the M3D proposal

  • Rohrer, Patrick Louis
  • Tütüncübasi, Ulya
  • Florit-Pons, Júlia
  • Vilà-Giménez, Ingrid
  • Esteve-Gibert, Núria
  • Ren-Mitchell, Ada
  • Shattuck-Hufnagel, Stefanie
  • Prieto Vives, Pilar, 1965-
Communication is multimodal in that speakers use not only their voices, but also co-speech gestures to communicate. Recent insights suggest that gestural behavior has a strong association with prosodic structure and that a single gesture can communicate various semantic and pragmatic meanings. This highlights the importance of developing a comprehensive, flexible, and transparent approach to gesture annotation that accounts for multiple dimensions of gesture, including a gesture's form, prosodic properties, and semantic and pragmatic contributions. To address this need for an increasingly dimensionalized approach to multimodal data annotation, the main goal of this paper is to present and describe a novel labeling system for manual gestures. The MultiModal MultiDimensional (M3D) system consists of an open access package that has been developed in coordination with five different labs working on gesture and its interaction with speech. The package includes a set of reliable annotation guidelines, a validated training program, and two annotated audiovisual corpora that represent over 60 minutes of lecture-style speech., This work was partially funded by the Spanish Ministry of Science, Innovation and Universities (Grants numbers PID2021-123823NB-I00, PID2020-115385GA-I00, PGC2018-097007-B-I00), the Pla de Recerca de Catalunya from the Generalitat de Catalunya (Grant numbers 2017 SGR_971 and 2021 SGR 00922) and the GEHM (Gesture and Head Movements in Language) Research Network, funded by the Independent Research Fund Denmark (grant number 9055-00004B).
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Preschoolers mark focus types through multimodal prominence: further evidence for the precursor role of gestures

  • Coego, Sara
  • Esteve Gibert, Núria
  • Prieto Vives, Pilar, 1965-
The present cross-sectional study assessed the role of multimodal cues in marking focus types during early childhood, focusing on prosodic prominence, gesture presence, and gestural prominence. A total of 116 Catalan-speaking three-, four- and five-year-olds participated in a semi-controlled interactive task eliciting words in three focus conditions: information, contrastive, and corrective. The data were coded manually using holistic assessments for all three measures. The results indicated, first, that children’s prosodic and gestural behavior was key in marking corrective focus. A significant tendency to use more gestures and increase both prosodic and gestural prominence was found in the corrective focus condition across the three age groups. Second, a developmental difference emerged in the acquisition of contrastive focus. Three-year-olds relied solely on gesture presence to encode contrastive focus, being unable to differentiate it prosodically from information focus. In turn, four- and five-year-olds used both gestures and prosody, with contrastive focus not only receiving more gestures than information focus but also increased prosodic prominence. This finding shows that gesture presence is a precursor to prosodic prominence in marking contrastive focus in Catalan, thus supporting the idea that gesture production can bootstrap the expression of focus type distinctions., This research was funded by grant PID2020-115385GA-I00, financed by MCIN/AEI/10.13039/501100011033 and by the European Regional Development Fund (ERDF), by the European Union. It was also supported by the grant AGAUR-FI Joan Oró 2025FI_B 00342, funded by the Agency for Management of University and Research Grants (Generalitat de Catalunya) together with the European Social Fund (ESF), by the European Union. The APC was funded by grants PID2021-123823NB-I00 and PID2020-115385GA-I00, financed by MCIN/AEI/10.13039/501100011033 and by the European Regional Development Fund (ERDF), by the European Union.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Multimodal skills, but not motor skills, predict narrative and expressive pragmatic skills in children with typical development and neurodevelopmental disorders

  • Florit Pons, Júlia
  • Pronina, Mariia
  • Igualada, Alfonso
  • Prieto Vives, Pilar, 1965-
  • Norbury, Courtenay
To see whether communicative-based multimodal skills (compared to non-communicative motor skills) predicted complex language skills, this study examined the predictive power of multimodal and motor skills on narrative and expressive pragmatic abilities across two groups. Participants were children with typical development (N=88, Mage=5.34, 48% female) and with neurodevelopmental disorders (N = 51, Mage = 5.01, 25% female) mostly of white ethnicity (86.3%). We evaluated children's multimodal accuracy, motor skills, core language skills, and narrative and pragmatic skills. Results revealed that, in all groups, both multimodal skills and core language significantly predicted narrative (R2 = 0.569) and pragmatic skills (R2 = 0.621), while motor skills did not. These findings highlight the relevance of multimodality in the assessment of children's complex language skills., The authors would like to acknowledge the financial support awarded by the Spanish Ministry of Science, Innovation and Universities (PID2021-123823NB-I00: "Multimodal Communication: The Integration of Prosody and Gesture in Human Communication and in Language Learning"; MICIU/AEI/10.13039/501100011033) and awarded by the Generalitat de Catalunya (2021 SGR_922). Also, the first author would like to acknowledge a Ph.D. grant, awarded by the Generalitat de Catalunya (2021 FI_B 00778) and the Department of Translation and Language Sciences at Universitat Pompeu Fabra.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

An individualized multimodal narrative intervention for young children with neurodevelopmental disorders

  • Florit-Pons, Júlia
  • Prieto Vives, Pilar, 1965-
  • Igualada Pérez, Alfonso
Data de publicació electrònica: 11-04-2026, This study examines whether an individualized narrative intervention can boost the narrative (macrostructure and perspec-tive-taking) abilities of children with neurodevelopmental disorders (NDD). An experimental group of 16 children with NDD from Catalonia, Spain, received 9 MultiModal Narrative intervention sessions. Children's narrative skills were mea-sured pre- and post-intervention with a retelling task. Their performance was compared to two control groups not receiving the intervention (17 children with NDD and 17 with typical development). After each session, children in the experimental group underwent two Dynamic Assessment measures (retelling task and graduated prompting comprehension questions about the story), tracking their learning. Results showed an improvement in narrative macrostructure but not perspective-taking. The Dynamic Assessment revealed faster learning of macrostructure, individual differences in the prompts needed for successful responses, and that fewer prompts predicted better macrostructure outcomes post-intervention. These find-ings underscore the relevance of diverse measures in evaluating continuous learning in interventions., Open Access funding provided thanks to the CRUE-CSIC agreement with Springer Nature. This research received financial support awarded by the Spanish Ministry of Science and Innovation, Agencia Estatal de Investigación and Fondo Europeo de Desarrollo Regional (PID2021-123823NB-I00); by the Generalitat de Catalunya (2021 SGR_922). Also, the first author would like to acknowledge a Ph.D. grant, awarded by the Generalitat de Catalunya (2021 FI_B00778) and by the Department of Translation and Language Sciences (Universitat Pompeu Fabra).
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Multimodal mitigation: how facial and body cues index politeness in Catalan requests

  • Hübscher, Iris
  • Sánchez-Conde, Cristina
  • Borràs-Comes, Joan
  • Vincze, Laura
  • Prieto Vives, Pilar, 1965-
Recent cross-linguistic research has demonstrated that speakers use a prosodic mitigation strategy when addressing higher status interlocutors by talking more slowly, reducing the intensity and lowering the overall fundamental frequency (F0). Much less is known, however, about how politeness-related meaning is expressed multimodally (i.e., combining verbal and multimodal channels). The present study investigates how Catalan native speakers encode politeness-related meanings through facial and body cues. We test whether speakers apply a gestural mitigation strategy and use specific hedging devices in socially distant situations (e.g., when asking an older person of higher status for a favor). Twenty Catalan speakers were video-recorded while participating in a discourse elicitation task where they were required to produce requests in polite and non-polite contexts. In the resulting recordings, a set of 21 facial and body cues associated with speech were coded and analyzed. The results show that politeness-related meanings are expressed through gestural mitigation strategies that go hand-in-hand with previously reported prosodic mitigation strategies., This work was financed by the Spanish Ministry of Science, Innovation and Universities, Agencia Estatal de Investigacion, and Fondo Europeo ´ de Desarrollo Regional, grant numbers PGC2018-097007-B-100 and PID2021- 123823NB-I00; and by a grant awarded by the Generalitat de Catalunya, grant number 2017 SGR_971 to the Prosodic Studies Group.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Encouraging participant embodiment during VR-assisted public speaking training improves persuasiveness and charisma and reduces anxiety in secondary school students

  • Valls-Ratés, Ïo
  • Niebuhr, Oliver
  • Prieto Vives, Pilar, 1965-
Practicing public speaking to simulated audiences created in virtual reality environments is reported to be effective for reducing public speaking anxiety. However, little is known about whether this effect can be enhanced by encouraging the use of gestures during VR-assisted public speaking training. In the present study two groups of secondary schools underwent a three-session public speaking training program in which they delivered short speeches to VR-simulated audiences. One group was encouraged to “embody” their speeches through gesture while the other was given no instructions regarding the use of gesture. Before and after the training sessions participants underwent respectively a pre- and a post-training session, which consisted of delivering a similar short speech to a small live audience. At pre- and post-training sessions, participants’ levels of anxiety were self-assessed, their speech performances were rated for persuasiveness and charisma by independent raters, and their verbal output was analyzed for prosodic features and gesture rate. Results showed that both groups significantly reduced their self-assessed anxiety between the pre- and post-training sessions. Persuasiveness and charisma ratings increased for both groups, but to a significantly greater extent in the gesture-using group. However, the prosodic and gestural features analyzed showed no significant differences across groups or from pre-to post-training speeches. Thus, our results seem to indicate that encouraging the use of gesture in VR-assisted public speaking practice can help students be more charismatic and their delivery more persuasive before presenting in front of a live audience., This work benefited from funding awarded by the Spanish Ministry of Economy and Competitiveness (PGC2018-097007-B-I00 and PID2021-123823NB-I00) and the Generalitat de Catalunya (2017 SGR_971). We also acknowledge support from the Recercaixa Project (RecerCaixa 2017ACUP 00249) and the Department of Translation at Universitat Pompeu Fabra through a 1-year doctoral grant to the first author.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Capítulo o parte de un libro (BookPart)

Gesture shape and gesture–speech alignment predict simultaneous L2 sound production accuracy

  • Li, Peng
  • Baills, Florence
  • Xi, Xiaotong
  • Prieto, Pilar
Previous research suggests hand gestures aid L2 pronunciation learning, but the underlying mechanisms remain unclear. This study examines if accurate gesture imitation during phonetic training predicts the accuracy of non-native sounds produced while doing a pedagogical hand gesture. Twenty-nine Catalan speakers, unfamiliar with Chinese, learned to pronounce Chinese aspirated plosives while mimicking gestures representing the aspirated sounds. Results revealed that more precise gesture shape and better gesture–speech alignment predicted more accurate speech imitation, emphasizing the significance of the spatiotemporal gesture–speech coupling in embodied L2 phonetic training., This study is funded by “Multimodal Communication: The integration of prosody and gesture in human communication and in language learning” [PID2021-123823NB-I00] awarded by the Ministerio de Ciencia e Innovación and “Multimodal language learning: Prosodic and Gestural Integration in Pragmatic and Phonological Development” [PGC2018- 097007-B-l00], awarded by the Ministerio de Ciencia, Innovación y Universidades, Agencia Estatal de Investigación, and Fondo Europeo de Desarrollo Regional. PL is supported by the Research Council of Norway through its Centres of Excellence funding scheme [223265]. FB acknowledges a Margarita Salas grant funded by European Union– NextGenerationEU, Ministry of Universities and Recovery, Transformation, and Resilience Plan, through a call from Pompeu Fabra University. XX is supported by the Secretaria d’Universitats i Recerca de la Generalitat de Catalunya and the European Social Fund under the Grant for the recruitment of early-stage research staff [2021FI_B 00137].
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Musical perception skills predict speech imitation skills: differences between speakers of tone and intonation languages

  • Li, Peng
  • Zhang, Yuan
  • Baills, Florence
  • Prieto Vives, Pilar, 1965-
The ability to imitate speech is linked to individual cognitive abilities such as working memory and the auditory processing of music. However, little research has focused on the role of specific components of musical perception aptitude in relation to an individual’s native language from a crosslinguistic perspective. This study explores the predictive role of four components of musical perception skills and working memory on phonetic language abilities for speakers of two typologically different languages, Catalan (an intonation language) and Chinese (a tone language). Sixty-one Catalan and 144 Chinese participants completed four subtests (accent, melody, pitch and rhythm) of the Profile of Music Perception Skills, a forward digit span task and a speech imitation task. The results showed that for both groups of participants, musical perception skills predicted speech imitation accuracy but working memory did not. Importantly, among the components of musical perception skills, accent was the only predictive factor for Chinese speakers, whereas melody was the only predictive factor for Catalan speakers. These findings suggest that speech imitation ability is predicted by musical perception skills rather than working memory and that the predictive role of specific musical components may depend on the phonological properties of the native language., This study is funded by ‘Multimodal Communication: The integration of prosody and gesture in human communication and in language learning’ (PID2021-123823NB-I00) awarded by the Ministerio de Ciencia e Innovación and ‘Multimodal language learning: Prosodic and Gestural Integration in Pragmatic and Phonological Development’ (PGC2018-097007-B-l00), awarded by the Ministerio de Ciencia, Innovación y Universidades, Agencia Estatal de Investigación, and Fondo Europeo de Desarrollo Regional. P.L. is supported by the Research Council of Norway through its Centres of Excellence funding scheme (223265). F.B. acknowledges a Margarita Salas grant funded by the European Union-NextGenerationEU, Ministry of Universities and Recovery, Transformation and Resilience Plan, through a call from Pompeu Fabra University.