Pasar al contenido principal

APRENDIZAJE MULTIMODAL DEL LENGUAJE (MLL). INTEGRACION PROSODICA Y GESTUAL EN EL DESARROLLO PRAGMATICO Y FONOLOGICO

PGC2018-097007-B-I00

•
Nombre agencia financiadora Agencia Estatal de Investigación
Acrónimo agencia financiadora AEI
Programa Programa Estatal de Generación de Conocimiento y Fortalecimiento Científico y Tecnológico del Sistema de I+D+i
Subprograma Subprograma Estatal de Generación de Conocimiento
Convocatoria Proyectos de I+D (Excelencia / Generación de Conocimiento)
Año convocatoria 2018
Plan Estatal 2017-2020 Plan Estatal de Investigación Científica y Técnica y de Innovación
Centro beneficiario UNIVERSITAT POMPEU FABRA CCT

Resultados relacionados

Resultados totales (Incluyendo duplicados): 57
Encontrada(s) 1 página(s)

Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Encouraging participant embodiment during VR-assisted public speaking training improves persuasiveness and charisma and reduces anxiety in secondary school students

  • Valls-Ratés, Ïo
  • Niebuhr, Oliver
  • Prieto Vives, Pilar, 1965-
Practicing public speaking to simulated audiences created in virtual reality environments is reported to be effective for reducing public speaking anxiety. However, little is known about whether this effect can be enhanced by encouraging the use of gestures during VR-assisted public speaking training. In the present study two groups of secondary schools underwent a three-session public speaking training program in which they delivered short speeches to VR-simulated audiences. One group was encouraged to “embody” their speeches through gesture while the other was given no instructions regarding the use of gesture. Before and after the training sessions participants underwent respectively a pre- and a post-training session, which consisted of delivering a similar short speech to a small live audience. At pre- and post-training sessions, participants’ levels of anxiety were self-assessed, their speech performances were rated for persuasiveness and charisma by independent raters, and their verbal output was analyzed for prosodic features and gesture rate. Results showed that both groups significantly reduced their self-assessed anxiety between the pre- and post-training sessions. Persuasiveness and charisma ratings increased for both groups, but to a significantly greater extent in the gesture-using group. However, the prosodic and gestural features analyzed showed no significant differences across groups or from pre-to post-training speeches. Thus, our results seem to indicate that encouraging the use of gesture in VR-assisted public speaking practice can help students be more charismatic and their delivery more persuasive before presenting in front of a live audience., This work benefited from funding awarded by the Spanish Ministry of Economy and Competitiveness (PGC2018-097007-B-I00 and PID2021-123823NB-I00) and the Generalitat de Catalunya (2017 SGR_971). We also acknowledge support from the Recercaixa Project (RecerCaixa 2017ACUP 00249) and the Department of Translation at Universitat Pompeu Fabra through a 1-year doctoral grant to the first author.
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Unguided virtual-reality training can enhance the oral presentation skills of high-school students

  • Valls-Ratés, Ïo
  • Niebuhr, Oliver
  • Prieto Vives, Pilar, 1965-
Public speaking is fundamental in our daily life, and it happens to be challenging for many people. Like all aspects of language, these skills should be encouraged early on in educational settings. However, the high number of students per class and the extensive curriculum both limit the possibilities of the training and, moreover, entail that students give short in-class presentations under great time pressure. Virtual Reality (VR) environments can help speakers and teachers meet these challenges and foster oral skills. This experimental study employs a between-subjects pre- and post-training design with four Catalan high-school student groups, a VR group (N = 30) and a Non-VR group (N = 20). Both groups gave a 2-min speech in front of a live audience before (pre-training) and after (post-training) 3 training sessions (one session per week) in which they practiced public speaking either in front of a VR audience or alone in a classroom (Non-VR). Students assessed their anxiety measures right before performing every speech and filled out a satisfaction questionnaire at the end. Pre- and post-training speeches were assessed by 15 raters, who analyzed the persuasiveness of the message and the charisma of the presenter. Speeches were also analyzed for prosodic features and gesture rate. First, results showed that self-assessed anxiety got significantly reduced at post-training for both conditions. Second, acoustic analyses of both groups' speeches show that the VR group has, unlike the Non-VR group, developed a more clear and resonant voice quality in the post-training speeches, in terms of higher cepstral-peak prominence (CPP) (although no significant differences in f0- related parameters as a function of training were obtained), as well as significantly less erosion effects than the Non-VR group. However, these differences across groups did not trigger a direct improvement on the participants' gesture rate, persuasiveness and charisma at post-training. Furthermore, students perceived the training in the VR environment to be more useful and beneficial for their future oral presentations than the Non-VR group. All in all, short unguided VR training sessions can help students feel less anxious, promote a more clear and resonant voice style, and can prevent them from experiencing an erosion effect while practicing speeches in front of a real audience., This work benefited from funding awarded by the Spanish Ministry of Economy and Competitiveness (PGC2018-097007-B-I00, PID2021-123823NB-I00) and the Generalitat de Catalunya (2017 SGR_971). We also acknowledge the support from the Recercaixa Project (RecerCaixa 2017ACUP 00249) and the Department of Translation at Universitat Pompeu Fabra through a 1-year doctoral grant to ÏV-R.
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

The predictive value of non-referential beat gestures: early use in parent–child interactions predicts narrative abilities at 5 years of age

  • Vilà-Giménez, Ingrid
  • Dowling, Natalie
  • Demir-Lira, Ö Ece
  • Prieto Vives, Pilar, 1965-
  • Goldin-Meadow, Susan
A longitudinal study with 45 children (Hispanic, 13%; non-Hispanic, 87%) investigated whether the early production of non-referential beat and flip gestures, as opposed to referential iconic gestures, in parent-child naturalistic interactions from 14 to 58 months old predicts narrative abilities at age 5. Results revealed that only non-referential beats significantly (p < .01) predicted later narrative productions. The pragmatic functions of the children’s speech that accompany these gestures were also analyzed in a representative sample of 18 parent-child dyads, revealing that beats were typically associated with biased assertions or questions. These findings show that the early use of beats predicts narrative abilities later in development, and suggest that this relation is likely due to the pragmatic– structuring function that beats reflect in early discourse., Funding Information: - Spanish Ministry of Science, Innovation and Universities (MCIU), Agencia Estatal de Investigación (AEI), and Fondo Europeo de Desarrollo Regional (FEDER): PGC2018-097007-B-100, “Multimodal Language Learning (MLL): Prosodic and Gestural Integration in Pragmatic and Phonological Development.” - Generalitat de Catalunya: 2017 SGR_971 - Generalitat de Catalunya, Agència de Gestió d’Ajuts Universitaris i de Recerca: 2019FI_B2_00125 - Eunice Kennedy Shriver National Institute of Child Health & Human Development of the National Institutes of Health: P01HD040605.
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Expressive pragmatics and prosody in young preschoolers are more closely related to structural language than to mentalizing

  • Pronina, Mariia
  • Prieto Vives, Pilar, 1965-
  • Bischetti, Luca
  • Bambini, Valentina
Pragmatics lies at the point where language meets the social world and encompasses both the linguistic and the social dimensions of communication. However, the relationship between pragmatic abilities, other language skills, and socio-cognitive aspects such as mentalizing is still a matter of wide debate. This study sets out to investigate the status of pragmatic abilities by testing from a developmental angle their relationship with other linguistic skills and mentalizing. We examined the role of structural language and mentalizing on both expressive pragmatic and prosodic skills in typically developing preschool children. A total of 105 3-to 4-year-old children were assessed on pragmatics and prosody with the Audiovisual Pragmatic Test, as well as on structural language skills (vocabulary and syntax) and a series of mentalizing measures (false belief, emotion understanding, and metacognitive vocabulary). A combined approach including correlations, regressions, and structural equation modeling (SEM) was used. Structural language was a strong positive predictor of both pragmatic and prosodic scores, while mentalizing predicted neither pragmatic nor prosodic performance. We suggest that in preschool years, expressive pragmatics and prosodic skills are more closely related to structural language skills than to mentalizing., This work was supported by the Generalitat de Catalunya [2017 SGR_971,2020 FI_B2 00192]; Ministerio de Ciencia, Innovación y Universidades [PGC2018-097007-B-I00].
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

The effectiveness of embodied prosodic training in L2 accentedness and vowel accuracy

  • Li, Peng
  • Baills, Florence
  • Baqué, Lorraine
  • Prieto Vives, Pilar, 1965-
This study explores the effects of embodied prosodic training on the production of non-native French front rounded vowels (i.e. /y, ø, œ/) and the overall pronunciation proficiency. Fifty-seven Catalan learners of French practiced pronunciation in one of two conditions: one group observed hand gestures embodying prosodic features of the sentences they were listening to, while the other group did not see any such gestures. The learning outcome was assessed in a pretest, posttest, and delayed posttest through a dialogue-reading task and a sentence imitation task in terms of accentedness, comprehensibility and fluency scores, and through formant analysis of participant-produced target vowels. The results showed that compared to non-embodied training, embodied prosodic training yielded continuous improvement in accentedness in both tasks and improved the F2 values of French front rounded vowels (more fronted). As for comprehensibility and fluency scores, both groups showed similar levels of significant improvement. This study highlights the interaction between prosodic and segmental features of speech by showing that training with embodied prosodic features benefitted accentedness and the production accuracy of non-native vowels., This work was funded by the Ministerio de Ciencia, Innovación y Universidades, Agencia Estatal de Investigación and European Regional Development Fund under Multimodal language learning (MLL): Prosodic and Gestural Integration in Pragmatic and Phonological Development [PGC2018-097007-B-I00]. This work is also supported by the Agència de Gestió d'Ajuts Universitaris i de Recerca (AGAUR) under Pla de Recerca de Catalunya [2017 SGR_971], and the Ministerio de Economía y Competitividad under Cognipros project [FFI2013-40419-P]. The second author acknowledges a grant from the Department of Translation and Language Sciences at Universitat Pompeu Fabra.
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Improving second language vowel production with hand gestures encoding visible articulation: evidence from picture‐naming and paragraph‐reading tasks

  • Xi, Xiaotong
  • Li, Peng
  • Prieto Vives, Pilar, 1965-
Includes supplementary materials for the online appendix., This study investigates whether audiovisual phonetic training with hand gestures encoding visible or nonvisible articulation features has a differential impact on learning second language sounds. Ninety-nine Catalan–Spanish bilingual students were trained to differentiate English /æ/ and /ʌ/, which differ in the visible lip aperture and nonvisible tongue position, with training involving no gestures, gestures representing the lip aperture, or gestures representing the tongue position. Before, immediately after, and 1 week after the training, participants’ perception of the targets was assessed through a word-identification task, and their production was tested through paragraph-reading, picture-naming, and word-imitation tasks. Although all participants improved in perception and production, the lip hand gesture was more effective in adjusting lip aperture than the other two conditions in the paragraph-reading and picture-naming tasks. These results suggest that hand gestures encoding visible rather than nonvisible articulation features are more effective for improving second language pronunciation., The study was supported by the Ministerio de Ciencia, Innovación y Universidades, Agencia Estatal de Investigación, and Fondo Europeo de Desarrollo Regional [PGC2018-097007-B-l00], and by the Ministerio de Ciencia e Innovación [PID2021-123823NB-I00]. The first author is supported by the Department d'Empresa i Coneixement de la Generalitat de Catalunya and the European Social Fund under a grant for the recruitment of early-stage research staff [2021FI_B 00137]. The second author is supported by the Research Council of Norway through its Centres of Excellence funding scheme [223265].
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Bridging the gap between prosody and pragmatics: the acquisition of pragmatic prosody in the preschool years and Its relation with theory of mind

  • Pronina, Mariia
  • Hübscher, Iris
  • Vilà-Giménez, Ingrid
  • Prieto Vives, Pilar, 1965-
While it is well known that prosodic features are central in the conveyance of pragmatic meaning across languages, developmental research has assessed a narrow set of pragmatic functions of prosody. Research on prosodic development has focused on early infancy, with the subsequent preschool ages and beyond having received less attention. This study sets out to explore how young preschoolers develop the ability to use prosody to express pragmatic meanings while taking into account children’s Theory of Mind (ToM) development. Though ToM has been suggested to be linked to the development of receptive prosody, little is known about its relationship with expressive prosodic skills. A total of 102 3- to 4-year-old Catalan-speaking children were assessed for their pragmatic prosody skills using 35 picture-supported prompts revolving around a variety of social scenarios, as well as for their ToM skills. The responses were analyzed for prosodic appropriateness. The analyses revealed that 3- to 4-year-olds successfully produced prosody to encode basic expressive acts and unbiased speech acts such as information-seeking questions. Yet they had more trouble with complex expressive acts and biased speech acts such as the ones that convey speakers’ beliefs. Further analyses showed that ToM alone is not sufficient to explain children’s prosodic score, but the prosodic performance in some pragmatic areas (unbiased pragmatic meanings) was predicted by the interaction between ToM and age. Overall, this evidence for the acquisition of pragmatic prosody by young preschoolers demonstrates the importance of bridging the gap between prosody and pragmatics when accounting for prosodic developmental profiles, as well as taking into account the potential influence of ToM and other socio-cognitive and language skills in this development., This work benefited from funding awarded by the Spanish Ministry of Science, Innovation and Universities (MCIU), Agencia Estatal de Investigación (AEI), Fondo Europeo de Desarrollo Regional (FEDER) (PGC2018-097007-B-I00), and the Generalitat de Catalunya (2017 SGR_971). IH was supported by the University of Zurich’s University Research Priority Program “Language and Space.” MP also acknowledges support from the Secretariat for Universities and Research of the Ministry of Business and Knowledge of the Government of Catalonia and the European Social Fund (2020 FI_B2 00192).
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Visualizing prosodic structure: manual gestures as highlighters of prosodic heads and edges in English academic discourses

  • Rohrer, Patrick Louis
  • Delais-Roussarie, Elisabeth
  • Prieto Vives, Pilar, 1965-
Research has shown a close temporal relationship between prominence-lending tonal movements in speech and prominence in manual gesture. However, prosodic structure consists of not only prosodic heads (i.e., pitch accentuation) but also prosodic edges. To our knowledge, no previous studies have assessed the value of prosodic edges (nuclear vs. phrase-initial prenuclear pitch accents) as anchoring sites for different types of gestures (i.e., referential vs. non-referential) while independently controlling for the relative degree of prominence associated with the pitch accent. The English M3D-TED corpus, which contains over 23 minutes of multimodal speech, was analyzed in terms of prosody and gesture. Results showed that while the majority of manual gesture strokes overlapped a pitch accented syllable (85.99%), apex alignment occurred at a relatively low rate (50.4%) and alignment rates did not significantly differ between referential and non-referential gestures. At the phrasal level, crucially our results also showed that strokes align with phrase-initial prenuclear pitch accents over nuclear accents, and this relationship is not driven by relative prominence. These findings show that both prosodic heads and prosodic edges (i.e., phrase initial and final positions) are key sites for both referential and non-referential gesture production., The authors would like to acknowledge the financial support awarded by the Spanish Ministry of Science, Innovation and Universities (MCIU), Agencia Estatal de Investigación (AEI), and Fondo Europeo de Desarrollo Regional (FEDER) [grant numbers PGC2018-097007-B-I00 'Multimodal Language Learning (MLL): Prosodic and Gestural Integration in Pragmatic and Phonological Development'; PID2021-123823NB-I00 'Multimodal Communication (MCOM): The integration of prosody and gesture in human communication and in language learning'], by the Generalitat de Catalunya [grant number 2017 SGR_971], and by the GEHM (Gesture and Head Movements in Language) Research Network, funded by the Independent Research Fund Denmark [grant number 9055-00004B]. The first author would like to acknowledge a joint Ph.D. grant, awarded by the Department of Translation and Language Sciences, Universitat Pompeu Fabra, and SGR Grant, Generalitat de Catalunya, [grant number 2017 SGR_971]. We would like to express our gratitude to Ulya Tütücünbasi for her help with the gestural annotation of the corpus. Many thanks are also extended to Dr. Stefanie Shattuck-Hufnagel and Ada Ren (Massachusetts Institute of Technology), as well as the members of the GrEP research lab for their feedback provided on early versions of this study. We would also like to thank Steve Dunne for his help with proofreading the final manuscript.
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Multimodal mitigation: how facial and body cues index politeness in Catalan requests

  • Hübscher, Iris
  • Sánchez-Conde, Cristina
  • Borràs-Comes, Joan
  • Vincze, Laura
  • Prieto Vives, Pilar, 1965-
Recent cross-linguistic research has demonstrated that speakers use a prosodic mitigation strategy when addressing higher status interlocutors by talking more slowly, reducing the intensity and lowering the overall fundamental frequency (F0). Much less is known, however, about how politeness-related meaning is expressed multimodally (i.e., combining verbal and multimodal channels). The present study investigates how Catalan native speakers encode politeness-related meanings through facial and body cues. We test whether speakers apply a gestural mitigation strategy and use specific hedging devices in socially distant situations (e.g., when asking an older person of higher status for a favor). Twenty Catalan speakers were video-recorded while participating in a discourse elicitation task where they were required to produce requests in polite and non-polite contexts. In the resulting recordings, a set of 21 facial and body cues associated with speech were coded and analyzed. The results show that politeness-related meanings are expressed through gestural mitigation strategies that go hand-in-hand with previously reported prosodic mitigation strategies., This work was financed by the Spanish Ministry of Science, Innovation and Universities, Agencia Estatal de Investigacion, and Fondo Europeo ´ de Desarrollo Regional, grant numbers PGC2018-097007-B-100 and PID2021- 123823NB-I00; and by a grant awarded by the Generalitat de Catalunya, grant number 2017 SGR_971 to the Prosodic Studies Group.
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Encouraging kids to beat: children's beat gesture production boosts their narrative performance

  • Vilà-Giménez, Ingrid
  • Prieto Vives, Pilar, 1965-
Gesture is an integral part of language development. While recent evidence shows that observing a speaker who is simultaneously producing beat gestures helps preschoolers remember and understand information and also improves the production of oral narratives, little is known about the potential value of encouraging children to produce beat gestures—as opposed to merely observing them. In this between‐subjects pretest–posttest training study we examine whether encouraging children to produce beats can boost their narrative performance. A total of 47 5‐ to 6‐year‐old children were divided into two groups and exposed to a training session in which a total of six stories were presented under one of two experimental conditions: (a) the children merely observed video‐recordings of a storyteller who used beat gestures and were then asked to retell the narratives; or (b) the children observed the same video‐recordings and then retold the narratives but were encouraged to simultaneously use their hands in the same way the storytellers did. Pretests and posttests consisting of children's narrations of short animated cartoons were analysed for narrative structure and fluency. A comparison of scores showed that children in the group that had been encouraged to use beat gestures in the training phase performed better in both narrative structure and fluency than the group of children who were simply asked to retell the story without gesture instruction. These findings suggest that linguistically relevant body movements serve to boost language development and that embodied storytelling can be of help in narrative training., This study was approved by the ethics committee of the Universitat Pompeu Fabra and benefited from funding awarded by the Spanish Ministry of Science, Innovation and Universities (MCIU), Agencia Estatal de Investigación (AEI), and Fondo Europeo de Desarrollo Regional (FEDER) (PGC2018- 097007-B-100 “Multimodal Language Learning (MLL): Prosodic and Gestural Integration in Pragmatic and Phonological Development”) and by the Generalitat de Catalunya (2017 SGR_971) to the Prosodic Studies Group. The first author also acknowledges an FI grant from the Generalitat de Catalunya (ref. 2019FI_B2_00125).
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Three-year-olds infer polite stance from intonation and facial cues

  • Prieto Vives, Pilar, 1965-
  • Hübscher, Iris
  • Wagner, Laura
Despite the evidence that infants are sensitive to facial cues and prosody for the detection of emotion, we have contradictory evidence regarding the use of these cues by older preschool and school children when inferring both emotional and politeness stance. This study assessed preschool aged children’s sensitivity to intonational and facial cues signalling a speaker’s polite stance in requestive speech acts with controlled lexical and contextual materials. Thirty-six 3-yearold American English-speaking children performed a forced-choice decision task which investigated whether children at this age use pitch and/or facial cues to infer a speaker’s affective stance in either audio-only, visual-only or audio-visual presentation modalities, when lexical cues are controlled for. Results showed that (a) children at three years can infer a speaker’s polite stance equally well in all three conditions (audio-only, visual-only and audio-visual) and thereby (b) unlike previous research, in the present task both intonation and facial cues are equally strong cues in children’s understanding of a speaker’s polite stance in requestive speech acts. The authors discuss especially the implications of this early use of intonation to detect politeness, relating it to other previous research on children’s ability to infer meaning from pitch.
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Children use non-referential gestures in narrative speech to mark discourse elements which update common ground

  • Rohrer, Patrick Louis
  • Florit-Pons, Júlia
  • Vilà-Giménez, Ingrid
  • Prieto Vives, Pilar, 1965-
While recent studies have claimed that non-referential gestures (i.e., gestures that do not visually represent any semantic content in speech) are used to mark discourse-new and/or -accessible referents and focused information in adult speech, to our knowledge, no prior investigation has studied the relationship between information structure (IS) and gesture referentiality in children’s narrative speech from a developmental perspective. A longitudinal database consisting of 332 narratives performed by 83 children at two different time points in development was coded for IS and gesture referentiality (i.e., referential and non-referential gestures). Results revealed that at both time points, both referential and non-referential gestures were produced more with information that moves discourse forward (i.e., focus) and predication (i.e., comment) rather than topical or background information. Further, at 7–9 years of age, children tended to use more nonreferential gestures to mark focus and comment constituents than referential gestures. In terms of the marking of the newness of discourse referents, non-referential gestures already seem to play a key role at 5–6 years old, whereas referential gestures did not show any patterns. This relationship was even stronger at 7–9 years old. All in all, our findings offer supporting evidence that in contrast with referential gestures, nonreferential gestures have been found to play a key role in marking IS, and that the development of this relationship solidifies at a period in development that coincides with a spurt in non-referential gesture production., PLR would like to acknowledge a joint Ph.D. grant, awarded by the Department of Translation and Language Sciences, Universitat Pompeu Fabra, and SGR AGAUR Grant, Generalitat de Catalunya, Award number: 2017 SGR_971, while JF-P would like to acknowledge a Ph.D. grant, awarded by the Generalitat de Catalunya, Award number: 2021 FI_B 00778. We would also like to acknowledge the financial support awarded by the Spanish Ministry of Science, Innovation and Universities (MCIU), Agencia Estatal de Investigación (AEI), and Fondo Europeo de Desarrollo Regional (FEDER) [PGC2018- 097007-B-100 “Multimodal Language Learning (MLL): Prosodic and Gestural Integration in Pragmatic and Phonological Development”], by the Spanish Ministry of Economy and Competitiveness (MINECO) (FFI2015-66533-P “Intonational and gestural meaning in language”), by the Generalitat de Catalunya (2014 SGR_925, 2017 SGR_971), and by the GEHM (Gesture and Head Movements in Language) Research Network, funded by the Independent Research Fund Denmark (9055- 00004B)
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Hand gestures facilitate the acquisition of novel phonemic contrasts when they appropriately mimic target phonetic features

  • Xi, Xiaotong
  • Li, Peng
  • Baills, Florence
  • Prieto Vives, Pilar, 1965-
Purpose: Research has shown that observing hand gestures mimicking pitch movements or rhythmic patterns can improve the learning of second language (L2) suprasegmental features. However, less is known about the effects of hand gestures on the learning of novel phonemic contrasts. This study examines (a) whether hand gestures mimicking phonetic features can boost L2 segment learning by naive learners and (b) whether a mismatch between the hand gesture form and the target phonetic feature influences the learning effect. Method: Fifty Catalan native speakers undertook a short multimodal training session on two types of Mandarin Chinese consonants (plosives and affricates) in either of two conditions: Gesture and No Gesture. In the Gesture condition, a fist-to-open-hand gesture was used to mimic air burst, while the No Gesture condition included no such use of gestures. Crucially, while the hand gesture appropriately mimicked the air burst produced in plosives, this was not the case for affricates. Before and after training, participants were tested on two tasks, namely, the identification task and the imitation task. Participants' speech output was rated by five Chinese native speakers. Results: The perception results showed that training with or without gestures yielded similar degrees of improvement for the identification of aspiration contrasts. By contrast, the production results showed that, while training without gestures did not help improve L2 pronunciation, training with gestures improved pronunciation, but only when the given gestures appropriately mimicked the phonetic properties they represented. Conclusions: Results revealed that the efficacy of observing hand gestures on the learning of nonnative phonemes depends on the appropriateness of the form of those gestures relative to the target phonetic features. That is, hand gestures seem to be more useful when they appropriately mimic phonetic features., This research was supported by funding from the Spanish Ministry of Science, Innovation and Universities (PGC2018-097007-B-I00) and the Generalitat de Catalunya projects (2017 SGR-971). The third author would like to acknowledge a predoctoral research grant awarded by the Department of Translation and Language Sciences, Universitat Pompeu Fabra.
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Beat gestures for comprehension and recall: differential effects of language learners and native listeners

  • Rohrer, Patrick Louis
  • Delais-Roussarie, Elisabeth
  • Prieto Vives, Pilar, 1965-
Previous work has shown how native listeners benefit from observing iconic gestures during speech comprehension tasks of both degraded and non-degraded speech. By contrast, effects of the use of gestures in non-native listener populations are less clear and studies have mostly involved iconic gestures. The current study aims to complement these findings by testing the potential beneficial effects of beat gestures (non-referential gestures which are often used for information- and discourse marking) on language recall and discourse comprehension using a narrative-drawing task carried out by native and non-native listeners. Using a within-subject design, 51 French intermediate learners of English participated in a narrative-drawing task. Each participant was assigned 8 videos to watch, where a native speaker describes the events of a short comic strip. Videos were presented in random order, in four conditions: in Native listening conditions with frequent, naturally-modeled beat gestures, in Native listening conditions without any gesture, in Non-native listening conditions with frequent, naturally-modeled beat gestures, and in Non-native listening conditions without any gesture. Participants watched each video twice and then immediately recreated the comic strip through their own drawings. Participants’ drawings were then evaluated for discourse comprehension (via their ability to convey the main goals of the narrative through their drawings) and recall (via the number of gesturally-marked elements in the narration that were included in their drawings). Results showed that for native listeners, beat gestures had no significant effect on either recall or comprehension. In non-native speech, however, beat gestures led to significantly lower comprehension and recall scores. These results suggest that frequent, naturally-modeled beat gestures in longer discourses may increase cognitive load for language learners, resulting in negative effects on both memory and language understanding. These findings add to the growing body of literature that suggests that gesture benefits are not a “one-sizefits-all” solution, but rather may be contingent on factors such as language proficiency and gesture rate, particularly in that whenever beat gestures are repeatedly used in discourse, they inherently lose their saliency as markers of important information., The first author would like to acknowledge a joint PhD grant, awarded by the Department of Translation and Language Sciences, Universitat Pompeu Fabra, and SGR AGAUR Grant, Generalitat de Catalunya, Award number: 2017 SGR_971. Spanish Ministry of Science, Innovation and Universities, Grant/Award Number: PGC2018-097007-B-I00. SGR AGAUR Grant, Generalitat de Catalunya, Award number: 2017 SGR_971. The GEHM (Gesture and Head Movements in Language) Research Network, funded by the Independent Research Fund Denmark, Grant/Award number: 9055-00004B.
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

The Encoding of epistemic operations in two Romance languages: the interplay between intonation and discourse markers

  • Prieto Vives, Pilar, 1965-
  • Roseano, Paolo
For years, linguists have noted that intonation patterns and discourse markers encode similar pragmatic meanings across languages. The present study investigates whether a functional compensatory distribution can be documented across languages by focusing on the expression of epistemic commitment in two Romance languages which have been reported to have either a rich or a very reduced inventory of intonational patterns (e.g., Catalan vs. Friulian). A total of 30 speakers (15 per language) participated in an oral Discourse Completion Task designed to elicit assertions with three degrees of speaker commitment. The results showed that while Catalan used specific intonation patterns for the expression of low and intensified commitment statements, Friulian speakers used only one type of pitch contour to express both types of epistemic commitment. In contrast, Friulian speakers made more frequent use of a more varied set of epistemic discourse markers for the two types of biased statements than their Catalan-speaking peers. This result suggests that a trade-off strategy can be observed between intonation and discourse markers across these two languages. This ultimately shows the need to integrate the study of intonational meaning with other parts of the grammar inside a more unified approach in comparative analyses of language., This research has been funded by two research grants awarded by the Spanish Ministry of Science, Innovation and Technologies (PGC2018-097007-B-I00 and FFI2015-64859-P MINECO/FEDER) to the Prosodic Studies Group at UPF and to the Phonetics Laboratory at UB, and by a grant awarded by the Generalitat de Catalunya (2017 SGR _ 971) to the Prosodic Studies Group.
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Gestural and prosodic development act as sister systems and jointly pave the way for children’s sociopragmatic development

  • Hübscher, Iris
  • Prieto Vives, Pilar, 1965-
Children might combine gesture and prosody to express a pragmatic meaning such as a request, information focus, uncertainty or politeness, before they can convey these meanings in speech. However, little is known about the developmental trajectories of gestural and prosodic patterns and how they relate to a child’s growing understanding and propositional use of these sociopragmatic meanings. Do gesture and prosody act as sister systems in pragmatic development? Do children acquire these components of language before they are able to express themselves through spoken language, thus acting as forerunners in children’s pragmatic development? This review article assesses empirical evidence that demonstrates that gesture and prosody act as intimately related systems and, importantly, pave the way for pragmatic acquisition at different developmental stages. The review goes on to explore how the integration of gesture and prosody with semantics and syntax can impact language acquisition and how multimodal interventions can be used effectively in educational settings. Our review findings support the importance of simultaneously assessing both the prosodic and the gestural components of language in the fields of language development, language learning, and language intervention., This article is partly based on, and was inspired by, the results of research funded by the Spanish Ministry of Science and Innovation (Grant No. FFI2015-66533-P “Intonational and gestural meaning in language” and Grant No. PGC2018-097007-B-I00 “Multimodal Language Learning”) and a grant awarded by the Generalitat de Catalunya (Grant No. 2017SGR-925) to the Prosodic Studies Group.
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Singing songs facilitates L2 pronunciation and vocabulary learning: a study with Chinese adolescent ESL learners

  • Zhang, Yuan
  • Baills, Florence
  • Prieto Vives, Pilar, 1965-
The present study assesses the effect of a three-session classroom-based training program involving singing songs with familiar melodies on second-language pronunciation and vocabulary learning. Ninety-five adolescent Chinese ESL learners (M = 14.04 years) were assigned to one of two groups. Participants learned the lyrics in English of three songs whose melodies were familiar to them either by singing or reciting the lyrics, following a native English singer/instructor. Before and after training, participants performed two vocabulary tasks (picture-naming and word meaning recall tasks) and two pronunciation tasks (word and sentence oral-reading tasks). The results revealed that although both groups showed gains in vocabulary and pronunciation after training, the singing group outperformed the speech group. These findings support the value of using songs with familiar melodies to teach second languages at the early stages of learning in an ESL classroom context., This research was supported by the Generalitat de Catalunya [2017 SGR_971]; Ministerio de Ciencia, Innovación y Universidades [PGC2018-097007-B-I00]; Ministerio de Ciencia e Innovación [PID2021-123823NB-I00]. The second author acknowledges funding by the European Union-NextGenerationEU, the Spanish Ministry of Universities and Recovery, Transformation and Resilience Plan, through a call from Pompeu Fabra University (Barcelona).
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Multidimensional labeling of gesture in communication: the M3D proposal

  • Rohrer, Patrick Louis
  • Tütüncübasi, Ulya
  • Florit-Pons, Júlia
  • Vilà-Giménez, Ingrid
  • Esteve-Gibert, Núria
  • Ren-Mitchell, Ada
  • Shattuck-Hufnagel, Stefanie
  • Prieto Vives, Pilar, 1965-
Communication is multimodal in that speakers use not only their voices, but also co-speech gestures to communicate. Recent insights suggest that gestural behavior has a strong association with prosodic structure and that a single gesture can communicate various semantic and pragmatic meanings. This highlights the importance of developing a comprehensive, flexible, and transparent approach to gesture annotation that accounts for multiple dimensions of gesture, including a gesture's form, prosodic properties, and semantic and pragmatic contributions. To address this need for an increasingly dimensionalized approach to multimodal data annotation, the main goal of this paper is to present and describe a novel labeling system for manual gestures. The MultiModal MultiDimensional (M3D) system consists of an open access package that has been developed in coordination with five different labs working on gesture and its interaction with speech. The package includes a set of reliable annotation guidelines, a validated training program, and two annotated audiovisual corpora that represent over 60 minutes of lecture-style speech., This work was partially funded by the Spanish Ministry of Science, Innovation and Universities (Grants numbers PID2021-123823NB-I00, PID2020-115385GA-I00, PGC2018-097007-B-I00), the Pla de Recerca de Catalunya from the Generalitat de Catalunya (Grant numbers 2017 SGR_971 and 2021 SGR 00922) and the GEHM (Gesture and Head Movements in Language) Research Network, funded by the Independent Research Fund Denmark (grant number 9055-00004B).
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Epistemic intonation and epistemic gesture are mutually co-expressive: empirical results from two intonation-gesture matching tasks

  • Borràs Comes, Joan Manel, 1984-
  • Kiagia, Evangelia
  • Prieto Vives, Pilar, 1965-
While several studies have investigated the temporal relationship between co-speech gestures and prosodic structure, little is known about their potential interaction at the level of their encoding of pragmatic meaning. Here we report the results of two complementary intonation-gesture matching tasks which investigate the potential co-dependencies between intonation patterns related to epistemic commitment operators and their associated gestures in Catalan. In Experiment 1, participants were shown audio-muted videos in which a speaker performed gestures conveying epistemic information of certainty or uncertainty while uttering statements and questions. The subjects were then asked to produce a stipulated target word, the goal being to examine whether they would produce the word with a tune that was semantically consistent with the gestures they had seen. In Experiment 2, participants were primed by hearing intonation patterns conveying epistemic information (certainty-uncertainty) and were then asked to select one of two silent videos which seemed to best match the intonation they had heard. The results suggest converging positive effects in both matching tasks and suggest a close interrelation between the pragmatic representations of intonation and gesture that needs to be taken into account when investigating multimodal pragmatic encoding., This research has been funded by a research grant awarded by the Spanish Ministry of Science, Innovation and Universities (PGC2018-097007-B-I00, “Multimodal Language Learning (MLL): Prosodic and Gestural Integration in Pragmatic and Phonological Development”), and by a grant awarded by the Generalitat de Catalunya (2017 SGR 971) to the Prosodic Studies Group. Also awarded by the Generalitat de Catalunya, the first author acknowledges a research group grant to “LLETRA, Linguistic and Literary Education” (2017 SGR 1728).
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Multimodal development in children’s narrative speech: evidence for tight gesture–speech temporal alignment patterns as early as 5 years old

  • Florit-Pons, Júlia
  • Vilà-Giménez, Ingrid
  • Rohrer, Patrick Louis
  • Prieto Vives, Pilar, 1965-
The authors would like to acknowledge the financial support awarded by the Spanish Ministry of Science, Innovation and Universities, Agencia Estatal de Investigación, and Fondo Europeo de Desarrollo Regional (PGC2018-097007-B-100: “Multimodal Language Learning: Prosodic and Gestural Integration in Pragmatic and Phonological Development,” and PID2021-123823NB-I00: “Multimodal Communication: The Integration of Prosody and Gesture in Human Communication and in Language Learning”); by the Generalitat de Catalunya (2017 SGR_971 and 2021 SGR_922); and by the GEstures and Head Movements in language research network, funded by the Independent Research Fund Denmark (9055-00004B). The first author would also like to acknowledge a PhD grant awarded by the Agency for Management of University and Research Grants from the Generalitat de Catalunya, (Award no. 2021 FI_B 00778), and the third author would like to acknowledge a joint PhD grant awarded by the Department of Translation and Language Sciences, Universitat Pompeu Fabra, and SGR AGAUR Grant, Generalitat de Catalunya (Award no. 2017 SGR_971). Also, the second author would like to acknowledge a postdoctoral fellowship funded by European Union-NextGenerationEU, Ministry of Universities and Recovery, Transformation and Resilience Plan, through a call from Universitat Pompeu Fabra (Barcelona).
Repositori Obert UdL
Publicación Capítulo o parte de un libro (BookPart)

The effects of prosodic training with logatomes and prosodic gestures on L2 spontaneous speech

  • Baills, Florence
  • Santiago, Fabián
  • Mairano, Paolo
  • Prieto, Pilar
Training L2 suprasegmental features benefits pronunciation accuracy and comprehensibility, especially through the use of hand gestures. However, studies have mainly looked at the effect of prosodic training in controlled tasks and less is known about spontaneous speech. The present study explores the effect of prosodic training with and without gestures depicting prosody on several dimensions of pronunciation in spontaneous speech. Fifty Catalan learners of French practiced oral reading and sentence-by-sentence imitation with short dialogues during three 30-minute sessions in one of three conditions: speech-only, repeating a selection of sentences from the dialogues; logatome, repeating logatome sequences (series of same consonant-vowel syllable leaving out lexical information) corresponding to the prosody of target sentences; and logatome+gesture, additionally mimicking gestures representing phrasal prosodic patterns. Perceptive evaluations of learners’ spontaneous speech at pre- and posttest revealed that training prosody did not have any significant effect on participants’ comprehensibility, accentedness, and suprasegmental accuracy in spontaneous speech. Acoustic analyses further showed null effects of prosodic training on fluency and the pronunciation of difficult vowel contrasts. Our findings suggest that L2 prosodic training should be better tailored for spontaneous speech. Alternatively, learners may need more time to transfer beneficial effects from controlled tasks to spontaneous speech., This research was supported by funding from the Spanish Ministry of Economy and Competitiveness (PGC2018-097007-B-I00) and the Generalitat de Catalunya (2017 SGR _ 971).
Repositori Obert UdL
Publicación Capítulo o parte de un libro (BookPart)

Training with embodied musical activities has positive effects on unfamiliar language imitation skills

  • Zhang, Yuan
  • Baills, Florence
  • Prieto, Pilar
Research shows that musical expertise benefits second language (L2) phonological learning [1], however little is known on the potential effects of training musical melodic and rhythmic skills on language production skills. This study investigated the role of training musical features such as melody, rhythm, and accent with embodied activities [2] (e.g., without any spoken input) on imitation skills. We hypothesized that embodied musical learning would activate participants' melodic and rhythmic musical skills, which will transfer to their ability to imitate speech. Fifty Chinese adolescents participated in three 45-minute classroom training sessions. They were randomly assigned to one of two conditions: the Embodied Music group followed rhythmic and melodic activities involving body movements; the Non-Embodied Music group followed traditional music classes. Before and after training, participants took part in a sentence-imitation task involving six unfamiliar languages. For each language, imitation skills were evaluated in terms of accentedness by three native speakers. Results showed a significantly higher improvement in the Embodied Music group. These findings demonstrate for the first time the beneficial effects of training students with embodied musical activities on speech imitation skills and add evidence to the relationship between musical rhythmic and melodic features and speech., This research is funded by the PGC2018-097007-B-l00 project awarded by the Ministerio de Ciencia, Innovación y Universidades (MCIU), and Pla de Recerca de Catalunya: 2017 SGR_971, awarded by Agència de Gestió d'Ajuts Universitaris i de Recerca.
Repositori Obert UdL
Publicación Artículo científico (JournalArticle)

Hand Gestures Facilitate the Acquisition of Novel Phonemic Contrasts When They Appropriately Mimic Target Phonetic Features

  • Xi, Xiaotong
  • Li, Peng
  • Baills, Florence
  • Prieto, Pilar
Purpose: Research has shown that observing hand gestures mimicking pitch movements or rhythmic patterns can improve the learning of second language (L2) suprasegmental features. However, less is known about the effects of hand gestures on the learning of novel phonemic contrasts. This study examines (a) whether hand gestures mimicking phonetic features can boost L2 segment learning by naive learners and (b) whether a mismatch between the hand gesture form and the target phonetic feature influences the learning effect. Method: Fifty Catalan native speakers undertook a short multimodal training session on two types of Mandarin Chinese consonants (plosives and affricates) in either of two conditions: Gesture and No Gesture. In the Gesture condition, a fist-to-open-hand gesture was used to mimic air burst, while the No Gesture condition included no such use of gestures. Crucially, while the hand gesture appropriately mimicked the air burst produced in plosives, this was not the case for affricates. Before and after training, participants were tested on two tasks, namely, the identification task and the imitation task. Participants' speech output was rated by five Chinese native speakers. Results: The perception results showed that training with or without gestures yielded similar degrees of improvement for the identification of aspiration contrasts. By contrast, the production results showed that, while training without gestures did not help improve L2 pronunciation, training with gestures improved pronunciation, but only when the given gestures appropriately mimicked the phonetic properties they represented. Conclusions: Results revealed that the efficacy of observing hand gestures on the learning of nonnative phonemes depends on the appropriateness of the form of those gestures relative to the target phonetic features. That is, hand gestures seem to be more useful when they appropriately mimic phonetic features., This research was supported by funding from the Spanish Ministry of Science, Innovation and Universities (PGC2018-097007-B-I00) and the Generalitat de Catalunya projects (2017 SGR-971).
Repositori Obert UdL
Publicación Artículo científico (JournalArticle)

Singing Songs Facilitates L2 Pronunciation and Vocabulary Learning: A Study with Chinese Adolescent ESL Learners

  • Zhang, Yuan
  • Baills, Florence
  • Prieto, Pilar
The present study assesses the effect of a three-session classroom-based training program involving singing songs with familiar melodies on second-language pronunciation and vocabulary learning. Ninety-five adolescent Chinese ESL learners (M = 14.04 years) were assigned to one of two groups. Participants learned the lyrics in English of three songs whose melodies were familiar to them either by singing or reciting the lyrics, following a native English singer/instructor. Before and after training, participants performed two vocabulary tasks (picture-naming and word meaning recall tasks) and two pronunciation tasks (word and sentence oral-reading tasks). The results revealed that although both groups showed gains in vocabulary and pronunciation after training, the singing group outperformed the speech group. These findings support the value of using songs with familiar melodies to teach second languages at the early stages of learning in an ESL classroom context., This research was supported by the Generalitat de Catalunya [2017 SGR_971]; Ministerio de Ciencia, Innovación y Universidades [PGC2018-097007-B-I00]; Ministerio de Ciencia e Innovación [PID2021-123823NB-I00].
Repositori Obert UdL
Publicación Capítulo o parte de un libro (BookPart)

Melodic perception skills predict Catalan speakers’ imitation abilities of unfamiliar languages

  • Li, Peng
  • Zhang, Yuan
  • Fu, Xianqiang
  • Baills, Florence
  • Prieto, Pilar
Musical perception skills have been shown to influence second language speech production. Likewise, working memory may also affect nonnative speech production abilities. However, very few studies have assessed their respective role in speech imitation abilities. The present study thus investigates the predictive role of musical perception skills and working memory on speech imitation abilities of unfamiliar languages. Sixty-one adult Catalan speakers imitated twelve sentences in six languages that were unfamiliar to them. Participants’ music perception skills were tested by four PROMS-S subsets, namely accent, melody, pitch, and rhythm, and their working memory was measured by a forward digit span test. A linear regression analysis revealed that melodic perception skills were the unique predictor among all four musical perception subtests and that working memory was not a significant predictor. Our findings show that melodic perception skills are key in predicting the capability in imitating unfamiliar speech and thus may be important for learning foreign language pronunciation., This study is funded by the PGC2018-097007-B-l00 project awarded by the Ministerio de Ciencia, Innovación y Universidades (MCIU), Agencia Estatal de Investigación (AEI) and Fondo Europeo de Desarrollo Regional (FEDER), Pla de Recerca de Catalunya: 2017 SGR_971, awarded by Agència de Gestió d'Ajuts Universitaris i de Recerca (AGAUR).
Repositori Obert UdL
Publicación Artículo científico (JournalArticle)

The effectiveness of embodied prosodic training in L2 accentedness and vowel accuracy

  • Li, Peng
  • Baills, Florence
  • Baqué, L.
  • Prieto, Pilar
This study explores the effects of embodied prosodic training on the production of non-native French front rounded vowels (i.e. /y, ø, œ/) and the overall pronunciation proficiency. Fifty-seven Catalan learners of French practiced pronunciation in one of two conditions: one group observed hand gestures embodying prosodic features of the sentences they were listening to, while the other group did not see any such gestures. The learning outcome was assessed in a pretest, posttest, and delayed posttest through a dialogue-reading task and a sentence imitation task in terms of accentedness, comprehensibility and fluency scores, and through formant analysis of participant-produced target vowels. The results showed that compared to non-embodied training, embodied prosodic training yielded continuous improvement in accentedness in both tasks and improved the F2 values of French front rounded vowels (more fronted). As for comprehensibility and fluency scores, both groups showed similar levels of significant improvement. This study highlights the interaction between prosodic and segmental features of speech by showing that training with embodied prosodic features benefitted accentedness and the production accuracy of non-native vowels., The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was funded by the Ministerio de Ciencia, Innovación y Universidades, Agencia Estatal de Investigación and European Regional Development Fund under Multimodal language learning (MLL): Prosodic and Gestural Integration in Pragmatic and Phonological Development [PGC2018-097007-B-I00]. This work is also supported by the Agència de Gestió d'Ajuts Universitaris i de Recerca (AGAUR) under Pla de Recerca de Catalunya [2017 SGR_971], and the Ministerio de Economía y Competitividad under Cognipros project [FFI2013-40419-P]. The second author acknowledges a grant from the Department of Translation and Language Sciences at Universitat Pompeu Fabra.
Repositori Obert UdL
Publicación Capítulo o parte de un libro (BookPart)

Gesture shape and gesture–speech alignment predict simultaneous L2 sound production accuracy

  • Li, Peng
  • Baills, Florence
  • Xi, Xiaotong
  • Prieto, Pilar
Previous research suggests hand gestures aid L2 pronunciation learning, but the underlying mechanisms remain unclear. This study examines if accurate gesture imitation during phonetic training predicts the accuracy of non-native sounds produced while doing a pedagogical hand gesture. Twenty-nine Catalan speakers, unfamiliar with Chinese, learned to pronounce Chinese aspirated plosives while mimicking gestures representing the aspirated sounds. Results revealed that more precise gesture shape and better gesture–speech alignment predicted more accurate speech imitation, emphasizing the significance of the spatiotemporal gesture–speech coupling in embodied L2 phonetic training., This study is funded by “Multimodal Communication: The integration of prosody and gesture in human communication and in language learning” [PID2021-123823NB-I00] awarded by the Ministerio de Ciencia e Innovación and “Multimodal language learning: Prosodic and Gestural Integration in Pragmatic and Phonological Development” [PGC2018- 097007-B-l00], awarded by the Ministerio de Ciencia, Innovación y Universidades, Agencia Estatal de Investigación, and Fondo Europeo de Desarrollo Regional. PL is supported by the Research Council of Norway through its Centres of Excellence funding scheme [223265]. FB acknowledges a Margarita Salas grant funded by European Union– NextGenerationEU, Ministry of Universities and Recovery, Transformation, and Resilience Plan, through a call from Pompeu Fabra University. XX is supported by the Secretaria d’Universitats i Recerca de la Generalitat de Catalunya and the European Social Fund under the Grant for the recruitment of early-stage research staff [2021FI_B 00137].
Repositori Obert UdL
Publicación Artículo científico (JournalArticle)

Musical perception skills predict speech imitation skills: differences between speakers of tone and intonation languages

  • Li, Peng
  • Zhang, Yuan
  • Baills, Florence
  • Prieto, Pilar
The ability to imitate speech is linked to individual cognitive abilities such as working memory and the auditory processing of music. However, little research has focused on the role of specific components of musical perception aptitude in relation to an individual’s native language from a crosslinguistic perspective. This study explores the predictive role of four components of musical perception skills and working memory on phonetic language abilities for speakers of two typologically different languages, Catalan (an intonation language) and Chinese (a tone language). Sixty-one Catalan and 144 Chinese participants completed four subtests (accent, melody, pitch and rhythm) of the Profile of Music Perception Skills, a forward digit span task and a speech imitation task. The results showed that for both groups of participants, musical perception skills predicted speech imitation accuracy but working memory did not. Importantly, among the components of musical perception skills, accent was the only predictive factor for Chinese speakers, whereas melody was the only predictive factor for Catalan speakers. These findings suggest that speech imitation ability is predicted by musical perception skills rather than working memory and that the predictive role of specific musical components may depend on the phonological properties of the native language., This study is funded by ‘Multimodal Communication: The integration of prosody and gesture in human communication and in language learning’ (PID2021-123823NB-I00) awarded by the Ministerio de Ciencia e Innovación and ‘Multimodal language learning: Prosodic and Gestural Integration in Pragmatic and Phonological Development’ (PGC2018-097007-B-l00), awarded by the Ministerio de Ciencia, Innovación y Universidades, Agencia Estatal de Investigación, and Fondo Europeo de Desarrollo Regional. P.L. is supported by the Research Council of Norway through its Centres of Excellence funding scheme (223265). F.B. acknowledges a Margarita Salas grant funded by the European Union-NextGenerationEU, Ministry of Universities and Recovery, Transformation and Resilience Plan, through a call from Pompeu Fabra University.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Singing Songs Facilitates L2 Pronunciation and Vocabulary Learning: A Study with Chinese Adolescent ESL Learners

  • Zhang, Yuan
  • Baills, Florence
  • Prieto, Pilar
The present study assesses the effect of a three-session classroom-based training program involving singing songs with familiar melodies on second-language pronunciation and vocabulary learning. Ninety-five adolescent Chinese ESL learners (M = 14.04 years) were assigned to one of two groups. Participants learned the lyrics in English of three songs whose melodies were familiar to them either by singing or reciting the lyrics, following a native English singer/instructor. Before and after training, participants performed two vocabulary tasks (picture-naming and word meaning recall tasks) and two pronunciation tasks (word and sentence oral-reading tasks). The results revealed that although both groups showed gains in vocabulary and pronunciation after training, the singing group outperformed the speech group. These findings support the value of using songs with familiar melodies to teach second languages at the early stages of learning in an ESL classroom context., This research was supported by the Generalitat de Catalunya [2017 SGR_971]; Ministerio de Ciencia, Innovación y Universidades [PGC2018-097007-B-I00]; Ministerio de Ciencia e Innovación [PID2021-123823NB-I00].
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Multimodal development in children’s narrative speech: evidence for tight gesture–speech temporal alignment patterns as early as 5 years old

  • Florit-Pons, Júlia
  • Vilà-Giménez, Ingrid
  • Rohrer, Patrick Louis
  • Prieto Vives, Pilar, 1965-
The authors would like to acknowledge the financial support awarded by the Spanish Ministry of Science, Innovation and Universities, Agencia Estatal de Investigación, and Fondo Europeo de Desarrollo Regional (PGC2018-097007-B-100: “Multimodal Language Learning: Prosodic and Gestural Integration in Pragmatic and Phonological Development,” and PID2021-123823NB-I00: “Multimodal Communication: The Integration of Prosody and Gesture in Human Communication and in Language Learning”); by the Generalitat de Catalunya (2017 SGR_971 and 2021 SGR_922); and by the GEstures and Head Movements in language research network, funded by the Independent Research Fund Denmark (9055-00004B). The first author would also like to acknowledge a PhD grant awarded by the Agency for Management of University and Research Grants from the Generalitat de Catalunya, (Award no. 2021 FI_B 00778), and the third author would like to acknowledge a joint PhD grant awarded by the Department of Translation and Language Sciences, Universitat Pompeu Fabra, and SGR AGAUR Grant, Generalitat de Catalunya (Award no. 2017 SGR_971). Also, the second author would like to acknowledge a postdoctoral fellowship funded by European Union-NextGenerationEU, Ministry of Universities and Recovery, Transformation and Resilience Plan, through a call from Universitat Pompeu Fabra (Barcelona).
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Improving second language vowel production with hand gestures encoding visible articulation: evidence from picture‐naming and paragraph‐reading tasks

  • Xi, Xiaotong
  • Li, Peng
  • Prieto Vives, Pilar, 1965-
Includes supplementary materials for the online appendix., This study investigates whether audiovisual phonetic training with hand gestures encoding visible or nonvisible articulation features has a differential impact on learning second language sounds. Ninety-nine Catalan–Spanish bilingual students were trained to differentiate English /æ/ and /ʌ/, which differ in the visible lip aperture and nonvisible tongue position, with training involving no gestures, gestures representing the lip aperture, or gestures representing the tongue position. Before, immediately after, and 1 week after the training, participants’ perception of the targets was assessed through a word-identification task, and their production was tested through paragraph-reading, picture-naming, and word-imitation tasks. Although all participants improved in perception and production, the lip hand gesture was more effective in adjusting lip aperture than the other two conditions in the paragraph-reading and picture-naming tasks. These results suggest that hand gestures encoding visible rather than nonvisible articulation features are more effective for improving second language pronunciation., The study was supported by the Ministerio de Ciencia, Innovación y Universidades, Agencia Estatal de Investigación, and Fondo Europeo de Desarrollo Regional [PGC2018-097007-B-l00], and by the Ministerio de Ciencia e Innovación [PID2021-123823NB-I00]. The first author is supported by the Department d'Empresa i Coneixement de la Generalitat de Catalunya and the European Social Fund under a grant for the recruitment of early-stage research staff [2021FI_B 00137]. The second author is supported by the Research Council of Norway through its Centres of Excellence funding scheme [223265].
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

The Encoding of epistemic operations in two Romance languages: the interplay between intonation and discourse markers

  • Prieto Vives, Pilar, 1965-
  • Roseano, Paolo
For years, linguists have noted that intonation patterns and discourse markers encode similar pragmatic meanings across languages. The present study investigates whether a functional compensatory distribution can be documented across languages by focusing on the expression of epistemic commitment in two Romance languages which have been reported to have either a rich or a very reduced inventory of intonational patterns (e.g., Catalan vs. Friulian). A total of 30 speakers (15 per language) participated in an oral Discourse Completion Task designed to elicit assertions with three degrees of speaker commitment. The results showed that while Catalan used specific intonation patterns for the expression of low and intensified commitment statements, Friulian speakers used only one type of pitch contour to express both types of epistemic commitment. In contrast, Friulian speakers made more frequent use of a more varied set of epistemic discourse markers for the two types of biased statements than their Catalan-speaking peers. This result suggests that a trade-off strategy can be observed between intonation and discourse markers across these two languages. This ultimately shows the need to integrate the study of intonational meaning with other parts of the grammar inside a more unified approach in comparative analyses of language., This research has been funded by two research grants awarded by the Spanish Ministry of Science, Innovation and Technologies (PGC2018-097007-B-I00 and FFI2015-64859-P MINECO/FEDER) to the Prosodic Studies Group at UPF and to the Phonetics Laboratory at UB, and by a grant awarded by the Generalitat de Catalunya (2017 SGR _ 971) to the Prosodic Studies Group.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Beat gestures for comprehension and recall: differential effects of language learners and native listeners

  • Rohrer, Patrick Louis
  • Delais-Roussarie, Elisabeth
  • Prieto Vives, Pilar, 1965-
Previous work has shown how native listeners benefit from observing iconic gestures during speech comprehension tasks of both degraded and non-degraded speech. By contrast, effects of the use of gestures in non-native listener populations are less clear and studies have mostly involved iconic gestures. The current study aims to complement these findings by testing the potential beneficial effects of beat gestures (non-referential gestures which are often used for information- and discourse marking) on language recall and discourse comprehension using a narrative-drawing task carried out by native and non-native listeners. Using a within-subject design, 51 French intermediate learners of English participated in a narrative-drawing task. Each participant was assigned 8 videos to watch, where a native speaker describes the events of a short comic strip. Videos were presented in random order, in four conditions: in Native listening conditions with frequent, naturally-modeled beat gestures, in Native listening conditions without any gesture, in Non-native listening conditions with frequent, naturally-modeled beat gestures, and in Non-native listening conditions without any gesture. Participants watched each video twice and then immediately recreated the comic strip through their own drawings. Participants’ drawings were then evaluated for discourse comprehension (via their ability to convey the main goals of the narrative through their drawings) and recall (via the number of gesturally-marked elements in the narration that were included in their drawings). Results showed that for native listeners, beat gestures had no significant effect on either recall or comprehension. In non-native speech, however, beat gestures led to significantly lower comprehension and recall scores. These results suggest that frequent, naturally-modeled beat gestures in longer discourses may increase cognitive load for language learners, resulting in negative effects on both memory and language understanding. These findings add to the growing body of literature that suggests that gesture benefits are not a “one-sizefits-all” solution, but rather may be contingent on factors such as language proficiency and gesture rate, particularly in that whenever beat gestures are repeatedly used in discourse, they inherently lose their saliency as markers of important information., The first author would like to acknowledge a joint PhD grant, awarded by the Department of Translation and Language Sciences, Universitat Pompeu Fabra, and SGR AGAUR Grant, Generalitat de Catalunya, Award number: 2017 SGR_971. Spanish Ministry of Science, Innovation and Universities, Grant/Award Number: PGC2018-097007-B-I00. SGR AGAUR Grant, Generalitat de Catalunya, Award number: 2017 SGR_971. The GEHM (Gesture and Head Movements in Language) Research Network, funded by the Independent Research Fund Denmark, Grant/Award number: 9055-00004B.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Three-year-olds infer polite stance from intonation and facial cues

  • Prieto Vives, Pilar, 1965-
  • Hübscher, Iris
  • Wagner, Laura
Despite the evidence that infants are sensitive to facial cues and prosody for the detection of emotion, we have contradictory evidence regarding the use of these cues by older preschool and school children when inferring both emotional and politeness stance. This study assessed preschool aged children’s sensitivity to intonational and facial cues signalling a speaker’s polite stance in requestive speech acts with controlled lexical and contextual materials. Thirty-six 3-yearold American English-speaking children performed a forced-choice decision task which investigated whether children at this age use pitch and/or facial cues to infer a speaker’s affective stance in either audio-only, visual-only or audio-visual presentation modalities, when lexical cues are controlled for. Results showed that (a) children at three years can infer a speaker’s polite stance equally well in all three conditions (audio-only, visual-only and audio-visual) and thereby (b) unlike previous research, in the present task both intonation and facial cues are equally strong cues in children’s understanding of a speaker’s polite stance in requestive speech acts. The authors discuss especially the implications of this early use of intonation to detect politeness, relating it to other previous research on children’s ability to infer meaning from pitch.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

The effectiveness of embodied prosodic training in L2 accentedness and vowel accuracy

  • Li, Peng
  • Baills, Florence
  • Baqué, Lorraine
  • Prieto Vives, Pilar, 1965-
This study explores the effects of embodied prosodic training on the production of non-native French front rounded vowels (i.e. /y, ø, œ/) and the overall pronunciation proficiency. Fifty-seven Catalan learners of French practiced pronunciation in one of two conditions: one group observed hand gestures embodying prosodic features of the sentences they were listening to, while the other group did not see any such gestures. The learning outcome was assessed in a pretest, posttest, and delayed posttest through a dialogue-reading task and a sentence imitation task in terms of accentedness, comprehensibility and fluency scores, and through formant analysis of participant-produced target vowels. The results showed that compared to non-embodied training, embodied prosodic training yielded continuous improvement in accentedness in both tasks and improved the F2 values of French front rounded vowels (more fronted). As for comprehensibility and fluency scores, both groups showed similar levels of significant improvement. This study highlights the interaction between prosodic and segmental features of speech by showing that training with embodied prosodic features benefitted accentedness and the production accuracy of non-native vowels., This work was funded by the Ministerio de Ciencia, Innovación y Universidades, Agencia Estatal de Investigación and European Regional Development Fund under Multimodal language learning (MLL): Prosodic and Gestural Integration in Pragmatic and Phonological Development [PGC2018-097007-B-I00]. This work is also supported by the Agència de Gestió d'Ajuts Universitaris i de Recerca (AGAUR) under Pla de Recerca de Catalunya [2017 SGR_971], and the Ministerio de Economía y Competitividad under Cognipros project [FFI2013-40419-P]. The second author acknowledges a grant from the Department of Translation and Language Sciences at Universitat Pompeu Fabra.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Musical perception skills predict speech imitation skills: differences between speakers of tone and intonation languages

  • Li, Peng
  • Zhang, Yuan
  • Baills, Florence
  • Prieto, Pilar
The ability to imitate speech is linked to individual cognitive abilities such as working memory and the auditory processing of music. However, little research has focused on the role of specific components of musical perception aptitude in relation to an individual’s native language from a crosslinguistic perspective. This study explores the predictive role of four components of musical perception skills and working memory on phonetic language abilities for speakers of two typologically different languages, Catalan (an intonation language) and Chinese (a tone language). Sixty-one Catalan and 144 Chinese participants completed four subtests (accent, melody, pitch and rhythm) of the Profile of Music Perception Skills, a forward digit span task and a speech imitation task. The results showed that for both groups of participants, musical perception skills predicted speech imitation accuracy but working memory did not. Importantly, among the components of musical perception skills, accent was the only predictive factor for Chinese speakers, whereas melody was the only predictive factor for Catalan speakers. These findings suggest that speech imitation ability is predicted by musical perception skills rather than working memory and that the predictive role of specific musical components may depend on the phonological properties of the native language., This study is funded by ‘Multimodal Communication: The integration of prosody and gesture in human communication and in language learning’ (PID2021-123823NB-I00) awarded by the Ministerio de Ciencia e Innovación and ‘Multimodal language learning: Prosodic and Gestural Integration in Pragmatic and Phonological Development’ (PGC2018-097007-B-l00), awarded by the Ministerio de Ciencia, Innovación y Universidades, Agencia Estatal de Investigación, and Fondo Europeo de Desarrollo Regional. P.L. is supported by the Research Council of Norway through its Centres of Excellence funding scheme (223265). F.B. acknowledges a Margarita Salas grant funded by the European Union-NextGenerationEU, Ministry of Universities and Recovery, Transformation and Resilience Plan, through a call from Pompeu Fabra University.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Singing songs facilitates L2 pronunciation and vocabulary learning: a study with Chinese adolescent ESL learners

  • Zhang, Yuan
  • Baills, Florence
  • Prieto Vives, Pilar, 1965-
The present study assesses the effect of a three-session classroom-based training program involving singing songs with familiar melodies on second-language pronunciation and vocabulary learning. Ninety-five adolescent Chinese ESL learners (M = 14.04 years) were assigned to one of two groups. Participants learned the lyrics in English of three songs whose melodies were familiar to them either by singing or reciting the lyrics, following a native English singer/instructor. Before and after training, participants performed two vocabulary tasks (picture-naming and word meaning recall tasks) and two pronunciation tasks (word and sentence oral-reading tasks). The results revealed that although both groups showed gains in vocabulary and pronunciation after training, the singing group outperformed the speech group. These findings support the value of using songs with familiar melodies to teach second languages at the early stages of learning in an ESL classroom context., This research was supported by the Generalitat de Catalunya [2017 SGR_971]; Ministerio de Ciencia, Innovación y Universidades [PGC2018-097007-B-I00]; Ministerio de Ciencia e Innovación [PID2021-123823NB-I00]. The second author acknowledges funding by the European Union-NextGenerationEU, the Spanish Ministry of Universities and Recovery, Transformation and Resilience Plan, through a call from Pompeu Fabra University (Barcelona).
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Unguided virtual-reality training can enhance the oral presentation skills of high-school students

  • Valls-Ratés, Ïo
  • Niebuhr, Oliver
  • Prieto Vives, Pilar, 1965-
Public speaking is fundamental in our daily life, and it happens to be challenging for many people. Like all aspects of language, these skills should be encouraged early on in educational settings. However, the high number of students per class and the extensive curriculum both limit the possibilities of the training and, moreover, entail that students give short in-class presentations under great time pressure. Virtual Reality (VR) environments can help speakers and teachers meet these challenges and foster oral skills. This experimental study employs a between-subjects pre- and post-training design with four Catalan high-school student groups, a VR group (N = 30) and a Non-VR group (N = 20). Both groups gave a 2-min speech in front of a live audience before (pre-training) and after (post-training) 3 training sessions (one session per week) in which they practiced public speaking either in front of a VR audience or alone in a classroom (Non-VR). Students assessed their anxiety measures right before performing every speech and filled out a satisfaction questionnaire at the end. Pre- and post-training speeches were assessed by 15 raters, who analyzed the persuasiveness of the message and the charisma of the presenter. Speeches were also analyzed for prosodic features and gesture rate. First, results showed that self-assessed anxiety got significantly reduced at post-training for both conditions. Second, acoustic analyses of both groups' speeches show that the VR group has, unlike the Non-VR group, developed a more clear and resonant voice quality in the post-training speeches, in terms of higher cepstral-peak prominence (CPP) (although no significant differences in f0- related parameters as a function of training were obtained), as well as significantly less erosion effects than the Non-VR group. However, these differences across groups did not trigger a direct improvement on the participants' gesture rate, persuasiveness and charisma at post-training. Furthermore, students perceived the training in the VR environment to be more useful and beneficial for their future oral presentations than the Non-VR group. All in all, short unguided VR training sessions can help students feel less anxious, promote a more clear and resonant voice style, and can prevent them from experiencing an erosion effect while practicing speeches in front of a real audience., This work benefited from funding awarded by the Spanish Ministry of Economy and Competitiveness (PGC2018-097007-B-I00, PID2021-123823NB-I00) and the Generalitat de Catalunya (2017 SGR_971). We also acknowledge the support from the Recercaixa Project (RecerCaixa 2017ACUP 00249) and the Department of Translation at Universitat Pompeu Fabra through a 1-year doctoral grant to ÏV-R.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

The effectiveness of embodied prosodic training in L2 accentedness and vowel accuracy

  • Li, Peng
  • Baills, Florence
  • Baqué, L.
  • Prieto, Pilar
This study explores the effects of embodied prosodic training on the production of non-native French front rounded vowels (i.e. /y, ø, œ/) and the overall pronunciation proficiency. Fifty-seven Catalan learners of French practiced pronunciation in one of two conditions: one group observed hand gestures embodying prosodic features of the sentences they were listening to, while the other group did not see any such gestures. The learning outcome was assessed in a pretest, posttest, and delayed posttest through a dialogue-reading task and a sentence imitation task in terms of accentedness, comprehensibility and fluency scores, and through formant analysis of participant-produced target vowels. The results showed that compared to non-embodied training, embodied prosodic training yielded continuous improvement in accentedness in both tasks and improved the F2 values of French front rounded vowels (more fronted). As for comprehensibility and fluency scores, both groups showed similar levels of significant improvement. This study highlights the interaction between prosodic and segmental features of speech by showing that training with embodied prosodic features benefitted accentedness and the production accuracy of non-native vowels., The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was funded by the Ministerio de Ciencia, Innovación y Universidades, Agencia Estatal de Investigación and European Regional Development Fund under Multimodal language learning (MLL): Prosodic and Gestural Integration in Pragmatic and Phonological Development [PGC2018-097007-B-I00]. This work is also supported by the Agència de Gestió d'Ajuts Universitaris i de Recerca (AGAUR) under Pla de Recerca de Catalunya [2017 SGR_971], and the Ministerio de Economía y Competitividad under Cognipros project [FFI2013-40419-P]. The second author acknowledges a grant from the Department of Translation and Language Sciences at Universitat Pompeu Fabra.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Visualizing prosodic structure: manual gestures as highlighters of prosodic heads and edges in English academic discourses

  • Rohrer, Patrick Louis
  • Delais-Roussarie, Elisabeth
  • Prieto Vives, Pilar, 1965-
Research has shown a close temporal relationship between prominence-lending tonal movements in speech and prominence in manual gesture. However, prosodic structure consists of not only prosodic heads (i.e., pitch accentuation) but also prosodic edges. To our knowledge, no previous studies have assessed the value of prosodic edges (nuclear vs. phrase-initial prenuclear pitch accents) as anchoring sites for different types of gestures (i.e., referential vs. non-referential) while independently controlling for the relative degree of prominence associated with the pitch accent. The English M3D-TED corpus, which contains over 23 minutes of multimodal speech, was analyzed in terms of prosody and gesture. Results showed that while the majority of manual gesture strokes overlapped a pitch accented syllable (85.99%), apex alignment occurred at a relatively low rate (50.4%) and alignment rates did not significantly differ between referential and non-referential gestures. At the phrasal level, crucially our results also showed that strokes align with phrase-initial prenuclear pitch accents over nuclear accents, and this relationship is not driven by relative prominence. These findings show that both prosodic heads and prosodic edges (i.e., phrase initial and final positions) are key sites for both referential and non-referential gesture production., The authors would like to acknowledge the financial support awarded by the Spanish Ministry of Science, Innovation and Universities (MCIU), Agencia Estatal de Investigación (AEI), and Fondo Europeo de Desarrollo Regional (FEDER) [grant numbers PGC2018-097007-B-I00 'Multimodal Language Learning (MLL): Prosodic and Gestural Integration in Pragmatic and Phonological Development'; PID2021-123823NB-I00 'Multimodal Communication (MCOM): The integration of prosody and gesture in human communication and in language learning'], by the Generalitat de Catalunya [grant number 2017 SGR_971], and by the GEHM (Gesture and Head Movements in Language) Research Network, funded by the Independent Research Fund Denmark [grant number 9055-00004B]. The first author would like to acknowledge a joint Ph.D. grant, awarded by the Department of Translation and Language Sciences, Universitat Pompeu Fabra, and SGR Grant, Generalitat de Catalunya, [grant number 2017 SGR_971]. We would like to express our gratitude to Ulya Tütücünbasi for her help with the gestural annotation of the corpus. Many thanks are also extended to Dr. Stefanie Shattuck-Hufnagel and Ada Ren (Massachusetts Institute of Technology), as well as the members of the GrEP research lab for their feedback provided on early versions of this study. We would also like to thank Steve Dunne for his help with proofreading the final manuscript.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Children use non-referential gestures in narrative speech to mark discourse elements which update common ground

  • Rohrer, Patrick Louis
  • Florit-Pons, Júlia
  • Vilà-Giménez, Ingrid
  • Prieto Vives, Pilar, 1965-
While recent studies have claimed that non-referential gestures (i.e., gestures that do not visually represent any semantic content in speech) are used to mark discourse-new and/or -accessible referents and focused information in adult speech, to our knowledge, no prior investigation has studied the relationship between information structure (IS) and gesture referentiality in children’s narrative speech from a developmental perspective. A longitudinal database consisting of 332 narratives performed by 83 children at two different time points in development was coded for IS and gesture referentiality (i.e., referential and non-referential gestures). Results revealed that at both time points, both referential and non-referential gestures were produced more with information that moves discourse forward (i.e., focus) and predication (i.e., comment) rather than topical or background information. Further, at 7–9 years of age, children tended to use more nonreferential gestures to mark focus and comment constituents than referential gestures. In terms of the marking of the newness of discourse referents, non-referential gestures already seem to play a key role at 5–6 years old, whereas referential gestures did not show any patterns. This relationship was even stronger at 7–9 years old. All in all, our findings offer supporting evidence that in contrast with referential gestures, nonreferential gestures have been found to play a key role in marking IS, and that the development of this relationship solidifies at a period in development that coincides with a spurt in non-referential gesture production., PLR would like to acknowledge a joint Ph.D. grant, awarded by the Department of Translation and Language Sciences, Universitat Pompeu Fabra, and SGR AGAUR Grant, Generalitat de Catalunya, Award number: 2017 SGR_971, while JF-P would like to acknowledge a Ph.D. grant, awarded by the Generalitat de Catalunya, Award number: 2021 FI_B 00778. We would also like to acknowledge the financial support awarded by the Spanish Ministry of Science, Innovation and Universities (MCIU), Agencia Estatal de Investigación (AEI), and Fondo Europeo de Desarrollo Regional (FEDER) [PGC2018- 097007-B-100 “Multimodal Language Learning (MLL): Prosodic and Gestural Integration in Pragmatic and Phonological Development”], by the Spanish Ministry of Economy and Competitiveness (MINECO) (FFI2015-66533-P “Intonational and gestural meaning in language”), by the Generalitat de Catalunya (2014 SGR_925, 2017 SGR_971), and by the GEHM (Gesture and Head Movements in Language) Research Network, funded by the Independent Research Fund Denmark (9055- 00004B)
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

The predictive value of non-referential beat gestures: early use in parent–child interactions predicts narrative abilities at 5 years of age

  • Vilà-Giménez, Ingrid
  • Dowling, Natalie
  • Demir-Lira, Ö Ece
  • Prieto Vives, Pilar, 1965-
  • Goldin-Meadow, Susan
A longitudinal study with 45 children (Hispanic, 13%; non-Hispanic, 87%) investigated whether the early production of non-referential beat and flip gestures, as opposed to referential iconic gestures, in parent-child naturalistic interactions from 14 to 58 months old predicts narrative abilities at age 5. Results revealed that only non-referential beats significantly (p < .01) predicted later narrative productions. The pragmatic functions of the children’s speech that accompany these gestures were also analyzed in a representative sample of 18 parent-child dyads, revealing that beats were typically associated with biased assertions or questions. These findings show that the early use of beats predicts narrative abilities later in development, and suggest that this relation is likely due to the pragmatic– structuring function that beats reflect in early discourse., Funding Information: - Spanish Ministry of Science, Innovation and Universities (MCIU), Agencia Estatal de Investigación (AEI), and Fondo Europeo de Desarrollo Regional (FEDER): PGC2018-097007-B-100, “Multimodal Language Learning (MLL): Prosodic and Gestural Integration in Pragmatic and Phonological Development.” - Generalitat de Catalunya: 2017 SGR_971 - Generalitat de Catalunya, Agència de Gestió d’Ajuts Universitaris i de Recerca: 2019FI_B2_00125 - Eunice Kennedy Shriver National Institute of Child Health & Human Development of the National Institutes of Health: P01HD040605.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Multidimensional labeling of gesture in communication: the M3D proposal

  • Rohrer, Patrick Louis
  • Tütüncübasi, Ulya
  • Florit-Pons, Júlia
  • Vilà-Giménez, Ingrid
  • Esteve-Gibert, Núria
  • Ren-Mitchell, Ada
  • Shattuck-Hufnagel, Stefanie
  • Prieto Vives, Pilar, 1965-
Communication is multimodal in that speakers use not only their voices, but also co-speech gestures to communicate. Recent insights suggest that gestural behavior has a strong association with prosodic structure and that a single gesture can communicate various semantic and pragmatic meanings. This highlights the importance of developing a comprehensive, flexible, and transparent approach to gesture annotation that accounts for multiple dimensions of gesture, including a gesture's form, prosodic properties, and semantic and pragmatic contributions. To address this need for an increasingly dimensionalized approach to multimodal data annotation, the main goal of this paper is to present and describe a novel labeling system for manual gestures. The MultiModal MultiDimensional (M3D) system consists of an open access package that has been developed in coordination with five different labs working on gesture and its interaction with speech. The package includes a set of reliable annotation guidelines, a validated training program, and two annotated audiovisual corpora that represent over 60 minutes of lecture-style speech., This work was partially funded by the Spanish Ministry of Science, Innovation and Universities (Grants numbers PID2021-123823NB-I00, PID2020-115385GA-I00, PGC2018-097007-B-I00), the Pla de Recerca de Catalunya from the Generalitat de Catalunya (Grant numbers 2017 SGR_971 and 2021 SGR 00922) and the GEHM (Gesture and Head Movements in Language) Research Network, funded by the Independent Research Fund Denmark (grant number 9055-00004B).
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Gestural and prosodic development act as sister systems and jointly pave the way for children’s sociopragmatic development

  • Hübscher, Iris
  • Prieto Vives, Pilar, 1965-
Children might combine gesture and prosody to express a pragmatic meaning such as a request, information focus, uncertainty or politeness, before they can convey these meanings in speech. However, little is known about the developmental trajectories of gestural and prosodic patterns and how they relate to a child’s growing understanding and propositional use of these sociopragmatic meanings. Do gesture and prosody act as sister systems in pragmatic development? Do children acquire these components of language before they are able to express themselves through spoken language, thus acting as forerunners in children’s pragmatic development? This review article assesses empirical evidence that demonstrates that gesture and prosody act as intimately related systems and, importantly, pave the way for pragmatic acquisition at different developmental stages. The review goes on to explore how the integration of gesture and prosody with semantics and syntax can impact language acquisition and how multimodal interventions can be used effectively in educational settings. Our review findings support the importance of simultaneously assessing both the prosodic and the gestural components of language in the fields of language development, language learning, and language intervention., This article is partly based on, and was inspired by, the results of research funded by the Spanish Ministry of Science and Innovation (Grant No. FFI2015-66533-P “Intonational and gestural meaning in language” and Grant No. PGC2018-097007-B-I00 “Multimodal Language Learning”) and a grant awarded by the Generalitat de Catalunya (Grant No. 2017SGR-925) to the Prosodic Studies Group.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Hand gestures facilitate the acquisition of novel phonemic contrasts when they appropriately mimic target phonetic features

  • Xi, Xiaotong
  • Li, Peng
  • Baills, Florence
  • Prieto Vives, Pilar, 1965-
Purpose: Research has shown that observing hand gestures mimicking pitch movements or rhythmic patterns can improve the learning of second language (L2) suprasegmental features. However, less is known about the effects of hand gestures on the learning of novel phonemic contrasts. This study examines (a) whether hand gestures mimicking phonetic features can boost L2 segment learning by naive learners and (b) whether a mismatch between the hand gesture form and the target phonetic feature influences the learning effect. Method: Fifty Catalan native speakers undertook a short multimodal training session on two types of Mandarin Chinese consonants (plosives and affricates) in either of two conditions: Gesture and No Gesture. In the Gesture condition, a fist-to-open-hand gesture was used to mimic air burst, while the No Gesture condition included no such use of gestures. Crucially, while the hand gesture appropriately mimicked the air burst produced in plosives, this was not the case for affricates. Before and after training, participants were tested on two tasks, namely, the identification task and the imitation task. Participants' speech output was rated by five Chinese native speakers. Results: The perception results showed that training with or without gestures yielded similar degrees of improvement for the identification of aspiration contrasts. By contrast, the production results showed that, while training without gestures did not help improve L2 pronunciation, training with gestures improved pronunciation, but only when the given gestures appropriately mimicked the phonetic properties they represented. Conclusions: Results revealed that the efficacy of observing hand gestures on the learning of nonnative phonemes depends on the appropriateness of the form of those gestures relative to the target phonetic features. That is, hand gestures seem to be more useful when they appropriately mimic phonetic features., This research was supported by funding from the Spanish Ministry of Science, Innovation and Universities (PGC2018-097007-B-I00) and the Generalitat de Catalunya projects (2017 SGR-971). The third author would like to acknowledge a predoctoral research grant awarded by the Department of Translation and Language Sciences, Universitat Pompeu Fabra.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Epistemic intonation and epistemic gesture are mutually co-expressive: empirical results from two intonation-gesture matching tasks

  • Borràs Comes, Joan Manel, 1984-
  • Kiagia, Evangelia
  • Prieto Vives, Pilar, 1965-
While several studies have investigated the temporal relationship between co-speech gestures and prosodic structure, little is known about their potential interaction at the level of their encoding of pragmatic meaning. Here we report the results of two complementary intonation-gesture matching tasks which investigate the potential co-dependencies between intonation patterns related to epistemic commitment operators and their associated gestures in Catalan. In Experiment 1, participants were shown audio-muted videos in which a speaker performed gestures conveying epistemic information of certainty or uncertainty while uttering statements and questions. The subjects were then asked to produce a stipulated target word, the goal being to examine whether they would produce the word with a tune that was semantically consistent with the gestures they had seen. In Experiment 2, participants were primed by hearing intonation patterns conveying epistemic information (certainty-uncertainty) and were then asked to select one of two silent videos which seemed to best match the intonation they had heard. The results suggest converging positive effects in both matching tasks and suggest a close interrelation between the pragmatic representations of intonation and gesture that needs to be taken into account when investigating multimodal pragmatic encoding., This research has been funded by a research grant awarded by the Spanish Ministry of Science, Innovation and Universities (PGC2018-097007-B-I00, “Multimodal Language Learning (MLL): Prosodic and Gestural Integration in Pragmatic and Phonological Development”), and by a grant awarded by the Generalitat de Catalunya (2017 SGR 971) to the Prosodic Studies Group. Also awarded by the Generalitat de Catalunya, the first author acknowledges a research group grant to “LLETRA, Linguistic and Literary Education” (2017 SGR 1728).
Recercat. Dipósit de la Recerca de Catalunya
Publicación Capítulo o parte de un libro (BookPart)

Melodic perception skills predict Catalan speakers’ imitation abilities of unfamiliar languages

  • Li, Peng
  • Zhang, Yuan
  • Fu, Xianqiang
  • Baills, Florence
  • Prieto, Pilar
Musical perception skills have been shown to influence second language speech production. Likewise, working memory may also affect nonnative speech production abilities. However, very few studies have assessed their respective role in speech imitation abilities. The present study thus investigates the predictive role of musical perception skills and working memory on speech imitation abilities of unfamiliar languages. Sixty-one adult Catalan speakers imitated twelve sentences in six languages that were unfamiliar to them. Participants’ music perception skills were tested by four PROMS-S subsets, namely accent, melody, pitch, and rhythm, and their working memory was measured by a forward digit span test. A linear regression analysis revealed that melodic perception skills were the unique predictor among all four musical perception subtests and that working memory was not a significant predictor. Our findings show that melodic perception skills are key in predicting the capability in imitating unfamiliar speech and thus may be important for learning foreign language pronunciation., This study is funded by the PGC2018-097007-B-l00 project awarded by the Ministerio de Ciencia, Innovación y Universidades (MCIU), Agencia Estatal de Investigación (AEI) and Fondo Europeo de Desarrollo Regional (FEDER), Pla de Recerca de Catalunya: 2017 SGR_971, awarded by Agència de Gestió d'Ajuts Universitaris i de Recerca (AGAUR).
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Multimodal mitigation: how facial and body cues index politeness in Catalan requests

  • Hübscher, Iris
  • Sánchez-Conde, Cristina
  • Borràs-Comes, Joan
  • Vincze, Laura
  • Prieto Vives, Pilar, 1965-
Recent cross-linguistic research has demonstrated that speakers use a prosodic mitigation strategy when addressing higher status interlocutors by talking more slowly, reducing the intensity and lowering the overall fundamental frequency (F0). Much less is known, however, about how politeness-related meaning is expressed multimodally (i.e., combining verbal and multimodal channels). The present study investigates how Catalan native speakers encode politeness-related meanings through facial and body cues. We test whether speakers apply a gestural mitigation strategy and use specific hedging devices in socially distant situations (e.g., when asking an older person of higher status for a favor). Twenty Catalan speakers were video-recorded while participating in a discourse elicitation task where they were required to produce requests in polite and non-polite contexts. In the resulting recordings, a set of 21 facial and body cues associated with speech were coded and analyzed. The results show that politeness-related meanings are expressed through gestural mitigation strategies that go hand-in-hand with previously reported prosodic mitigation strategies., This work was financed by the Spanish Ministry of Science, Innovation and Universities, Agencia Estatal de Investigacion, and Fondo Europeo ´ de Desarrollo Regional, grant numbers PGC2018-097007-B-100 and PID2021- 123823NB-I00; and by a grant awarded by the Generalitat de Catalunya, grant number 2017 SGR_971 to the Prosodic Studies Group.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Encouraging participant embodiment during VR-assisted public speaking training improves persuasiveness and charisma and reduces anxiety in secondary school students

  • Valls-Ratés, Ïo
  • Niebuhr, Oliver
  • Prieto Vives, Pilar, 1965-
Practicing public speaking to simulated audiences created in virtual reality environments is reported to be effective for reducing public speaking anxiety. However, little is known about whether this effect can be enhanced by encouraging the use of gestures during VR-assisted public speaking training. In the present study two groups of secondary schools underwent a three-session public speaking training program in which they delivered short speeches to VR-simulated audiences. One group was encouraged to “embody” their speeches through gesture while the other was given no instructions regarding the use of gesture. Before and after the training sessions participants underwent respectively a pre- and a post-training session, which consisted of delivering a similar short speech to a small live audience. At pre- and post-training sessions, participants’ levels of anxiety were self-assessed, their speech performances were rated for persuasiveness and charisma by independent raters, and their verbal output was analyzed for prosodic features and gesture rate. Results showed that both groups significantly reduced their self-assessed anxiety between the pre- and post-training sessions. Persuasiveness and charisma ratings increased for both groups, but to a significantly greater extent in the gesture-using group. However, the prosodic and gestural features analyzed showed no significant differences across groups or from pre-to post-training speeches. Thus, our results seem to indicate that encouraging the use of gesture in VR-assisted public speaking practice can help students be more charismatic and their delivery more persuasive before presenting in front of a live audience., This work benefited from funding awarded by the Spanish Ministry of Economy and Competitiveness (PGC2018-097007-B-I00 and PID2021-123823NB-I00) and the Generalitat de Catalunya (2017 SGR_971). We also acknowledge support from the Recercaixa Project (RecerCaixa 2017ACUP 00249) and the Department of Translation at Universitat Pompeu Fabra through a 1-year doctoral grant to the first author.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Capítulo o parte de un libro (BookPart)

Training with embodied musical activities has positive effects on unfamiliar language imitation skills

  • Zhang, Yuan
  • Baills, Florence
  • Prieto, Pilar
Research shows that musical expertise benefits second language (L2) phonological learning [1], however little is known on the potential effects of training musical melodic and rhythmic skills on language production skills. This study investigated the role of training musical features such as melody, rhythm, and accent with embodied activities [2] (e.g., without any spoken input) on imitation skills. We hypothesized that embodied musical learning would activate participants' melodic and rhythmic musical skills, which will transfer to their ability to imitate speech. Fifty Chinese adolescents participated in three 45-minute classroom training sessions. They were randomly assigned to one of two conditions: the Embodied Music group followed rhythmic and melodic activities involving body movements; the Non-Embodied Music group followed traditional music classes. Before and after training, participants took part in a sentence-imitation task involving six unfamiliar languages. For each language, imitation skills were evaluated in terms of accentedness by three native speakers. Results showed a significantly higher improvement in the Embodied Music group. These findings demonstrate for the first time the beneficial effects of training students with embodied musical activities on speech imitation skills and add evidence to the relationship between musical rhythmic and melodic features and speech., This research is funded by the PGC2018-097007-B-l00 project awarded by the Ministerio de Ciencia, Innovación y Universidades (MCIU), and Pla de Recerca de Catalunya: 2017 SGR_971, awarded by Agència de Gestió d'Ajuts Universitaris i de Recerca.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Capítulo o parte de un libro (BookPart)

Gesture shape and gesture–speech alignment predict simultaneous L2 sound production accuracy

  • Li, Peng
  • Baills, Florence
  • Xi, Xiaotong
  • Prieto, Pilar
Previous research suggests hand gestures aid L2 pronunciation learning, but the underlying mechanisms remain unclear. This study examines if accurate gesture imitation during phonetic training predicts the accuracy of non-native sounds produced while doing a pedagogical hand gesture. Twenty-nine Catalan speakers, unfamiliar with Chinese, learned to pronounce Chinese aspirated plosives while mimicking gestures representing the aspirated sounds. Results revealed that more precise gesture shape and better gesture–speech alignment predicted more accurate speech imitation, emphasizing the significance of the spatiotemporal gesture–speech coupling in embodied L2 phonetic training., This study is funded by “Multimodal Communication: The integration of prosody and gesture in human communication and in language learning” [PID2021-123823NB-I00] awarded by the Ministerio de Ciencia e Innovación and “Multimodal language learning: Prosodic and Gestural Integration in Pragmatic and Phonological Development” [PGC2018- 097007-B-l00], awarded by the Ministerio de Ciencia, Innovación y Universidades, Agencia Estatal de Investigación, and Fondo Europeo de Desarrollo Regional. PL is supported by the Research Council of Norway through its Centres of Excellence funding scheme [223265]. FB acknowledges a Margarita Salas grant funded by European Union– NextGenerationEU, Ministry of Universities and Recovery, Transformation, and Resilience Plan, through a call from Pompeu Fabra University. XX is supported by the Secretaria d’Universitats i Recerca de la Generalitat de Catalunya and the European Social Fund under the Grant for the recruitment of early-stage research staff [2021FI_B 00137].
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Expressive pragmatics and prosody in young preschoolers are more closely related to structural language than to mentalizing

  • Pronina, Mariia
  • Prieto Vives, Pilar, 1965-
  • Bischetti, Luca
  • Bambini, Valentina
Pragmatics lies at the point where language meets the social world and encompasses both the linguistic and the social dimensions of communication. However, the relationship between pragmatic abilities, other language skills, and socio-cognitive aspects such as mentalizing is still a matter of wide debate. This study sets out to investigate the status of pragmatic abilities by testing from a developmental angle their relationship with other linguistic skills and mentalizing. We examined the role of structural language and mentalizing on both expressive pragmatic and prosodic skills in typically developing preschool children. A total of 105 3-to 4-year-old children were assessed on pragmatics and prosody with the Audiovisual Pragmatic Test, as well as on structural language skills (vocabulary and syntax) and a series of mentalizing measures (false belief, emotion understanding, and metacognitive vocabulary). A combined approach including correlations, regressions, and structural equation modeling (SEM) was used. Structural language was a strong positive predictor of both pragmatic and prosodic scores, while mentalizing predicted neither pragmatic nor prosodic performance. We suggest that in preschool years, expressive pragmatics and prosodic skills are more closely related to structural language skills than to mentalizing., This work was supported by the Generalitat de Catalunya [2017 SGR_971,2020 FI_B2 00192]; Ministerio de Ciencia, Innovación y Universidades [PGC2018-097007-B-I00].
Recercat. Dipósit de la Recerca de Catalunya
Publicación Tesis de maestría (MasterThesis)

Visual effects on the perception of persuasive speech

  • Vilà Calvo, Paula
Treball de fi de màster en Discurs. Tutora: Dra. Pilar Prieto Vives, Human communication is multimodal, e.g. it is composed of a set of verbal and non-verbal (visual) cues produced by the speaker with the aim to persuade the hearer’s thoughts and actions. However, the actual impact of the visual channel on the recipients’ impressions of persuasiveness is still not clear. The purpose of this investigation is to assess how the visual channel can impact on the perceived persuasiveness of public speeches. A set of 45 persuasive speeches delivered by a group of high school students in front of a real audience was assessed by a group of 15 raters who evaluated the persuasive effect of the speech on a 1 to 7 Likert scale. Crucially, they first assessed the isolated audio-only version of the discourses (Audio-Only condition) and a week later the audio- visual recordings of the same speeches (Audio-Visual condition). A quantitative analysis of the rater’s persuasive assessments revealed that AV presentations were perceived to be more persuasive than AO presentations of the same discourses. In order to assess why the visual presentation of the discourses triggered a more positive effect on hearers, a multimodal analysis was carried out for prosodic and non-verbal features of these discourses. The results showed that the speakers that obtained a mean improvement of 0.4 in their AV persuasiveness score used significantly more positive multimodal stance indicators (e.g., direct gaze at the audience, more communicative gestures, and more smiling faces) than the speakers that did not obtain an improvement. All in all, the present study shows that access to the visual channel tends to impact positively on the audience's perception of persuasive speech., This study has been partly funded by a research grant awarded by the Spanish Ministry of Science, Innovation, and Universities (PGC2018-097007-B-I00, ‘Multimodal Language Learning (MLL): Prosodic and Gestural Integration in Pragmatic and Phonological Development’).
Recercat. Dipósit de la Recerca de Catalunya
Publicación Capítulo o parte de un libro (BookPart)

The effects of prosodic training with logatomes and prosodic gestures on L2 spontaneous speech

  • Baills, Florence
  • Santiago, Fabián
  • Mairano, Paolo
  • Prieto, Pilar
Training L2 suprasegmental features benefits pronunciation accuracy and comprehensibility, especially through the use of hand gestures. However, studies have mainly looked at the effect of prosodic training in controlled tasks and less is known about spontaneous speech. The present study explores the effect of prosodic training with and without gestures depicting prosody on several dimensions of pronunciation in spontaneous speech. Fifty Catalan learners of French practiced oral reading and sentence-by-sentence imitation with short dialogues during three 30-minute sessions in one of three conditions: speech-only, repeating a selection of sentences from the dialogues; logatome, repeating logatome sequences (series of same consonant-vowel syllable leaving out lexical information) corresponding to the prosody of target sentences; and logatome+gesture, additionally mimicking gestures representing phrasal prosodic patterns. Perceptive evaluations of learners’ spontaneous speech at pre- and posttest revealed that training prosody did not have any significant effect on participants’ comprehensibility, accentedness, and suprasegmental accuracy in spontaneous speech. Acoustic analyses further showed null effects of prosodic training on fluency and the pronunciation of difficult vowel contrasts. Our findings suggest that L2 prosodic training should be better tailored for spontaneous speech. Alternatively, learners may need more time to transfer beneficial effects from controlled tasks to spontaneous speech., This research was supported by funding from the Spanish Ministry of Economy and Competitiveness (PGC2018-097007-B-I00) and the Generalitat de Catalunya (2017 SGR _ 971).
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Hand Gestures Facilitate the Acquisition of Novel Phonemic Contrasts When They Appropriately Mimic Target Phonetic Features

  • Xi, Xiaotong
  • Li, Peng
  • Baills, Florence
  • Prieto, Pilar
Purpose: Research has shown that observing hand gestures mimicking pitch movements or rhythmic patterns can improve the learning of second language (L2) suprasegmental features. However, less is known about the effects of hand gestures on the learning of novel phonemic contrasts. This study examines (a) whether hand gestures mimicking phonetic features can boost L2 segment learning by naive learners and (b) whether a mismatch between the hand gesture form and the target phonetic feature influences the learning effect. Method: Fifty Catalan native speakers undertook a short multimodal training session on two types of Mandarin Chinese consonants (plosives and affricates) in either of two conditions: Gesture and No Gesture. In the Gesture condition, a fist-to-open-hand gesture was used to mimic air burst, while the No Gesture condition included no such use of gestures. Crucially, while the hand gesture appropriately mimicked the air burst produced in plosives, this was not the case for affricates. Before and after training, participants were tested on two tasks, namely, the identification task and the imitation task. Participants' speech output was rated by five Chinese native speakers. Results: The perception results showed that training with or without gestures yielded similar degrees of improvement for the identification of aspiration contrasts. By contrast, the production results showed that, while training without gestures did not help improve L2 pronunciation, training with gestures improved pronunciation, but only when the given gestures appropriately mimicked the phonetic properties they represented. Conclusions: Results revealed that the efficacy of observing hand gestures on the learning of nonnative phonemes depends on the appropriateness of the form of those gestures relative to the target phonetic features. That is, hand gestures seem to be more useful when they appropriately mimic phonetic features., This research was supported by funding from the Spanish Ministry of Science, Innovation and Universities (PGC2018-097007-B-I00) and the Generalitat de Catalunya projects (2017 SGR-971).
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Encouraging kids to beat: children's beat gesture production boosts their narrative performance

  • Vilà-Giménez, Ingrid
  • Prieto Vives, Pilar, 1965-
Gesture is an integral part of language development. While recent evidence shows that observing a speaker who is simultaneously producing beat gestures helps preschoolers remember and understand information and also improves the production of oral narratives, little is known about the potential value of encouraging children to produce beat gestures—as opposed to merely observing them. In this between‐subjects pretest–posttest training study we examine whether encouraging children to produce beats can boost their narrative performance. A total of 47 5‐ to 6‐year‐old children were divided into two groups and exposed to a training session in which a total of six stories were presented under one of two experimental conditions: (a) the children merely observed video‐recordings of a storyteller who used beat gestures and were then asked to retell the narratives; or (b) the children observed the same video‐recordings and then retold the narratives but were encouraged to simultaneously use their hands in the same way the storytellers did. Pretests and posttests consisting of children's narrations of short animated cartoons were analysed for narrative structure and fluency. A comparison of scores showed that children in the group that had been encouraged to use beat gestures in the training phase performed better in both narrative structure and fluency than the group of children who were simply asked to retell the story without gesture instruction. These findings suggest that linguistically relevant body movements serve to boost language development and that embodied storytelling can be of help in narrative training., This study was approved by the ethics committee of the Universitat Pompeu Fabra and benefited from funding awarded by the Spanish Ministry of Science, Innovation and Universities (MCIU), Agencia Estatal de Investigación (AEI), and Fondo Europeo de Desarrollo Regional (FEDER) (PGC2018- 097007-B-100 “Multimodal Language Learning (MLL): Prosodic and Gestural Integration in Pragmatic and Phonological Development”) and by the Generalitat de Catalunya (2017 SGR_971) to the Prosodic Studies Group. The first author also acknowledges an FI grant from the Generalitat de Catalunya (ref. 2019FI_B2_00125).
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Bridging the gap between prosody and pragmatics: the acquisition of pragmatic prosody in the preschool years and Its relation with theory of mind

  • Pronina, Mariia
  • Hübscher, Iris
  • Vilà-Giménez, Ingrid
  • Prieto Vives, Pilar, 1965-
While it is well known that prosodic features are central in the conveyance of pragmatic meaning across languages, developmental research has assessed a narrow set of pragmatic functions of prosody. Research on prosodic development has focused on early infancy, with the subsequent preschool ages and beyond having received less attention. This study sets out to explore how young preschoolers develop the ability to use prosody to express pragmatic meanings while taking into account children’s Theory of Mind (ToM) development. Though ToM has been suggested to be linked to the development of receptive prosody, little is known about its relationship with expressive prosodic skills. A total of 102 3- to 4-year-old Catalan-speaking children were assessed for their pragmatic prosody skills using 35 picture-supported prompts revolving around a variety of social scenarios, as well as for their ToM skills. The responses were analyzed for prosodic appropriateness. The analyses revealed that 3- to 4-year-olds successfully produced prosody to encode basic expressive acts and unbiased speech acts such as information-seeking questions. Yet they had more trouble with complex expressive acts and biased speech acts such as the ones that convey speakers’ beliefs. Further analyses showed that ToM alone is not sufficient to explain children’s prosodic score, but the prosodic performance in some pragmatic areas (unbiased pragmatic meanings) was predicted by the interaction between ToM and age. Overall, this evidence for the acquisition of pragmatic prosody by young preschoolers demonstrates the importance of bridging the gap between prosody and pragmatics when accounting for prosodic developmental profiles, as well as taking into account the potential influence of ToM and other socio-cognitive and language skills in this development., This work benefited from funding awarded by the Spanish Ministry of Science, Innovation and Universities (MCIU), Agencia Estatal de Investigación (AEI), Fondo Europeo de Desarrollo Regional (FEDER) (PGC2018-097007-B-I00), and the Generalitat de Catalunya (2017 SGR_971). IH was supported by the University of Zurich’s University Research Priority Program “Language and Space.” MP also acknowledges support from the Secretariat for Universities and Research of the Ministry of Business and Knowledge of the Government of Catalonia and the European Social Fund (2020 FI_B2 00192).