Pasar al contenido principal

Resultados relacionados

Resultados totales (Incluyendo duplicados): 14
Encontrada(s) 1 página(s)

Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Synchronization by the hand: the sight of gestures modulates low-frequency activity in brain responses to continuous speech

  • Biau, Emmanuel, 1985-
  • Soto-Faraco, Salvador, 1970-
During social interactions, speakers often produce spontaneous gestures to accompany their speech. These coordinated body movements convey communicative intentions, and modulate how listeners perceive the message in a subtle, but important way. In the present perspective, we put the focus on the role that congruent non-verbal information from beat gestures may play in the neural responses to speech. Whilst delta-theta oscillatory brain responses reflect the time-frequency structure of the speech signal, we argue that beat gestures promote phase resetting at relevant word onsets. This mechanism may facilitate the anticipation of associated acoustic cues relevant for prosodic/syllabic-based segmentation in speech perception. We report recently published data supporting this hypothesis, and discuss the potential of beats (and gestures in general) for further studies investigating continuous AV speech processing through low-frequency oscillations., This research was supported by the Spanish Ministry of Science and Innovation (PSI2013-42626-P), AGAUR Generalitat de Catalunya (2014SGR856) and the European Research Council (StG-2010 263145).
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Hand gestures as visual prosody: BOLD responses to audio–visual alignment are modulated by the communicative nature of the stimuli

  • Biau, Emmanuel, 1985-
  • Morís Fernández, Luis, 1982-
  • Holle, Henning
  • Ávila, César
  • Soto-Faraco, Salvador, 1970-
During public addresses, speakers accompany their discourse with spontaneous hand gestures (beats) that are tightly synchronized with the prosodic contour of the discourse. It has been proposed that speech and beat gestures originate from a common underlying linguistic process whereby both speech prosody and beats serve to emphasize relevant information. We hypothesized that breaking the consistency between beats and prosody by temporal desynchronization, would modulate activity of brain areas sensitive to speech–gesture integration. To this aim, we measured BOLD responses as participants watched a natural discourse where the speaker used beat gestures./nIn order to identify brain areas specifically involved in processing hand gestures with communicative intention, beat synchrony was evaluated against arbitrary visual cues bearing equivalent rhythmic and spatial properties as the gestures. Our results revealed that left MTG and IFG were specifically sensitive to speech synchronized with beats, compared to the arbitrary vision–speech pairing. Our results suggest that listeners confer beats a function of visual prosody, complementary to the prosodic structure of speech. We conclude that the emphasizing function of beat gestures in speech perception is instantiated through a specialized brain network sensitive to the communicative intent conveyed by a speaker with his/her hands., This research was supported by the Ministerio de Economia y Competitividad (PSI2013-42626-P), AGAUR Generalitat de Catalunya (2014SGR856), and the European Research Council (StG-2010 263145).
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Speaker's hand gestures modulate speech perception through phase resetting of ongoing neural oscillations

  • Biau, Emmanuel, 1985-
  • Torralba, Mireia
  • Fuentemilla, Lluis
  • de Diego-Balaguer, Ruth
  • Soto-Faraco, Salvador, 1970-
Speakers often accompany speech with spontaneous beat gestures in natural spoken communication. These gestures are usually aligned with lexical stress and can modulate the saliency of their affiliate words. Here we addressed the consequences of beat gestures on the neural correlates of speech perception. Previous studies have highlighted the role played by theta oscillations in temporal prediction of speech. We hypothesized that the sight of beat gestures may influence ongoing low-frequency neural oscillations around the onset of the corresponding words. Electroencephalographic (EEG) recordings were acquired while participants watched a continuous, naturally recorded discourse. The phase-locking/nvalue (PLV) at word onset was calculated from the EEG from pairs of identical words that had been pronounced with and without a concurrent beat gesture in the discourse. We observed an increase in PLV in the 5e6 Hz theta range as well as a desynchronization in the 8e10 Hz alpha band around the onset of words preceded by a beat gesture. These findings suggest that beats help tune low-frequency oscillatory activity at relevant moments during natural speech perception, providing a new insight of how speech and paralinguistic information are integrated., This research was supported by the Spanish Ministry of Science and Innovation (PSI2013-42626-P), AGAUR Generalitat de Catalunya (2014SGR856) and, the European Research Council (StG-2010 263145).
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

The COGs (context, object, and goals) in multisensory processing

  • ten Oever, Sanne
  • Romei, Vincenzo
  • van Atteveldt, Nienke
  • Soto-Faraco, Salvador, 1970-
  • Murray, Micah M.
  • Matusz, Pawel J.
Our understanding of how perception operates in real-world environments has been substantially advanced by studying both multisensory processes and “top-down” control processes influencing sensory processing via activity from higher-order brain areas, such as attention, memory, and expectations. As the two topics have been traditionally studied separately, the mechanisms orchestrating real-world multisensory processing remain unclear. Past work has revealed that the observer’s goals gate the influence of many multisensory processes on brain and behavioural responses, whereas some other multisensory processes might occur independently of these goals. Consequently, other forms of top-down control beyond goal dependence are necessary to explain the full range of multisensory effects currently reported at the brain and the cognitive level. These forms of control include sensitivity to stimulus context as well as the detection of matches (or lack thereof) between a multisensory stimulus and categorical attributes of naturalistic objects (e.g. tools, animals). In this review we discuss and integrate the existing findings that demonstrate the importance of such goal-, object- and context-based top-down control over multisensory processing. We then put forward a few principles emerging from this literature review with respect to the mechanisms underlying multisensory processing and discuss their possible broader implications., This research was supported by grants from the Ministerio de Economia y Competitividad (PSI2013-42626-P), AGAUR Generalitat de Catalunya (2014SGR856), and the European Research Council (StG-2010 263145) to S.S-F, and the Swiss National Science Foundation (Grant #320030-149982 as well as the National Centre of Competence in Research project “SYNAPSY, The Synaptic Bases of Mental Disease” [Project 51AU40-125759]) and the Swiss Brain League (2014 Research Prize) to MMM. StO receives support from the Dutch Organisation for Scientific Research (Grant 406-11-068).
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Visual limitations shape audio-visual integration

  • Pérez-Bellido, Alexis
  • Ernst, Marc O.
  • Soto-Faraco, Salvador, 1970-
  • López-Moliner, Joan
Recent studies have proposed that some cross-modal illusions might be expressed in what were previously thought of as sensory-specific brain areas. Therefore, one interesting question is whether auditory-driven visual illusory percepts respond to manipulations of low-level visual attributes (such as luminance or chromatic contrast) in the same way as their nonillusory analogs. Here, we addressed this question using the double flash illusion (DFI), whereby one brief flash can be perceived as two when combined with two beeps presented in rapid succession. Our results showed that the perception of two illusory flashes depended on luminance contrast, just as the temporal resolution for two real flashes did. Specifically we found that the higher the luminance contrast, the stronger the DFI. Such a pattern seems to contradict what would be predicted from a maximum likelihood estimation perspective, and can be explained by considering that low-level visual stimulus attributes similarly modulate the perception of sound-induced visual phenomena and "real" visual percepts. This finding provides psychophysical support for the involvement of sensory-specific brain areas in the expression of the DFI. On the other hand, the addition of chromatic contrast failed to produce a change in the strength of the DFI despite it improved visual sensitivity to real flashes. The null impact of chromaticity on the cross-modal illusion might suggest a weaker interaction of the parvocellular visual pathway with the auditory system for cross-modal illusions., This research was supported by the Spanish Ministry of Science and Innovation (PSI2013-41568-P, PSI2013-42626-P), Comissionat per a Universitats i Recerca del DIUEGeneralitat de Catalunya (2014SGR856, SGR2014-079), ICREA-Academy and the European Research Council (StG-2010263145).
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Top-down attention regulates the neural expression of audiovisual integration

  • Morís Fernández, Luis, 1982-
  • Visser, Maya
  • Ventura Campos, Noelia
  • Ávila, César
  • Soto-Faraco, Salvador, 1970-
The interplay between attention and multisensory integration has proven to be a difficult question to tackle./nThere are almost as many studies showing that multisensory integration occurs independently from the focus/nof attention as studies implying that attention has a profound effect on integration. Addressing the neural/nexpression of multisensory integration for attended vs. unattended stimuli can help disentangle this apparent/ncontradiction. In the present study, we examine if selective attention to sound pitch influences the expression/nof audiovisual integration in both behavior and neural activity. Participants were asked to attend to one of two/nauditory speech streams while watching a pair of talking lips that could be congruent or incongruent with the/nattended speech stream. We measured behavioral and neural responses (fMRI) to multisensory stimuli under/nattended and unattended conditions while physical stimulation was kept constant. Our results indicate that participants/nrecognized words more accurately from an auditory stream that was both attended and audiovisually/n(AV) congruent, thus reflecting a benefit due to AV integration. On the other hand, no enhancement was found/nfor AV congruency when it was unattended. Furthermore, the fMRI results indicated that activity in the superior/ntemporal sulcus (an area known to be related to multisensory integration) was contingent on attention as well as/non audiovisual congruency. This attentional modulation extended beyond heteromodal areas to affect processing/nin areas classically recognized as unisensory, such as the superior temporal gyrus or the extrastriate cortex, and to/nnon-sensory areas such as the motor cortex. Interestingly, attention to audiovisual incongruence triggered responses/nin brain areas related to conflict processing (i.e., the anterior cingulate cortex and the anterior insula)./nBased on these results, we hypothesize that AV speech integration can take place automatically only when/nboth modalities are sufficiently processed, and that if a mismatch is detected between the AV modalities,/nfeedback from conflict areas minimizes the influence of this mismatch by reducing the processing of the least/ninformative modality., This research was supported by the Ministerio de Economía y/nCompetitividad (PSI2013-42626-P), AGAUR Generalitat de Catalunya/n(2014SGR856), and the European Research Council (StG-2010 263145).
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Watching subtitled films can help learning foreign languages

  • Birulés Muntané, Joan
  • Soto-Faraco, Salvador, 1970-
Watching English-spoken films with subtitles is becoming increasingly popular throughout the world. One reason for this trend is the assumption that perceptual learning of the sounds of a foreign language, English, will improve perception skills in non-English speakers. Yet, solid proof for this is scarce. In order to test the potential learning effects derived from watching subtitled media, a group of intermediate Spanish students of English as a foreign language watched a 1h-long episode of a TV drama in its original English version, with English, Spanish or no subtitles overlaid. Before and after the viewing, participants took a listening and vocabulary test to evaluate their speech perception and vocabulary acquisition in English, plus a final plot comprehension test. The results of the listening skills tests revealed that after watching the English subtitled version, participants improved these skills significantly more than after watching the Spanish subtitled or no-subtitles versions. The vocabulary test showed no reliable differences between subtitled conditions. Finally, as one could expect, plot comprehension was best under native, Spanish subtitles. These learning effects with just 1 hour exposure might have major implications with longer exposure times., This research was supported by the/nMinisterio de Economía y Competitividad (PSI2013-42626-P), AGAUR Generalitat de Catalunya/n(2014SGR856), and the European Research Council/n(StG-2010 263145).
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Synchronization by the hand: the sight of gestures modulates low-frequency activity in brain responses to continuous speech

  • Biau, Emmanuel, 1985-
  • Soto-Faraco, Salvador, 1970-
During social interactions, speakers often produce spontaneous gestures to accompany their speech. These coordinated body movements convey communicative intentions, and modulate how listeners perceive the message in a subtle, but important way. In the present perspective, we put the focus on the role that congruent non-verbal information from beat gestures may play in the neural responses to speech. Whilst delta-theta oscillatory brain responses reflect the time-frequency structure of the speech signal, we argue that beat gestures promote phase resetting at relevant word onsets. This mechanism may facilitate the anticipation of associated acoustic cues relevant for prosodic/syllabic-based segmentation in speech perception. We report recently published data supporting this hypothesis, and discuss the potential of beats (and gestures in general) for further studies investigating continuous AV speech processing through low-frequency oscillations., This research was supported by the Spanish Ministry of Science and Innovation (PSI2013-42626-P), AGAUR Generalitat de Catalunya (2014SGR856) and the European Research Council (StG-2010 263145).
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

The COGs (context, object, and goals) in multisensory processing

  • ten Oever, Sanne
  • Romei, Vincenzo
  • van Atteveldt, Nienke
  • Soto-Faraco, Salvador, 1970-
  • Murray, Micah M.
  • Matusz, Pawel J.
Our understanding of how perception operates in real-world environments has been substantially advanced by studying both multisensory processes and “top-down” control processes influencing sensory processing via activity from higher-order brain areas, such as attention, memory, and expectations. As the two topics have been traditionally studied separately, the mechanisms orchestrating real-world multisensory processing remain unclear. Past work has revealed that the observer’s goals gate the influence of many multisensory processes on brain and behavioural responses, whereas some other multisensory processes might occur independently of these goals. Consequently, other forms of top-down control beyond goal dependence are necessary to explain the full range of multisensory effects currently reported at the brain and the cognitive level. These forms of control include sensitivity to stimulus context as well as the detection of matches (or lack thereof) between a multisensory stimulus and categorical attributes of naturalistic objects (e.g. tools, animals). In this review we discuss and integrate the existing findings that demonstrate the importance of such goal-, object- and context-based top-down control over multisensory processing. We then put forward a few principles emerging from this literature review with respect to the mechanisms underlying multisensory processing and discuss their possible broader implications., This research was supported by grants from the Ministerio de Economia y Competitividad (PSI2013-42626-P), AGAUR Generalitat de Catalunya (2014SGR856), and the European Research Council (StG-2010 263145) to S.S-F, and the Swiss National Science Foundation (Grant #320030-149982 as well as the National Centre of Competence in Research project “SYNAPSY, The Synaptic Bases of Mental Disease” [Project 51AU40-125759]) and the Swiss Brain League (2014 Research Prize) to MMM. StO receives support from the Dutch Organisation for Scientific Research (Grant 406-11-068).
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Visual limitations shape audio-visual integration

  • Pérez-Bellido, Alexis
  • Ernst, Marc O.
  • Soto-Faraco, Salvador, 1970-
  • López-Moliner, Joan
Recent studies have proposed that some cross-modal illusions might be expressed in what were previously thought of as sensory-specific brain areas. Therefore, one interesting question is whether auditory-driven visual illusory percepts respond to manipulations of low-level visual attributes (such as luminance or chromatic contrast) in the same way as their nonillusory analogs. Here, we addressed this question using the double flash illusion (DFI), whereby one brief flash can be perceived as two when combined with two beeps presented in rapid succession. Our results showed that the perception of two illusory flashes depended on luminance contrast, just as the temporal resolution for two real flashes did. Specifically we found that the higher the luminance contrast, the stronger the DFI. Such a pattern seems to contradict what would be predicted from a maximum likelihood estimation perspective, and can be explained by considering that low-level visual stimulus attributes similarly modulate the perception of sound-induced visual phenomena and "real" visual percepts. This finding provides psychophysical support for the involvement of sensory-specific brain areas in the expression of the DFI. On the other hand, the addition of chromatic contrast failed to produce a change in the strength of the DFI despite it improved visual sensitivity to real flashes. The null impact of chromaticity on the cross-modal illusion might suggest a weaker interaction of the parvocellular visual pathway with the auditory system for cross-modal illusions., This research was supported by the Spanish Ministry of Science and Innovation (PSI2013-41568-P, PSI2013-42626-P), Comissionat per a Universitats i Recerca del DIUEGeneralitat de Catalunya (2014SGR856, SGR2014-079), ICREA-Academy and the European Research Council (StG-2010263145).
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Watching subtitled films can help learning foreign languages

  • Birulés Muntané, Joan
  • Soto-Faraco, Salvador, 1970-
Watching English-spoken films with subtitles is becoming increasingly popular throughout the world. One reason for this trend is the assumption that perceptual learning of the sounds of a foreign language, English, will improve perception skills in non-English speakers. Yet, solid proof for this is scarce. In order to test the potential learning effects derived from watching subtitled media, a group of intermediate Spanish students of English as a foreign language watched a 1h-long episode of a TV drama in its original English version, with English, Spanish or no subtitles overlaid. Before and after the viewing, participants took a listening and vocabulary test to evaluate their speech perception and vocabulary acquisition in English, plus a final plot comprehension test. The results of the listening skills tests revealed that after watching the English subtitled version, participants improved these skills significantly more than after watching the Spanish subtitled or no-subtitles versions. The vocabulary test showed no reliable differences between subtitled conditions. Finally, as one could expect, plot comprehension was best under native, Spanish subtitles. These learning effects with just 1 hour exposure might have major implications with longer exposure times., This research was supported by the/nMinisterio de Economía y Competitividad (PSI2013-42626-P), AGAUR Generalitat de Catalunya/n(2014SGR856), and the European Research Council/n(StG-2010 263145).
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Top-down attention regulates the neural expression of audiovisual integration

  • Morís Fernández, Luis, 1982-
  • Visser, Maya
  • Ventura Campos, Noelia
  • Ávila, César
  • Soto-Faraco, Salvador, 1970-
The interplay between attention and multisensory integration has proven to be a difficult question to tackle./nThere are almost as many studies showing that multisensory integration occurs independently from the focus/nof attention as studies implying that attention has a profound effect on integration. Addressing the neural/nexpression of multisensory integration for attended vs. unattended stimuli can help disentangle this apparent/ncontradiction. In the present study, we examine if selective attention to sound pitch influences the expression/nof audiovisual integration in both behavior and neural activity. Participants were asked to attend to one of two/nauditory speech streams while watching a pair of talking lips that could be congruent or incongruent with the/nattended speech stream. We measured behavioral and neural responses (fMRI) to multisensory stimuli under/nattended and unattended conditions while physical stimulation was kept constant. Our results indicate that participants/nrecognized words more accurately from an auditory stream that was both attended and audiovisually/n(AV) congruent, thus reflecting a benefit due to AV integration. On the other hand, no enhancement was found/nfor AV congruency when it was unattended. Furthermore, the fMRI results indicated that activity in the superior/ntemporal sulcus (an area known to be related to multisensory integration) was contingent on attention as well as/non audiovisual congruency. This attentional modulation extended beyond heteromodal areas to affect processing/nin areas classically recognized as unisensory, such as the superior temporal gyrus or the extrastriate cortex, and to/nnon-sensory areas such as the motor cortex. Interestingly, attention to audiovisual incongruence triggered responses/nin brain areas related to conflict processing (i.e., the anterior cingulate cortex and the anterior insula)./nBased on these results, we hypothesize that AV speech integration can take place automatically only when/nboth modalities are sufficiently processed, and that if a mismatch is detected between the AV modalities,/nfeedback from conflict areas minimizes the influence of this mismatch by reducing the processing of the least/ninformative modality., This research was supported by the Ministerio de Economía y/nCompetitividad (PSI2013-42626-P), AGAUR Generalitat de Catalunya/n(2014SGR856), and the European Research Council (StG-2010 263145).
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Speaker's hand gestures modulate speech perception through phase resetting of ongoing neural oscillations

  • Biau, Emmanuel, 1985-
  • Torralba, Mireia
  • Fuentemilla, Lluis
  • de Diego-Balaguer, Ruth
  • Soto-Faraco, Salvador, 1970-
Speakers often accompany speech with spontaneous beat gestures in natural spoken communication. These gestures are usually aligned with lexical stress and can modulate the saliency of their affiliate words. Here we addressed the consequences of beat gestures on the neural correlates of speech perception. Previous studies have highlighted the role played by theta oscillations in temporal prediction of speech. We hypothesized that the sight of beat gestures may influence ongoing low-frequency neural oscillations around the onset of the corresponding words. Electroencephalographic (EEG) recordings were acquired while participants watched a continuous, naturally recorded discourse. The phase-locking/nvalue (PLV) at word onset was calculated from the EEG from pairs of identical words that had been pronounced with and without a concurrent beat gesture in the discourse. We observed an increase in PLV in the 5e6 Hz theta range as well as a desynchronization in the 8e10 Hz alpha band around the onset of words preceded by a beat gesture. These findings suggest that beats help tune low-frequency oscillatory activity at relevant moments during natural speech perception, providing a new insight of how speech and paralinguistic information are integrated., This research was supported by the Spanish Ministry of Science and Innovation (PSI2013-42626-P), AGAUR Generalitat de Catalunya (2014SGR856) and, the European Research Council (StG-2010 263145).
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Hand gestures as visual prosody: BOLD responses to audio–visual alignment are modulated by the communicative nature of the stimuli

  • Biau, Emmanuel, 1985-
  • Morís Fernández, Luis, 1982-
  • Holle, Henning
  • Ávila, César
  • Soto-Faraco, Salvador, 1970-
During public addresses, speakers accompany their discourse with spontaneous hand gestures (beats) that are tightly synchronized with the prosodic contour of the discourse. It has been proposed that speech and beat gestures originate from a common underlying linguistic process whereby both speech prosody and beats serve to emphasize relevant information. We hypothesized that breaking the consistency between beats and prosody by temporal desynchronization, would modulate activity of brain areas sensitive to speech–gesture integration. To this aim, we measured BOLD responses as participants watched a natural discourse where the speaker used beat gestures./nIn order to identify brain areas specifically involved in processing hand gestures with communicative intention, beat synchrony was evaluated against arbitrary visual cues bearing equivalent rhythmic and spatial properties as the gestures. Our results revealed that left MTG and IFG were specifically sensitive to speech synchronized with beats, compared to the arbitrary vision–speech pairing. Our results suggest that listeners confer beats a function of visual prosody, complementary to the prosodic structure of speech. We conclude that the emphasizing function of beat gestures in speech perception is instantiated through a specialized brain network sensitive to the communicative intent conveyed by a speaker with his/her hands., This research was supported by the Ministerio de Economia y Competitividad (PSI2013-42626-P), AGAUR Generalitat de Catalunya (2014SGR856), and the European Research Council (StG-2010 263145).