Pasar al contenido principal

Resultados relacionados

Resultados totales (Incluyendo duplicados): 62
Encontrada(s) 1 página(s)

Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Modality-switching in the Simon task: The clash of reference frames

  • Ruzzoli, Manuela
  • Soto-Faraco, Salvador, 1970-
The representation of spatial information related to an event can influence behavior even when location is task-irrelevant, as in the case of Stimulus–Response (S-R) compatibility effects on the Simon task. However, unlike single-modality situations, which are often used to study the Simon effect, in real-life scenarios various sensory modalities provide spatial information coded in different coordinate systems. Here, we address the expression of S-R compatibility effects in mixed-modality contexts, where events can occur in 1 of various sensory modalities (i.e., vision, touch or audition). The results confirm that, in single-modality cases, Simon effects in vision are expressed in an external spatial frame of reference, while touch information is coded anatomically. Remarkably, when mixing visual and tactile trials in an unpredictable way, the Simon effect disappeared in vision whereas tactile Simon effects remained expressed in their own (anatomical) frame of reference. Mixing visual and auditory stimuli did not obliterate the visual Simon effect and S-R compatibility effects in an external reference frame were evident for both modalities. The extinction of visual Simon effects as a result of mixing visual and tactile modalities can be interpreted as a consequence of the dynamic reorganization of the weights associated to the different sources of spatial information at play., This research was supported by the Ministerio de Economia y Competitividad (PSI2016-75558-P), the AGAUR Generalitat de Catalunya (2014SGR856) and the European Research Council (StG-2010 263145) to SSF. M.R. was supported by the Juan de la Cierva postdoctoral fellowship (JCI-2012-12335) from the Ministerio de Economia y Competitividad.
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Effect of attentional load on audiovisual speech perception: evidence from ERPs

  • Alsius, Agnès
  • Möttönen, Riikka
  • Sams, Mikko E.
  • Soto-Faraco, Salvador, 1970-
  • Tiippana, Kaisa
Seeing articulatory movements influences perception of auditory speech. This is often reflected in a shortened latency of auditory event-related potentials (ERPs) generated in the auditory cortex. The present study addressed whether this early neural correlate of audiovisual interaction is modulated by attention. We recorded ERPs in 15 subjects while they were presented with auditory, visual, and audiovisual spoken syllables. Audiovisual stimuli consisted of incongruent auditory and visual components known to elicit a McGurk effect, i.e., a visually driven alteration in the auditory speech percept. In a Dual task condition, participants were asked to identify spoken syllables whilst monitoring a rapid visual stream of pictures for targets, i.e., they had to divide their attention. In a Single task condition, participants identified the syllables without any other tasks, i.e., they were asked to ignore the pictures and focus their attention fully on the spoken syllables. The McGurk effect was weaker in the Dual task than in the Single task condition, indicating an effect of attentional load on audiovisual speech perception. Early auditory ERP components, N1 and P2, peaked earlier to audiovisual stimuli than to auditory stimuli when attention was fully focused on syllables, indicating neurophysiological audiovisual interaction. This latency decrement was reduced when attention was loaded, suggesting that attention influences early neural processing of audiovisual speech. We conclude that reduced attention weakens the interaction between vision and audition in speech., This research was supported by grants from Spanish Ministry of Science and Innovation (PSI2010-15426) and the European Research Council (StG-2010 263145) to Salvador Soto-Faraco and Agnès Alsius.
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Synchronization by the hand: the sight of gestures modulates low-frequency activity in brain responses to continuous speech

  • Biau, Emmanuel, 1985-
  • Soto-Faraco, Salvador, 1970-
During social interactions, speakers often produce spontaneous gestures to accompany their speech. These coordinated body movements convey communicative intentions, and modulate how listeners perceive the message in a subtle, but important way. In the present perspective, we put the focus on the role that congruent non-verbal information from beat gestures may play in the neural responses to speech. Whilst delta-theta oscillatory brain responses reflect the time-frequency structure of the speech signal, we argue that beat gestures promote phase resetting at relevant word onsets. This mechanism may facilitate the anticipation of associated acoustic cues relevant for prosodic/syllabic-based segmentation in speech perception. We report recently published data supporting this hypothesis, and discuss the potential of beats (and gestures in general) for further studies investigating continuous AV speech processing through low-frequency oscillations., This research was supported by the Spanish Ministry of Science and Innovation (PSI2013-42626-P), AGAUR Generalitat de Catalunya (2014SGR856) and the European Research Council (StG-2010 263145).
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Age-related sensitive periods influence visual language discrimination in adults

  • Weikum, Whitney M.
  • Vouloumanos, Athena
  • Navarra, Jordi
  • Soto-Faraco, Salvador, 1970-
  • Sebastián Gallés, Núria
  • Werker, Janet F.
Adults as well as infants have the capacity to discriminate languages based on visual speech alone. Here, we investigated whether adults' ability to discriminate languages based on visual speech cues is influenced by the age of language acquisition. Adult participants who had all learned English (as a first or second language) but did not speak French were shown faces of bilingual (French/English) speakers silently reciting sentences in either language. Using only visual speech information, adults who had learned English from birth or as a second language before the age of 6 could discriminate between French and English significantly better than chance. However, adults who had learned English as a second language after age 6 failed to discriminate these two languages, suggesting that early childhood exposure is crucial for using relevant visual speech information to separate languages visually. These findings raise the possibility that lowered sensitivity to non-native visual speech cues may contribute to the difficulties encountered when learning a new language in adulthood., This research was supported by Research grants from the Natural Sciences and Engineering Research Council (NSERC) and the Social Sciences and Humanities Research Council (SSHRC) to Janet F. Werker, by NSERC and SSHRC Fellowships to Whitney M. Weikum, and by grants PSI2009-12859, PSI2012-39149 and RYC-2008-03672 from Ministerio de Economía y Competitividad (Spanish Government), and the European COST action TD0904 to Jordi Navarra. Salvador Soto-Faraco was supported by ERC (StG-2010263145), MICINN (PSI2010-15426 and Consolider INGENIO CSD2007-00012) and AGAUR (SGR2009-092).
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Alpha stimulation of the human parietal cortex attunes tactile perception to external space

  • Ruzzoli, Manuela
  • Soto-Faraco, Salvador, 1970-
An intriguing question in neuroscience concerns how somatosensory events on the skin are represented in the human brain. Since Head and Holmes’ [1] neuropsychological dissociation between localizing touch on the skin and localizing body parts in external space, touch is considered to operate in a variety of spatial reference frames [2]. At least two representations of space are in competition during orienting to touch: a somatotopic one, reflecting the organization of the somatosensory cortex (S1) [3], and a more abstract, external reference frame that factors postural changes in relation to body parts and/or external space [4, 5]. Previous transcranial magnetic stimulation (TMS) studies suggest that the posterior parietal cortex (PPC) plays a key role in supporting representations as well as orienting attention in an external reference frame [4, 6]. Here, we capitalized on the TMS entrainment approach [7, 8], targeting the intraparietal sulcus (IPS). We found that frequency-specific (10 Hz) tuning of the PPC induced spatially specific enhancement of tactile detection that was expressed in an external reference frame. This finding establishes a tight causal link between a concrete form of brain activity (10 Hz oscillation) and a specific type of spatial representation, revealing a fundamental property of how the parietal cortex encodes information., This work was supported by grants from the European Research Council (StG-2010 263145), the Spanish Ministry of Science and Innovation (PSI2010-15426), and the Comissionat per a Universitats i Recerca del DIUE-Generalitat de Catalunya (SRG2009-092) to S.S.-F.
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Beat gestures and syntactic parsing: an ERP study

  • Biau, Emmanuel, 1985-
  • Fromont, Lauren A.
  • Soto-Faraco, Salvador, 1970-
We tested the prosodic hypothesis that the temporal alignment of a speaker's beat gestures in a sentence influences syntactic parsing by driving the listener's attention. Participants chose between two possible interpretations of relative-clause (RC) ambiguous sentences, while their electroencephalogram (EEG) was recorded. We manipulated the alignment of the beat within sentences where auditory prosody was removed. Behavioral performance showed no effect of beat placement on the sentences’ interpretation, while event-related potentials (ERPs) revealed a positive shift of the signal in the windows corresponding to N100 and P200 components. Additionally, post hoc analyses of the ERPs time locked to the RC revealed a modulation of the P600 component as a function of gesture. These results suggest that beats modulate early processing of affiliate words in continuous speech and potentially have a global impact at the level of sentence-parsing components. We speculate that beats must be synergistic with auditory prosody to be fully consequential in behavior., This research was supported by the Ministerio de Econom´ıa y Competitividad (PSI2016-75558-P), AGAUR Generalitat de Catalunya (2014SGR856), and the European Research Council (StG-2010 263145). EB was supported by a postdoctoral fellowship from the European Union’s Horizon 2020 research and innovation programme, under the Marie Sklodowska-Curie grant agreement No. 707727
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Selective attention modulates the direction of audio-visual temporal recalibration

  • Ikumi Montserrat, Nara, 1986-
  • Soto-Faraco, Salvador, 1970-
Temporal recalibration of cross-modal synchrony has been proposed as a mechanism to compensate for timing differences between sensory modalities. However, far from the rich complexity of everyday life sensory environments, most studies to date have examined recalibration on isolated cross-modal pairings. Here, we hypothesize that selective attention might provide an effective filter to help resolve which stimuli are selected when multiple events compete for recalibration. We addressed this question by testing audio-visual recalibration following an adaptation phase where two opposing audio-visual asynchronies were present. The direction of voluntary visual attention, and therefore to one of the two possible asynchronies (flash leading or flash lagging), was manipulated using colour as a selection criterion. We found a shift in the point of subjective audio-visual simultaneity as a function of whether the observer had focused attention to audio-then-flash or to flash-then-audio groupings during the adaptation phase. A baseline adaptation condition revealed that this effect of endogenous attention was only effective toward the lagging flash. This hints at the role of exogenous capture and/or additional endogenous effects producing an asymmetry toward the leading flash. We conclude that selective attention helps promote selected audio-visual pairings to be combined and subsequently adjusted in time but, stimulus organization exerts a strong impact on recalibration. We tentatively hypothesize that the resolution of recalibration in complex scenarios involves the orchestration of top-down selection mechanisms and stimulus-driven processes., This research was supported by the Spanish Ministry of Science and Innovation (PSI2010-15426), Comissionat per a Universitats i Recerca del DIUE-Generalitat de Catalunya (SRG2009-092), and the European Research Council (StG-2010 263145).
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Cross-modal decoupling in temporal attention

  • Mühlberg, Stefanie, 1986-
  • Oriolo, Giovanni
  • Soto-Faraco, Salvador, 1970-
Prior studies have repeatedly reported behavioural benefits to events occurring at attended, compared to unattended, points in time. It has been suggested that, as for spatial orienting, temporal orienting of attention spreads across sensory modalities in a synergistic fashion. However, the consequences of cross-modal temporal orienting of attention remain poorly understood. One challenge is that the passage of time leads to an increase in event predictability throughout a trial, thus making it difficult to interpret possible effects (or lack thereof). Here we used a design that avoids complete temporal predictability to investigate whether attending to a sensory modality (vision or touch) at a point in time confers beneficial access to events in the other, non-attended, sensory modality (touch or vision, respectively). In contrast to previous studies and to what happens with spatial attention, we found that events in one (unattended) modality do not automatically benefit from happening at the time point when another modality is expected. Instead, it seems that attention can be deployed in time with relative independence for different sensory modalities. Based on these findings, we argue that temporal orienting of attention can be cross-modally decoupled in order to flexibly react according to the environmental demands, and that the efficiency of this selective decoupling unfolds in time., This research was funded by the Spanish Ministry of Science and Innovation (PSI2010-15426), the Comissionat per a Universitats i Recerca del DIUE-Generalitat de Catalunya (SRG2009-092) and the European Research Council (StG-2010 263145) to S.S.F.
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Visual information constrains early and late stages of spoken-word recognition in sentence context

  • Brunellière, Angèle
  • Sánchez García, Carolina, 1984-
  • Ikumi Montserrat, Nara, 1986-
  • Soto-Faraco, Salvador, 1970-
Audiovisual speech perception has been frequently studied considering phoneme, syllable and word processing levels. Here, we examined the constraints that visual speech information might exert during the recognition of words embedded in a natural sentence context. We recorded event-related potentials (ERPs) to words that could be either strongly or weakly predictable on the basis of the prior semantic sentential context and, whose initial phoneme varied in the degree of visual saliency from lip movements. When the sentences were presented audio-visually (Experiment 1), words weakly predicted from semantic context elicited a larger long-lasting N400, compared to strongly predictable words. This semantic effect interacted with the degree of visual saliency over a late part of the N400. When comparing audio-visual versus auditory alone presentation (Experiment 2), the typical amplitude-reduction effect over the auditory-evoked N100 response was observed in the audiovisual modality. Interestingly, a specific benefit of high- versus low-visual saliency constraints occurred over the early N100 response and at the late N400 time window, confirming the result of Experiment 1. Taken together, our results indicate that the saliency of visual speech can exert an influence over both auditory processing and word recognition at relatively late stages, and thus suggest strong interactivity between audio-visual integration and other (arguably higher) stages of information processing during natural speech comprehension., This research was supported by the Spanish Ministry of Science and Innovation (PSI2010-15426 and Consolider INGENIO CSD2007-00012), Comissionat per a Universitats i Recerca del DIUE-Generalitat de Catalunya (SGR2009-092), and the European Research Council (StG-2010263145).
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Hand gestures as visual prosody: BOLD responses to audio–visual alignment are modulated by the communicative nature of the stimuli

  • Biau, Emmanuel, 1985-
  • Morís Fernández, Luis, 1982-
  • Holle, Henning
  • Ávila, César
  • Soto-Faraco, Salvador, 1970-
During public addresses, speakers accompany their discourse with spontaneous hand gestures (beats) that are tightly synchronized with the prosodic contour of the discourse. It has been proposed that speech and beat gestures originate from a common underlying linguistic process whereby both speech prosody and beats serve to emphasize relevant information. We hypothesized that breaking the consistency between beats and prosody by temporal desynchronization, would modulate activity of brain areas sensitive to speech–gesture integration. To this aim, we measured BOLD responses as participants watched a natural discourse where the speaker used beat gestures./nIn order to identify brain areas specifically involved in processing hand gestures with communicative intention, beat synchrony was evaluated against arbitrary visual cues bearing equivalent rhythmic and spatial properties as the gestures. Our results revealed that left MTG and IFG were specifically sensitive to speech synchronized with beats, compared to the arbitrary vision–speech pairing. Our results suggest that listeners confer beats a function of visual prosody, complementary to the prosodic structure of speech. We conclude that the emphasizing function of beat gestures in speech perception is instantiated through a specialized brain network sensitive to the communicative intent conveyed by a speaker with his/her hands., This research was supported by the Ministerio de Economia y Competitividad (PSI2013-42626-P), AGAUR Generalitat de Catalunya (2014SGR856), and the European Research Council (StG-2010 263145).
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Cross-modal prediction in speech perception

  • Sánchez García, Carolina, 1984-
  • Alsius, Agnès
  • Enns, James T.
  • Soto-Faraco, Salvador, 1970-
Speech perception often benefits from vision of the speaker's lip movements when they are available. One potential mechanism underlying this reported gain in perception arising from audio-visual integration is on-line prediction. In this study we address whether the preceding speech context in a single modality can improve audiovisual processing and whether this improvement is based on on-line information-transfer across sensory modalities. In the experiments presented here, during each trial, a speech fragment (context) presented in a single sensory modality (voice or lips) was immediately continued by an audiovisual target fragment. Participants made speeded judgments about whether voice and lips were in agreement in the target fragment. The leading single sensory context and the subsequent audiovisual target fragment could be continuous in either one modality only, both (context in one modality continues into both modalities in the target fragment) or neither modalities (i.e., discontinuous). The results showed quicker audiovisual matching responses when context was continuous with the target within either the visual or auditory channel (Experiment 1). Critically, prior visual context also provided an advantage when it was cross-modally continuous (with the auditory channel in the target), but auditory to visual cross-modal continuity resulted in no advantage (Experiment 2). This suggests that visual speech information can provide an on-line benefit for processing the upcoming auditory input through the use of predictive mechanisms. We hypothesize that this benefit is expressed at an early level of speech analysis., This research was supported by the Spanish Ministry of Science and Innovation (PSI2010-15426 and Consolider INGENIO CSD2007-00012), Comissionat per a Universitats i Recerca del DIUE-Generalitat de Catalunya (SRG2009-092 and PIV2009-00122), and the European Research Council (StG-2010 263145).
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Somatosensory saccades reveal the timing of tactile spatial remapping

  • Overvliet, Krista E.
  • Azañón Gracia, Elena
  • Soto-Faraco, Salvador, 1970-
Remapping tactile events from skin to external space is an essential process for human behaviour. It allows us to refer tactile sensations to their actual externally based location, by combining anatomically based somatosensory information with proprioceptive information about the current body posture. We examined the time course of tactile remapping by recording speeded saccadic responses to somatosensory stimuli delivered to the hands. We conducted two experiments in which arm posture varied (crossed or uncrossed), so that anatomical and external frames of reference were either put in spatial conflict or were aligned. The data showed that saccade onset latencies in the crossed hands conditions were slower than in the uncrossed hands condition, suggesting that, in the crossed hands condition, remapping had to be completed before a correct saccade could be executed. Saccades to tactile stimuli when the hands were crossed were sometimes initiated to the wrong direction and then corrected in-flight, resulting in a turn-around saccade. These turn-around saccades were more likely to occur in short-latency responses, compared to onset latencies of saccades that went straight to target. The latter suggests that participants were postponing their saccade until the time the tactile event was represented according to the current body posture. We propose that the difference between saccade onset latencies of crossed and uncrossed hand postures, and between the onset of a turn-around saccade and a straight saccade in the crossed hand posture, reveal the timing of tactile spatial remapping., This research was supported by Spanish Ministry of Science and Innovation (PSI2010-15426 and Consolider INGENIOCSD2007-00012), Comissionat per a Universitats i Recerca del DIUE-Generalitat de Catalunya (SRG2009-092), and European Research Council (StG-2010 263145).
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Sound-driven enhancement of vision: disentangling detection from decisional level contributions

  • Pérez-Bellido, Alexis
  • Soto-Faraco, Salvador, 1970-
  • López-Moliner, Joan
Cross-modal enhancement can be mediated both by higher-order effects due to attention and decision making and by detection-level stimulus-driven interactions. However, the contribution of each of these sources to behavioral improvements has not been conclusively determined and quantified separately. Here, we apply psychophysical analysis based on Piéron functions in order to separate stimulus-dependent changes from those accounted by decision-level contributions. Participants performed a simple visual speeded detection task on Gabor patches of different spatial frequencies and contrast values, presented with and without accompanying sounds. On one hand, we identified an additive cross-modal improvement in mean reaction times across all types of visual stimuli that would be well explained by interactions not strictly based on stimulus-driven modulations (e.g., due to reduction of temporal uncertainty and motor times). On the other hand, we singled out an audio-visual benefit that strongly depended on stimulus features such as frequency and contrast. This particular enhancement was selective to low-visual spatial frequency stimuli, optimized for magnocellular sensitivity. We therefore conclude that interactions at detection stages and at decisional processes in response selection that contribute to audio-visual enhancement can be separated online and express on partly different aspects of visual processing., This research was supported by the Spanish Ministry of Science and Innovation (PSI2010-15867, PSI2010-15426, and Consolider INGENIO CSD2007-00012), Comissionat per a Universitats i Recerca del DIUE-Generalitat de Catalunya (SGR2009-092, SGR2009-308), and the European Research Council (StG-2010263145).
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Speaker's hand gestures modulate speech perception through phase resetting of ongoing neural oscillations

  • Biau, Emmanuel, 1985-
  • Torralba, Mireia
  • Fuentemilla, Lluis
  • de Diego-Balaguer, Ruth
  • Soto-Faraco, Salvador, 1970-
Speakers often accompany speech with spontaneous beat gestures in natural spoken communication. These gestures are usually aligned with lexical stress and can modulate the saliency of their affiliate words. Here we addressed the consequences of beat gestures on the neural correlates of speech perception. Previous studies have highlighted the role played by theta oscillations in temporal prediction of speech. We hypothesized that the sight of beat gestures may influence ongoing low-frequency neural oscillations around the onset of the corresponding words. Electroencephalographic (EEG) recordings were acquired while participants watched a continuous, naturally recorded discourse. The phase-locking/nvalue (PLV) at word onset was calculated from the EEG from pairs of identical words that had been pronounced with and without a concurrent beat gesture in the discourse. We observed an increase in PLV in the 5e6 Hz theta range as well as a desynchronization in the 8e10 Hz alpha band around the onset of words preceded by a beat gesture. These findings suggest that beats help tune low-frequency oscillatory activity at relevant moments during natural speech perception, providing a new insight of how speech and paralinguistic information are integrated., This research was supported by the Spanish Ministry of Science and Innovation (PSI2013-42626-P), AGAUR Generalitat de Catalunya (2014SGR856) and, the European Research Council (StG-2010 263145).
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Sounds can boost the awareness of visual events through attention without cross-modal integration

  • Soto-Faraco, Salvador, 1970-
  • Pápai, Márta Szabina, 1987-
Cross-modal interactions can lead to enhancement of visual perception, even for visual events below awareness. However, the underlying mechanism is still unclear. Can purely bottom-up cross-modal integration break through the threshold of awareness? We used a binocular rivalry paradigm to measure perceptual switches after brief flashes or sounds which, sometimes, co-occurred. When flashes at the suppressed eye coincided with sounds, perceptual switches occurred the earliest. Yet, contrary to the hypothesis of cross-modal integration, this facilitation never surpassed the assumption of probability summation of independent sensory signals. A follow-up experiment replicated the same pattern of results using silent gaps embedded in continuous noise, instead of sounds. This manipulation should weaken putative sound-flash integration, although keep them salient as bottom-up attention cues. Additional results showed that spatial congruency between flashes and sounds did not determine the effectiveness of cross-modal facilitation, which was again not better than probability summation. Thus, the present findings fail to fully support the hypothesis of bottom-up cross-modal integration, above and beyond the independent contribution of two transient signals, as an account for cross-modal enhancement of visual events below level of awareness., This study was funded by a Ministerio de Educación, Cultura y Deporte, Ref. FPU/1206471 to MSP, and ERC Starting Grant, Ref. 263145, Spanish Ministry of Science and Innovation (PSI2016-75558-P), and Comissionat per a Universitats i Recerca del DIUE- Generalitat de Catalunya (2014SGR856) to SS-F.
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Discriminating speech rhythms in audition, vision, and touch

  • Navarra, Jordi
  • Soto-Faraco, Salvador, 1970-
  • Spence, Charles
We investigated the extent to which people can discriminate between languages on the basis of their characteristic temporal, rhythmic information, and the extent to which this ability generalizes across sensory modalities. We used rhythmical patterns derived from the alternation of vowels and consonants in English and Japanese, presented in audition, vision, both audition and vision at the same time, or touch. Experiment 1 confirmed that discrimination is possible on the basis of auditory rhythmic patterns, and extended it to the case of vision, using ‘aperture-close’ mouth movements of a schematic face. In Experiment 2, language discrimination was demonstrated using visual and auditory materials that did not resemble spoken articulation. In a combined analysis including data from Experiments 1 and 2, a beneficial effect was also found when the auditory rhythmic information was available to participants. Despite the fact that discrimination could be achieved using vision alone, auditory performance was nevertheless better. In a final experiment, we demonstrate that the rhythm of speech can also be discriminated successfully by means of vibrotactile patterns delivered to the fingertip. The results of the present study therefore demonstrate that discrimination between language's syllabic rhythmic patterns is possible on the basis of visual and tactile displays., This research was supported by grants PSI2012-39149, PSI2009-12859 and RYC-2008-03672 from Ministerio de Economía y Competitividad (MINECO, Spain) to J. Navarra, the European COST action TD0904, and by grants PSI2010-15426 and CDS00012 from MINECO (Spain), 2009SGR-292 from Comissionat per a Universitats i Recerca del DIUE-Generalitat de Catalunya, and European Research Council (StG-2010 263145) to S. Soto-Faraco.
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Searching for audiovisual correspondence in multiple speaker scenarios

  • Alsius, Agnès
  • Soto-Faraco, Salvador, 1970-
A critical question in multisensory processing is how the constant information flow that arrives to our different senses is organized in coherent representations. Some authors claim that pre-attentive detection of inter-sensory correlations supports crossmodal binding, whereas other findings indicate that attention plays a crucial role. We used visual and auditory search tasks for speaking faces to address the role of selective spatial attention in audiovisual binding. Search efficiency amongst faces for the match with a voice declined with the number of faces being monitored concurrently, consistent with an attentive search mechanism. In contrast, search amongst auditory speech streams for the match with a face was independent of the number of streams being monitored concurrently, as long as localization was not required. We suggest that the fundamental differences in the way in which auditory and visual information is encoded play a limiting role in crossmodal binding. Based on these unisensory limitations, we provide a unified explanation for several previous apparently contradictory findings., This work was supported by grants PSI2010-15426 and Consolider INGENIO CSD2007-00012 (MICINN), Generalitat de Catalunya (SRG2009-092), and European Research Council (StG-2010 263145).
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Neural correlates of audiovisual speech processing in a second language

  • Barrós Loscertales, Alfonso
  • Ventura Campos, Noelia
  • Visser, Maya
  • Alsius, Agnès
  • Pallier, Christophe
  • Ávila, César
  • Soto-Faraco, Salvador, 1970-
Neuroimaging studies of audiovisual speech processing have exclusively addressed listeners’ native language (L1). Yet, several behavioural studies now show that AV processing plays an important role in non-native (L2) speech perception. The current fMRI study measured brain activity during auditory, visual, audiovisual congruent and audiovisual incongruent utterances in L1 and L2. BOLD responses to congruent AV speech in the pSTS were stronger than in either unimodal condition in both L1 and L2. Yet no differences in AV processing were expressed according to the language background in this area. Instead, the regions in the bilateral occipital lobe had a stronger congruency effect on the BOLD response (congruent higher than incongruent) in L2 as compared to L1. According to these results, language background differences are predominantly expressed in these unimodal regions, whereas the pSTS is similarly involved in AV integration regardless of language dominance., This research has been supported by the Spanish Ministry of Science and Innovation (PSI2010-15426, PSI2010-20168, and Consolider INGENIO CSD2007-00012), Comissionat per a Universitats i Recerca del DIUE-Generalitat de Catalunya (SRG2009-092), and the European Research Council (StG-2010 263145).
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Theta oscillations reflect conflict processing in the perception of the McGurk illusion

  • Morís Fernández, Luis, 1982-
  • Torralba, Mireia
  • Soto-Faraco, Salvador, 1970-
The McGurk illusion is one of the most famous illustrations of cross‐modal integration in human perception. It has been often used as a proxy of audiovisual (AV) integration and to infer the properties of the integration process in natural (congruent) AV conditions. Nonetheless, a blatant difference between McGurk stimuli and natural, congruent, AV speech is the conflict between the auditory and the visual information in the former. Here, we hypothesized that McGurk stimuli (and any AV incongruency) engage brain responses similar to those found in more general cases of perceptual conflict (e.g., Stroop), and propose that the McGurk illusion arises as a result of the resolution of such conflict. We used electroencephalography to measure variations in the power of theta, a well‐known marker of the brain response to conflict. The results showed that perception of AV McGurk stimuli, just like AV incongruence in general, induces an increase in activity in the theta band. This response was similar to that evoked by Stroop stimuli, as measured in the same participants. This finding suggests that the McGurk illusion is mediated by general‐purpose conflict mechanisms, and calls for caution in generalizing findings obtained using the McGurk illusion, to the general case of multisensory integration., This research was supported by the Ministerio de Economia y Competitividad (PSI2016‐75558‐P AEI/FEDER), AGAUR Generalitat de Catalunya (2014SGR856) and the European Research Council (StG‐2010 263145).
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Isolating shape from semantics in haptic-visual priming

  • Pesquita, Ana
  • Brennan, Allison A.
  • Enns, James T.
  • Soto-Faraco, Salvador, 1970-
The exploration of a familiar object by hand can benefit its identification by eye. What is unclear is how much this multisensory cross-talk reflects shared shape representations versus generic semantic associations. Here, we compare several simultaneous priming conditions to isolate the potential contributions of shape and semantics in haptic-to-visual priming. Participants explored a familiar object manually (haptic prime) while trying to name a visual object that was gradually revealed in increments of spatial resolution. Shape priming was isolated in a comparison of identity priming (shared semantic category and shape) with category priming (same category, but different shapes). Semantic priming was indexed by the comparisons of category priming with unrelated haptic primes. The results showed that both factors mediated priming, but that their relative weights depended on the reliability of the visual information. Semantic priming dominated in Experiment 1, when participants were free to use high-resolution visual information, but shape priming played a stronger role in Experiment 2, when participants were forced to respond with less reliable visual information. These results support the structural description hypothesis of haptic-visual priming (Reales and Ballesteros in J Exp Psychol Learn Mem Cogn 25:644–663, 1999) and are also consistent with the optimal integration theory (Ernst and Banks in Nature 415:429–433, 2002), which proposes a close coupling between the reliability of sensory signals and their weight in decision making., This work was supported by a PhD scholarship to author AP from the Portuguese Fundação para a Ciência e Tecnologia (SFRH/BD/76087/2011), an Exchange Fellowship to author AAB from the Dr. Michael Quinn Memorial Fund, Department of Psychology, University of British Columbia, an NSERC (Canada) Discovery Grant to author JTE, and Grants to author SS-F from the Spanish Ministry of Science and Innovation (PSI2010-15426 and Consolider INGENIO CSD2007-00012 Grants, the Comissionat per a Universitats i Recerca del DIUE-Generalitat de Catalunya (SRG2009- 092), and the European Research Council (StG-2010 263145).
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Electrophysiological correlates of tactile remapping

  • Soto-Faraco, Salvador, 1970-
  • Azañón Gracia, Elena
Orienting our gaze or attention to the location of tactile events in the skin feels natural and effortless. However, this process requires combining somatosensory and proprioceptive information in a non-trivial, time consuming, fashion. Here we address the time course of tactile remapping, from somatotopically-based representations, to a spatiotopic reference frame. We compared electrical responses at the scalp evoked by touch at one finger as participants held their arms straight or else, crossed about the body midline. This postural manipulation creates a conflict between frames of reference, which can be used to reveal the consequences of spatial remapping. Behavioural performance was gauged online for crossed and uncrossed postures using bimanual temporal order judgment (TOJ) trials occurring occasionally during the recording session. The first electrophysiological signs of tactile remapping were observed around 70 ms after the tactile event, overlapping in time with the somatosensory component N80. This electrophysiological effect was strongly lateralized to the left scalp and independent of the hand being stimulated. Moreover, participants who manifested a stronger behavioural conflict between frames of reference in the TOJ task displayed a larger electrophysiological effect. Based on these findings, and the known properties of the somatosensory network, we argue that remapping of tactile space depends on fast feedback projections from association areas of the parietal cortex, and encompasses a left-lateralized fronto-parietal network supporting the selection of guided actions., This research was supported by the Spanish Ministry of Science and Innovation (PSI2010-15426 and Consolider INGENIO CSD2007-00012), Comissionat per a Universitats i Recerca del DIUE-Generalitat de Catalunya (SRG2009-092), and the European Research Council (StG-2010 263145).
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Acoustic facilitation of object movement detection during self-motion

  • Calabro, Finnegan
  • Soto-Faraco, Salvador, 1970-
  • Vaina, Lucia M.
In humans, as well as most animal species, perception of object motion is critical to successful interaction with the surrounding environment. Yet, as the observer also moves, the retinal projections of the various motion components add to each other and extracting accurate object motion becomes computationally challenging. Recent psychophysical studies have demonstrated that observers use a flow-parsing mechanism to estimate and subtract self-motion from the optic flow field. We investigated whether concurrent acoustic cues for motion can facilitate visual flow parsing, thereby enhancing the detection of moving objects during simulated self-motion. Participants identified an object (the target) that moved either forward or backward within a visual scene containing nine identical textured objects simulating forward observer translation. We found that spatially co-localized, directionally congruent, moving auditory stimuli enhanced object motion detection. Interestingly, subjects who performed poorly on the visual-only task benefited more from the addition of moving auditory stimuli. When auditory stimuli were not co-localized to the visual target, improvements in detection rates were weak. Taken together, these results suggest that parsing object motion from self-motion-induced optic flow can operate on multisensory object representations., L.M.V. and F.J.C. were supported by NIH grant RO1NS064100 to L.M.V. S.S.F. was supported by grants from the Ministerio de Ciencia e Innovación (PSI2010-15426 and CSD2007-00012), by the Comissionat per a Universitats i Recerca del DIUE (SRG2009-092) and by the European Research Council (StG-2010 263145).
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

The COGs (context, object, and goals) in multisensory processing

  • ten Oever, Sanne
  • Romei, Vincenzo
  • van Atteveldt, Nienke
  • Soto-Faraco, Salvador, 1970-
  • Murray, Micah M.
  • Matusz, Pawel J.
Our understanding of how perception operates in real-world environments has been substantially advanced by studying both multisensory processes and “top-down” control processes influencing sensory processing via activity from higher-order brain areas, such as attention, memory, and expectations. As the two topics have been traditionally studied separately, the mechanisms orchestrating real-world multisensory processing remain unclear. Past work has revealed that the observer’s goals gate the influence of many multisensory processes on brain and behavioural responses, whereas some other multisensory processes might occur independently of these goals. Consequently, other forms of top-down control beyond goal dependence are necessary to explain the full range of multisensory effects currently reported at the brain and the cognitive level. These forms of control include sensitivity to stimulus context as well as the detection of matches (or lack thereof) between a multisensory stimulus and categorical attributes of naturalistic objects (e.g. tools, animals). In this review we discuss and integrate the existing findings that demonstrate the importance of such goal-, object- and context-based top-down control over multisensory processing. We then put forward a few principles emerging from this literature review with respect to the mechanisms underlying multisensory processing and discuss their possible broader implications., This research was supported by grants from the Ministerio de Economia y Competitividad (PSI2013-42626-P), AGAUR Generalitat de Catalunya (2014SGR856), and the European Research Council (StG-2010 263145) to S.S-F, and the Swiss National Science Foundation (Grant #320030-149982 as well as the National Centre of Competence in Research project “SYNAPSY, The Synaptic Bases of Mental Disease” [Project 51AU40-125759]) and the Swiss Brain League (2014 Research Prize) to MMM. StO receives support from the Dutch Organisation for Scientific Research (Grant 406-11-068).
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

The Interplay between semantic and phonological constraints during spoken-word comprehension

  • Soto-Faraco, Salvador, 1970-
  • Brunellière, Angèle
This study addresses how top-down predictions driven by phonological and semantic information interact on spokenword comprehension. To do so, we measured event-related potentials to words embedded in sentences that varied in the degree of semantic constraint (high or low) and in regional accent (congruent or incongruent) with respect to the target word pronunciation. The data showed a negative amplitude shift following phonological mismatch (target pronunciation incongruent with respect to sentence regional accent). Here, we show that this shift is modulated by sentence-level semantic constraints over latencies encompassing auditory (N100) and lexical (N400) components. These findings suggest a fast influence of top-down predictions and the interplay with bottom-up processes at sublexical and lexical levels of analysis, This research was supported by the Spanish Ministry of Science and Innovation (PSI2010-15426 and Consolider INGENIO CSD2007-00012), Comissionat per a Universitats i Recerca del DIUE-Generalitat de/nCatalunya (SGR2009-092), and the European Research Council (StG-2010263145). ERP analyses were performed with the Cartool software (supported by the Center for Biomedical Imaging of Geneva and Lausanne).
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Top-down attention regulates the neural expression of audiovisual integration

  • Morís Fernández, Luis, 1982-
  • Visser, Maya
  • Ventura Campos, Noelia
  • Ávila, César
  • Soto-Faraco, Salvador, 1970-
The interplay between attention and multisensory integration has proven to be a difficult question to tackle./nThere are almost as many studies showing that multisensory integration occurs independently from the focus/nof attention as studies implying that attention has a profound effect on integration. Addressing the neural/nexpression of multisensory integration for attended vs. unattended stimuli can help disentangle this apparent/ncontradiction. In the present study, we examine if selective attention to sound pitch influences the expression/nof audiovisual integration in both behavior and neural activity. Participants were asked to attend to one of two/nauditory speech streams while watching a pair of talking lips that could be congruent or incongruent with the/nattended speech stream. We measured behavioral and neural responses (fMRI) to multisensory stimuli under/nattended and unattended conditions while physical stimulation was kept constant. Our results indicate that participants/nrecognized words more accurately from an auditory stream that was both attended and audiovisually/n(AV) congruent, thus reflecting a benefit due to AV integration. On the other hand, no enhancement was found/nfor AV congruency when it was unattended. Furthermore, the fMRI results indicated that activity in the superior/ntemporal sulcus (an area known to be related to multisensory integration) was contingent on attention as well as/non audiovisual congruency. This attentional modulation extended beyond heteromodal areas to affect processing/nin areas classically recognized as unisensory, such as the superior temporal gyrus or the extrastriate cortex, and to/nnon-sensory areas such as the motor cortex. Interestingly, attention to audiovisual incongruence triggered responses/nin brain areas related to conflict processing (i.e., the anterior cingulate cortex and the anterior insula)./nBased on these results, we hypothesize that AV speech integration can take place automatically only when/nboth modalities are sufficiently processed, and that if a mismatch is detected between the AV modalities,/nfeedback from conflict areas minimizes the influence of this mismatch by reducing the processing of the least/ninformative modality., This research was supported by the Ministerio de Economía y/nCompetitividad (PSI2013-42626-P), AGAUR Generalitat de Catalunya/n(2014SGR856), and the European Research Council (StG-2010 263145).
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Reversing the Colavita visual dominance effect

  • Ngo, Mary Kim
  • Cadieux, Michelle L.
  • Sinnet, Scott
  • Soto-Faraco, Salvador, 1970-
  • Spence, Charles
Many researchers have taken the Colavita effect to represent a paradigm case of visual dominance. Broadly defined, the effect occurs when people fail to respond to an auditory target if they also have to respond to a visual target presented at the same time. Previous studies have revealed the remarkable resilience of this effect to various manipulations. In fact, a reversal of the Colavita visual dominance effect (i.e., auditory dominance) has never been reported. Here, we present a series of experiments designed to investigate whether it is possible to reverse the Colavita effect when the target stimuli consist of repetitions embedded in simultaneously presented auditory and visual streams of stimuli. In line with previous findings, the Colavita effect was still observed for an immediate repetition task, but when an n-1 repetition detection task was used, a reversal of visual dominance was demonstrated. These results suggest that masking from intervening stimuli between n-1 repetition targets was responsible for the elimination and reversal of the Colavita visual dominance effect. They further suggest that varying the presence of a mask (pattern, conceptual, or absent) in the repetition detection task gives rise to different patterns of sensory dominance (i.e., visual dominance, an elimination of the Colavita effect, or even auditory dominance)., SS-F was supported by grants SEJ 2007-64103/PSIC and CDS00012 from the Spanish Ministerio de Ciencia e Innovación, ERC StG 263145, and grant 2009SGR-292 from DURSI.
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Watching subtitled films can help learning foreign languages

  • Birulés Muntané, Joan
  • Soto-Faraco, Salvador, 1970-
Watching English-spoken films with subtitles is becoming increasingly popular throughout the world. One reason for this trend is the assumption that perceptual learning of the sounds of a foreign language, English, will improve perception skills in non-English speakers. Yet, solid proof for this is scarce. In order to test the potential learning effects derived from watching subtitled media, a group of intermediate Spanish students of English as a foreign language watched a 1h-long episode of a TV drama in its original English version, with English, Spanish or no subtitles overlaid. Before and after the viewing, participants took a listening and vocabulary test to evaluate their speech perception and vocabulary acquisition in English, plus a final plot comprehension test. The results of the listening skills tests revealed that after watching the English subtitled version, participants improved these skills significantly more than after watching the Spanish subtitled or no-subtitles versions. The vocabulary test showed no reliable differences between subtitled conditions. Finally, as one could expect, plot comprehension was best under native, Spanish subtitles. These learning effects with just 1 hour exposure might have major implications with longer exposure times., This research was supported by the/nMinisterio de Economía y Competitividad (PSI2013-42626-P), AGAUR Generalitat de Catalunya/n(2014SGR856), and the European Research Council/n(StG-2010 263145).
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Audiovisual integration as conflict resolution: The conflict of the McGurk illusion

  • Morís Fernández, Luis, 1982-
  • Macaluso, Emiliano
  • Soto-Faraco, Salvador, 1970-
There are two main behavioral expressions of multisensory integration (MSI) in speech; the perceptual enhancement produced by the sight of the congruent lip movements of the speaker, and the illusory sound perceived when a speech syllable is dubbed with incongruent lip movements, in the McGurk effect. These two models have been used very often to study MSI. Here, we contend that, unlike congruent audiovisually (AV) speech, the McGurk effect involves brain areas related to conflict detection and resolution. To test this hypothesis, we used fMRI to measure blood oxygen level dependent responses to AV speech syllables. We analyzed brain activity as a function of the nature of the stimuli—McGurk or non‐McGurk—and the perceptual outcome regarding MSI—integrated or not integrated response—in a 2 × 2 factorial design. The results showed that, regardless of perceptual outcome, AV mismatch activated general‐purpose conflict areas (e.g., anterior cingulate cortex) as well as specific AV speech conflict areas (e.g., inferior frontal gyrus), compared with AV matching stimuli. Moreover, these conflict areas showed stronger activation on trials where the McGurk illusion was perceived compared with non‐illusory trials, despite the stimuli where physically identical. We conclude that the AV incongruence in McGurk stimuli triggers the activation of conflict processing areas and that the process of resolving the cross‐modal conflict is critical for the McGurk illusion to arise. Hum Brain Mapp 38:5691–5705, 2017., This research was supported by the Ministerio de Economía y Competitividad (PSI2016–75558‐PAEI/FEDER), AGAUR Generalitat de Catalunya (2014SGR856 and 2012BE100392), and the European Research Council (StG‐2010 263145). The Neuroimaging Laboratory of the Fondazione Santa Lucia is supported by the Italian Ministry of Health. EM is supported by the program “Investissements d'Avenir” (ANR‐11‐IDEX‐0007) and by the program “BQR Accueil EC 16” of University Claude Bernard Lyon 1.
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Cross-modal decoupling in temporal attention between audition and touch

  • Mühlberg, Stefanie, 1986-
  • Soto-Faraco, Salvador, 1970-
Temporal orienting leads to well-documented behavioural benefits for sensory events occurring at the anticipated moment. However, the consequences of temporal orienting in cross-modal contexts are still unclear. On the one hand, some studies using audio-tactile paradigms suggest that attentional orienting in time and modality are a closely coupled system, in which temporal orienting dominates modality orienting, similar to what happens in cross-modal spatial attention. On the other hand, recent findings using a visuo-tactile paradigm suggest that attentional orienting in time can unfold independently in each modality, leading to cross-modal decoupling. In the present study, we investigated if cross-modal decoupling in time can be extrapolated to audio-tactile contexts. If so, decoupling might represent a general property of cross-modal attention in time. To this end, we used a speeded discrimination task in which we manipulated the probability of target presentation in time and modality. In each trial, a manipulation of time-based expectancy was used to guide participants’ attention to task-relevant events, either tactile or auditory, at different points in time. In two experiments, we found that participants generally showed enhanced behavioural performance at the most likely onset time of each modality and no evidence for coupling. This pattern supports the hypothesis that cross-modal decoupling could be a general phenomenon in temporal orienting., This research was funded by the Spanish Ministry of Science and Innovation (PSI2016-75558-P(AEI/FEDER)), the Comissionat per a Universitats i Recerca del DIUE-Generalitat de Catalunya (2014SGR856), and the European Research Council (StG-2010 263145) to S.S.F.
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Beat gestures modulate auditory integration in speech perception

  • Biau, Emmanuel, 1985-
  • Soto-Faraco, Salvador, 1970-
Spontaneous beat gestures are an integral part of the paralinguistic context during face-to-face conversations. Here we investigated the time course of beat-speech integration in speech perception by measuring ERPs evoked by words pronounced with or without an accompanying beat gesture, while participants watched a spoken discourse. Words accompanied by beats elicited a positive shift in ERPs at an early sensory stage (before 100 ms) and at a later time window coinciding with the auditory component P2. The same word tokens produced no ERP differences when participants listened to the discourse without view of the speaker. We conclude that beat gestures are integrated with speech early on in time and modulate sensory/phonological levels of processing. The present results support the possible role of beats as a highlighter, helping the listener to direct the focus of attention to important information and modulate the parsing of the speech stream., This research was supported by the Spanish Ministry of Science and Innovation (PSI2010-15426 and Consolider INGENIO CSD2007-00012), Comissionat per a Universitats i Recerca del DIUE-Generalitat de Catalunya (SRG2009-092), and the European Research Council (StG-2010 263145).
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

The speakers’ accent shapes the listeners’ phonological predictions during speech perception

  • Brunellière, Angèle
  • Soto-Faraco, Salvador, 1970-
This study investigates the specificity of predictive coding in spoken word comprehension using event-related potentials (ERPs). We measured word-evoked ERPs in Catalan speakers listening to semantically constraining sentences produced in their native regional accent (Experiment 1) or in a non-native accent (Experiment 2). Semantically anomalous words produced long-lasting negative shift (N400) starting as early as 250 ms, thus reflecting phonological as well as semantic mismatch. Semantically expected but phonologically unexpected (non-native forms embedded in a native context) produced only an early (∼250 ms) negative difference. In contrast, this phonological expectancy effect failed for native albeit phonologically unexpected target words embedded in a non-native context. These results suggest phonologically precise expectations when operating over native input, whilst phonologically less specified expectations in a non-native context. Our findings shed light on contextual influence during word recognition, suggesting that word form prediction based on context is sensitive and adaptive to phonological variability., his research was supported by the Spanish Ministry of Science and Innovation (PSI2010-15426 and Consolider INGENIO CSD2007-00012), Comissionat per a Universitats i Recerca del DIUE-Generalitat de Catalunya (SGR2009-092), and the European Research Council (StG-2010263145).
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Synchronization by the hand: the sight of gestures modulates low-frequency activity in brain responses to continuous speech

  • Biau, Emmanuel, 1985-
  • Soto-Faraco, Salvador, 1970-
During social interactions, speakers often produce spontaneous gestures to accompany their speech. These coordinated body movements convey communicative intentions, and modulate how listeners perceive the message in a subtle, but important way. In the present perspective, we put the focus on the role that congruent non-verbal information from beat gestures may play in the neural responses to speech. Whilst delta-theta oscillatory brain responses reflect the time-frequency structure of the speech signal, we argue that beat gestures promote phase resetting at relevant word onsets. This mechanism may facilitate the anticipation of associated acoustic cues relevant for prosodic/syllabic-based segmentation in speech perception. We report recently published data supporting this hypothesis, and discuss the potential of beats (and gestures in general) for further studies investigating continuous AV speech processing through low-frequency oscillations., This research was supported by the Spanish Ministry of Science and Innovation (PSI2013-42626-P), AGAUR Generalitat de Catalunya (2014SGR856) and the European Research Council (StG-2010 263145).
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Theta oscillations reflect conflict processing in the perception of the McGurk illusion

  • Morís Fernández, Luis, 1982-
  • Torralba, Mireia
  • Soto-Faraco, Salvador, 1970-
The McGurk illusion is one of the most famous illustrations of cross‐modal integration in human perception. It has been often used as a proxy of audiovisual (AV) integration and to infer the properties of the integration process in natural (congruent) AV conditions. Nonetheless, a blatant difference between McGurk stimuli and natural, congruent, AV speech is the conflict between the auditory and the visual information in the former. Here, we hypothesized that McGurk stimuli (and any AV incongruency) engage brain responses similar to those found in more general cases of perceptual conflict (e.g., Stroop), and propose that the McGurk illusion arises as a result of the resolution of such conflict. We used electroencephalography to measure variations in the power of theta, a well‐known marker of the brain response to conflict. The results showed that perception of AV McGurk stimuli, just like AV incongruence in general, induces an increase in activity in the theta band. This response was similar to that evoked by Stroop stimuli, as measured in the same participants. This finding suggests that the McGurk illusion is mediated by general‐purpose conflict mechanisms, and calls for caution in generalizing findings obtained using the McGurk illusion, to the general case of multisensory integration., This research was supported by the Ministerio de Economia y Competitividad (PSI2016‐75558‐P AEI/FEDER), AGAUR Generalitat de Catalunya (2014SGR856) and the European Research Council (StG‐2010 263145).
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Audiovisual integration as conflict resolution: The conflict of the McGurk illusion

  • Morís Fernández, Luis, 1982-
  • Macaluso, Emiliano
  • Soto-Faraco, Salvador, 1970-
There are two main behavioral expressions of multisensory integration (MSI) in speech; the perceptual enhancement produced by the sight of the congruent lip movements of the speaker, and the illusory sound perceived when a speech syllable is dubbed with incongruent lip movements, in the McGurk effect. These two models have been used very often to study MSI. Here, we contend that, unlike congruent audiovisually (AV) speech, the McGurk effect involves brain areas related to conflict detection and resolution. To test this hypothesis, we used fMRI to measure blood oxygen level dependent responses to AV speech syllables. We analyzed brain activity as a function of the nature of the stimuli—McGurk or non‐McGurk—and the perceptual outcome regarding MSI—integrated or not integrated response—in a 2 × 2 factorial design. The results showed that, regardless of perceptual outcome, AV mismatch activated general‐purpose conflict areas (e.g., anterior cingulate cortex) as well as specific AV speech conflict areas (e.g., inferior frontal gyrus), compared with AV matching stimuli. Moreover, these conflict areas showed stronger activation on trials where the McGurk illusion was perceived compared with non‐illusory trials, despite the stimuli where physically identical. We conclude that the AV incongruence in McGurk stimuli triggers the activation of conflict processing areas and that the process of resolving the cross‐modal conflict is critical for the McGurk illusion to arise. Hum Brain Mapp 38:5691–5705, 2017., This research was supported by the Ministerio de Economía y Competitividad (PSI2016–75558‐PAEI/FEDER), AGAUR Generalitat de Catalunya (2014SGR856 and 2012BE100392), and the European Research Council (StG‐2010 263145). The Neuroimaging Laboratory of the Fondazione Santa Lucia is supported by the Italian Ministry of Health. EM is supported by the program “Investissements d'Avenir” (ANR‐11‐IDEX‐0007) and by the program “BQR Accueil EC 16” of University Claude Bernard Lyon 1.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Sounds can boost the awareness of visual events through attention without cross-modal integration

  • Soto-Faraco, Salvador, 1970-
  • Pápai, Márta Szabina, 1987-
Cross-modal interactions can lead to enhancement of visual perception, even for visual events below awareness. However, the underlying mechanism is still unclear. Can purely bottom-up cross-modal integration break through the threshold of awareness? We used a binocular rivalry paradigm to measure perceptual switches after brief flashes or sounds which, sometimes, co-occurred. When flashes at the suppressed eye coincided with sounds, perceptual switches occurred the earliest. Yet, contrary to the hypothesis of cross-modal integration, this facilitation never surpassed the assumption of probability summation of independent sensory signals. A follow-up experiment replicated the same pattern of results using silent gaps embedded in continuous noise, instead of sounds. This manipulation should weaken putative sound-flash integration, although keep them salient as bottom-up attention cues. Additional results showed that spatial congruency between flashes and sounds did not determine the effectiveness of cross-modal facilitation, which was again not better than probability summation. Thus, the present findings fail to fully support the hypothesis of bottom-up cross-modal integration, above and beyond the independent contribution of two transient signals, as an account for cross-modal enhancement of visual events below level of awareness., This study was funded by a Ministerio de Educación, Cultura y Deporte, Ref. FPU/1206471 to MSP, and ERC Starting Grant, Ref. 263145, Spanish Ministry of Science and Innovation (PSI2016-75558-P), and Comissionat per a Universitats i Recerca del DIUE- Generalitat de Catalunya (2014SGR856) to SS-F.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

The COGs (context, object, and goals) in multisensory processing

  • ten Oever, Sanne
  • Romei, Vincenzo
  • van Atteveldt, Nienke
  • Soto-Faraco, Salvador, 1970-
  • Murray, Micah M.
  • Matusz, Pawel J.
Our understanding of how perception operates in real-world environments has been substantially advanced by studying both multisensory processes and “top-down” control processes influencing sensory processing via activity from higher-order brain areas, such as attention, memory, and expectations. As the two topics have been traditionally studied separately, the mechanisms orchestrating real-world multisensory processing remain unclear. Past work has revealed that the observer’s goals gate the influence of many multisensory processes on brain and behavioural responses, whereas some other multisensory processes might occur independently of these goals. Consequently, other forms of top-down control beyond goal dependence are necessary to explain the full range of multisensory effects currently reported at the brain and the cognitive level. These forms of control include sensitivity to stimulus context as well as the detection of matches (or lack thereof) between a multisensory stimulus and categorical attributes of naturalistic objects (e.g. tools, animals). In this review we discuss and integrate the existing findings that demonstrate the importance of such goal-, object- and context-based top-down control over multisensory processing. We then put forward a few principles emerging from this literature review with respect to the mechanisms underlying multisensory processing and discuss their possible broader implications., This research was supported by grants from the Ministerio de Economia y Competitividad (PSI2013-42626-P), AGAUR Generalitat de Catalunya (2014SGR856), and the European Research Council (StG-2010 263145) to S.S-F, and the Swiss National Science Foundation (Grant #320030-149982 as well as the National Centre of Competence in Research project “SYNAPSY, The Synaptic Bases of Mental Disease” [Project 51AU40-125759]) and the Swiss Brain League (2014 Research Prize) to MMM. StO receives support from the Dutch Organisation for Scientific Research (Grant 406-11-068).
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

The speakers’ accent shapes the listeners’ phonological predictions during speech perception

  • Brunellière, Angèle
  • Soto-Faraco, Salvador, 1970-
This study investigates the specificity of predictive coding in spoken word comprehension using event-related potentials (ERPs). We measured word-evoked ERPs in Catalan speakers listening to semantically constraining sentences produced in their native regional accent (Experiment 1) or in a non-native accent (Experiment 2). Semantically anomalous words produced long-lasting negative shift (N400) starting as early as 250 ms, thus reflecting phonological as well as semantic mismatch. Semantically expected but phonologically unexpected (non-native forms embedded in a native context) produced only an early (∼250 ms) negative difference. In contrast, this phonological expectancy effect failed for native albeit phonologically unexpected target words embedded in a non-native context. These results suggest phonologically precise expectations when operating over native input, whilst phonologically less specified expectations in a non-native context. Our findings shed light on contextual influence during word recognition, suggesting that word form prediction based on context is sensitive and adaptive to phonological variability., his research was supported by the Spanish Ministry of Science and Innovation (PSI2010-15426 and Consolider INGENIO CSD2007-00012), Comissionat per a Universitats i Recerca del DIUE-Generalitat de Catalunya (SGR2009-092), and the European Research Council (StG-2010263145).
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Somatosensory saccades reveal the timing of tactile spatial remapping

  • Overvliet, Krista E.
  • Azañón Gracia, Elena
  • Soto-Faraco, Salvador, 1970-
Remapping tactile events from skin to external space is an essential process for human behaviour. It allows us to refer tactile sensations to their actual externally based location, by combining anatomically based somatosensory information with proprioceptive information about the current body posture. We examined the time course of tactile remapping by recording speeded saccadic responses to somatosensory stimuli delivered to the hands. We conducted two experiments in which arm posture varied (crossed or uncrossed), so that anatomical and external frames of reference were either put in spatial conflict or were aligned. The data showed that saccade onset latencies in the crossed hands conditions were slower than in the uncrossed hands condition, suggesting that, in the crossed hands condition, remapping had to be completed before a correct saccade could be executed. Saccades to tactile stimuli when the hands were crossed were sometimes initiated to the wrong direction and then corrected in-flight, resulting in a turn-around saccade. These turn-around saccades were more likely to occur in short-latency responses, compared to onset latencies of saccades that went straight to target. The latter suggests that participants were postponing their saccade until the time the tactile event was represented according to the current body posture. We propose that the difference between saccade onset latencies of crossed and uncrossed hand postures, and between the onset of a turn-around saccade and a straight saccade in the crossed hand posture, reveal the timing of tactile spatial remapping., This research was supported by Spanish Ministry of Science and Innovation (PSI2010-15426 and Consolider INGENIOCSD2007-00012), Comissionat per a Universitats i Recerca del DIUE-Generalitat de Catalunya (SRG2009-092), and European Research Council (StG-2010 263145).
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Isolating shape from semantics in haptic-visual priming

  • Pesquita, Ana
  • Brennan, Allison A.
  • Enns, James T.
  • Soto-Faraco, Salvador, 1970-
The exploration of a familiar object by hand can benefit its identification by eye. What is unclear is how much this multisensory cross-talk reflects shared shape representations versus generic semantic associations. Here, we compare several simultaneous priming conditions to isolate the potential contributions of shape and semantics in haptic-to-visual priming. Participants explored a familiar object manually (haptic prime) while trying to name a visual object that was gradually revealed in increments of spatial resolution. Shape priming was isolated in a comparison of identity priming (shared semantic category and shape) with category priming (same category, but different shapes). Semantic priming was indexed by the comparisons of category priming with unrelated haptic primes. The results showed that both factors mediated priming, but that their relative weights depended on the reliability of the visual information. Semantic priming dominated in Experiment 1, when participants were free to use high-resolution visual information, but shape priming played a stronger role in Experiment 2, when participants were forced to respond with less reliable visual information. These results support the structural description hypothesis of haptic-visual priming (Reales and Ballesteros in J Exp Psychol Learn Mem Cogn 25:644–663, 1999) and are also consistent with the optimal integration theory (Ernst and Banks in Nature 415:429–433, 2002), which proposes a close coupling between the reliability of sensory signals and their weight in decision making., This work was supported by a PhD scholarship to author AP from the Portuguese Fundação para a Ciência e Tecnologia (SFRH/BD/76087/2011), an Exchange Fellowship to author AAB from the Dr. Michael Quinn Memorial Fund, Department of Psychology, University of British Columbia, an NSERC (Canada) Discovery Grant to author JTE, and Grants to author SS-F from the Spanish Ministry of Science and Innovation (PSI2010-15426 and Consolider INGENIO CSD2007-00012 Grants, the Comissionat per a Universitats i Recerca del DIUE-Generalitat de Catalunya (SRG2009- 092), and the European Research Council (StG-2010 263145).
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Discriminating speech rhythms in audition, vision, and touch

  • Navarra, Jordi
  • Soto-Faraco, Salvador, 1970-
  • Spence, Charles
We investigated the extent to which people can discriminate between languages on the basis of their characteristic temporal, rhythmic information, and the extent to which this ability generalizes across sensory modalities. We used rhythmical patterns derived from the alternation of vowels and consonants in English and Japanese, presented in audition, vision, both audition and vision at the same time, or touch. Experiment 1 confirmed that discrimination is possible on the basis of auditory rhythmic patterns, and extended it to the case of vision, using ‘aperture-close’ mouth movements of a schematic face. In Experiment 2, language discrimination was demonstrated using visual and auditory materials that did not resemble spoken articulation. In a combined analysis including data from Experiments 1 and 2, a beneficial effect was also found when the auditory rhythmic information was available to participants. Despite the fact that discrimination could be achieved using vision alone, auditory performance was nevertheless better. In a final experiment, we demonstrate that the rhythm of speech can also be discriminated successfully by means of vibrotactile patterns delivered to the fingertip. The results of the present study therefore demonstrate that discrimination between language's syllabic rhythmic patterns is possible on the basis of visual and tactile displays., This research was supported by grants PSI2012-39149, PSI2009-12859 and RYC-2008-03672 from Ministerio de Economía y Competitividad (MINECO, Spain) to J. Navarra, the European COST action TD0904, and by grants PSI2010-15426 and CDS00012 from MINECO (Spain), 2009SGR-292 from Comissionat per a Universitats i Recerca del DIUE-Generalitat de Catalunya, and European Research Council (StG-2010 263145) to S. Soto-Faraco.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Watching subtitled films can help learning foreign languages

  • Birulés Muntané, Joan
  • Soto-Faraco, Salvador, 1970-
Watching English-spoken films with subtitles is becoming increasingly popular throughout the world. One reason for this trend is the assumption that perceptual learning of the sounds of a foreign language, English, will improve perception skills in non-English speakers. Yet, solid proof for this is scarce. In order to test the potential learning effects derived from watching subtitled media, a group of intermediate Spanish students of English as a foreign language watched a 1h-long episode of a TV drama in its original English version, with English, Spanish or no subtitles overlaid. Before and after the viewing, participants took a listening and vocabulary test to evaluate their speech perception and vocabulary acquisition in English, plus a final plot comprehension test. The results of the listening skills tests revealed that after watching the English subtitled version, participants improved these skills significantly more than after watching the Spanish subtitled or no-subtitles versions. The vocabulary test showed no reliable differences between subtitled conditions. Finally, as one could expect, plot comprehension was best under native, Spanish subtitles. These learning effects with just 1 hour exposure might have major implications with longer exposure times., This research was supported by the/nMinisterio de Economía y Competitividad (PSI2013-42626-P), AGAUR Generalitat de Catalunya/n(2014SGR856), and the European Research Council/n(StG-2010 263145).
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Searching for audiovisual correspondence in multiple speaker scenarios

  • Alsius, Agnès
  • Soto-Faraco, Salvador, 1970-
A critical question in multisensory processing is how the constant information flow that arrives to our different senses is organized in coherent representations. Some authors claim that pre-attentive detection of inter-sensory correlations supports crossmodal binding, whereas other findings indicate that attention plays a crucial role. We used visual and auditory search tasks for speaking faces to address the role of selective spatial attention in audiovisual binding. Search efficiency amongst faces for the match with a voice declined with the number of faces being monitored concurrently, consistent with an attentive search mechanism. In contrast, search amongst auditory speech streams for the match with a face was independent of the number of streams being monitored concurrently, as long as localization was not required. We suggest that the fundamental differences in the way in which auditory and visual information is encoded play a limiting role in crossmodal binding. Based on these unisensory limitations, we provide a unified explanation for several previous apparently contradictory findings., This work was supported by grants PSI2010-15426 and Consolider INGENIO CSD2007-00012 (MICINN), Generalitat de Catalunya (SRG2009-092), and European Research Council (StG-2010 263145).
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Cross-modal decoupling in temporal attention

  • Mühlberg, Stefanie, 1986-
  • Oriolo, Giovanni
  • Soto-Faraco, Salvador, 1970-
Prior studies have repeatedly reported behavioural benefits to events occurring at attended, compared to unattended, points in time. It has been suggested that, as for spatial orienting, temporal orienting of attention spreads across sensory modalities in a synergistic fashion. However, the consequences of cross-modal temporal orienting of attention remain poorly understood. One challenge is that the passage of time leads to an increase in event predictability throughout a trial, thus making it difficult to interpret possible effects (or lack thereof). Here we used a design that avoids complete temporal predictability to investigate whether attending to a sensory modality (vision or touch) at a point in time confers beneficial access to events in the other, non-attended, sensory modality (touch or vision, respectively). In contrast to previous studies and to what happens with spatial attention, we found that events in one (unattended) modality do not automatically benefit from happening at the time point when another modality is expected. Instead, it seems that attention can be deployed in time with relative independence for different sensory modalities. Based on these findings, we argue that temporal orienting of attention can be cross-modally decoupled in order to flexibly react according to the environmental demands, and that the efficiency of this selective decoupling unfolds in time., This research was funded by the Spanish Ministry of Science and Innovation (PSI2010-15426), the Comissionat per a Universitats i Recerca del DIUE-Generalitat de Catalunya (SRG2009-092) and the European Research Council (StG-2010 263145) to S.S.F.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Top-down attention regulates the neural expression of audiovisual integration

  • Morís Fernández, Luis, 1982-
  • Visser, Maya
  • Ventura Campos, Noelia
  • Ávila, César
  • Soto-Faraco, Salvador, 1970-
The interplay between attention and multisensory integration has proven to be a difficult question to tackle./nThere are almost as many studies showing that multisensory integration occurs independently from the focus/nof attention as studies implying that attention has a profound effect on integration. Addressing the neural/nexpression of multisensory integration for attended vs. unattended stimuli can help disentangle this apparent/ncontradiction. In the present study, we examine if selective attention to sound pitch influences the expression/nof audiovisual integration in both behavior and neural activity. Participants were asked to attend to one of two/nauditory speech streams while watching a pair of talking lips that could be congruent or incongruent with the/nattended speech stream. We measured behavioral and neural responses (fMRI) to multisensory stimuli under/nattended and unattended conditions while physical stimulation was kept constant. Our results indicate that participants/nrecognized words more accurately from an auditory stream that was both attended and audiovisually/n(AV) congruent, thus reflecting a benefit due to AV integration. On the other hand, no enhancement was found/nfor AV congruency when it was unattended. Furthermore, the fMRI results indicated that activity in the superior/ntemporal sulcus (an area known to be related to multisensory integration) was contingent on attention as well as/non audiovisual congruency. This attentional modulation extended beyond heteromodal areas to affect processing/nin areas classically recognized as unisensory, such as the superior temporal gyrus or the extrastriate cortex, and to/nnon-sensory areas such as the motor cortex. Interestingly, attention to audiovisual incongruence triggered responses/nin brain areas related to conflict processing (i.e., the anterior cingulate cortex and the anterior insula)./nBased on these results, we hypothesize that AV speech integration can take place automatically only when/nboth modalities are sufficiently processed, and that if a mismatch is detected between the AV modalities,/nfeedback from conflict areas minimizes the influence of this mismatch by reducing the processing of the least/ninformative modality., This research was supported by the Ministerio de Economía y/nCompetitividad (PSI2013-42626-P), AGAUR Generalitat de Catalunya/n(2014SGR856), and the European Research Council (StG-2010 263145).
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Modality-switching in the Simon task: The clash of reference frames

  • Ruzzoli, Manuela
  • Soto-Faraco, Salvador, 1970-
The representation of spatial information related to an event can influence behavior even when location is task-irrelevant, as in the case of Stimulus–Response (S-R) compatibility effects on the Simon task. However, unlike single-modality situations, which are often used to study the Simon effect, in real-life scenarios various sensory modalities provide spatial information coded in different coordinate systems. Here, we address the expression of S-R compatibility effects in mixed-modality contexts, where events can occur in 1 of various sensory modalities (i.e., vision, touch or audition). The results confirm that, in single-modality cases, Simon effects in vision are expressed in an external spatial frame of reference, while touch information is coded anatomically. Remarkably, when mixing visual and tactile trials in an unpredictable way, the Simon effect disappeared in vision whereas tactile Simon effects remained expressed in their own (anatomical) frame of reference. Mixing visual and auditory stimuli did not obliterate the visual Simon effect and S-R compatibility effects in an external reference frame were evident for both modalities. The extinction of visual Simon effects as a result of mixing visual and tactile modalities can be interpreted as a consequence of the dynamic reorganization of the weights associated to the different sources of spatial information at play., This research was supported by the Ministerio de Economia y Competitividad (PSI2016-75558-P), the AGAUR Generalitat de Catalunya (2014SGR856) and the European Research Council (StG-2010 263145) to SSF. M.R. was supported by the Juan de la Cierva postdoctoral fellowship (JCI-2012-12335) from the Ministerio de Economia y Competitividad.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Beat gestures modulate auditory integration in speech perception

  • Biau, Emmanuel, 1985-
  • Soto-Faraco, Salvador, 1970-
Spontaneous beat gestures are an integral part of the paralinguistic context during face-to-face conversations. Here we investigated the time course of beat-speech integration in speech perception by measuring ERPs evoked by words pronounced with or without an accompanying beat gesture, while participants watched a spoken discourse. Words accompanied by beats elicited a positive shift in ERPs at an early sensory stage (before 100 ms) and at a later time window coinciding with the auditory component P2. The same word tokens produced no ERP differences when participants listened to the discourse without view of the speaker. We conclude that beat gestures are integrated with speech early on in time and modulate sensory/phonological levels of processing. The present results support the possible role of beats as a highlighter, helping the listener to direct the focus of attention to important information and modulate the parsing of the speech stream., This research was supported by the Spanish Ministry of Science and Innovation (PSI2010-15426 and Consolider INGENIO CSD2007-00012), Comissionat per a Universitats i Recerca del DIUE-Generalitat de Catalunya (SRG2009-092), and the European Research Council (StG-2010 263145).
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Acoustic facilitation of object movement detection during self-motion

  • Calabro, Finnegan
  • Soto-Faraco, Salvador, 1970-
  • Vaina, Lucia M.
In humans, as well as most animal species, perception of object motion is critical to successful interaction with the surrounding environment. Yet, as the observer also moves, the retinal projections of the various motion components add to each other and extracting accurate object motion becomes computationally challenging. Recent psychophysical studies have demonstrated that observers use a flow-parsing mechanism to estimate and subtract self-motion from the optic flow field. We investigated whether concurrent acoustic cues for motion can facilitate visual flow parsing, thereby enhancing the detection of moving objects during simulated self-motion. Participants identified an object (the target) that moved either forward or backward within a visual scene containing nine identical textured objects simulating forward observer translation. We found that spatially co-localized, directionally congruent, moving auditory stimuli enhanced object motion detection. Interestingly, subjects who performed poorly on the visual-only task benefited more from the addition of moving auditory stimuli. When auditory stimuli were not co-localized to the visual target, improvements in detection rates were weak. Taken together, these results suggest that parsing object motion from self-motion-induced optic flow can operate on multisensory object representations., L.M.V. and F.J.C. were supported by NIH grant RO1NS064100 to L.M.V. S.S.F. was supported by grants from the Ministerio de Ciencia e Innovación (PSI2010-15426 and CSD2007-00012), by the Comissionat per a Universitats i Recerca del DIUE (SRG2009-092) and by the European Research Council (StG-2010 263145).
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Reversing the Colavita visual dominance effect

  • Ngo, Mary Kim
  • Cadieux, Michelle L.
  • Sinnet, Scott
  • Soto-Faraco, Salvador, 1970-
  • Spence, Charles
Many researchers have taken the Colavita effect to represent a paradigm case of visual dominance. Broadly defined, the effect occurs when people fail to respond to an auditory target if they also have to respond to a visual target presented at the same time. Previous studies have revealed the remarkable resilience of this effect to various manipulations. In fact, a reversal of the Colavita visual dominance effect (i.e., auditory dominance) has never been reported. Here, we present a series of experiments designed to investigate whether it is possible to reverse the Colavita effect when the target stimuli consist of repetitions embedded in simultaneously presented auditory and visual streams of stimuli. In line with previous findings, the Colavita effect was still observed for an immediate repetition task, but when an n-1 repetition detection task was used, a reversal of visual dominance was demonstrated. These results suggest that masking from intervening stimuli between n-1 repetition targets was responsible for the elimination and reversal of the Colavita visual dominance effect. They further suggest that varying the presence of a mask (pattern, conceptual, or absent) in the repetition detection task gives rise to different patterns of sensory dominance (i.e., visual dominance, an elimination of the Colavita effect, or even auditory dominance)., SS-F was supported by grants SEJ 2007-64103/PSIC and CDS00012 from the Spanish Ministerio de Ciencia e Innovación, ERC StG 263145, and grant 2009SGR-292 from DURSI.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Speaker's hand gestures modulate speech perception through phase resetting of ongoing neural oscillations

  • Biau, Emmanuel, 1985-
  • Torralba, Mireia
  • Fuentemilla, Lluis
  • de Diego-Balaguer, Ruth
  • Soto-Faraco, Salvador, 1970-
Speakers often accompany speech with spontaneous beat gestures in natural spoken communication. These gestures are usually aligned with lexical stress and can modulate the saliency of their affiliate words. Here we addressed the consequences of beat gestures on the neural correlates of speech perception. Previous studies have highlighted the role played by theta oscillations in temporal prediction of speech. We hypothesized that the sight of beat gestures may influence ongoing low-frequency neural oscillations around the onset of the corresponding words. Electroencephalographic (EEG) recordings were acquired while participants watched a continuous, naturally recorded discourse. The phase-locking/nvalue (PLV) at word onset was calculated from the EEG from pairs of identical words that had been pronounced with and without a concurrent beat gesture in the discourse. We observed an increase in PLV in the 5e6 Hz theta range as well as a desynchronization in the 8e10 Hz alpha band around the onset of words preceded by a beat gesture. These findings suggest that beats help tune low-frequency oscillatory activity at relevant moments during natural speech perception, providing a new insight of how speech and paralinguistic information are integrated., This research was supported by the Spanish Ministry of Science and Innovation (PSI2013-42626-P), AGAUR Generalitat de Catalunya (2014SGR856) and, the European Research Council (StG-2010 263145).
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Sound-driven enhancement of vision: disentangling detection from decisional level contributions

  • Pérez-Bellido, Alexis
  • Soto-Faraco, Salvador, 1970-
  • López-Moliner, Joan
Cross-modal enhancement can be mediated both by higher-order effects due to attention and decision making and by detection-level stimulus-driven interactions. However, the contribution of each of these sources to behavioral improvements has not been conclusively determined and quantified separately. Here, we apply psychophysical analysis based on Piéron functions in order to separate stimulus-dependent changes from those accounted by decision-level contributions. Participants performed a simple visual speeded detection task on Gabor patches of different spatial frequencies and contrast values, presented with and without accompanying sounds. On one hand, we identified an additive cross-modal improvement in mean reaction times across all types of visual stimuli that would be well explained by interactions not strictly based on stimulus-driven modulations (e.g., due to reduction of temporal uncertainty and motor times). On the other hand, we singled out an audio-visual benefit that strongly depended on stimulus features such as frequency and contrast. This particular enhancement was selective to low-visual spatial frequency stimuli, optimized for magnocellular sensitivity. We therefore conclude that interactions at detection stages and at decisional processes in response selection that contribute to audio-visual enhancement can be separated online and express on partly different aspects of visual processing., This research was supported by the Spanish Ministry of Science and Innovation (PSI2010-15867, PSI2010-15426, and Consolider INGENIO CSD2007-00012), Comissionat per a Universitats i Recerca del DIUE-Generalitat de Catalunya (SGR2009-092, SGR2009-308), and the European Research Council (StG-2010263145).
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Hand gestures as visual prosody: BOLD responses to audio–visual alignment are modulated by the communicative nature of the stimuli

  • Biau, Emmanuel, 1985-
  • Morís Fernández, Luis, 1982-
  • Holle, Henning
  • Ávila, César
  • Soto-Faraco, Salvador, 1970-
During public addresses, speakers accompany their discourse with spontaneous hand gestures (beats) that are tightly synchronized with the prosodic contour of the discourse. It has been proposed that speech and beat gestures originate from a common underlying linguistic process whereby both speech prosody and beats serve to emphasize relevant information. We hypothesized that breaking the consistency between beats and prosody by temporal desynchronization, would modulate activity of brain areas sensitive to speech–gesture integration. To this aim, we measured BOLD responses as participants watched a natural discourse where the speaker used beat gestures./nIn order to identify brain areas specifically involved in processing hand gestures with communicative intention, beat synchrony was evaluated against arbitrary visual cues bearing equivalent rhythmic and spatial properties as the gestures. Our results revealed that left MTG and IFG were specifically sensitive to speech synchronized with beats, compared to the arbitrary vision–speech pairing. Our results suggest that listeners confer beats a function of visual prosody, complementary to the prosodic structure of speech. We conclude that the emphasizing function of beat gestures in speech perception is instantiated through a specialized brain network sensitive to the communicative intent conveyed by a speaker with his/her hands., This research was supported by the Ministerio de Economia y Competitividad (PSI2013-42626-P), AGAUR Generalitat de Catalunya (2014SGR856), and the European Research Council (StG-2010 263145).
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Visual information constrains early and late stages of spoken-word recognition in sentence context

  • Brunellière, Angèle
  • Sánchez García, Carolina, 1984-
  • Ikumi Montserrat, Nara, 1986-
  • Soto-Faraco, Salvador, 1970-
Audiovisual speech perception has been frequently studied considering phoneme, syllable and word processing levels. Here, we examined the constraints that visual speech information might exert during the recognition of words embedded in a natural sentence context. We recorded event-related potentials (ERPs) to words that could be either strongly or weakly predictable on the basis of the prior semantic sentential context and, whose initial phoneme varied in the degree of visual saliency from lip movements. When the sentences were presented audio-visually (Experiment 1), words weakly predicted from semantic context elicited a larger long-lasting N400, compared to strongly predictable words. This semantic effect interacted with the degree of visual saliency over a late part of the N400. When comparing audio-visual versus auditory alone presentation (Experiment 2), the typical amplitude-reduction effect over the auditory-evoked N100 response was observed in the audiovisual modality. Interestingly, a specific benefit of high- versus low-visual saliency constraints occurred over the early N100 response and at the late N400 time window, confirming the result of Experiment 1. Taken together, our results indicate that the saliency of visual speech can exert an influence over both auditory processing and word recognition at relatively late stages, and thus suggest strong interactivity between audio-visual integration and other (arguably higher) stages of information processing during natural speech comprehension., This research was supported by the Spanish Ministry of Science and Innovation (PSI2010-15426 and Consolider INGENIO CSD2007-00012), Comissionat per a Universitats i Recerca del DIUE-Generalitat de Catalunya (SGR2009-092), and the European Research Council (StG-2010263145).
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Age-related sensitive periods influence visual language discrimination in adults

  • Weikum, Whitney M.
  • Vouloumanos, Athena
  • Navarra, Jordi
  • Soto-Faraco, Salvador, 1970-
  • Sebastián Gallés, Núria
  • Werker, Janet F.
Adults as well as infants have the capacity to discriminate languages based on visual speech alone. Here, we investigated whether adults' ability to discriminate languages based on visual speech cues is influenced by the age of language acquisition. Adult participants who had all learned English (as a first or second language) but did not speak French were shown faces of bilingual (French/English) speakers silently reciting sentences in either language. Using only visual speech information, adults who had learned English from birth or as a second language before the age of 6 could discriminate between French and English significantly better than chance. However, adults who had learned English as a second language after age 6 failed to discriminate these two languages, suggesting that early childhood exposure is crucial for using relevant visual speech information to separate languages visually. These findings raise the possibility that lowered sensitivity to non-native visual speech cues may contribute to the difficulties encountered when learning a new language in adulthood., This research was supported by Research grants from the Natural Sciences and Engineering Research Council (NSERC) and the Social Sciences and Humanities Research Council (SSHRC) to Janet F. Werker, by NSERC and SSHRC Fellowships to Whitney M. Weikum, and by grants PSI2009-12859, PSI2012-39149 and RYC-2008-03672 from Ministerio de Economía y Competitividad (Spanish Government), and the European COST action TD0904 to Jordi Navarra. Salvador Soto-Faraco was supported by ERC (StG-2010263145), MICINN (PSI2010-15426 and Consolider INGENIO CSD2007-00012) and AGAUR (SGR2009-092).
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Neural correlates of audiovisual speech processing in a second language

  • Barrós Loscertales, Alfonso
  • Ventura Campos, Noelia
  • Visser, Maya
  • Alsius, Agnès
  • Pallier, Christophe
  • Ávila, César
  • Soto-Faraco, Salvador, 1970-
Neuroimaging studies of audiovisual speech processing have exclusively addressed listeners’ native language (L1). Yet, several behavioural studies now show that AV processing plays an important role in non-native (L2) speech perception. The current fMRI study measured brain activity during auditory, visual, audiovisual congruent and audiovisual incongruent utterances in L1 and L2. BOLD responses to congruent AV speech in the pSTS were stronger than in either unimodal condition in both L1 and L2. Yet no differences in AV processing were expressed according to the language background in this area. Instead, the regions in the bilateral occipital lobe had a stronger congruency effect on the BOLD response (congruent higher than incongruent) in L2 as compared to L1. According to these results, language background differences are predominantly expressed in these unimodal regions, whereas the pSTS is similarly involved in AV integration regardless of language dominance., This research has been supported by the Spanish Ministry of Science and Innovation (PSI2010-15426, PSI2010-20168, and Consolider INGENIO CSD2007-00012), Comissionat per a Universitats i Recerca del DIUE-Generalitat de Catalunya (SRG2009-092), and the European Research Council (StG-2010 263145).
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

The Interplay between semantic and phonological constraints during spoken-word comprehension

  • Soto-Faraco, Salvador, 1970-
  • Brunellière, Angèle
This study addresses how top-down predictions driven by phonological and semantic information interact on spokenword comprehension. To do so, we measured event-related potentials to words embedded in sentences that varied in the degree of semantic constraint (high or low) and in regional accent (congruent or incongruent) with respect to the target word pronunciation. The data showed a negative amplitude shift following phonological mismatch (target pronunciation incongruent with respect to sentence regional accent). Here, we show that this shift is modulated by sentence-level semantic constraints over latencies encompassing auditory (N100) and lexical (N400) components. These findings suggest a fast influence of top-down predictions and the interplay with bottom-up processes at sublexical and lexical levels of analysis, This research was supported by the Spanish Ministry of Science and Innovation (PSI2010-15426 and Consolider INGENIO CSD2007-00012), Comissionat per a Universitats i Recerca del DIUE-Generalitat de/nCatalunya (SGR2009-092), and the European Research Council (StG-2010263145). ERP analyses were performed with the Cartool software (supported by the Center for Biomedical Imaging of Geneva and Lausanne).
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Cross-modal prediction in speech perception

  • Sánchez García, Carolina, 1984-
  • Alsius, Agnès
  • Enns, James T.
  • Soto-Faraco, Salvador, 1970-
Speech perception often benefits from vision of the speaker's lip movements when they are available. One potential mechanism underlying this reported gain in perception arising from audio-visual integration is on-line prediction. In this study we address whether the preceding speech context in a single modality can improve audiovisual processing and whether this improvement is based on on-line information-transfer across sensory modalities. In the experiments presented here, during each trial, a speech fragment (context) presented in a single sensory modality (voice or lips) was immediately continued by an audiovisual target fragment. Participants made speeded judgments about whether voice and lips were in agreement in the target fragment. The leading single sensory context and the subsequent audiovisual target fragment could be continuous in either one modality only, both (context in one modality continues into both modalities in the target fragment) or neither modalities (i.e., discontinuous). The results showed quicker audiovisual matching responses when context was continuous with the target within either the visual or auditory channel (Experiment 1). Critically, prior visual context also provided an advantage when it was cross-modally continuous (with the auditory channel in the target), but auditory to visual cross-modal continuity resulted in no advantage (Experiment 2). This suggests that visual speech information can provide an on-line benefit for processing the upcoming auditory input through the use of predictive mechanisms. We hypothesize that this benefit is expressed at an early level of speech analysis., This research was supported by the Spanish Ministry of Science and Innovation (PSI2010-15426 and Consolider INGENIO CSD2007-00012), Comissionat per a Universitats i Recerca del DIUE-Generalitat de Catalunya (SRG2009-092 and PIV2009-00122), and the European Research Council (StG-2010 263145).
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Alpha stimulation of the human parietal cortex attunes tactile perception to external space

  • Ruzzoli, Manuela
  • Soto-Faraco, Salvador, 1970-
An intriguing question in neuroscience concerns how somatosensory events on the skin are represented in the human brain. Since Head and Holmes’ [1] neuropsychological dissociation between localizing touch on the skin and localizing body parts in external space, touch is considered to operate in a variety of spatial reference frames [2]. At least two representations of space are in competition during orienting to touch: a somatotopic one, reflecting the organization of the somatosensory cortex (S1) [3], and a more abstract, external reference frame that factors postural changes in relation to body parts and/or external space [4, 5]. Previous transcranial magnetic stimulation (TMS) studies suggest that the posterior parietal cortex (PPC) plays a key role in supporting representations as well as orienting attention in an external reference frame [4, 6]. Here, we capitalized on the TMS entrainment approach [7, 8], targeting the intraparietal sulcus (IPS). We found that frequency-specific (10 Hz) tuning of the PPC induced spatially specific enhancement of tactile detection that was expressed in an external reference frame. This finding establishes a tight causal link between a concrete form of brain activity (10 Hz oscillation) and a specific type of spatial representation, revealing a fundamental property of how the parietal cortex encodes information., This work was supported by grants from the European Research Council (StG-2010 263145), the Spanish Ministry of Science and Innovation (PSI2010-15426), and the Comissionat per a Universitats i Recerca del DIUE-Generalitat de Catalunya (SRG2009-092) to S.S.-F.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Effect of attentional load on audiovisual speech perception: evidence from ERPs

  • Alsius, Agnès
  • Möttönen, Riikka
  • Sams, Mikko E.
  • Soto-Faraco, Salvador, 1970-
  • Tiippana, Kaisa
Seeing articulatory movements influences perception of auditory speech. This is often reflected in a shortened latency of auditory event-related potentials (ERPs) generated in the auditory cortex. The present study addressed whether this early neural correlate of audiovisual interaction is modulated by attention. We recorded ERPs in 15 subjects while they were presented with auditory, visual, and audiovisual spoken syllables. Audiovisual stimuli consisted of incongruent auditory and visual components known to elicit a McGurk effect, i.e., a visually driven alteration in the auditory speech percept. In a Dual task condition, participants were asked to identify spoken syllables whilst monitoring a rapid visual stream of pictures for targets, i.e., they had to divide their attention. In a Single task condition, participants identified the syllables without any other tasks, i.e., they were asked to ignore the pictures and focus their attention fully on the spoken syllables. The McGurk effect was weaker in the Dual task than in the Single task condition, indicating an effect of attentional load on audiovisual speech perception. Early auditory ERP components, N1 and P2, peaked earlier to audiovisual stimuli than to auditory stimuli when attention was fully focused on syllables, indicating neurophysiological audiovisual interaction. This latency decrement was reduced when attention was loaded, suggesting that attention influences early neural processing of audiovisual speech. We conclude that reduced attention weakens the interaction between vision and audition in speech., This research was supported by grants from Spanish Ministry of Science and Innovation (PSI2010-15426) and the European Research Council (StG-2010 263145) to Salvador Soto-Faraco and Agnès Alsius.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Electrophysiological correlates of tactile remapping

  • Soto-Faraco, Salvador, 1970-
  • Azañón Gracia, Elena
Orienting our gaze or attention to the location of tactile events in the skin feels natural and effortless. However, this process requires combining somatosensory and proprioceptive information in a non-trivial, time consuming, fashion. Here we address the time course of tactile remapping, from somatotopically-based representations, to a spatiotopic reference frame. We compared electrical responses at the scalp evoked by touch at one finger as participants held their arms straight or else, crossed about the body midline. This postural manipulation creates a conflict between frames of reference, which can be used to reveal the consequences of spatial remapping. Behavioural performance was gauged online for crossed and uncrossed postures using bimanual temporal order judgment (TOJ) trials occurring occasionally during the recording session. The first electrophysiological signs of tactile remapping were observed around 70 ms after the tactile event, overlapping in time with the somatosensory component N80. This electrophysiological effect was strongly lateralized to the left scalp and independent of the hand being stimulated. Moreover, participants who manifested a stronger behavioural conflict between frames of reference in the TOJ task displayed a larger electrophysiological effect. Based on these findings, and the known properties of the somatosensory network, we argue that remapping of tactile space depends on fast feedback projections from association areas of the parietal cortex, and encompasses a left-lateralized fronto-parietal network supporting the selection of guided actions., This research was supported by the Spanish Ministry of Science and Innovation (PSI2010-15426 and Consolider INGENIO CSD2007-00012), Comissionat per a Universitats i Recerca del DIUE-Generalitat de Catalunya (SRG2009-092), and the European Research Council (StG-2010 263145).
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Selective attention modulates the direction of audio-visual temporal recalibration

  • Ikumi Montserrat, Nara, 1986-
  • Soto-Faraco, Salvador, 1970-
Temporal recalibration of cross-modal synchrony has been proposed as a mechanism to compensate for timing differences between sensory modalities. However, far from the rich complexity of everyday life sensory environments, most studies to date have examined recalibration on isolated cross-modal pairings. Here, we hypothesize that selective attention might provide an effective filter to help resolve which stimuli are selected when multiple events compete for recalibration. We addressed this question by testing audio-visual recalibration following an adaptation phase where two opposing audio-visual asynchronies were present. The direction of voluntary visual attention, and therefore to one of the two possible asynchronies (flash leading or flash lagging), was manipulated using colour as a selection criterion. We found a shift in the point of subjective audio-visual simultaneity as a function of whether the observer had focused attention to audio-then-flash or to flash-then-audio groupings during the adaptation phase. A baseline adaptation condition revealed that this effect of endogenous attention was only effective toward the lagging flash. This hints at the role of exogenous capture and/or additional endogenous effects producing an asymmetry toward the leading flash. We conclude that selective attention helps promote selected audio-visual pairings to be combined and subsequently adjusted in time but, stimulus organization exerts a strong impact on recalibration. We tentatively hypothesize that the resolution of recalibration in complex scenarios involves the orchestration of top-down selection mechanisms and stimulus-driven processes., This research was supported by the Spanish Ministry of Science and Innovation (PSI2010-15426), Comissionat per a Universitats i Recerca del DIUE-Generalitat de Catalunya (SRG2009-092), and the European Research Council (StG-2010 263145).
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Cross-modal decoupling in temporal attention between audition and touch

  • Mühlberg, Stefanie, 1986-
  • Soto-Faraco, Salvador, 1970-
Temporal orienting leads to well-documented behavioural benefits for sensory events occurring at the anticipated moment. However, the consequences of temporal orienting in cross-modal contexts are still unclear. On the one hand, some studies using audio-tactile paradigms suggest that attentional orienting in time and modality are a closely coupled system, in which temporal orienting dominates modality orienting, similar to what happens in cross-modal spatial attention. On the other hand, recent findings using a visuo-tactile paradigm suggest that attentional orienting in time can unfold independently in each modality, leading to cross-modal decoupling. In the present study, we investigated if cross-modal decoupling in time can be extrapolated to audio-tactile contexts. If so, decoupling might represent a general property of cross-modal attention in time. To this end, we used a speeded discrimination task in which we manipulated the probability of target presentation in time and modality. In each trial, a manipulation of time-based expectancy was used to guide participants’ attention to task-relevant events, either tactile or auditory, at different points in time. In two experiments, we found that participants generally showed enhanced behavioural performance at the most likely onset time of each modality and no evidence for coupling. This pattern supports the hypothesis that cross-modal decoupling could be a general phenomenon in temporal orienting., This research was funded by the Spanish Ministry of Science and Innovation (PSI2016-75558-P(AEI/FEDER)), the Comissionat per a Universitats i Recerca del DIUE-Generalitat de Catalunya (2014SGR856), and the European Research Council (StG-2010 263145) to S.S.F.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Beat gestures and syntactic parsing: an ERP study

  • Biau, Emmanuel, 1985-
  • Fromont, Lauren A.
  • Soto-Faraco, Salvador, 1970-
We tested the prosodic hypothesis that the temporal alignment of a speaker's beat gestures in a sentence influences syntactic parsing by driving the listener's attention. Participants chose between two possible interpretations of relative-clause (RC) ambiguous sentences, while their electroencephalogram (EEG) was recorded. We manipulated the alignment of the beat within sentences where auditory prosody was removed. Behavioral performance showed no effect of beat placement on the sentences’ interpretation, while event-related potentials (ERPs) revealed a positive shift of the signal in the windows corresponding to N100 and P200 components. Additionally, post hoc analyses of the ERPs time locked to the RC revealed a modulation of the P600 component as a function of gesture. These results suggest that beats modulate early processing of affiliate words in continuous speech and potentially have a global impact at the level of sentence-parsing components. We speculate that beats must be synergistic with auditory prosody to be fully consequential in behavior., This research was supported by the Ministerio de Econom´ıa y Competitividad (PSI2016-75558-P), AGAUR Generalitat de Catalunya (2014SGR856), and the European Research Council (StG-2010 263145). EB was supported by a postdoctoral fellowship from the European Union’s Horizon 2020 research and innovation programme, under the Marie Sklodowska-Curie grant agreement No. 707727