Audio-Visual Sensory Interactions and the Statistical Covariance of the Natural Environment

by C. Zetzsche, F. Roehrbein, M. Hofbauer, K. Schill

Abstract:

We show that a mobile observer in a natural environment receives systematically co-varying signals in his different sensory modalities. An independent, modality-specific processing - as assumed in classical theories of perception - would hence be sub-optimal. Rather, information theory predicts that the system should use a statistically optimised joint processing strategy. We tested this by measuring the two-dimensional just-noticeable difference (jnd) curves for basic visual-auditory stimulus configurations (a patch of light combined with a 1kHz tone). The forcedchoice task was to detect any change in this configuration, irrespective of modality. The resulting two-dimensional jnd-curve cannot be explained by an independent, modality-specific processing. In particular, the sensitivity increase for the "ecologically relevant" joint auditoryvisual increments or decrements is much higher than the usual probability summation effects. This points to a direct neural integration of visual and auditory information at an early stage.

Reference:

Audio-Visual Sensory Interactions and the Statistical Covariance of the Natural Environment (C. Zetzsche, F. Roehrbein, M. Hofbauer, K. Schill), In Proc. Forum Acousticum 2002, 2002.

Bibtex Entry:

@InProceedings{Zetzsche2002, author = {C. Zetzsche and F. Roehrbein and M. Hofbauer and K. Schill}, title = {Audio-Visual Sensory Interactions and the Statistical Covariance of the Natural Environment}, booktitle = {Proc. Forum Acousticum 2002}, year = {2002}, abstract = {We show that a mobile observer in a natural environment receives systematically co-varying signals in his different sensory modalities. An independent, modality-specific processing - as assumed in classical theories of perception - would hence be sub-optimal. Rather, information theory predicts that the system should use a statistically optimised joint processing strategy. We tested this by measuring the two-dimensional just-noticeable difference (jnd) curves for basic visual-auditory stimulus configurations (a patch of light combined with a 1kHz tone). The forcedchoice task was to detect any change in this configuration, irrespective of modality. The resulting two-dimensional jnd-curve cannot be explained by an independent, modality-specific processing. In particular, the sensitivity increase for the "ecologically relevant" joint auditoryvisual increments or decrements is much higher than the usual probability summation effects. This points to a direct neural integration of visual and auditory information at an early stage.}, }