Actions speak no louder than words: Symmetrical cross-modal interference effects in the processing of verbal and gestural information

Abstract

Five experiments are reported that investigate the distribution of selective attention to verbal and nonverbal components of an utterance when conflicting information exists in these channels. A Stroop-type interference paradigm is adopted in which attributes from the verbal and nonverbal dimensions are placed into conflict. Static directional (deictic) gestures and corresponding spoken and written words show symmetrical interference (Experiments 1, 2, and 3), as do directional arrows and spoken words (Experiment 4). This symmetry is maintained when the task is switched from a manual keypress to a verbal naming response (Experiment 5), suggesting the mutual influence of the 2 dimensions is independent of spatial stimulus-response compatibility. It is concluded that the results are consistent with a model of interference in which information from pointing gestures and speech is integrated prior to the response selection stage of processing.