Annotated Bibliography 3: More on Multimodal Analysis

Zhao, S., Djonov, E., & van Leeuwen, T. (2014). Semiotic technology and practice: A multimodal social semiotic approach to PowerPoint. Text & Talk, 34(3), 349-375.

PowerPoint is an excellent example of a multimodal form of communication.  Not only do the slides combine the visual and the verbal, but live presentations are almost always accompanied by oral language and gestures.  In this study, Zhao, Djonov & van Leeuwen consider these multimodal elements of a PowerPoint presentation, but they also consider the semiotic nature of the software design and how the interface shapes the meaning-making process.  The authors used a social semiotics approach to examine the semiotic resources available through the software design, the ways these resources are used to design slides, and the speech and gestures of presentations.  As is the goal of a social semiotic approach, they also sought to uncover the social norms underlying each of these semiotic processes.  To investigate how PowerPoint shapes semiotic decisions, the researchers examined all versions of PowerPoint that had been released by the time of the study (i.e. from PowerPoint 3.0 to the 2007 version).   To examine the multimodal elements of the presentations, they compared video recordings of 27 PowerPoint presentations to the slideshow files.  Finally, they followed up by interviewing each presenter.  One of the goals of the research was to examine PowerPoint as a unified phenomenon including the software, the slideshow, and the presentation, or, in other words, as a genuinely multimodal form of communication.  Much previous research, for example, has been limited to slide design. When slides are examined as a standalone text, many semantic relations are impossible to determine.  On the other hand, when a PowerPoint presentation is examined as a multimodal event mapped over time, researchers were able to see how meaning was generated through a combination of oral language, visual design, and gesture. 

This study offers a strong argument for examining semiotic events multimodally.  It shows how important meanings and ways of making meaning are ignored when an analysis limits itself to any one mode.  Communication, of course, has always been multimodal.  Public speeches have always been accompanied by gestures and body language.  Written texts have always had a visuality on the printed page.  But contemporary technologies have multiplied the ways that modes can be combined to communicate a message.  I first became interested in multimodality several years ago when I encountered an article by Gunther Kress in a book on the teaching of grammar in school.  In my Ph.D. studies, one of my key research interests is genre theory, and it has become obvious to me that genre analysis needs to be multimodal.  Intuitively, it seems likely that users first make genre decisions based on the look and feel of an artefact.  In other words, visual elements and materiality are probably more salient to users than the organization of a text or its arguments, at least initially.  It seems strange, therefore, that most genre research is quite logocentric, although that may be changing, as, for example, in Batemen’s (2008) model of genre analysis. I don’t think Zhao, Djonov & van Leeuwen would deny the value of studies in discourse analysis or visual rhetoric that emphasize one mode.  After all, all research is necessarily selective.  Nevertheless, this article and other multimodal studies demonstrate that in semiotics or rhetoric, studies designed to investigate how multiple modes work together offer greater explanatory power.  Even when a research project is narrowed to focus on data within one mode, such as the written text or the page layout, it is always valuable to be aware that this is a strategic decision, and the whole semiotic package includes more.

References

Bateman, J. A. (2008). Multimodality and genre: A foundation for the systematic analysis of multimodal documents. Basingstoke, England: Palgrave Macmillan.

Kress, G. (2010.) A grammar for meaning-making. In T. Locke (Ed.), Beyond the grammar wars: A resource for teachers and students on developing language knowledge in the English/literacy classroom (233-253). New York: Routledge.