Timeline

This is not super complete but...

http://www.dipity.com/lauriellens/Digital-design-journey/

Annotated Bibliography 3: More on Multimodal Analysis

Zhao, S., Djonov, E., & van Leeuwen, T. (2014). Semiotic technology and practice: A multimodal social semiotic approach to PowerPoint. Text & Talk, 34(3), 349-375.

PowerPoint is an excellent example of a multimodal form of communication.  Not only do the slides combine the visual and the verbal, but live presentations are almost always accompanied by oral language and gestures.  In this study, Zhao, Djonov & van Leeuwen consider these multimodal elements of a PowerPoint presentation, but they also consider the semiotic nature of the software design and how the interface shapes the meaning-making process.  The authors used a social semiotics approach to examine the semiotic resources available through the software design, the ways these resources are used to design slides, and the speech and gestures of presentations.  As is the goal of a social semiotic approach, they also sought to uncover the social norms underlying each of these semiotic processes.  To investigate how PowerPoint shapes semiotic decisions, the researchers examined all versions of PowerPoint that had been released by the time of the study (i.e. from PowerPoint 3.0 to the 2007 version).   To examine the multimodal elements of the presentations, they compared video recordings of 27 PowerPoint presentations to the slideshow files.  Finally, they followed up by interviewing each presenter.  One of the goals of the research was to examine PowerPoint as a unified phenomenon including the software, the slideshow, and the presentation, or, in other words, as a genuinely multimodal form of communication.  Much previous research, for example, has been limited to slide design. When slides are examined as a standalone text, many semantic relations are impossible to determine.  On the other hand, when a PowerPoint presentation is examined as a multimodal event mapped over time, researchers were able to see how meaning was generated through a combination of oral language, visual design, and gesture. 

This study offers a strong argument for examining semiotic events multimodally.  It shows how important meanings and ways of making meaning are ignored when an analysis limits itself to any one mode.  Communication, of course, has always been multimodal.  Public speeches have always been accompanied by gestures and body language.  Written texts have always had a visuality on the printed page.  But contemporary technologies have multiplied the ways that modes can be combined to communicate a message.  I first became interested in multimodality several years ago when I encountered an article by Gunther Kress in a book on the teaching of grammar in school.  In my Ph.D. studies, one of my key research interests is genre theory, and it has become obvious to me that genre analysis needs to be multimodal.  Intuitively, it seems likely that users first make genre decisions based on the look and feel of an artefact.  In other words, visual elements and materiality are probably more salient to users than the organization of a text or its arguments, at least initially.  It seems strange, therefore, that most genre research is quite logocentric, although that may be changing, as, for example, in Batemen’s (2008) model of genre analysis. I don’t think Zhao, Djonov & van Leeuwen would deny the value of studies in discourse analysis or visual rhetoric that emphasize one mode.  After all, all research is necessarily selective.  Nevertheless, this article and other multimodal studies demonstrate that in semiotics or rhetoric, studies designed to investigate how multiple modes work together offer greater explanatory power.  Even when a research project is narrowed to focus on data within one mode, such as the written text or the page layout, it is always valuable to be aware that this is a strategic decision, and the whole semiotic package includes more.

References

Bateman, J. A. (2008). Multimodality and genre: A foundation for the systematic analysis of multimodal documents. Basingstoke, England: Palgrave Macmillan.

Kress, G. (2010.) A grammar for meaning-making. In T. Locke (Ed.), Beyond the grammar wars: A resource for teachers and students on developing language knowledge in the English/literacy classroom (233-253). New York: Routledge.

Reading Images: A Post-reading Activity

Connections between Kress and Van Leeuwen and other readings

• One idea that is key to Kress and van Leeuwen and that they share with all the authors interested in ideology, rhetoric, or critical theory is the ways that social attitudes and cultural memory are encoded visually.  Kress and van Leeuwen use the term social semiotics to describe their approach to decoding these messages.  Like Barthes and Williamson, they are concerned with tracing how signs are constructed through cultural references.  In the way that Barthes describes the Italianicity of the Panzani ad and Williamson notices the crucifixion reference in the body language of the body builder in the Soloflex ad, Kress and van Leeuwen pick up on visual relationships and interactions and what they mean in Western society.  A number of the authors include such meanings in their analyses. Kinross in his reference to modernism in timetables and Atzmon on the Vietnam Veteran’s Memorial are just two I might mention.
• Another connection that Kress and van Leeuwen share with Williamson is the goal of balancing cognitive and cultural explanations.  Williamson makes a point of acknowledging that design works psychologically and “below the level of conscious detection” (329) and, in his discussion of the 1937 billboard, “Watch the Fords Go By,” he talks about the role that visual processing and eye movement work in generating meaning.  Likewise, Kress and van Leeuwen consider cognitive processing along with cultural references in their analysis.  Another author strongly interested in cognitive processing, of course, is Donald Norman.
• One of Kress and van Leeuwen’s main goal in this book is to offer an analytical metalanguage, a “grammar” of visual design.  They are unique in their coverage, but some other authors do offer some metalanguage for specific types of analysis.  For example, in their Rhetorical Handbook, Lupton and Ehses offer a list of rhetorical operations and rhetorical figures and offer examples of how each of these can be used in design.  In “Seeing the Text,” Stephen Bernhardt gives terms related to gestalt theory that he uses in his analysis of the wetland flyer.  In his discussion of design narrative—an interest that he shares with Kress and van Leeuwen—Williamson offers the concepts of script, protagonist, props, flow of action, and what he calls, “experiential episode” (p. 328).  Barthes even offers a bit of metalanguage, such as anchor and relay.  An author that seems to disagree with this goal is Mitchell, who resists the idea of metalanguage, arguing that visual messages cannot really be verbally explicated.

Thought-provoking or provocative ideas

• One question that Kress and van Leeuwen ask has long interested me.  On page 31, they discuss the current shift away from verbal towards greater reliance on the visual in communication. They note that “implicit in this is a central question, which needs to be put openly, and debated seriously: is the move from the verbal to the visual a loss or a gain?” (p. 31). This is a question that I have been grappling with since I started my graduate work, and probably one of the reasons that I took this class was to explore this question more knowledgeably.  For my final project my first semester in Major Debates, in fact, I explored the question of whether literacy education in the English-speaking world should privilege the verbal.  Though my paper made a provisional argument that writing should be privileged in the literacy curriculum, whether a visual shift is ultimately a gain or a loss for society as a whole is something I feel much less certain about. I believe that visual rhetoric deserves a place in rhetorical education, but to what extent and at what levels and for what purposes exactly is a question that continues to interest me, and it clearly depends in part on what the visual shift means for our collective futures.
• On a more minor note, a specific idea that I found provocative was the idea of insider/outsider status being generated by angle of view in a photograph.  I found the analysis of the Aboriginal classroom to be quite interesting in the claim that the white teachers are displayed with a frontal angle as “like us, the viewer” whereas the Aboriginal children displayed through the oblique angle are being othered.  I can provisionally accept this analysis, but I would be interested in seeing more evidence for this claim.
• The idea of modality was also thought provoking, especially since it seems a little less usual in visual analyses.  The idea of interaction through which “social interactions and social relations can be encoded in images” (p. 115) and the idea of way a viewer can be confronted by the gaze of an image, for example, has interested a number of recent scholars.  Discussion of how elements in a layout interact and the semiotics of images, the way that ideology is encoded semiotically, both come through in a number of analyses.  On the other hand, the concept of modality may not be unique to Kress and van Leeuwen, but we have not seen this idea in any other article that we have read this semester.  It is also interesting to consider the ways that the modality continuum from not naturalistic (unreal) to maximally real to hyper-real has changed as technology has changed how reality can be visually represented, as, for example, with the development of photography and, more recently, the changes to modality with the development of digital photography and editing software.  It is also interesting that Kress and van Leeuwen recognize that modality differs in different contexts.  

Reading questions

• This was more of a post-reading question—although it did occur to me before reading the book as well—and that is what Kress and Van Leeuwen’s sources were for their terminology and concepts. Even before this semester, I noticed that Rudolf Arnheim’s works appeared in their bibliography, and that made sense to me because he did work on cognitive aspects of visual reasoning.  I was also aware of their theoretical debt to Barthes and Halliday, but I am curious to know the sources of the details of their system of analysis.  Did they gather ideas from a wide array of sources?  Did they take ideas from here and there and then invent their own terminology, or did quite a bit of the terminology come from other thinkers?  Anyway, if I was interviewing the authors I would ask for more details about where they got their ideas.  I just think it would be interesting to see what came from where (beyond, of course, just noticing each citation as it comes up during a reading of the book). 
• Kress and van Leeuwen suggest that in terms of social control, there is a trend towards “a decrease of control over language (e.g. the greater variety of accents allowed on the public media, the increasing problems in enforcing normative spelling), and towards an increase in codification and control over the visual (e.g. the use of image banks from which ready-made images can be drawn for the construction of visual texts, and generally, the effect of computer imaging technology).”  When I read this, my question was whether this was in fact the case or not.  Random thought:  Does the presence of cat videos and doge pictures prove or disprove this claim?
• Chapter 4 deals with interaction with and positioning in relation to the viewer.  John Berger talks about this in Ways of Seeing and lots of articles talk about “gaze”.  Would this be the most widely analyzed topic of the ones that Kress and van Leeuwen include?  As I said earlier, I didn’t think modality was discussed much, and Kress and van Leeuwen claim that “visual structuring… has been dealt with less satisfactorily” (p. 47).  Does that also mean it has not been discussed that often, or that it has been discussed frequently enough, but just not well?  I guess my question, then, has to do with a ranking of the topics covered in this book in terms of the research attention that each has garnered.

An infographic

Here is the infographic

Here is the data set that I used:

	Motifs	Anda	Plazas
Photos	31	11	16
Line drawings	54	47	20
Maps	0	3	5

Another version:

Heuristic for Analysis of Visuals in Modern Language Texts

Iconic language (descriptive; close reading)

• Take a census of visuals used in first 40 pages of three texts (What is it made of? What does it contain worth explaining?
• Photographs
• Representative speakers
• Celebrities
• Cultural representations (landscapes, architecture, festivals, art, etc.)
• General human interaction
• Representative genres (ads, posters, schedules, etc.)
• Thumbnails
• Background
• Line drawings
• Cartoons with dialogue
• Classification (i.e. scenes or things that are labeled with vocabulary)
• Concepts
• Maps

• Use Kress and van Leeuwen’s framework for examining representative visuals of first two tlypes, above (What do you see in the artefact?)
• Narrative representations
• Conceptual representations
• Position of and interaction with viewer
• Context of layout

The cultural language (links to larger culture/social context)

• Consider the layout and how representative visuals are being used (What is the context?)
• What appears to be the pedagogical purpose?
• What cultural allusions are being made?
• What universal (i.e. shared human values) are being evoked?

• Briefly define the audience, in this case is pretty straightforward (Who is the audience?)
•  The audience is university students in the United States, probably native speakers of English, and culturally mainstream.  Assumed to be 18-22 years of age.  (Visuals reinforce this)

The theoretical language (stance; lens for understanding)

• What do the findings reveal about how visuals are used in language pedagogy?  (What does it mean? How do we interpret it?)
• Do the visuals leverage cultural stereotypes to make meaning more predictable and therefore, enhance language learning?  In what ways, might this be a positive or negative thing?

• What does this say about disciplinary differences in the use of visuals in textbooks?

• What does this say about how visuals represent linguistic concepts at all?  (Nowhere is this more straightforward, after all, than in a language textbook.)

Artifact: Sample page and partial analysis

• This is a line drawing, a variation of cartoon with dialogue.  Notice that the purpose is to construct the situation in which the dialogue, above the pictures, would occur.

• Notice the interaction between participants within the picture.  In Kress and van Leeuwen's terms there is a transactional reaction in both scenes. That this should be the case is highly appropriate for a verbal dialogue between multiple participants, as in a language textbook.

• Notice that in both cases there is a small image of a house framed within, in the first picture, a computer screen, and in the second, a cell phone.  Not only does this establish the topic of the conversation in a very concrete way, but it can be assumed to appeal to the target audience, for whom images on a screen such as these would a normal conversation starter.