Annotated Bibliography 2: Multimodal Genres

Hiippala, T. (2013). The interface between rhetoric and layout in multimodal artefacts. Literary and Linguistic Computing, 28(3), 461-471.

In “The Interface between Rhetoric and Layout in Multimodal Artefacts,” Tuomo Hiippala argues that given genres have prototypical layouts.  In other words, genres are enacted not through templates where certain element occur in the same positions as a convention, but instead carry rhetorical commonalities which can be abstracted through analysis and consequently predicted.  Using a sample of tourist brochures published at different points in time, Hiippala notes how layouts have shifted over time and now most typically use a back and forth interplay between visual elements and text known as page- flow. Page flow can be contrasted with text-flow, a generally uninterrupted linear text, and image-flow, where the text is organized as a sequence of images.  However, perhaps a more important goal for Hiippala in this article is to improve multimodal genre analysis by demonstrating ways of modeling the prototypical structure of a given genre.  In other words, for users, “artefacts have certain prototypical characteristics, which enable their recognition and invoke particular models of inference and interpretation established during previous encounters with similar artefacts” (p. 464).  In a simple investigation of the presence of prototype clues, Hiippala gave a group of students three different multimodal artefacts, blurred to prevent recognition of words, symbols or visual specifics, but showing the layout and enabling them to distinguish visual and verbal elements.  The students easily picked out the genres for the artefacts from these clues, suggesting that each genre was associated with a prototypical layout.  In the specific analysis of the brochures modeled here, Hiippala used J.A. Bateman’s Genre and Multimodality model to demonstrate the underlying rhetorical relationships between visual and verbal elements in each artefact.

Hiippala’s article, in fact, is a preliminary analysis associated with a longer corpus analysis currently underway.  One of the goals of the ongoing project is to map the relationships between different semiotic modes and look for the ways different modes operate in parallel and through interaction with each other.   He urges further research on genres using a multimodal approach, an area of research of great interest to me.  I have read a number of excellent genre studies, from Devitt’s widely-cited 1991 study of genres of tax accounting to Giltrow and Stein’s (2009) Genres in the Internet, but all of these studies focused entirely on the linguistic elements of genres, even though all of these researchers employed definitions of genre in which multimodal elements were implicit.  Clearly, more research needs to employ a multimodal framework to accurately capture how genre operates.  Besides the general importance of doing this type of analysis, one other aspect of Hiippala’s article also resonated with me, and that is the concept of prototype.  In cognitive linguistics—for example, Lakoff and Johson’s work with conceptual metaphor theory (1980)—the idea of prototypes has been widely applied to semantics and sometimes to morphology or syntax.  It has seemed logical to me that, in a cognitive sense, genres also behave like prototypes.  But I have not seen many scholars mention prototypes in relation to genre, so the fact that Hiippala does so caught my attention.  That being said, Hiippala’s sample was too small for us to verify that his principles apply broadly even to the genre he was investigating, the tourist brochure.  It was also not fine-grained enough, in my opinion, to differentiate between tourist brochures and similar genres that might also use page-flow principles.  Nevertheless, I intend to take a closer look at Bateman’s work to see what his multimodal Genre and Multimodality modal might offer for my own research.

Devitt, A.J. (1991). Intertextuality in tax accounting: Generic, referential, and functional. In C. Bazerman & J. Paradis (Eds.), Textual dynamics of the profession (pp. 336-357). Madison, WI: University of Wisconsin Press.

Giltrow, J., & Stein, D. (2009). Genres in the Internet: Issues in the theory of genre. Amsterdam: John Benjamins Pub. Co.

Lakoff, G., & Johnson, M. (1980). Metaphors we live by. Chicago: University of Chicago Press.

Visual arguments: Are they possible?

From working through this visual argument exercise, I have come to a greater appreciation of the difficulty of creating a visual argument.  For example, with my entry, I was very pleased to see that my classmates had caught the concept that I was trying to convey.  Indeed, they caught some of subtle features that I had included intentionally as well as a feature or two that I admitted were present but hadn’t noticed myself.  On the other hand, I wondered to what extent I had cheated by including a headline.  Classmates who offered more purely visual entries probably created a truer test.  I will admit that I struggled to grasp the complete argument in some of these cases.  However, other responders did much better at generating careful analyses with a high degree of plausibility.  I felt like going back and trying to make my analyses more thorough and finer-grained.  I felt that my comments may have left the impression that the display was lacking when, in fact, it was my analysis that was inadequate.  Nevertheless, for me, the experience demonstrated the challenges of creating a purely visual argument. Before going through this process, I did believe that visual arguments were possible.  I felt that Birdsell and Groarke’s example of the anti-smoking poster was a good case in point (311).  In fact, the short headline plus image was a model for my visual argument.  I also felt that Blair offered a good criteria for measuring whether a visual could be deemed an argument or not.  Yet when I tried to make all the connections between images and their connotations in the various collages posted by my classmates and tried to grasp the key proposition of each, I found it much harder than I had imagined.  I believe it is possible to use visuals for argument, but I have come to believe that most effective visual arguments are probably multimodal, harnessing the visual along with other modes to make the argument clear.

A Visual Argument

Annotated Bibliography 1: Multimedia Genres and Traversals

Lemke, J. L. (2005). Multimedia Genres and Traversals. Folia Linguistica, 39(1), 45-56.

In “Multimedia Genres and Traversals”, Lemke considers two important questions related to genre theory: multimodality and the shifts users make between diverse genres over the long- and short-term.  The article is divided between discussions of these two questions.  The first question revolves around the intrinsic multimodality of genre and basically asks the question, “If genre is, and always has been, multimodal, then what changes must genre theory make to accommodate the ‘visual-spatial meaning systems’ (45) of texts?”  The second question considers attentional shifts of a user moving from one genre in one moment to another genre in another moment.  Like the hypertextual link that a user clicks on a webpage, these moves end up directing attention from one text to another and thus generate their own forms of cohesion: “Meaning is no longer internal to genre and institutions. It is also made across and between them, as we juxtapose, catenate, and traverse not just websites or television channels, but, on longer timescales, the sites and roles of our days, weeks, and lives” (51).

In this article Lemke uses no specific artifacts nor reports on any specific research.  Instead, the article argues the need to further conceptualize the multimodal aspects of genre, extends the analytical framework to better tackle visual-spatial meaning, and complicates the genre landscape by not only reinforcing generic diversity but proposing the idea of traversals as dynamic pathways of meaning external to genres themselves.

Lemke’s article added value to me by expanding my awareness of some of the elements of a multimodal genre analysis. Since it is precisely the “visual-spatial meaning” of genre that I hope to investigate in my research, a good starting point was a discussion of genre as multimodal.  Lemke begins the article by defining genre as a multimodal package and using examples to demonstrate how “orthography, typography and page layout” (45) include visual-spatial elements even when other images are not present.  A significant challenge of multimodal analysis, however, involves the question of sequentiality.  Having multiple elements as part of a page layout means that attention may not be directed in a linear fashion. Users have the freedom to switch attention between elements during the reading.  Genre theorists have often analyzed genre in terms of linear moves.  Lemke suggests that multimodal genres are “loosely constrained” in terms of sequence and offer “multi-sequential” pathways, yet at the same time genre conventions are still operating (48).  Researchers need to find ways to discuss both the possibilities and probabilities within a given genre.  Lemke also offers value in this article by posing three questions on verbal-visual relationships that need to be explicated in genre theory to guide analysis and explain how cohesion is created from text to images or figures, from images or figures to texts, and within texts and within visuals.  These questions showed me some things to demand from an analytical framework.  On the other hand, I found the discussion of traversals to be interesting but not immediately useful.  Lemke broadened my awareness of the generic landscape, a useful first step.  Nevertheless, if one is seeking specific analytical tools, this article is not the place to look.  

My mind map

Here is the link to my mind map

Laurie's mind map