Research talk on "From language as a multimodal phenomenon to multimodal phenomena as language? Theoretical and methodological issues for multimodal investigations"

Research talk on "From language as a multimodal phenomenon to multimodal phenomena as language? Theoretical and methodological issues for multimodal investigations"

Who: John Bateman, University of Bremen

When: 6 February 2015, 13:00-14:00

Where: Room: Baines Wing Miall LT (2.34)

Abstract

As the awareness that language use is fundamentally multimodal gains traction across the board, challenging issues arise concerning how this can most effectively be studied. The idea that verbal language and its theoretical treatment may provide a suitable ‘template’ for signifying practices in general has been discussed and practiced at least since Saussure. Many current approaches to multimodal phenomena follow in this vein. These range across assumptions such as the socio-cultural embedding of communicative practices will necessarily lead to metafunctionally-organised semiotic resources found in systemic-functionally inspired accounts (e.g., Kress, van Leeuwen, O’Toole, O’Halloran), that accounts of syntax and semantics may also offer sites for gestural and intonational components (e.g., Fricke), that differing means of physically producing and cognitively interpreting signs may each be subject to unified properties of the human cognitive architecture concerned with recursive structures and parallel sets of contributing constraints (e.g., Jackendoff, Cohn), and many more. Interestingly, although all drawing close connections between language and other modes of communication, the views of language assumed are themselves often quite different: we find social orientations, formal orientations, cognitive orientations but the traditional semiotic placement of verbal language and its properties as central remains. In this talk, I draw on our current explorations of the multimodal functioning of a variety of communicative forms – including examples from film, comics and graphic novels, illustrated documents, picturebooks and the like – to address questions concerning both the foundation of multimodal meaning-making and the resulting methodologies that are required to study it. As part of this, I will consider just which kinds of constructs may be usefully imported from our models of verbal language and which not; conversely, the talk will also consider aspects of the functioning primarily of visual media which are not covered within the linguistic models available. Whether this will motivate extensions of our notions of ‘language’, or better be treated by making explicit the very different affordances and capabilities involved, will be raised and discussed as an open research question.

Terms that will receive definition and illustration along the way will include genre, medium, semiotic resource, discourse, textuality, texturality, and, centrally, semiotic mode itself.