The linguistic foundations of a theory of multimodality 

Get Complete Project Material File(s) Now! »

CHAPTER 3 Modes and multimodality

INTRODUCTION

Multimodality has until recently largely been studied by linguists. However, the role of multimodality has not only had a profound influence on language practices, but also on contemporary visual art, locally as well as worldwide. While the visual arts have not as yet embraced multimodality by definition or included it in the teaching and training thereof, it is evident that contemporary artists are incorporating a wider repertoire of modes into their chosen media of expression. This is especially evident when looking at the annual entries to the Sasol New Signatures National Art Competition over the past six years.
In this chapter I first explore how scholars working in the field of multimodality delimit the range of possible modes available to the artist. Second, I give an overview of the characteristics of modes in order to justify ‘mode-status’ as it manifests in contemporary visual art. This is followed by a characterisation of multimodality, as primarily expounded in the work of Gunther Kress.

DEFINING AND DELIMITING MODES

In order to understand how multimodality is applied to meaning-making in contemporary visual art, it is firstly important to investigate what linguist scholars aimed to define as mode. According to Kress (2010, p. 54), Kress and Bezemer (2008, p. 169) and Andrew (2007, 2011) modes are culturally determined resources for making meaning. Kress (2010) and Kress and Bezemer (2008) add to this definition that the resources are socially shaped and culturally determined.
Although other scholars do not contest that a mode has these two components, a survey of the literature on multimodality, including seminal works by key social semiotic and multimodal scholars, has revealed that they delimit the range of modes they recognise in different ways, which may indicate that they conceptualise the notion of mode in different ways. This may be due to constraints imposed by the discipline or field of study, the interest of the scholar, and the fact that semiotic blurring has allowed modes to be combined across various disciplines and genres.
Kress and Van Leeuwen (1996) and Kress (2010) include images, writing, layout, music, gesture, speech, moving images and soundtracks as modes, while Kress (2010) lists modes as speech, still images, moving images, writing, gesture, music, 3D models, actions and colour. These ranges are elaborations of the modes that the New London Group originally listed in 2000: audio, spatial, visual, gestural and linguistic (Cope & Kalantzis, 2000, p. 25). Recently, even more specialisation has taken place to accommodate sub-disciplines and genres. Kress and Bezemer (2008, p. 169), who became interested in how modes have shaped the layout of text-making in the printed media, including textbooks and e-resources, list image, writing, layout, speech, moving images and gestures as modes. They suggest that moving images and speech are used alongside or instead of writing or text (2008, p. 174), for instance when viewing online tutorials on YouTube (Kress & Bezemer, 2008, p 176). Iedema, who also became interested in a multi-semiotic approach to modes as they became technology-assisted through television, computers and the internet, lists image, music, gesture, sound, and film or moving images as modes in the technological domain (Iedema, 2003, p. 36).
Andrew’s (2007, 2011) context is the visual art classroom, and thus he includes various genres and mediums in art-making. He draws on the work of Jewitt and Kress (2003, p. 1). For him, the resources for meaning-making include image, gaze, gesture, movement, music, speech and sound (Andrew, 2007, p. 16). O’Halloran et al. (2009) are particularly interested in the study of multimodality as it manifests in film, and how moving images transmediate into the computer or digital realm. Their interest in how short films (moving images) in the form of advertisements are digitally made, stored and re-played has afforded them the opportunity to assist in pioneering software for digital multimodal analysis through the retrieval of multimodal patterns (O’Halloran et al., 2009, p. 1).
They specify lighting, movement, gesture, gaze, dress and camera angles as her list of modes (O’Halloran et al., 2009, p. 2). O’Toole (2011, p. 24) applied various principles and elements of art (Gestalt) to Kress and Van Leeuwen’s (1996) visual grammar in order to analyse architectural structures in Australia. O’Toole (2011, p. 25) specifies gaze, rhythm, stance, characterisation, scale, perspective, line and form as modes for architectural expression. O’Halloran and O’Toole’s approaches differ from that of Kress and Van Leeuwen (1996) in that they regard language, mathematical symbols and images as semiotic resources rather than modes (Jewitt, 2007, pp. 21‒22). Stein’s (2008) main focus is applying multiple modes to teaching pedagogy in education and she includes colour, sound, movement, gesture, image and space as part of her list of modes. Kress identifies eight ‘modes of communication’, viz. writing, speech, moving images, still images, colour, action, gesture and 3D models (Kress, 2010, p. 21).
In 2002 Finnegan (p. 5) identified two further modes not mentioned by any of the other scholars. In addition to sound, sight, movement, embodied engagements and material objects, she lists touch and smell. What has, however, not been found in any of the sources consulted, is taste. This raises the question of whether taste does indeed manifests as a mode in visual art. Although not much research has been conducted on the inclusion of taste as a mode in visual art, there is evidence that visual artists have been including modes that draw upon all five senses. The sense of sight, and to some extent touch and hearing, have formed the basis for prototypical sensory modes in the
arts throughout history, but contemporary visual artists have started utilising touch, smell and taste in the format of edible art, olfactory art and tactile art, which will be covered in detail in later chapters of this study.
Thus, the list of primary modes have now come to eleven: writing, speech, moving images, still images, colour, action, gesture, 3D models, touch, smell and taste. Still images, colour and 3D models may be regarded as the primary discipline-specific modes at work in visual art.
A question that comes to mind is how we might decide which of these are ‘legitimate’ or ‘justified’, without any criteria that are more specific than ‘meaning-making resources’ and ‘culturally determined’. Andrew (2011, p. 106) draws on Stein (2008) and Schon (1999) who posit that modes occur on a social semiotic continuum where lexicogrammatical associations allow modes and the combination thereof to be free from fixed rules. Andrew (2011, p. 106) contends that the classification of modes may be difficult as their application to theorisation may seem ‘rule like’ but not ‘rule bound’ to its process of choice and combination.
Although Stein’s (2008) work precedes many of the publications referred to above, her exposition of mode moves us the closest to a comprehensive understanding of the notion. She regards the enumeration of the characteristics of modes as the most useful starting point for defining mode, and thus her range of modes, as well as the ranges identified by other scholars, could be tested with reference to the following six characteristics (Stein 2008, pp. 25‒26):

• Modes are an interface between the natural and cultural world.
• may be disseminated through technology.
•  Modes have grammars and communicative effects.
• Modes have materiality.
• Modes can be realised in multiple media.

Modes may be related to sensory possibilities of the body.
In the next section a brief explanation will be given of each of these characteristics before they are applied as criteria.

CHARACTERISTICS OF MODES

Modes are an interface between the natural and cultural world

Modes are the carriers of meaning. Within the constructivist paradigm of meaningmaking, modes transport information and make sense of the world, which in turn shapes social and cultural identity. As early as 1984 Charles Suhor, an educationist drawing on Vygotsky’s 1978 constructivist theory, became aware of the changes in the characteristics of post-modern meaning-making. He introduced the concept of ‘transmediation’, signalling an awareness of the translation of content from one sign system into another. The notion of ‘mode’ coined by the New London Group in 1996
offered the crossing point Suhor (1984) was referring to. Furthermore, when modes relate to sensory possibilities of the human body, they are seen as the ‘interface’ between natural and cultural world (Stein, 2008, p. 26). The senses thus offer us the connection between the real world and our understanding and ordering thereof. This ‘interface’ is what Andrew (2011, p. 102) refers to ‘makeshiftness’. Meaning-makers constantly shift between and adapt to different social and cultural circumstances, and are thus able to shift between different modal repertoires which are at their disposal in order to make the transportation of information more fluid and meaningful. Modal repertoires relating to all the senses of human experience offer a wider crossing for the transportation of meaning.

Modes have grammars

‘Grammar’ is seen as the system of a language which is governed by a certain set of rules. In other words, grammar is a set of structural rules that govern the composition of language into subject-verb-object word orders and relationships (Butterfield, 2008, p. 141). The key characteristic of grammar is that its function is more prescriptive than descriptive regarding the construction of the written and spoken word (Butterfield, 2008, p. 142). What Stein (2008, p. 25) seems to be stating is that most primary modes (visual and auditory), such as words, pictures, speech and sounds (music), all subscribe to their own ‘grammatical’ rule bases governed by their specific disciplines. Such modes are constructed from subscribed guidelines which govern their acceptance and validity.
To include touch, smell and taste to a list of modes operating in the visual arts would therefore be, according to Stein’s classification, problematic. The grammars (or lexicons) of touch, smell and taste operate more descriptively than prescriptively and are in most cases ambiguous, relying on individual or cultural preferences. What may smell sweetly or appealing to some individuals may not smell that appealing to others. The same may be said for touch and taste.
Smell and taste are more difficult to identify or to define than touch, since odours, fragrances and tastes are elusive. They cannot be measured in the same way other senses such as sight and sound may be measured, classified or identified. The sense of smell and taste are seen as more continuous than discrete. According to Finnegan (2002, p. 182) smells have no edges or boundaries and do not contain a beginning or an end. No matter how powerful the sense of smell, taste and touch may be, they are difficult to underpin. Finnegan (2002, p. 192) contends that smell does not necessarily reside in things but escapes and floats above them. Thus, smell and taste have not as yet been fully exploited as standard communication systems in the way that sound, and
visual senses have been, which may make it difficult to include and describe as modes. The modal status of the non-prototypical senses are further complicated by what Koster (2002, p. 27) claims to be a split between ‘high’ and ‘low’ sense classifications which started in the 19th century and still exists today. Touch, smell and taste fall within the ‘low’ sense category and despite advances in technology. Koster (2002, p. 28) believes that the status quo will remain due to the lack of rapid advances being made in technological developments.

CHAPTER 1 Purpose, background and problem statement 
1 PURPOSE AND BACKGROUND
1.1 National education policies and documents for arts education
1.2 Personal triggers
1.3 Origins and development of the notion of ‘multimodality’
1.4 Previous research
2 PROBLEM STATEMENT
3 RESEARCH QUESTIONS
4 OBJECTIVES
5 SUMMARY AND CHAPTER PREVIEW
CHAPTER 2 Theoretical underpinnings: The linguistic foundations of a theory of multimodality 
1 INTRODUCTION
2 HISTORICAL OVERVIEW OF MULTIMODALITY
3 STRUCTURALIST APPROACHES TO SEMIOTICS: SAUSSUREAN AND PEIRCEAN SEMIOTICS
4 SOCIAL AND CONSTRUCTIVIST APPROACHES TO SEMIOTICS
5 SUMMARY
CHAPTER 3 Modes and multimodality 
1 INTRODUCTION
2 DEFINING AND DELIMITING MODES
3 CHARACTERISTICS OF MODES
4 SUMMARY
CHAPTER 4 An overview of modalities and multimodality in the history of art 
1 INTRODUCTION
2 MULTIMODALITY IN VISUAL ART
3 ELEVEN MODES AT WORK IN VISUAL ART
4 SUMMARY
CHAPTER 5 Research methodology 
1 INTRODUCTION
2 PHILOSOPHICAL WORLDVIEW
3 RESEARCH APPROACH
4 RESEARCH STRATEGY: CASE STUDY
5 METHODS OF SAMPLING, DATA GATHERING AND DATA ANALYSIS
6 REPORT BACK
7 QUALITY CONTROL (STRATEGIES OF VALIDATION)
8 ETHICAL CONSIDERATIONS
9 SUMMARY
CHAPTER 6 Data analysis 
1 INTRODUCTION
2 BACKGROUND TO THE RESEARCH SITE
3 DATA COLLECTION AND ANALYSIS
4 APPLICATION OF THE FOUR THEMES IN AN ANALYSIS OF THE ARTWORKS
5 SYNOPSIS OF THE THEMATIC ANALYSIS
6 SUMMARY
CHAPTER 7 Framework for multimodal teaching and learning in visual art 
1 INTRODUCTION
2 OVERVIEW OF THE RESEARCH
3 FRAMEWORK FOR TEACHING MULTIMODALITY IN VISUAL ART
4 INSTRUCTIONAL DESIGN FRAMEWORK FOR MULTIMODAL ART PRACTICES BASED ON THE ADDIE MODEL
5 CONCLUDING THOUGHTS
LISTOF  REFERENCES
GET THE COMPLETE PROJECT