From collection to connection: Teaching and learning science in an interactive multimodal learning environment

Dr Sally Stephens, Director of Science and Mathematics

…the walls of the classroom are literally made irrelevant by the creation of communities of learners that span oceans, races, genders, and generations.
Richardson (2009, p.130) 

Take a snapshot of how students conduct their out-of-school lives and you will see them using a kaleidoscope of digital technologies to communicate, collaborate and form social constructs with the world around them. They might tweet, text, chat or blog, buy and sell online, Skype or use Facetime. Or they might play online games. On their Facebook page they might update their status, write on their friend’s wall, upload and view still and video images, sort their ‘friends’ into social groupings, make their likes and dislikes known, and play games with other Facebook users. They might watch a video clip of One Direction singing ‘What makes you beautiful’ over and over (preferably wearing their headphones) or even, like thousands of other people, create a video of themselves dancing to ‘I’m sexy and I know it’ to broadcast on YouTube.

Are they learning while they are playing in their digital world? If so, what developmental dimensions are being enhanced? While research results are inconclusive, there is MRI evidence that excessive gaming (more than sixty hours per week) may cause parts of teenagers’ brains to atrophy, reducing their inhibitions and affecting their concentration, memory, and ability to make decisions and set goals (Yuan et al., 2011). On the other hand, American pop science writer, Steven Johnson, author of Everything Bad is Good for You, drawing from fields as diverse as neuroscience, economics and literary theory, argues that computer games require concentration, forward planning, lateral thinking and sustained problem solving and, as such, offer intellectual demands that can benefit overall cognitive development (2005). Similarly, Gee (2003) claims that there are better learning theories embedded in video games than many students experience in the classroom because online video games encourage them to be critical, reflective and strategic. In terms of social development, a study by Ellison, Steinfield and Lampe (2007) investigated the relationship between Facebook usage and the formation and maintenance of social capital and found the use of this social networking site to be of particular benefit for users experiencing low self-esteem and low satisfaction with life.

Now take a snapshot of these same students in the classroom. What you find is a hive of activity as students and teachers wrestle with the challenges of a variety of rich learning experiences. There will be episodes of whole-class teaching, small group work, one-on-one student-teacher interactions and quiet or individual study. Students could be working from the textbook or completing a worksheet. They might be engaged in an audiovisual activity or information and communication technologies (ICT) work. In most classrooms, the online delivery of activities will be blended with more traditional forms of classroom instruction via an array of effective instructional strategies. Learning objectives are being achieved, but it is clear that despite our best efforts, there remains a sharp disconnect between the learning strategies used inside classrooms and how students are learning in their out-of-school time. According to Hramiak (2012), ‘many pupils go home to better computing facilities than they have at school … and have much fewer restrictions in what they can and cannot access at home compared with school’. Communication is increasingly digital so digital technology is an intrinsic part of students’ lives. Young people are heavily invested in multimodal technology, yet classrooms remain primarily print-based. Some social researchers fear that students who live multimodal lives will soon become disenchanted with formal school practices that employ only traditional literacies (Green & Hannon, 2007).

Being literate is a transformational experience; it is our window to the world. For many years, being literate was defined as the ability to read and write (Richardson, 2009) but, according to Leu, Kinzer, Coiro and Cammack (2004), prominent researchers in the field of new literacies, it is clear that we are entering a new era of literacy in which all forms of communication are themselves undergoing a transformation. Leu et al. contend that ‘as the medium of the message changes, comprehension processes, decoding processes, and what “counts” as literacy activities must change to reflect readers’ and authors’ present-day strategies for comprehension and response’. Thus, if we are to maximize the potential of continuously emerging ICTs, we will require new literacies, in both our personal and professional lives.

Leu et al (2004) believe that traditional definitions of literacy are inadequate and any attempts to frame a new definition must consider the rapid changes we are experiencing today and will continue to encounter as new ICTs regularly emerge. The definition of literacy by Luke and Freebody (2000, p.9) is ‘the flexible and sustainable mastery of a repertoire of practices with the texts of traditional and new communications technologies via spoken, print and multimedia’, which identifies the characteristics of a literate person that should endure in the face of emerging technologies.

New literacies combine letters, symbols, colours, sounds and graphics to expand the ways by which we communicate. It is clear that expanding the boundaries of literacy has implications for literacy instruction and curriculum development as we seek to prepare students for the new multiple literacies that will characterise their future. There is a need for schools to assist learners to bridge the gap between their informal practices and the demands of academic study that incorporate ICTs (Hramiak, 2012). They can do this by adopting and integrating the types of resources made possible by new technologies, such as electronic textbooks and the instructional strategies that complement them. Although research into these types of texts is still in its infancy, Larson (2010) reports that their use is accompanied by the sorts of positive attitudes and behaviours that can promote literacy development. Other research has found that teachers and students who have adopted new literacies have replaced language-based pedagogies with those that enable them to take full advantage of multimodal learning styles (Bull & Anstey, 2010).

According to the New London Group (1996), multimodal texts communicate meaning through the juxtaposition of two or more of the five possible patterns of meaning: linguistic (oral and written language), visual (still and moving images), audio (music and sound effects), gestural (movement expression and body language), and spatial (organisation of objects in a setting). From this definition it is clear that multimodality in texts is not new. Any science text with illustrations would fit the bill. What is new, however, is the interactivity that accompanies electronic delivery. According to Moreno and Mayer (2007), an interactive multimodal text is not just an electronic version of the textbook, but a resource that contains design elements that allow active engagement with the text that is not possible in print-based documents. What happens in an interactive multimodal learning environment depends on the actions of the learner. In fact, the defining attribute of interactivity is responsiveness to the learner’s actions: the text transforms as the student interacts with it. On the other hand, a non-interactive communication might include text and pictures, but they are presented in a predetermined way, irrespective of the actions of the learner.

Science teachers have long used multiple representations of concepts to appeal to the variety of learning styles favoured by students, but they are still largely print-centric and tied to language-based pedagogy. That is not to say that Science teachers do not use interactive resources, we do. During lesson planning we constantly search for electronic resources, such as apps, games, applets, and simulations, which we use to help students make meaning of both fundamental and complex concepts. The very best get incorporated into the teacher’s individual ‘collection’. Effectively we are supplementing traditional teaching methods with multimedia instruction but clearly we need to do more than this if we wish to bridge the technological divide and ‘leverage the power of emerging technologies for instructional gain’ (Klopfer, Osterwell, Groff, & Haas, 2009, p.3).

So where are we headed with the use of digital technology in science education at Brisbane Girls Grammar School? We know that multimodal technologies are already permeating the workplace as productivity and development tools and that there is a reason that these technologies are so pervasive (Klopfer et al., 2009). Industries, businesses, medical training facilities, government departments and scientific research facilities are recognising the advantages of these tools and are exploiting them to enhance their work. Research in educational settings has found that multimedia education improves both motivation and comprehension, which translates into positive learning outcomes (Brady, 2004). So, just as non-educational institutions are able to see the benefits of these tools for their core business, we, as science educators, see enough evidence of improved learning outcomes to warrant the integration of emerging technology-based resources and pedagogies into our new curricula. There is a new wave sweeping across the educational landscape and we want to make sure that we are on the front end of it. We cannot let the emergence of new ICTs outpace pedagogy.

So that we are better able to adapt to the dynamics of our changing world, science teachers need to redefine ourselves in ways that will transform our repertoire of teaching strategies. Richardson (2009) argues that this will only be achieved if teachers commit themselves to the five different roles described here: 1) to be able to teach multiliteracies successfully to students, we must be able to use them competently ourselves, so we must learn to use new technologies effectively enough to become creators of content; 2) we must join with our students to become true collaborators in the learning process; 3) as students gain real-time access to trustworthy, authoritative information, we can no longer claim to be the most expert voice in the classroom, so we must learn to think of ourselves as coaches, those who train students to find their own information, and to critically evaluate and make sense of it; 4) we need to update our view of what it means to be a connector, the one who, in this new era of emerging multiliteracies, continues to use appropriate pedagogies to make the mandated curriculum accessible to all students; 5) finally, to transform the print-centric classroom to a multiliterate one, teachers have to become agents of change (Richardson, 2009). The time has come to move from ‘collections’ of resources to ‘connections’, to exploit the connectedness of interactive multimodal texts with all of the benefits that they afford our students.

Today’s students live in a highly interactive multimodal world where they appear, to many, to be skilled and confident consumers of emerging digital technologies. But are they as savvy as they seem? Our challenge as science curriculum developers is to channel the skills students are developing in their out-of-school lives into the academic arena so that they can receive guidance in the development of the desired ‘repertoire of capabilities’ they might not be able to acquire on their own. Interactive multimodal texts can promote new literacies practices and strengthen the connection between learners and text as engagement with, and manipulation of, text is enabled through the tools and features associated with electronic media. The ultimate goal is to future proof students’ multiliteracies.



Brady, J. (2004). More than just fun and games? Applied Clinical Trials. Retrieved April 29, 2012, from

Bull, G., & Anstey, M. (2010). Evolving Pedagogies; Reading and Writing in a Multimodal World. Retrieved April 3, 2012, from

Ellison, N. B., Steinfield, C., & Lampe, C. (2007). The benefits of Facebook “friends”: Social capital and college students’ use of online social network sites. Journal of Computer-Mediated Communication, 12(4). Retrieved April 3, 2012, from

Gee, J. P. (2003). What Video Games Have to Teach Us About Learning and Literacy? New York: Palgrave/Macmillan.

Green, H. & Hannon, C. (2007). Their Space: Education for a digital generation. Retrieved April 29, 2012, from

Hramiak, A. (2012). What does it mean to be literate in 2012? Retrieved April 3, 2012, from

Johnson, S. (2005). Everything Bad Is Good For You: How Today’s Popular Culture Is Actually Making Us Smarter. New York: Riverhead Books.

Klopfer, E., Osterwell, S., Groff, J., & Haas, J. (2009). The instructional power of digital games, social networking and simulations and how teachers can leverage them. Retrieved April 29, 2012, from

Larson, L. C. (2010). Digital readers: The next chapter in e-book reading and response. The Reading Teacher, 64(1), 15-22. Retrieved April 3, 2012, from

Leu, D. J., Kinzer, C. K., Coiro, J. L., & Cammack, D. W. (2004). Toward a Theory of New Literacies Emerging From the Internet and Other Information and Communication Technologies. Retrieved April 3, 2012, from

Luke, A., & Freebody, P. (1999). Further notes the four resources model. Retrieved April 3, 2012, from

Moreno, R., & Mayer, R. E. (2007). Interactive multimodal learning environments. Educational Psychology Review, 19, 309-326.

New London Group. (Spring, 1996). A pedagogy of multiliteracies: Designing social futures. Harvard Educational Review, 66 (1), 60-92. Retrieved April 3, 2012, from

Richardson, W. (2009). Blogs, Wikis, Podcasts, and Other Powerful Web Tools for Classrooms. 2nd ed. California: Corwin Press.

Leave a Reply