Reading Multimodally: Designing and Developing Multimedia Literacy Projects through an Understanding of Eye Movement Miscue Analysis (EMMA)

Maria Perpetua Socorro U. Liwanag and Steve Dresbach, State University of New York at Geneseo

Pre-service teachers use Eye Movement Miscue Analysis in designing book trailers.

The influence of new technologies has set a new direction for our understanding of literacy and literacy teaching and learning. For this reason, teacher educators need to support pre-service teachers’ understanding of how the use of new technologies facilitates students’ literacy learning. Informed by comprehensive theory of reading (K. Goodman and Y. Goodman 2011) and multimodal theory (Kress and van Leeuwen 1996), we share in this article a multimedia literacy project, which is a book trailer, that allows pre-service teachers to apply their knowledge of what it means to read using 21st century technologies. A comprehensive theory of reading explains how reading works and how understanding language extends our knowledge of how the brain makes sense of the world (K. Goodman and Y. Goodman 2011, 27) while a multimodal perspective (Kress and van Leeuwen 1996) considers how different modes of representation, such as writing, speech, visuals, sounds, and music, help us make meaning of our world.

We describe in our inquiry how we extended the work of Kim, Duckett, Brown (2010) and others (Paulson and Freeman 2003; Kim, Knox, and Brown 2007) who have used eye movement miscue analysis (EMMA) data as an instructional tool to inform pre-service teachers of the complexities of the reading process. EMMA is a research methodology that combines miscue analysis with eye tracking research, allowing researchers to record a reader’s eye movements while reading (Paulson and Goodman 1999). We used EMMA as a teaching resource to help explain the complexities of the reading process. Then we utilized multimodal instruction to guide pre-service teachers on how to communicate their ideas in various modes as they design and develop a book trailer. A book trailer is a visual presentation of a book, similar to a movie trailer. The application of EMMA research and multimodal instruction enables us to integrate literacy and technology teaching. We are able to foster an understanding of how we read multimodally by engaging our students in discussing the dynamics of reading through the use of EMMA data and utilizing various sign systems to represent meaning in their book trailer projects. This article presents how we work together with pre-service teachers to understand how creating book trailers through knowledge of EMMA and multimodal instruction supports literacy teaching and learning.

Context

Pre-service teachers taking their second and final literacy course in teaching 4th-8th–grade students are involved in a variety of learning experiences to prepare them to be future teachers of literacy. An expected learner outcome for the course is to: “Demonstrate knowledge of and select quality resources (i.e., literature and technology), which enhance lesson planning and teaching skills in the English Language Arts for all students” (Course syllabus 2009, 1). To accomplish this, we have assigned a multimedia literacy project, the book trailer (Appendix A).. This book trailer project connects students’ learning from other course assignments like (1) giving booktalks to engage readers (Atwell 2007), (2) participating in online discussions of young adult literature, and (3) discussing and writing reflections on the reading process and eye movement research (Garan 2007).
We collaborated on all aspects of the book trailer project like planning, organizing, and providing instruction to help our students complete this assignment. Since we started work on this project three years ago, we have learned to improve our procedures and instruction. Figure 1 (below) shows how we have organized our literacy and technology instruction:

Figure 1. Support for pre-service teachers at different stages of the book trailer project.

Design and Development of a Book Trailer

To help our pre-service teachers design and develop a book trailer, we planned lessons on: (1) using selected software to create a book trailer, (2) selecting high-interest books, (3) identifying internet resources and using these in a responsible and scholarly manner, (4) understanding the reading process, and (5) integrating different modes of communication to represent meaning.

Lesson on software use

The book trailer project is only one of several learning experiences for the course, and we struggled to find the appropriate number of sessions to devote to learning the software. We eventually determined that allocating four instructional class sessions works well for our students. We use these four class sessions (50-60 minutes each) to introduce our students to MS PowerPoint, ProShow Producer, and iMovie. Steve, the technology instructor, begins the lesson with examples of book trailers created with each of the three tools. He lectures on how the structure, text, images, colors, animation, transitions, and sounds influence a book trailer’s impact depending on the selected software. Pre-service teachers actively identify and sample the different features of the software during these technology instructional sessions. Although we only have four technology sessions in class, pre-service teachers continue to consult with us outside class hours for any support or questions regarding this project.

Selecting high-interest books

We encourage our pre-service teachers to begin drafting their book trailers during the technology sessions while we observe and give feedback on their work. The first semester we implemented this project, we learned that pre-service teachers get more out of the technology instruction when they have already selected a piece of young adult literature (YAL) to develop into a book trailer. They are also more likely to try out the different features of the software when they have some knowledge of what their book is about, and even more likely to do so if they have already read and discussed their selected YAL with their peers. This confirms how a comprehensive model of reading works to support pre-service teachers as “learning…is integrated, unifying what is learned and how it is learned” (K. Goodman and Y. Goodman 2011, 20).
Part of the literacy course requirement is to read and discuss YAL materials. We do booktalk presentations several times throughout the semester. We follow Atwell’s (2007) recommendations in doing a booktalk which includes being short (lasts two to three minutes) and direct, be on books that one would recommend and share a bit of the storyline without revealing the story’s ending., Pre-service teachers find this procedure helpful in identifying high-interest books. Students’ expanded knowledge of a book provides opportunities to visualize the possibilities of their book trailer project.

Identifying Internet Resources

We wanted our pre-service teachers to have the resources to use internet materials (e.g., images and music) responsibly, so we asked our institutional librarians for assistance regarding issues of copyright and academic fair use. Our librarians conducted instructional sessions to discuss and share with our students ways to find, select, and cite internet materials in their book trailers.

Understanding the reading process

Critical to the development of a book trailer is an understanding of how we read and how technology can be used as a tool to enhance students’ literacy learning. We incorporated discussions about reading processes and strategies when we discussed how reading works with pre-service teachers (Liwanag and Kim, 2011). To extend their understanding of the complex nature of reading, we used eye movement miscue analysis (EMMA) data to visually demonstrate what readers do when they read. EMMA is a methodology that combines eye movement data, oral data, and retellings through miscue analysis procedures (Goodman, Watson, and Burke 2005) to provide a map of a reader’s reading (Paulson and Freeman 2003). Miscue analysis is a diagnostic tool for observing how miscues or observed responses in a reading differ from the expected response. It is also based on the understanding that “reading is an active search for meaning” (Goodman et al. 2005, 4).

Integrating different modes for meaning-making

One goal of this multimedia literacy project is for our pre-service teachers to expand their knowledge of what it means to read multimodally. Kress (2004) defines multimodality as “deal[ing] with all the means we have for making meanings—the modes of representation—and consider their specific way of configuring the world” (1). We addressed this by including lessons about multiple ways of knowing, like how to read art and other sign systems (Harste 2009; Albers 2007; Kress and van Leeuwen 1996). By combining these lessons with the understanding of the reading process gained through the EMMA data, pre-service teachers learned how the selection and characteristics of image, text, color, sounds, animation, and transitions in the book trailers all influence the meaning of a book.

Using EMMA Data and Multimodal Instruction as a Teaching Resource and Strategy

Using EMMA data serves as an invitation to examine the reading process (Kim et al. 2010) and to read multimodally by demonstrating how readers pay attention to different modes of communication for representing one’s meaning. Knowledge of multimodality also helps pre-service teachers understand and recognize that moving from one sign system to another encourages broad and critical learning (Harste 2009).
EMMA data consists of oral recordings and a recording of an individual’s eye movements. In one example of EMMA data we’ve shown, a fifth grader, Marina,¹ reads a book trailer for James and the Giant Peach (Dahl 1999) by Dianne (a pre-service teacher). Figure 2 shows a PowerPoint slide from this book trailer with Marina’s fixation points overlaid on the slide to show her eye tracks as she read the specified slide. Fixation points represent “what the reader is paying attention to on a millisecond-by-millisecond basis…what the reader looked at, where the reader went next [and] how long [a reader] looked at a word…” (Paulson and Freeman 2003, xii). Marina’s fixation points 226-235 (in blue dots) showed that she did not look at every letter or every word. She also reread and focused her attention on the center of the page.

Figure 2. Partial PowerPoint slide for Marina’s fixation points 226-235.

In another example of EMMA data, Andrea, a pre-service teacher, reads the book trailer for Keep Your Hair On (Vercoe 2004) developed by pre-service teachers Karina and Beth. In the selected slide example (see Figure 3), Andrea’s fixation points 334-358 are overlaid on the slide with text and image. The fixation data shows that she did not look at every letter or every word. Her eye track also shows eye movements that focus on the face in the image and move from text to image and back to text again.

Figure 3. Andrea’s fixation points of a slide with text and image.

We asked our pre-service teachers to reflect on a set of EMMA questions (see Appendix B) after showing selected EMMA book trailer data clips to the class. A sample section of an EMMA book trailer clip can be viewed at http://vimeo.com/35058799. The black cursor in the EMMA data clip represents where the eyes are looking. Sample responses below showpre-service teachers’ insights about what the viewers of the book trailers paid attention to, what they noticed about the use of text, images, and music, and how other features like timing, use of color, and words can enhance one’s understanding of the book trailer. They developed a deeper connection between elements of design (text, image, color, boundary, transition, and music) and how the readers viewed the book trailers.
Insights about the book trailers:

• “[Reader] looked very quickly at some spots and didn’t look at other parts of the frame.”
• “Students expect to see things in certain areas and then realize that there’s nothing there or what really appears there is not as important as it seems.”
• “Reader went back several times at some point in the trailer.”

Comments on the use of text, images, and music:

• “I find it interesting that the reader looked at the text first then the picture later.”
• “Readers tend to look at both the words and pictures to put the information together. It is key that we understand how both these features [words and pictures] help our readers.”
• “We will try to incorporate better music so no one gets turned off.”

Comments on timing, colors, and word choice:

• “The choice of single words is powerful…the simple phrase helps get across the message.
• “One thing EMMA helped us recognize is the importance of timing. There needs to be enough time given to readers…and getting rid of pictures that can be distracting…”

Pre-service teachers realize that reading book trailers also entails reading what viewers determine as important to them. Reading is more than recognizing the text and images. It is also what viewers take from what they visually perceive.
Using EMMA for instruction provides a powerful tool for demonstrating to pre-service teachers how reading works as they understand “how the brain uses the eyes to provide the input necessary to construct meaning” (K.Goodman and Y. Goodman, 2011, 28). Myths about reading (Meyer and Whitmore, 2011) are debunked as new inquiries about why readers read the way they do are further explored and examined. Explanations of reading, such as those that describe the reader sequentially reading all the letters in the words, are rejected, and the complexities of the reading process are recognized. Through the use of EMMA, pre-service teachers better understand why reading cannot be reduced or taught in isolation, as context plays a key role in comprehension. The EMMA data examples above serve as a catalyst for exploring the reading process and an invitation for reflecting on instructional strategies that support the complexities of how reading works. EMMA data helps pre-service teachers expand their definitions of literacy to include reading multimodally using 21st century technologies.

What We Have Learned: Supporting Pre-service Teachers in Creating Book Trailers

Creating book trailers enables pre-service teachers to gain knowledge about ways to provide students in grades 4-8 with a multimodal instructional experience and enhance their literacy learning. The need for content and technology expertise provides a great collaborative effort between an education professor and a library technology instructor. This collaboration helps improve technology instruction while addressing pre-service teachers’ academic development. We also were able to facilitate pre-service teachers’ understanding of meanings embedded in the use of text, images, colors, animation and timing, and sounds in the development of their book trailers. These elements help them understand the complexities of the reading process and better design their book trailer project.
EMMA data reflections provide an opportunity for our class to talk about ways to choose, develop, design, and critique the content and format of book trailers. Awareness of the role played by text, images, color, animation, and sound helps pre-service teachers move from one sign system to another to represent their meanings. This process helps them read multimodally and become critical users of texts (Harste 2009). It expands their knowledge of how technology can be utilized as an effective tool for literacy instruction and learning. In the final presentation of the book trailer project, pre-service teachers become part of a community with a shared digital literacy experience.

Reflections on our roles as collaborators

We took a reflective practitioner stance (Schon 1983) as we each recognize that our questions about the use of book trailers in our course became a point of learning for both of us. The questions we faced include identifying the appropriate number of sessions devoted to technology instruction, what software students can use to develop their book trailer, and how to provide information about responsible and scholarly use of images and music from the internet or other resources.
As a technology instructor, Steve knew we had only so much time to devote to all the learning experiences planned for the course, and asking for additional technology instructional sessions meant less time would be available for other course instruction. Steve explained the most effective way to cover the content and deferred to my decisions as the course instructor. We tried different numbers of sessions ranging from two to four instructional meeting dates as we were implementing the project and found that devoting four meeting sessions to technology instruction worked well for the course schedule, for the technology instruction, and for our students to get a good understanding of the technology involved in the project.
As a literacy professor, I (Maria Perpetua) also had to let go of my own technological insecurities. For instance, I was comfortable using MS PowerPoint so I required students to use this software to develop their book trailers in the first two years of the project. I eventually learned through Steve’s recommendation that other software, like ProShow Producer and iMovie, can be advantageous for our students. I had to let go of my hesitation to use unfamiliar tools and learn to use them, accepting that even if I was only a step ahead of my students, it was okay. On our third year implementing the book trailer project, we used MS PowerPoint, ProShow Producer, and iMovie to develop book trailers. Shifting my view of teaching and relying on Steve’s expertise allowed us both to collaborate and maximize our students’ learning experience.
To inform our pre-service teachers about copyright and academic fair use, we sought the help of our institutional librarians. They presented and held class workshops for our pre-service teachers and guided them in finding, selecting, and citing materials for inclusion in their book trailers.
Working on this project has enhanced joint commitment to our students, and we have benefited from working collaboratively with a colleague. We both did what we needed to do to make excellent and effective instruction for our students and make our mutual respect for each other’s expertise the cornerstone of our own inquiry as we endeavored to merge literacy and technology instruction for our pre-service teachers.

Conclusion

Using EMMA data helps us examine how content, structure, and design of multimedia technologies can support pre-service teachers’ use of technology. Through this multimedia project, pre-service teachers collaborate and evaluate different technology resources. They are able to critically examine how book trailer formats and content can motivate students to expand their own learning development and prepare them for the literacy challenges of the 21st century.

Appendix A: Multimedia Literacy Project on the Development of a YAL Book Trailer

Multimedia Literacy Project: YAL Book Trailers

Design and develop a book trailer using either MS PowerPoint, ProShow Producer and iMovie. The book trailers are a tool to:
a) Booktalk a YAL literature,
b) Expand students schema about high interest books of different genres, and
c) Engage students in a multimodal learning experience through the use of book trailers.
Book trailers should have text, images, animation, transitions, and audio. Materials included in the book trailers should follow copyright laws and be referenced using standard APA citations.

Grading for book trailer:
1. Content (short, direct, engaging, hints at climax but not revealing) 5 pts.
2. Creativity (adds own interpretation, does not overdo design, purposeful and
meaningful selection of text, images, animation, transitions, and audio) 5 pts.
3. Design (relevant to purpose of a digital booktalk, actually works, able to utilize the
abilities of the software to demonstrate an engaging book trailer) 5 pts.
4. Copyright laws, citations, and credits are used and properly recognized 5 pts.
5. Fair participation/contribution of ideas, time, and work 5 pts.
Total 25 pts.

Appendix B: Reflections on EMMA Book Trailer Data

1. What did the viewer pay attention to/ focus on/ draw their attention to as they were looking at the book trailer?
2. How long did they spend looking at text? Images?
3. Where do the eyes focus when objects (text and images) appear simultaneously?
4. How are viewers anticipating where text or an image appears next?
5. How much detail in an image influences where viewers look?
6. How long do viewers look at the text or image?
7. How does the duration of text and images influence viewers’ interest or comprehension?
8. How has the EMMA data helped you select the appropriate software for your project?
9. How can the software selected help improve or limit design?
10. What elements of the software did you consider important in developing your multimedia literacy project?