Online Learning /

Two people discuss by a computer and smartphone, gesticulating with their hands.

Authoring an Open-Source Game for a Faculty Open Educational Resources Workshop: A Case Study

Katherine Foshko Tsan, Baruch College-CUNY Center for Teaching and Learning

Abstract

While most college institutions running open educational resource (OER) conversion initiatives currently teach some online version of the OER faculty workshop, little scholarship exists on how this type of instruction can be optimized for best comprehension of, and engagement with, open resources, especially in a remote context. An overview of existing models of the workshop and their digital pedagogy philosophies is followed by a discussion of one initiative currently implemented by Baruch College’s Center of Teaching and Learning, which embraces online gaming as an effective modality to reinforce textual and auditory learning. The Old Régime and the OER Revolution, an online interactive tutorial created with the open tool Twine, is the first RPG for faculty to engage with open educational resource conversion issues. The game can be accessed in its current beta version. Because Twine encourages players to think in terms of choices and their implications while considering multiple scenarios, introducing this tool to college faculty via the OER workshop can also encourage its uptake in a plethora of other pedagogical contexts.

Introduction

The faculty development OER workshop is one of the most important, yet little-researched, links in the chain of adoption of open resources on campus. Most of the models that exist acknowledge the importance of concept learning for instructors encountering terms such as OER, ZTC, and open pedagogy for the first time. Moreover, teaching faculty about best practices in fair use and Creative Commons licensing—a related field—has a trickle-down effect to undergraduates. One way to achieve familiarity with open education is through hands-on learning, where instructors model examining and converting course materials to open or zero-cost and abiding by U.S. copyright law.

The digital game-based module in this case study was created as an alternative to the traditional workshop, to meet the needs of instructors as observed after several years of an OER initiative. It gives the users an opportunity to be active participants in a multimedia narrative and provides them with the flexibility of an asynchronous, iterative assignment. It also bases itself in scenario-based learning—problem-centered instructional solutions, which complement and cement the more straightforward directional learning in the regular faculty workshop. And, as a program made with a relatively uncomplicated open-source tool, it provides an attractive use case for both faculty and students.

The OER Faculty Development Workshop: Going Digital, Going Multimodal

The workshop in OER for faculty has not yet emerged as a subject of research, although a few on-campus initiatives have presented their versions, explaining decisions around context and organization. At a base level, the class primarily focuses on certain topics germane to the field: what are open resources and open education in general, what is meant by zero textbook cost, how do open licenses work, why follow best practices when pursuing free content, and how do instructors follow accessibility guidelines. Practice activities involve faculty participants searching for materials to replace assigned readings or, if the initiative is grounded in goal-based learning, determining the course objectives and the kinds of resources required to meet them. More than a workshop, this scope of subjects suggests a training program designed for specialist-facilitated, hands-on learning. However, given the relative novelty of the twenty-year-old field, no set rules exist and no single model prevails.

Several teaching bodies have seized upon the potential of the OER workshop to be taught remotely. In 2017, the City University of New York (CUNY) was awarded a $4 million package as part of a Scale Up! OER grant “to establish, sustain and enhance new and ongoing OER initiatives throughout CUNY” (CUNY Libraries 2017). Most of the work of converting a class to open involves searching across multiple databases, and variable concerns arise depending on the instructor’s discipline. The nature of instruction, even at these online-optimized institutions, thus remains of necessity hybrid, combining asynchronous content delivery with a synchronous mode for live discussion and troubleshooting. The possibility to take such courses remotely and for free has added convenience and flexibility; their parallel emergence at public commuter colleges in the CUNY and the Washington State Community college systems has been no accident.

In 2017, Lehman College-CUNY Director of Online Learning Olena Zhadko and Susan Ko, Director of Faculty Development at the NYU School of Professional Studies, developed an asynchronous, online faculty OER workshop to be offered for a two-week period several times during the year. The mini-course, designed for incorporation into the Blackboard LMS, scaffolds understanding of OERs through a module-based system, based on a tripartite organization of defining, finding, and integrating OER, as well as a discussion forum. The learners are asked to think about how open fits into their teaching and articulate their rationale for selecting a specific OER. Their final project in the workshop consists of a list of course materials replaced by open or zero-cost equivalents, with justification and description of intended use: adoption, adaptation, or creation. The library department at Lehman College has an accompanying final assignment for hands-on practice, which tests participants’ ability to link to websites, images, and video, apply attribution, and choose Creative Commons licenses (Cohen 2018).

Simultaneously with Lehman, the Washington State Board for Technical and Community Colleges—the creator of the Open Attribution Builder—developed a free, self-paced course made up of ten modules, including one on accessibility and a reflection on “Why OER Matters.” The website likewise includes “Personal Journeys Using OER,” a series of videos by on-campus practitioners. This version appears less hands-on yet also more personal than Lehman’s, emphasizing the human element of course conversions. It co-exists in a face-to-face format, offered over a two-week period at set points throughout the academic year. These inroads in digital education have opened up the possibility of yet more interactive ways for faculty to engage with the material in concert with the existing learning structure.

Creating a New Mode for the OER Workshop at Baruch College

The Baruch College-CUNY Center for Teaching and Learning joined the OER initiative the same year as Lehman and created an in-person faculty seminar accompanied by a set of slides since made publicly available on the Center’s OER pedagogy site. Baruch’s seminar, conducted in a mix of presentation and workshop elements, features backward course design (What outcomes am I looking for? How will I measure if these outcomes have been achieved?) as well as choice of platform (mainly between the Blackboard LMS and the proprietary, though open, WordPress site Blogs@Baruch). The most recent addition to the topics covered in the workshop have been open-source digital tools (Hypothes.is, StoryMap JS, Timeline JS, Voyant, and Twine), which are simple to use and versatile enough to create OERs for both teaching demonstrations and final assignments, i.e. teacher- or student-led work.

In 2019, on the suggestion of Center for Teaching and Learning Director Allison Lehr Samuels, I began creating a future addition to the workshop. In the spirit of providing opportunities for faculty to educate themselves about OER and related topics, the game, like the slides, was to be made openly available on the TeachOER website and licensed under a Creative Commons Attribution-ShareAlike license (CC-BY-SA). Four goals for this project had materialized: (a) showcasing one of the digital tools, (b) adding another mode to the regular workshop to cater to different kinds of learners, (c) creating a fully asynchronous engagement experience with the workshop, which could also serve as review material, and (d) incorporating recurrent questions that had come from workshop participants for the past three years of the OER initiative.

The technology to carry out these diverse functions had to align with the Center’s principles of using open-source tools to support interactive digital solutions whenever possible. The college had already delved into creating single-player digital training with the 2009 Interactive Guide to Producing Copyrighted Media, developed by Baruch’s Computing and Technology Center and Kognito Interactive, a paid tool used for health simulations, owned and managed by college alumni. The experience took the user through an interactive maze consisting of various copyright and fair-use related situations drawing on such resources as U.S. copyright law, copyright guidelines for CUNY libraries, and public domain information. While this work was produced before any of today’s more common open-source tools and predated the state grant-funded initiative by several years, the Baruch library’s concern for adhering to best practices in decisions around intellectual property has been behind much of its OER programming as well.

The ultimate decision to make the new OER faculty workshop model with Twine, a non-linear, interactive storytelling tool for making single-player games based on decision-making, was grounded in Baruch’s longtime support for classroom technology as well as the Center for Teaching and Learning’s embrace of game-based pedagogy. According to the Center’s team,

The prevailing theory is that games enhance and mobilize internal desires and motivations for learning…[W]hen the rules of a game are tailored around learning outcomes, students are self-motivated to learn in order to ‘win’. Role-playing games in particular have been shown to increase student engagement and motivation to read, enhance persuasive writing and speaking, and increase critical and analytical thinking. (Baruch College CTL 2020)

Pedagogy games have in fact been shown to support classroom learning and stimulate greater engagement from students (Clark 2016; Gross 2007; Hamari, Koivisto, and Sarsa 2014). Internationally renowned game designer Jane McGonigal points out the salutary effect of games on students who experience the “satisfactions of achievement and mastery" (McGonigal 2016, 231). And a 2012 study by the Mathematics Education Research Journal showed 93% of class time was spent on task when using game-based learning, compared to only 72% without it (Bragg 2012). As the anti-rote memorization approach, game-playing is directly associated with active learning and has been a part of a general pedagogical shift towards fostering creativity by working through challenges and coming up with innovative solutions (Cheka-Romero and Gómez 2018; Lameras et al. 2015).

The Baruch CTL, already experienced in active learning practices, therefore considered the gaming approach optimal for the dual task of creating an alternative mode for faculty instruction and modeling new teaching techniques. And, with the target audience being the college’s largely contingent and overextended faculty, the team’s hope was that this exercise’s immersive approach would provide a high engagement—as well as an edutainment—quotient.

Twine and Digital Pedagogy

“Games should be created by everyone” is a motto that Chris Klimas, the creator of Twine, abides by in his work (Klimas 2020). The award-winning, Baltimore-based web developer and game designer published his text-authoring tool in 2009. The target audience was writers, and the experience of creating with Twine was meant to resemble a brainstorming exercise. Paying tribute to this legacy, its home page, www.twinery.org, to this day represents a corkboard with multi-colored notes attached to it with pushpins—a working surface of inspiration.

Passages in Twine’s editor interface appear in a simple, visual flowchart, a map of text nodes connected by arrows, to be filled out, then moved and rearranged at will. The starting point is a single node, and new ones are made when a user encloses the name of another passage in double brackets to continue the story. The scripting does not require any programming experience, although basic HTML skills are needed to change the style sheet and embed images, audio, and video. These technical features add up to a flexible, user-friendly tool with a low barrier to uptake, as well as quick connection capabilities due to its text-based nature. Scholar Anastasia Salter (2016) explains that an additional strength of Twine is its deregulated publication process, meaning that games can be distributed as quickly as they are created—unlike most products of subscription-based gaming websites, which go through an extensive review process before they can go live. This affordance of the open-source tool has also encouraged a greater immediacy and unselfconsciousness of the products made with it, as will be discussed later.

The main vehicle for moving through a Twine game has traditionally been the decision made by the user at the end of each passage between two or three options, each leading to a different fork in the narrative (occasionally, for the purposes of simply proceeding with the story, there may be just one choice). The origins of this decision-tree setup can be traced to “Choose Your Own Adventure,” a series of interactive fiction created in the 1970s by Edward Packard. Klimas (2020) fondly remembers devouring the CYOA books during regular visits to the public library and, later on, trying to “repurpose the genre for something more adult.” A manual on Twine for which Klimas served as the technical editor has recommendations for strong character arcs and narrative devices such as gifts and secret powers (Ford 2016). The author, who envisioned games like Dungeons and Dragons in book form, has since seen his stripped-down writing tool take on a superhero’s cloak—many of those using Twine build role-playing games that feature fighting fantasies. Another influence was hypertext in early Macs and the culture of object-oriented programming in graphical environments.

In a surprising development, yet one that its creator has heartily welcomed, the indie gaming community originally put Twine on the map in 2012–13, crafting a whole body of “vignette” stories with messages that are often unsettling or subversive. In a field dominated by males, its association with female voices and the experience of queer and marginalized groups has lasted to this day (Harvey 2014). Titles made with Twine range from the offbeat Cat Petting Simulator to 2014’s The Depression Quest, a work initially controversial for its very existence as a “game” yet now hailed for raising awareness of clinical depression by including crossed-out “non-choices”—regular actions impossible for someone afflicted with the condition. Salter (2016) refers to this feature as “metaphors of limitations” and praises the power of Twine’s economy of means as helpful for conveying emotion and raising awareness of mental health issues. (She also stresses that it is one of the few vehicles for games with non-mainstream/taboo subjects.)

While Twine has largely flown under the radar, the darling of a niche creative community, greater exposure has recently come to it from popular artistic culture—and, to some extent, the educational environment. The new-media artist Porpentine Charity Heartscape, whose unsettling, subversive games are made largely with the tool, had her works, With Those We Love Alive and howling dogs, featured in the 2017 Whitney Biennial (Klimas 2017). The following year, Charlie Brooker, the creator of the interactive Netflix film Black Mirror: Bandersnatch (2018) announced that its hundred-plus page script was written entirely in Twine (Aggarwal 2019). Around the same time, instructors began discussing their own teaching projects made with the tool and posting assignments made for humanities courses on the open web (McGrath 2019) (McCall 2017, 2019) although the practice remains limited to select classes by the enthusiastic few.

Salter credits Twine’s orientation toward the agency of the main character and lack of a peer-reviewed publishing model with its rising prominence: “Compared to the intensive team-based productions of 2D and 3D games, text-based games continue to offer a space for individual production.” In an industry known for technological extravagance, Twine is the plucky underdog that can achieve inspired—and pedagogically sound—results within a surprising economy of means.

The Baruch Twine Game: Scenario-Based Learning in a “CYOA” Format

According to its reviewers, Twine’s greatest affordance is that it deepens understanding and engagement through exploring alternatives. The Old Régime and the OER Revolution game also aims to mimic the branching, decision-making process of selecting the materials for a class: why and how should faculty abide by best practices? What are the consequences and trade-offs of certain decisions one makes in choosing freely distributed over proprietary copies—for legal repercussions, plain peace of mind, pedagogical freedom? And, more generally, why should instructors care about these issues if they do not yet teach with open resources? The title, juxtaposing “the old ways” and the “OER turn,” alludes to the main character’s specialization: in the game, “you” are a professor of French history. Yet more broadly, it references the pedagogical transformation involved in switching to open resources, which cover a wide range of mediums and modalities and involve not just a replacement of materials but also an examination of the way one teaches.

The decision that launches the game is not to teach in the usual way, as the introduction says of the main character:

You have always struggled with how unsatisfactory and expensive textbooks on the subject are…Given that your students come from so many diverse and international backgrounds, you wish there were some texts that talked more at length about their histories. (Tsan 2020)

As explained in a footnote, the prototype for this hero is Helmut Loeffler, a professor in the History Department of Queensborough Community College, CUNY, who authored a text, Introduction to Ancient Civilizations (2015), with a small grant, creating a book that understandably eschewed timelines, charts, or illustrations (Loeffler 2019). The non-fiction aspect of this “origin story” of the interactive fiction game was intentional, since the overarching goal of any faculty OER workshop is to inspire by actual example.

The Old Régime and the OER Revolution game follows Scenario-Based Learning (SBL), a popular instructional strategy in online training. In this approach, facilitators use situations close to ones that might come up and introduce applicable problems, to which the participants try to figure out the answers. Because SBL is grounded in the premise of applicability for the learners, the role-playing game aspect of many Twine products does not apply here. There are no magic friends, secret weapons or helpful gifts picked up, and the experience stays in the realm of the believable. (However, more than one aside, such as the French history professor’s waxing eloquent over a War and Peace passage she is choosing for her open syllabus, or the character of a curmudgeonly faculty member determined to resist the incursions of OER content, have been included for the purposes of levity or edutainment).

The single-player Old Régime game was designed to be non-competitive and low-stakes, a “game” in the sense of an adventure rather than a quest that results in “winning” or “losing.” In this sense, it is inspired by Jane McGonigal’s assertion that “good game developers know that the emotional experience itself is the reward” (McGonigal 2011, 244). Like most Twine tutorials created for learning and instruction, it follows the simple mode of SBL, “used to validate the learner’s recall and basic comprehension” and “good for basic problem solving” (Pandey 2018). In a similar vein, a Twine game recently made at Hostos Community College, CUNY, “encourages college-level English Language Learners to practice grammar concepts as they play the game narrative” (Lyons and Lundberg 2018). Making the wrong choice of several options sends the user back to do the exercise again until they get it right and can proceed with a visit to the Metropolitan College of Art (Met)—a relatable exercise for Hostos ELL students who frequently go to the real-world Met on field trips. The Old Régime aims for a more branching narrative, since, due to variations in the availability and permissions of course material, the choices taken by an instructor can only sometimes be classified in the right or wrong category.

While The Old Régime does not frequently correct the user’s choices, it has built-in capabilities for addressing decisions that may result in an impasse. If the answer is blatantly incorrect (“copy a 1992 book in its entirety and distribute the PDF to your class”) or fraught (“embed the YouTube link to a culinary video directly in your course website”), it leads to an “explainer”—a tactic common to Twine narratives where the user can explore links embedded within the body of the text for extra information and then go back to the main narrative. The explainers used in the game align almost exactly with the principal concepts being communicated: OER, zero-cost materials, saving money on textbooks, copyright/fair use doctrine, Creative Commons licenses, and open pedagogy (see Figure 2). Since the premise is that participants go through this experience to learn rather than be tested on their knowledge, the game aims to take them through the real-life consequences of certain actions. For instance, one storyline explains that, in a rare case, you may receive a DMCA (Digital Millennium Copyright Act) notice to take down illegal digital content, and another shows how always having a rationale for why you are obeying the fair use doctrine in a specific case might save trouble and keep you on the right side of copyright law.

The story map of the game, which shows the entire story projected on one screen, divided into boxes, or passage links and connected by arrows. — Figure 1. The “story map” view of the *Old Régime and the OER Revolution* game.

A screenshot of the game, showing a page from the Syllabus 1 story path. — Figure 2. A screenshot of the game.

A screenshot of the game, showing an explainer for DMCA (Digital Millennium Copyright Act) notices. — Figure 3. A screenshot of the game.

Scenario-based learning is often explained in the context of scaffolding the skills being taught to the learners (Clark and Mayer 2012). As the cognitive psychologist Jerome Bruner, who first coined the term while inspired by Lev Vygotsky’s Zone of Promixal Development theory, has posited, the goal here is to provide a series of instructional techniques and support to increase the learner’s understanding, with the end result of increased independence (Wood, Bruner, and Ross 1976). The Old Régime begins with an explanation of the main concepts in the open community, then goes on to offer the user the choice of working through a trio of syllabi types: (a) a traditional list of copyrighted resources, (b) a set of zero-cost multimedia resources, some of them of questionable provenance, and (c) a non-traditional syllabus focused on assignment creation, i.e. open pedagogy (a section paying tribute to recent developments and publications in the field). The issues within escalate in terms of complexity, from questions such as “Can I distribute a PDF I got off the Internet?” to “What digital tools can students use to create multimedia assignments?” While faculty can choose which syllabus to start with, depending on their interests and background, as well as go back and forth between the syllabi, there is a natural progression built into the game from less to more open permissions.

Although most Twine games are text-based, with visuals, if any, used for illustrative purposes, the images, audio, and video in The Old Régime serve an integral purpose: to demonstrate the variety of resources—and issues—a faculty member might come across in their OER course-conversion journey. One page, for instance, features a poster for the Andrzej Wajda-directed film Danton (1983) on Wikipedia, discussed in the game as an example of a fair-use argument (according to the Wikimedia Foundation, the commercial poster had to be used on the website since it was the only available visual of the film). Even on pages where the image cited is not directly related to an argument over openness, the purpose of including it is to reinforce correct attribution practices—each picture in the game comes from Wikimedia Commons and carries either a public domain or a Creative Commons license. A visualized broken link message reinforces a discussion of permanence, and a video from the New York Times’ cooking section is embedded next to a paragraph about the potential dangers of “link rot.”

A screenshot of the game discussing fair use using the example of the poster for the movie Danton on Wikipedia. — Figure 4. A screenshot of the game.

In recognition of the range of curricular decisions available to faculty, the game is signposted throughout with a system of symbols that explain to the learner whether they are being told about a resource that is fully open (green O—“this resource can be used, modified and redistributed freely”), open with restrictions (yellow O—“look out for restrictions on modification and/or redistribution”), and zero textbook cost (yellow Z—“look out for restrictions on use, modification and/or redistribution”). This iconography was originally developed by CTL staff member Pamela Thielman to classify the teaching materials posted on the TeachOER website, as well as to appeal to audiovisual learners, in a sustained commitment to multimodality.

The home page, or title page, of the game. — Figure 5. The home page of the game.

The ending page of the game showing the player having completed the path to OER conversion. — Figure 6. The ending page of the game.

Reflections and Potential Applications

The Old Régime and the OER Revolution game had its planned launch during the Spring 2021 semester OER faculty development workshop, running remotely for the first time with two sections of fourteen participants in total. Due to other priorities that emerged during the COVID-19 pandemic, this initiative did not involve gathering feedback on the game. Moreover, the small sample size has made it difficult to draw causal inferences between game playing and seminar completion. Nonetheless, the anecdotal evidence from having faculty participants play the game—for many, their first encounter with the topic of open-source education—has been positive. The number of questions and points of confusion that the seminar leaders get—especially about OER vs. ZTC and where to look for open-source images—has lessened significantly. One area that needed clarification from a number of seminar participants is the meaning of a Creative Commons license and the mechanism for creating one. This observation is being used to create an update (i.e. a sub-scenario) to the game that focuses specifically on these questions.

Because of the comparative ease of screen sharing and the availability of virtual help, Baruch’s CTL has launched a concomitant initiative to encourage faculty to create their own assignments made with open-source tools. Despite the predictable roadblocks to this effort in pandemic times, it has met with success so far: several faculty members have authored interactive course resources predicated on input from students, which they are currently using to teach their classes. Twine has not featured among their chosen platforms so far, most likely because it still enjoys less visibility on an institutional level and is also perceived as a “more involved” and time-consuming choice. Knight Lab tools, which do not require CSS stylesheet language, especially StoryMap JS, are currently a bigger draw among Baruch instructors. Although more hands-on tutorials are needed to dispel the understandable hesitation many faculty members still feel about creating their own resources, the ensemble of these efforts has gradually led to a greater engagement with, and appreciation of, OER on campus.

In-person survey of the game with pie charts. Part 1. Have you had previous training about OER and/or Open Pedagogy? Yes-both: 60% Yes-OER: 20% No training: 20% — Figure 7. Results to question regarding previous OER training from in-person survey of the game from Baruch-CUNY OER Showcase 2020.

In-person survey of the game with pie charts. Do you feel that you understand how to navigate the game and could play through it on your own? Yes: 55.6% I don’t know: 33.3% No: 11.1% — Figure 8. Results to question regarding game navigation from in-person survey of the game from Baruch-CUNY OER Showcase 2020.

In-person survey of the game with pie charts.Part 3. What concept do you think you’ve learned most about during this brief, guided experience with the game? Copyright law: 20% Twine: 20% OER: 20% Zero cost materials: 20% CC licensing: 10% Open Pedagogy: 10% — Figure 9. Results to question regarding key learning concepts from in-person survey of the game from Baruch-CUNY OER Showcase 2020.

Feedback on the game from instructional designers and Baruch faculty who playtested the game during the 2020 Baruch OER Faculty Showcase, featured in Figures 7–9, has been used to edit the current version for greater clarity and playability. A more robust set of instructions is being created and guidelines for helping players navigate through all three scenarios developed. Moreover, the game developer has set up an apparatus with learning objectives and specific questions to better target faculty engagement. A “how we made it” section and detailed instructions for making one’s own basic Twine resource will eventually accompany the game: information on where to find tutorials, what some examples with the best interactivity/greatest emotional impact are, and how the game can be hosted with campus technology. Such a content-based approach in the supplementary material means to combine subject expertise and instructional design, taking its example from openly published resources on making history simulation games and brainstorming creative writing with Twine (McCall 2016, 2017, and 2019; McGrath 2019). More educational solutions—using Twine for process modeling in entrepreneurship classes, to consider alternate plot twists in literature, or to follow scientific processes—are being considered.

These efforts are taking place alongside promising developments in Twine itself. Emily Christina Murphy and Lai-Tze Fan have announced they are assembling EnTwine: A Critical and Creative Companion to Teaching with Twine, a proposed companion to the new and invaluable Twining: Critical and Creative Approaches to Hypertext Narratives (2021) by Anastasia Salter and Stuart Moulthrop, with the latter published on Amherst College Press’s OER platform. In the past year, Chris Klimas has also released a new story format, Chapbook, which makes it easier to work with multimedia and create and play mobile-friendly games (The Old Régime was written in the default format, Harlowe, authored by Leon Arnott). As stated in a presentation to the Interactive Fiction Technology Foundation’s annual event, NarraScope, the Twine team hopes to add a collaborative authorship function in the future (Klimas 2019), making the tool a natural choice not just for assignment creation, but also for in-class work, either in the face-in-face or asynchronous format.

Conclusion

Instructional designers, pedagogy specialists, and librarians who wish to create their own faculty workshop for teaching about open resources are encouraged to think about building both asynchronous and interactive content into their programming. The changing content of the open education field highlights the importance of providing materials in flexible ways that permit add-ons and variations. Easy-to-use open-source tools constitute a convenient and pedagogically sound resource for the faculty workshop. Twine, with its decision-tree setup and built-in interactivity, presents one possible solution to faculty handling their scenario-based learning independently and remotely alongside other face-to-face or synchronous modalities and asynchronous discussions. The other affordance of “teaching OER with OERs” is the example it provides to faculty who might introduce tools such as Twine to their own classes, creating assignments and perpetuating a culture of best practices in online publishing. The ongoing improvements in the technology suggest that gaming pedagogy can indeed become the vehicle for teaching about openness while promoting active learning and student initiative. However, a concerted initiative that combines instructional design and subject-specific pedagogy is needed to make this vision a reality.

References

Aggarwal, Raghav. 2019. “Bandersnatch: The Marketing Weapon.” June 28, 2019. https://medium.com/@raghavaggarwal0089/bandersnatch-the-marketing-weapon-f8eb6020ed04.

Baruch College Center for Teaching and Learning. n.d. Teach OER (website). Accessed November 8, 2020. http://teachoer.org/.

Bragg, Leicha. 2012. “Testing the Effectiveness of Mathematical Games as a Pedagogical Tool for Children’s Learning.” International Journal of Science and Math Education 10: 1445–1467. https://doi.org/10.1007/s10763-012-9349-9.

Checa-Romero, Mirian and Isabel Pascual Gómez. 2018. “Minecraft and Machinima in Action: Development of Creativity in the Classroom.” Technology, Pedagogy and Education, 27:5, 625–637. https://doi.org/10.1080/1475939X.2018.1537933.

Clark, Douglas, Emily E. Tanner-Smith, and Stephen S. Killingsworth. 2015. “Digital Games, Design, and Learning: A Systematic Review and Meta-Analysis.” Review of Educational Research 86, no. 1: 79–122. https://doi.org/10.3102/0034654315582065.

Clark, Ruth C. and Richard E. Mayer. 2012. Scenario-Based e-Learning: Evidence-Based Guidelines for Online Workforce Learning. San Francisco: Pfeiffer.

Cohen, Madeline. 2018. Responsible Use of Materials for OER: A Hands-On Workshop for Faculty. CUNY Lehman College Publications and Research, CUNY Academic Works. https://academicworks.cuny.edu/le_pubs/241/.

CUNY Libraries. 2017. “CUNY Open Educational Resources.” Accessed November 8, 2020. (2017), https://www.cuny.edu/libraries/open-educational-resources/.

Davidson, Cathy N. 2011. Now You See It: How Technology and Brain Science Will Transform Schools and Business for the 21st Century. New York: Penguin Books.

Ford, Melissa. 2016. Writing Interactive Fiction with Twine. London: Que Publishing.

Friedhoff, Jane. 2014. “Untangling Twine: A Platform Study.” DiGRA ’13 – Proceedings of the 2013 DiGRA International Conference: DeFragging Game Studies, August 2014, Volume 7.

Gross, Begoña. 2007. “Digital Games in Education: The Design of Games-Based Learning Environments.” Journal of Research on Technology in Education 40, no. 1: 23–38.

Hamari, Juho, Jonna Koivisto, and Harri Sarsa. 2014. “Does Gamification Work? A Literature Review of Empirical Studies on Gamification.” Proceedings from the 47th Hawaii International Conference on System Science 2014.

Harvey, Allison. 2014. “Twine’s Revolution: Democratization, Depoliticization and the Queering of Game Design.” Game: The Italian Journal of Game Studies (3).

Hayles, Katherine. 2002. Writing Machines. Cambridge, Massachusetts: MIT Press (Mediawork Pamphlet Series, 1). Interactive Fiction Technology Foundation (2020).

Hostos Community College. 2018. “Grammar Adventure Story.” Hostos Community College, CUNY Commons. Accessed November 8, 2020. https://commons.hostos.cuny.edu/esl/.

Huang, Wendy Hsin-Yuan and Dilip Soman. 2013. A Practitioner’s Guide to Gamification of Education, Research Report Series Behavioral Economics in Action, University of Toronto.

Jackson, Brianne and Virginia Thompson. 2018. “Exploring Twine and Its Use in Active Learning Online.” In Proceedings of Society for Information Technology & Teacher Education International Conference, edited by Elizabeth Langran and J. Borup. Washington, DC: Association for the Advancement of Computing in Education (AACE): 1533–1536. https://www.learntechlib.org/primary/p/182731/.

Klimas, Chris. 2017. “Interactive Fiction Appears in the Whitney Biennial.” The Interactive Fiction Technology Foundation blog, April 2, 2017. Accessed November 8, 2020. https://blog.iftechfoundation.org/2017-04-02-interactive-fiction-appears-at-the-whitney-biennial.html.

———. 2019. “Twine: Past, Present and Future.” Blog sharing of NarraScope presentation at the Massachusetts Institute of Technology, Cambridge, MA, 21 June 2019. https://chrisklimas.com/blog/2019-06-21-twine-past-present-future/.

———. 2020. Interview by Katherine Tsan, December 21, 2020.

Kumar, M. S. Vijay. 2012. “The New Landscape for the Innovative Transformation of Education.” Social Research: An International Quarterly 79, no. 3: 619–630. https://muse.jhu.edu/article/528185.

Lameras, Petros, Thrasyvoulos Tsiatsos, Panagiotis Petridis, Dimitris Tolis, Fotis Liarokapis, Despina Anastasiadou, Aristidis Protopsaltis, Maurice Hendrix, and Sylvester Arnab. 2015. “Creative Thinking Experimentations for Entrepreneurship with a Disruptive, Personalised and Mobile Game-Based Learning Ecosystem.” 2015 International Conference on Interactive Mobile Communication Technologies and Learning (IMCL), Thessaloniki: 348–352. http://dx.doi.org/10.1109/IMCTL.2015.7359617.

Lehr Samuels, Allison, Pamela Thielman, and Katherine Tsan. 2019. “Helping Faculty Navigate OER and Copyright Law: Two Approaches (Instructional and Curricular Design).” Presentation at the 18^th Annual CUNY IT Conference, John Jay College of Criminal Justice, New York, NY.

Loeffler, Helmut. 2019. “Experience with Authoring and Using an OER Textbook.” The 16^th Annual Open Education Conference, Phoenix, AZ, October 30, 2019.

Lundberg, Karin and Catherine Lyons. 2018. “Using Twine to Deliver a Grammar-Linked Creative Writing Assignment in a Hybrid ESL Course.” The Free Library, November 1, 2018. Accessed November 8, 2020. https://www.thefreelibrary.com/Using+Twine+to+Deliver+a+Grammar-Linked+Creative+Writing+Assignment…-a0563684036.

Lyons, Kate and Karin Lundberg. 2018. “Twine—Using Non-Linear Storytelling in Your Pedagogy,” Hostos Community College Commons. Accessed November 8, 2020. https://commons.hostos.cuny.edu/edtech/2018/02/05/twine-using-non-linear-storytelling-in-your-pedagogy/.

McCall, Jeremiah. 2016. “Twine, Inform and Designing Interactive History Texts.” Play the Past (website). Accessed November 8, 2020. http://www.playthepast.org/?p=5739.

———. 2017. “Historical Twine Project Rubric.” Gaming the Past: Historical Video Games in the Classroom and Beyond (website). Accessed November 8, 2020. https://gamingthepast.net/simulation-design/twine-interactive-fiction-tool/twine-rubric-for-interactive-history-projects/.

———. 2019. “Twine Hist Rubric 9^th Grade History 1.5.” Gaming the Past: Historical Video Games in the Classroom and Beyond (website). Accessed November 9, 2020. https://gamingthepast.net/simulation-design/twine-interactive-fiction-tool/twine-rubric-for-interactive-history-projects/.

McGonigal, Jane. 2011. Reality Is Broken: Why Games Are Better and How They Can Change the World. New York: Penguin Random House.

———. 2016. SuperBetter: The Power of Living Gamefully. New York: Penguin Random House.

McGrath, Jim. 2019. “Digital Storytelling.” Spring 2019 syllabus, Brown University, Provincetown. Accessed November 8, 2020. http://www.digitalstorytelling2019.jimmcgrath.us/.

Ohler, Jason. 2006. “The World of Digital Storytelling.” Educational Leadership: 63 (4) 44–47.

Pandey, Asha. 2018. “7 Examples on Scenario-Based Learning for Formal and Informal Learning.” EI Design. Accessed November 16, 2020. https://www.eidesign.net/7-examples-scenario-based-learning-sbl-formal-informal-learning/.

Piedra, Nelson, Janneth Chicaiza, Jorge López, and Edmundo Tovar Caro. 2016. “Integrating OER in the Design of Educational Material: Blended Learning and Linked-Open-Educational-Resources-Data Approach," 2016 IEEE Global Engineering Education Conference (EDUCON), Abu Dhabi, 2016: 1179–1187. https://doi.org/10.1109/EDUCON.2011.5773299.

Queneau, Raymond. 1981. Conte à votre façon. Contes et propos. Paris: Gallimard.

Rizvić, Selma, Dusanka Boskovic, Vensada Okanovic, Sanda Slijvo, and Merima Zukić. 2019. “Interactive Digital Storytelling: Bringing Cultural Heritage in a Classroom.” Journal of Computer Education: 143–166. https://doi.org/10.1007/s40692-018-0128-7.

Salter, Anastasia. 2016. “Playing at Empathy: Representing and Experiencing Emotional Growth through Twine Games.” 2016 IEEE International Conference on Serious Games and Applications for Health (SeGAH), Orlando, FL, 2016: 1–8. https://doi.org/10.1109/SeGAH.2016.7586272.

Salter, Anastasia and Scott Moulthrop. 2021. Twining: Critical and Creative Approaches to Hypertext Narratives. Amherst: Amherst College Press. http://dx.doi.org/10.3998/mpub.12255695

Tsan, Katherine. 2020. “The Old Régime and the OER Revolution.” TeachOER (website). Accessed November 8, 2020. http://teachoer.org/the-old-regime-and-the-oer-revolution-game/.

Tytler, Sarah. 2017. “‘TwitFic’, Twine, and Student-Centred Learning: Combining Creativity and Coding in the Classroom.” Africa International Journal of Management Education and Governance: 21–34.

Wilson, Rebecca, Jon Saklofske, and INKE Research team. 2019. “Playful Lenses: Using Twine to Facilitate Open Social Scholarship through Game-Based Inquiry, Research, and Scholarly Communication. KULA: Knowledge Creation, Dissemination, and Preservation Studies 3, no. 1: 5. http://doi.org/10.5334/kula.11.

Wood, David, Jerome S. Bruner, and Gail Ross. 1976. “The Role of Tutoring in Problem Solving.” Journal of Child Psychiatry and Psychology 17, no. 2: 89–100. https://doi.org/10.1111/j.1469-7610.1976.tb00381.x.

Zhadko, Olena and Ko, Susan. 2017. “OER Sustainable Scale Up: Faculty Development as Key Strategy (OER).” Presentation at the 16^th Annual CUNY IT Conference: Bridging the Gaps, John Jay College of Criminal Justice, New York, NY. https://docs.google.com/presentation/d/1u4JIGhu6b08l_gwaIWH8kpmy9KD10yR9yJE_iTz2Zv8/edit?usp=sharing.

———. 2019. “Refresh your Course with Open Educational Resources.” The Teaching Professor (blog). Accessed November 8, 2020. https://www.teachingprofessor.com/topics/online-learning/refresh-your-course-with-open-educational-resources/.

Acknowledgments

The author/game designer would like to thank Chris Klimas for the insights and background knowledge shared in his interview, Hamad Sindhi and Dimitrios Papadopoulos for sharing useful bibliographical sources, Pamela Thielman for creating the pie charts based on faculty responses in Figures 7–9 and suggesting necessary changes to the game, and Allison Lehr Samuels and the whole staff of the Baruch Center for Teaching and Learning for their invaluable help and support throughout the creative and editorial process.

About the Author

Katherine Foshko Tsan (PhD, MLS) focuses on open educational resources and the digital humanities at the Center for Teaching and Learning (CTL), Baruch College, CUNY. Her scholarly background is in Academic Libraries and Modern European History. She designed The Old Régime and the OER Revolution game for Baruch CTL’s TeachOER website with input from her colleagues and is now completing edits and additions to the resource. She welcomes comments at katherine.tsan@baruch.cuny.edu.

Two charts compare interaction by videoconference meeting participants.

Issue Nineteen

Assessing the Effectiveness of Using Live Interactions and Feedback to Increase Engagement in Online Learning

Beth Porter, Riff Analytics

John Doucette, Riff Analytics

Andrew Reilly, New College of Florida

Dan Calacci, MIT Media Lab

Burcin Bozkaya, New College of Florida

Alex Pentland, MIT Media Lab

Abstract

Traditional, in-person instruction provides a social environment and immediate feedback mechanisms that typically ensure participants are successful. In comparison, online mediums for instruction lack these mechanisms and rely on the motivation and persistence of each individual learner, often resulting in low completion rates. In this study, we examined the effect of introducing enabling tools and live feedback into an online learning experience on learner performance and persistence in the course, as well as election to complete supplemental readings and assignments. The findings from our experiments show positive correlations with strong statistical significance between live interactions and all performance measures studied. Specifically, we find that students who consistently participated (at least once a week on average) received final grades 80% higher than those who did not and were twice as likely to earn a certificate of completion. Furthermore, the odds of a student receiving higher grades are increased by a factor of 23% for each additional hour spent using our enabling tools and the odds of passing the course by 35%. We contend that further exploration and implementation of such enabling tools will aid both instructors and learners, as the online platforms may continue to be the preferred choice of instruction even after the pandemic subsides.

Introduction

Achieving and maintaining consistent student engagement can be difficult in online learning contexts (Kizilcec and Halawa 2015; Allen and Seaman 2015, 61; Greene et al. 2015). There are many factors that impact learner engagement: course structure, amount of lecture time, user experience of the learning management system, and countless other variables. However, online educators and scholars of online learning have consistently found evidence that peer interaction and learner-instructor interaction are strongly related to learner engagement (Bryson 2016; Garrison and Cleveland-Innes 2005; Greene et al. 2015; Richardson and Swan 2003; Yamada 2009). Several studies have found that measures of social presence, a theoretical construct originating from communications studies, are significantly related to learners’ overall outcomes and perceived experience (Richardson and Swan 2003). Social presence refers to how “salient” an interpersonal interaction is—how readily one feels like they are “present” with another.

To achieve high social presence between students, researchers have discovered that the modality of online communication is extremely important (Yamada 2009; Kuyath 2008; Johnson 2006). The modality of communication refers to the types of channels used: text chat, forums, voice chat, video conference, etc. In communications, human-computer interaction, and online education literature, it has been shown that richer modes of communication between conversing members lead to higher rates of trust, cooperation, engagement, and social presence (Bos et al. 2002). For example, video conference, which includes real time video and voice, is a richer mode of communication than asynchronous forums, and has been shown to lead to higher rates of cooperation than text chat (Bos et al. 2002).

These advances provide strong evidence that the interaction patterns of people carry rich signals that correlate with a variety of outcomes important to personal growth, learning, and team success. By using modern machine learning methods and statistics of conversation patterns—who spoke when, for how long, etc.—even without knowing the content of these conversations, researchers have been able to predict speed dating outcomes, group brainstorming success, and group achievement on a test of “collective intelligence” (Pentland et al. 2006, Woolley et al. 2010; Dong and Pentland 2010; Jayagopi et al. 2010; Kim et al. 2008; Dong et al. 2012a, 2012b). By developing more sophisticated computational models of human interaction, it is possible to infer social roles, such as group leaders or followers (Dong et al. 2013).

Current study

To show the effectiveness of online learning systems that contribute to strong learning outcomes and learner performance via tools that foster participation and interaction, we present an experimental study in this paper conducted with an online class. Our learning platform, which is used to conduct our experiments, allows students to interact using video chat and text chat. In addition, students get continuous feedback on their participation and engagement in the video chat via a tool we named Meeting Mediator (MM). MM essentially provides feedback on how the class participant is doing in terms of measures of engagement such as speaking time, turn-taking, influencing or affirming other participants, and interrupting other speakers.

The Meeting Mediator integrated into our online learning platform is a further enhancement of the concepts and ideas introduced by Kim et al. (2008), who implemented a similar user feedback mechanism in a face-to-face setting with data collected via sociometric badges worn by users interacting in small groups. Our platform includes a new version of the MM, which introduces new metrics for key conversational events such as interruptions and affirmations. In this study, we validate the effectiveness of this online MM through a number of experiments we conduct.

Part of the innovation at the core of our study is the burgeoning science of quantitatively analyzing human social interaction. This small but exciting subfield of computational social science has shown immense promise in developing machine understanding of human interaction (Woolley et al. 2010). By instrumenting many different types of communication, such as face-to-face physical conversation, video conference, and text chat, researchers have been able to predict group performance on a variety of tasks.

Though this study was conducted prior to the 2020 pandemic, the challenge presented to the team carrying out the experiment was to understand the factors contributing to successful transition of programs to online environments. The platform was designed to foster virtual engagement, providing course developers and instructors new tools for offering online experiences likely to capture and keep learner interest. This is especially critical in a time when the pandemic has forced millions of students and professional learners around the globe to use online platforms, replacing the traditional in-person instruction.

An important point to emphasize is that we do not make a claim to replace in-person learning with technology-enabled online learning. We acknowledge that remote instruction may present certain technology-driven features and advantages, but there are many ways in-person instruction provides a superior learning experience. Through the experiments we conduct in this study and our findings, we contend that the remote learning experience can be enhanced and the gap between in-person versus online learning can be bridged to some extent using real-time feedback mechanisms such as the Meeting Mediator and interaction metrics. This is clearly good news in times of a pandemic like COVID-19, where instructors as well as learners are challenged in many unique ways.

Materials and Methods

We first describe our user-facing applications, supporting backend tools and the metrics we designed and employed for performance assessment. Then, we present our experiment design.

The Riff Platform and its user-facing applications

Our team developed three foundational user-facing applications and supporting backend tools as part of the Riff Platform. These are:

Video Chat (with Meeting Mediator)
Post-Meeting Metrics
Text Chat (which includes Video Chat and Post-Meeting Metrics)

These applications and tools were built for both commercial and research purposes, cloud-deployed in standardized ways, but customized over time to meet the changing needs of Riff customers and modified research goals. (An open version of Riff Video Chat, called Riff Remote, is available here: https://my.riffremote.com/.)

Video chat

Riff Video Chat (Figure 1) is an in-browser video chat application that uses the computer’s microphone to collect voice data for analysis and is present in all manifestations of the Riff Platform. Riff Video does not require plug-ins or a local application in order to run—participants navigate to a URL (either by entering it directly into the address bar or through an authenticated redirect from another application), enable their camera and microphone, and then enter the video chat room. Other features of the video chat include screen sharing, microphone muting, and the ability to load a document (typically one with shared access) side-by-side with the video on screen.

A key element of Riff Video is the Meeting Mediator (MM) (Figure 2), which gives participants of the video chat real-time feedback about their speaking time. Specifically, it provides three metrics:

Figure 1: The Riff Video browser-based video conferencing
client with Meeting Mediator shown. — Figure 1. Riff Video browser-based video conferencing client.

Engagement, as indicated by the shade of purple of the node in the middle of the visualization, which shows the total number of turns taken in running five–minute intervals throughout the chat—dark purple means many turns; light purple means fewer turns;
Influence, as indicated by the location of the purple node, which moves toward the person who has taken the largest number of turns in running five–minute intervals throughout the chat;
Dominance, as indicated by the thickness of the grey “spokes” running from the central purple node out to each of the participant nodes, which indicates the number of turns people have taken in running five–minute intervals.

Figure 2: The Riff Meeting Mediator, with hover information
“Name”, “Speaking Time” and “Turn Count” shown. — Figure 2. Hover states of the Meeting Mediator showing different metrics.

These metrics are continuously updated throughout the video chat, lagging one or two seconds behind the live conversation. In a meeting, this tool acts as a nudging mechanism, which is essentially an intervention by the system aimed at increasing a participant’s self and situational awareness, influencing a participant’s behavior towards a more rewarding learning experience and outcomes. The MM indirectly encourages users to maintain conversational balance by speaking up (if the user sees they have not spoken recently) or to engage others who have not been as active in the conversation.

Post-meeting metrics

Immediately after a Riff Video Chat meeting, Metrics (Figure 3) appear on screen to show participants measures associated with the meeting, specifically:

Speaking Time—a chart showing the percentages of turns taken by each participant;
Pairwise Comparisons—each of the following charts shows the pairwise comparisons of the individual participant with respect to each other participant, to show the balance of each pair in their interactions:
- Influences—a chart showing who spoke after whom in the video chat, which aggregates data across all types of spoken events;
- Interruptions—a chart showing interruptions (when someone cuts off another person’s speaking turn);
- Affirmations—a chart showing affirmations (when someone has a verbalization, but does not cut off another person’s speaking turn);
Timeline—an interactive view of the video chat meeting over time, on top showing what time each participant spoke and below when each pairwise interaction took place (“your” and “their” interruptions, affirmations, and influences).

Figure 3: Meeting metrics (timeline, speaking time, influences,
interruptions and affirmations). — Figure 3. Meeting metrics (timeline, speaking time, influences, interruptions and affirmations).

Additionally, each user can access their past history of meetings at any time, independent of the video chat itself (Figure 4).

Figure 4: Meeting history, in chronological order. — Figure 4. Meeting history, in chronological order.

Text chat

Riff Text Chat (also known as Riff EDU) is an environment that allows people to text chat with one another in open channels, private channels, and direct messages (Figure 5).

Riff EDU is built on top of the Mattermost platform, which is an open-source project (https://mattermost.com/). The Riff team developed a customized version of this platform which has three key additions (and several smaller modifications):

LTI integration, which allows the learner to move seamlessly from their learning environment (Canvas, Open edX, etc.) using the learning tools interoperability (LTI) protocol for authentication;
Video integration, which allows the learner to start an ad hoc video chat session from the Chat environment;
Metrics integration, which shows the learner views of their video meeting metrics within the Chat environment, as well as additional text chat data and recommendations.

Learner data privacy

At the outset of the course, learners were asked to consent to have data collected about their activity and performance. No learners declined to have data collected about them. Data collected during this experiment was shared directly with learners through the Meeting Mediator and Post-Meeting Metrics page. Everyone in a video chat got access to the data about their conversation, but video and text chat data was not shared with anyone else during the experience. Data analyzed for the purpose of the experiment was anonymized to preserve the privacy of participants.

Supporting accessibility

The experimental environment was designed to account for participants with accessibility needs. Following the WCAG 2.0 AA accessibility standard from the W3C (World Wide Web Consortium 2021) the software underwent an accessibility audit to identify potential issues, which were then remediated before the platform was released to learners. Accessibility compliance was documented using a publicly available Voluntary Participation Accessibility Template (Riff Analytics 2021). Though we did not have any students who let us know that they had specific accessibility needs, we were prepared for that eventuality.

Experiment design

To show the effectiveness of our video conferencing environment and our nudging tool, the Meeting Mediator (MM), we conducted an experiment with a major client of Riff Learning Inc. in Canada. In this experiment, test subjects were recruited to participate in an online course called “AI Strategy and Application” from a government subsidized incubator of tech companies headquartered in Toronto, Ontario, Canada. Students elected to take the course as a professional development experience, driven by interest in learning how to start an AI initiative either as an intrapreneurial or entrepreneurial activity. In most cases, the learners were paying for the course, but in some cases, either the course had been paid for by their employer or they were given free access to the course. Learners were either students in advanced degree programs or full-time professionals taking the course as a supplemental learning experience.

The eight-week course was delivered on the Open edX platform with the Riff Platform serving as the communications and collaboration platform for instructor and learner interactions, both structured and ad-hoc. Learners had opportunities to view videos and read original materials on how AI is applied to business problems, tackle hands-on coding exercises to learn basic machine learning techniques in a sandbox environment, and take assessments at the end of each learning unit.

The participants were divided into groups throughout the course to facilitate the following two types of activities:

Brainstorming activities (in pre-set groups of four to six people)—Each week during the first four weeks of the course, people met on video in their Peer Learning Groups (PLG) to collaborate on the assignment, and then create a shared submission.
Capstone project collaboration (in self-selected groups of four to six people, plus a mentor)—Each week during the last four weeks of the course, people met on video in their Capstone Groups to collaborate on developing a venture plan based on the winning pitches submitted by their peers in the course.

Learners were left to schedule video meetings with their group members on their own, and the course was time-released (one week of material at a time), but self-paced. However, the course support staff did participate in the first meeting of each group, to ensure that people didn’t have any technical issues and could make the video chats happen without issue.

The main goal of our experiment was to analyze the relationship between using video and text chat, and various outcomes among learners in the course. Furthermore, we aimed to analyze differences, if any, between early users (those who completed at least the first half of the course) versus the rest, to understand the effects of early usage on retention and grades.

Note that in this experiment, the Meeting Mediator (MM) was always present for video chats. Our rationale for that choice was that if our hypotheses that video with MM is positively correlated with any of performance, persistence, or satisfaction in online courses, then having it present for some students and not for others in a paid, graded learning experience might disadvantage the control population and be deemed inequitable.

In our experiment, we set out to test a number of hypotheses and hence show the (positive) relationship (or its lack thereof) between the use of video and various outcomes. Specifically, we explored the following questions:

Did students who used Riff Video Chat (and Meeting Mediator) more often receive higher grades in the course?
Did students who used Riff Video Chat (and Meeting Mediator) more often complete the course at a higher rate?
Is there a strong positive association between participating in additional Riff Video Chats during the first four weeks of the course and earning a certificate?
Is there a strong positive association between participating in additional Riff Video Chats during the first four weeks of the course and receiving higher grades at the end of the course?
Is there a strong positive association between participating in additional Riff Video Chats during the first four weeks of the course and completing more of the optional course assignments?
Is there a strong positive association between participating in additional Riff Video Chats during the first four weeks of the course and pitching a capstone project topic?

The first two research questions are explored for students who completed the entire course (n=62); we removed the chat records and performance output results for students who dropped part way. The remaining questions are explored for the entire cohort students of n=83. These students all started the course, but some of them dropped out and, hence, did not receive a final grade for completion or a certificate.

In our analysis of our experiment results, we used the following three constructs for reporting and interpreting our figures: correlations, odds ratios, and significance. For definitions of these metrics and their implications for our study, we refer the reader to Appendix B. With correlation analysis, we would like to further note that while one may observe the degree to which people who do one thing more (e.g., use video chat) also tend to do something else more (e.g., receive higher grades), such a relationship cannot be directly used to imply causality; they are merely indicators that two phenomena occur together (in a positive or negative relationship) or not.

Finally, we conducted an exit survey for all students who completed the course and a separate survey for those who failed to complete it. While the results of this survey are not directly related to the quantitative analysis we report in the next section, they do provide some qualitative feedback from users on the efficacy of the Riff Platform. We have included the list of survey questions and some key results we collected from these surveys in the Appendix.

Results

In this section, we report the results of our experimental study for students who participated in the course for its entire duration (questions 1–2, n=62) and for all students who started with the course, even if they dropped out (questions 3–6, n=83) separately.

Results for questions 1–2 for students who completed the course

To answer the first question in our study (“Did students who used Riff Video Chat more often receive higher grades in the course?”), we considered the following output variables:

Final grade earned
Coding exercise grade
Capstone exercise grade
Collaboration exercise grade
Pitch video completion

For the second question (“Did students who used Riff Video Chat more often complete the course at a higher rate?”), we simply considered whether or not the student earned his/her certificate of completion in the course.

The following table, Table 1, shows the correlation between each output variable listed above and the input variable of “video calls made,” which is the number of times a student connected to the video chat. The p-values were further corrected using Holm’s method to maintain a FWER of 0.05. We find that all p-values are significant at 95% confidence level or better, and correlation values point to reasonably high levels of relationship with most of the output variables.

Attribute	Correlation to # Riff Calls Made	n	p	Significance
Final Grades	0.50	62	1.56e-04	***
Coding Exercise Grades	0.41	62	2.54e-03	**
Capstone Exercise Grades	0.49	62	1.56e-04	***
Collaborative Exercise Grades	0.27	62	2.94e-02	*
Pitch Video Completion	0.37	62	5.25e-03	**
Certificate Earned	0.50	62	1.56e-04	***

Table 1. Correlation between Riff Video usage and various performance variables.
*p < 0.5, **p < 0.01, ***p < 0.001

We further calculated the odds ratio for two of these variables, as shown in Table 2. Here, in addition to the “Certificate Earned” binary variable, the “Grades” variable was created as another binary variable indicating the student has received (or not) a passing grade based on his/her final grade.

Attribute	Odds Ratio	n	p	Significance
Grades	1.23	62	16.92e-03	**
Certificate Earned	1.35	62	3.964-04	***

Table 2. Odds ratios for the effect of attending one additional Riff Video call on final grade and certificate attainment obtained by fitting a logistic regression model.
*p < 0.5, **p < 0.01, ***p < 0.001

Table 2 with odds ratios greater than 1.0 suggests that spending additional time with video chat increases the likelihood of earning a passing grade and a final certificate of completion (see the Discussion section for more details).

We further illustrate the relationship between these two variables and our input variable (“video calls made”) in Figures 6 and 7. The figures display positively sloped trends which confirm the positive correlations reported in Table 1. That is, as the learners spend more time with video chat, they are more likely to earn higher grades and a certificate of completion.

Figure 6: A scatter plot showing the relationship between the students’ total Riff video usage and final grade. There is a regression line included showing a positive relationship between the two variables. — Figure 6. Relationship between Riff Video usage and students’ final grades.

Figure 7: A scatter plot showing the relationship between the students’ total Riff video usage and whether the student received a certificate for the course. There is a regression line included showing a positive relationship between the two variables. — Figure 7. Relationship between Riff Video usage and certificate achievement.

Results for questions 3–6 for all students

We explored the remaining research questions regarding the early usage (first four weeks) of Riff Video using data we collected from all students in this course. We note again that some students completed the course only partially and dropped out. They did, however, complete the first four weeks of the course. Here we used the start of the Capstone Project as the cutoff date for tracking how often the students were present in channels in which video chats were offered. We correlated this variable to the same performance variables we considered in the previous section to explore the research questions 3–6:

Tables 3 and 4 show the correlation values between the variables similar to the way they are presented in the previous section. All correlations were computed with Pearson’s method, and p values were corrected again using Holm’s method to maintain a FWER of 0.05. We find that not only are the correlation values in Table 3 generally higher than those in Table 1, the p-value significance levels are also stronger. Furthermore, higher odds ratios in Table 4 suggest that early video chat usage is even more determinant in achieving successful performance in the course.

Attribute	Correlation to # Riff Calls Made	n	p	Significance
Final Grades	0.54	83	5.40e-07	****
Coding Exercise Grades	0.42	83	5.22e-05	****
Capstone Exercise Grades	0.50	83	3.89e-06	****
Collaboration Exercise Grades	0.52	83	2.07e-06	****
Pitch Video Completion	0.45	83	3.95e-05	****
Certificate Earned	0.50	83	3.89e-06	****

Table 3. Correlation between early Riff Video usage and various performance variables.
*p < 0.5, **p < 0.01, ***p < 0.001, ****p<0.0001

Attribute	Odds Ratio	n	p	Significance
Grades	1.79	83	1.03e-04	***
Certificate Earned	2.00	83	1.96e-05	****

Table 4. Odds ratios for the effect of attending one additional Riff Video call on final grade and certificate attainment obtained by fitting a logistic regression model.
*p < 0.5, **p < 0.01, ***p < 0.001, ****p<0.0001

The relationship between these two variables and our input variable (“Early video usage”) is further illustrated in Figures 8 and 9. Red points in these figures indicate students who dropped the course. Note that in one case, a student indicated they were dropping the course, but then completed all work and earned a certificate. Similar to Figures 6 and 7, we also find that as the early video chat usage increases, the likelihood of earning higher grades and a certificate of completion also increases.

Figure 8: A scatter plot showing the relationship between the students’ early Riff video usage and final grade. There is a regression line included showing a positive relationship between the two variables. — Figure 8. Relationship between early Riff Video usage and students’ final grades.

Figure 9: A scatter plot showing the relationship between the students’ early Riff video usage and whether the student received a certificate for the course. There is a regression line included showing a positive relationship between the two variables. — Figure 9. Relationship between early Riff Video usage and certificate achievement.

Summary, Discussion, and Conclusions

Summary

The results of the experiment may be summarized as follows:

Each additional Riff Video Chat made during the first four weeks of the course predicts a doubling of the odds that the student will receive a certificate of course completion. (Fig. 9)
Each additional Riff Video Chat made during the first four weeks of the course predicts an increase in the odds of receiving a high grade by 79%. (Fig. 8)
Most benefits are accrued after participating in the first four to five video chats. Students who participated in more than four calls (an average of one per week) received final grades 80% higher than those who did not and were twice as likely to earn a certificate. (Tables 3–4 and Fig. 8–9)
We find significantly stronger evidence, as reported in Tables 1–4 and Figures 6–9, of measured metrics linked to user performance, than those reported in the literature by Kim et al. (2008).

Discussion and Conclusions

In our analysis of experiment results, for the first group that completed the entire course, we found strong correlations between the time (in minutes) spent using video to communicate with peers, and every outcome of interest as shown in Table 1. Particularly strong correlations were found between students’ final grades and attainment of course certificates, which has significant commercial implications. Historically, open online courses have had very low completion rates around 5% (Lederman 2019); hence the use of novel tools within online courses can increase engagement, raise the rates of persistence, and improve outcomes. Because this is an observational design, these results do not establish a causal relationship between usage of the Riff Platform and these outcomes, but they suggest that such a relationship may exist. We found that the relationship between video usage and both of our response variables (grades and course completion) was well fit by a logistic curve, as shown in Table 2 and Figure 7. The upward pattern observed in Figures 6–7 simply indicates that the participants are more likely to achieve higher grades and earn a certificate of completion if they more frequently use Riff Video chat.

The reported odds ratios as extracted (reported in Table 2) from the coefficients of the logistic regression model fit suggest results that are statistically significant. The reported values correspond to increases in the odds of a student receiving higher grades by a factor of 23% for each additional hour spent using Riff Video chat and in the odds of passing the course by 35%. The analysis suggests that potential improvements in outcomes are realized with a total exposure of no more than approximately 15 hours over the course.

With the second group of students who remained enrolled during at least the first quarter of the course, the results summarized in Tables 3 and 4 demonstrate that early usage of Riff Platform has an even stronger relationship to the outcomes of interest than usage over the course as a whole. Particularly notable is the effect size for each additional hour of exposure to Riff Video during the first half of the course, which is associated with a doubling of the odds that the student completes the course and earns a certificate. Although this is an observational experiment, again, it suggests that the Riff Platform may be having a significant impact on students’ course completion rates and participation. Similar to Figures 6 and 7, we also observe an upward pattern in Figures 8 and 9, which indicates that this second group of students are also more likely to achieve higher grades in the course and earn a certificate of completion if they more frequently use video. We further note that the high positive correlation values relating measured metrics to user performance in the course are significantly better than those reported in the literature by Kim et al. (2008).

These results suggest that the platform and its nudging mechanisms ultimately benefit student learning as assessed via several exercises completed by the students. The high correlations described above suggest that there is a statistically significant difference in student performance when nudging features such as the MM are used versus not. We also contend that the Riff Platform helps students with self and situational awareness, as it presents the student with live as well as post-session information about their individual and group performance. This helps students adjust themselves in terms of balancing active contribution versus listening to others, and their level of engagement, via the real time feedback loop, when the course requires engagement to meet the learning goals.

These experiments and the corresponding results, though they signal strong correlations between the usage of Riff Platform and all measured output variables, are not without limitations. The results are based only on this set of experiments with this cohort of students in a single course. One could repeat these experiments in a variety of settings (using different cohorts, different countries, and different learning environments and tasks) to see if the results can be generalized. Secondly, we did not have any control variables (such as age, gender, job experience or other variables like prior online learning experience or performance in similar online or offline courses) on the subjects for whom we collected the data. It is possible that students who are successful using the Riff Platform would have been successful anyway because they have prior experience using online learning platforms or they are already high-performing students regardless of the learning environment or tools. Or some students may afford extra time outside the class to review and digest the material, explore additional resources and/or complete additional exercises while others may not. These effects should definitely be included in a more detailed future experimental study. This line of research would highly benefit from an A/B testing approach (e.g., using randomized controlled trials) where one would control for various factors. This requires a different learning environment to be set up where participants are not disadvantaged due to the parameters of the experiment in a legitimate online course where they expect to be provided with equal opportunities for learning.

Another limitation is the fact that the platform only keeps track of audio signals, not the actual content of what is spoken, and it classifies them into three categories as influences, interruptions or affirmations. It is possible that these classifications may not always be accurate, despite the rules we apply, due to backchannel noise. Future work could refine current techniques for identifying the three conversation events with additional hand-labeling, which could then be used to confirm the validity of the rules currently used in the platform.

To capture content being exchanged by participants, there is further opportunity to process spoken words using Natural Language Processing libraries and collect information on the scope and “quality” of the content (e.g., new topics or ideas introduced; sentiment of the words used). The platform could then reflect not only on the frequency or duration of one’s verbal participation, but also its quality.

One may also be concerned about the cost of using such a platform or integrating it with an existing learning management system (LMS). We note that because the platform readily comes as a complete platform solution, there is no additional cost of integrating it with existing LMS. Users simply login and use the features of the platform as needed. Some non-financial “costs” may exist, but they are limited to time and effort needed for training on how to use the system and understand its features such as the MM and Riff Metrics. We believe these costs are quite reasonable as the user interface has been designed with usability in mind.

Finally, though this experiment was within a course for adults interested in expanding their understanding of AI, the participants came from a variety of backgrounds including undergraduate and graduate students, early and mid-career professionals, and government employees. We see potential for this technology to address engagement in virtual settings for any type of collaborative work and any age group. (In all cases, it is important for people meeting in small groups to have an appropriate motivation for collaborating, such as brainstorming or problem solving.)

Even with the limitations noted, our results are a solid first step towards showing that online learning platforms with the right user-facing components providing relevant and prompt feedback to participants are indeed likely to be effective in enhancing the learning experience. This is important especially in the wake of growing demand for online learning platforms due to the COVID-19 pandemic. We believe tools like the Riff Platform and the Meeting Mediator will play an increasingly important role in educational and professional settings where online platforms foster more and more interactions.

Bibliography

Allen, Isabel Elaine, and Jeff Seaman. 2015. Tracking Online Education in the United States. Babson Survey Research Group. https://onlinelearningconsortium.org/read/online-report-card-tracking-online-education-united-states-2015/

Bos, Nathan, Judy Olson, Darren Gergle, Gary Olson, and Zach Wright. 2002. “Effects of Four Computer-Mediated Communications Channels on Trust Development.” Proceedings of CHI ‘02, SIGCHI Conference on Human Factors in Computing Systems: 135–40.

Bryson, Colin. 2016. Review of Engagement through Partnership: Students as Partners in Learning and Teaching in Higher Education, by Mick Healey, Abbi Flint and Kathy Harrington. International Journal for Academic Development 21, no. 1: 84–86.

Dong, Wen, and Alex “Sandy” Pentland. 2010. “Quantifying Group Problem Solving with Stochastic Analysis.” International Conference on Multimodal Interfaces and the Workshop on Machine Learning for Multimodal Interaction (ICMI-MLMI ’10), Article 40: 1–4.

Dong, Wen, Bruno Lepri, and Alex “Sandy” Pentland. 2012a. “Automatic Prediction of Small Group Performance in Information Sharing Tasks.” Proceedings of Collective Intelligence Conference, arXiv: 1204.2991.

Dong, Wen, Bruno Lepri, Taemie Kim, Fabio Pianesi, and Alex “Sandy” Pentland. 2012b. “Modeling Conversational Dynamics and Performance in a Social Dilemma Task.” Proceedings of the 5th International Symposium on Communications Control and Signal Processing, 1–4.

Dong, Wen, Bruno Lepri, Fabio Pianesi, and Alex “Sandy” Pentland. 2013. “Modeling Functional Roles Dynamics in Small Group Interactions.” IEEE Transactions on Multimedia 15, no. 1: 83–95.

Garrison, Donn Randy, and Martha Cleveland-Innes. 2005. “Facilitating Cognitive Presence in Online Learning: Interaction Is Not Enough.” American Journal of Distance Education 19, no. 3: 133–148.

Greene, Jeffrey A., Christopher A. Oswald, and Jeffrey Pomerantz. 2015. “Predictors of Retention and Achievement in a Massive Open Online Course.” American Educational Research Journal 52, no. 5: 925–955.

Jayagopi, Dinesh Babu, Taemie Kim, Alex “Sandy” Pentland, and Daniel Gatica-Perez. 2010. “Recognizing Conversational Context in Group Interaction using Privacy-Sensitive Mobile Sensors.” Proceedings of the 9th International Conference on Mobile and Ubiquitous Multimedia (MUM ’10), Article 8: 1–4.

Johnson, Genevieve Marie. 2006. “Synchronous and Asynchronous Text-Based CMC in Educational Contexts: A Review of Recent Research.” TechTrends 50, no. 4: 46–53.

Kim, Taemie, Agnes Chang, Lindsey Holland, and Alex “Sandy” Pentland. 2008. “Meeting Mediator: Enhancing Group Collaboration using Sociometric Feedback.” Proceedings of the 2008 ACM conference on Computer Supported Cooperative Work: 457–466.

Kizilcec, René F., and Sherif Halawa. 2015. “Attrition and Achievement Gaps in Online Learning.” Proceedings of the Second (2015) ACM Conference on Learning @ Scale – L@S ’15: 57–66.

Kuyath, Stephen James. 2008. “The Social Presence of Instant Messaging: Effects on Student Satisfaction, Perceived Learning, and Performance in Distance Education.” PhD diss., The University of North Carolina at Charlotte.

Lederman, Doug. 2019. “Why MOOCs Didn’t Work, in 3 Data Points.” Inside HigherEd, Jan. 16, 2019. Accessed March 15, 2021. https://www.insidehighered.com/digital-learning/article/2019/01/16/study-offers-data-show-moocs-didnt-achieve-their-goals.

Pentland, Alex “Sandy,” Anmol Madan, and Jon Gips. 2006. “Perception of Social Interest: A Computational Model of Social Signaling.” Proceedings of the 18th International Conference on Pattern Recognition: 1080–1083.

Richardson, Jennifer C., and Karen Swan. 2003. “Examining Social Presence in Online Courses in Relation to Students’ Perceived Learning and Satisfaction.” Journal of Asynchronous Learning Network 7, no. 1: 68–88.

Riff Analytics. 2021. “Riff Learning Accessibility Conformance Report.” Accessed March 15, 2021. https://riffanalytics.ai/riff-edu-vpats-3/.

Sun, Huaiyi “Cici,” Dan Calacci, and Alex “Sandy” Pentland. 2018. “Patterns of Interaction Predict Social Role in Online Learning.” 2018 International Conference on Social Computing, Behavioral-Cultural Modeling & Prediction and Behavior Representation in Modeling and Simulation. Accessed April 21, 2021. http://sbp-brims.org/2018/proceedings/papers/latebreaking_papers/LB_11.pdf.

Woolley, Anita Williams, Christopher F. Chabris, Alex ‘Sandy’ Pentland, Nada Hashmi, and Thomas W. Malone. 2010. “Evidence for a Collective Intelligence Factor in the Performance of Human Groups.” Science 330, no. 6004: 686–688.

World Wide Web Consortium. 2021. “Web Content Accessibility Guidelines (WCAG) 2 Level AA Conformance.” Accessed March 15, 2021. https://www.w3.org/WAI/WCAG2AA-Conformance.

Yamada, Masanori. 2009. “The Role of Social Presence in Learner-Centered Communicative Language Learning using Synchronous Computer-Mediated Communication: Experimental Study.” Computers & Education 52, no. 4: 820–833.

Appendix A—Exit Survey Questions and Key Results

A1. Exit Survey Questions

How well did you understand the lessons at the start of the course?
How well did you understand the subject matter at the midpoint of the course?
How well did you understand the subject matter at the end of the course?
How satisfied were you with what you learned throughout the course?
How well did the instructional materials convey course expectations?
How well did course assignments test your understanding of course materials?
How effectively did the activities and assignments used in this course aid your learning?
Would you recommend this course to other students?
How well did course staff support your learning throughout the course?
How well were your technical questions addressed?
How well were your questions about the coding assignments addressed?
How well were your questions about the business assignments addressed?
How effectively did course staff direct you toward resources?
How well did the course staff support you overall?
How useful did you find the real-time feedback during video calls?
How clear was the real-time feedback during video calls?
Thinking about the video calls for your Peer Learning Group and your Capstone team, to what degree do you agree with the following statements?

The feedback caused me to participate more.
The feedback caused me to give other team members more of an opportunity to speak.
The feedback caused me to participate less.
The feedback caused the other team members to talk more.
The feedback caused the other team members to give other team members more opportunities to speak.
The feedback caused the other team members to talk less.

On average, approximately what percentage of time were you speaking during meetings?
How would you describe how the conversations played out?
After using the video tool and seeing the real-time feedback, to what degree do you agree with the following statements?

I found this to be a useful tool.
I will try to talk more in similar activities in the future.
I will try to make sure other people have an equal opportunity to talk.
I will try to talk less in similar activities in the future.
I would like to use this tool in the future (for learning or work).

Do you have any suggestions for how we can improve the real-time feedback?
How useful was the Riff Stats feedback?
How easy was it to interpret the Riff Stats feedback?
How did Riff Stats feedback change your behavior?
How useful was the Riff messaging feature (the channels where you connected with other team members)?
Considering all the aspects of the Riff Platform (video calling, messaging, real-time feedback, and post-meeting feedback) to what degree do you agree with these statements?

The Riff Platform helped me connect with my team members.
The Riff Platform helped me connect with other people in the course.
The Riff Platform helped me connect with course staff.
The Riff Platform enabled my team to work effectively together.
The Riff Platform enabled me to be more successful in the course.

The Riff Platform was easy to use.
The Riff Platform did not present technical challenges.
The course environment was easy to use.
The course environment did not present technical challenges.
The coding environment was easy to use.
The coding environment did not present technical challenges.
I found it easy to navigate between the course environment, the coding environment, and the Riff Platform.
I was able to use all the tools in the course without any difficulty.

A2. Key Results

General

73% of course close survey respondents would recommend the course to a friend
89% reported being Satisfied or Very Satisfied with what they learned in the course
63% felt that activities and assignments aided their learning
92% felt well or very well supported by course staff and mentors
Net Promoter Score (NPS) = 62

Collaborative exercises

85% of survey respondents reported that the exercises helped them connect with peers
65% of survey respondents reported that the exercises aided their business understanding

Appendix B—Definitions

Correlations: A correlation value quantifies the relationship between the values of two numeric variables. It ranges between -1 and +1, indicating negative and positive relationships, respectively. A correlation of 0 (or near-zero) means there is no discernable relationship between the two outcomes that are measured. As an example, if the correlation between Riff Video usage and passing a course is 0.70, that means the more a participant uses Riff Video, the more likely it is that this person will pass the course.

Odds Ratios: The odds ratio ranges from 0 to infinitely large. A ratio above 1 means that the odds improve when the change is made. A ratio below 1 means the odds get worse.
A ratio of 1 means the odds didn’t change at all. As an example, if the odds of passing the course for a student who never used the video were 1 to 5, and the odds of passing among students who used the video exactly once were 3 to 7, then the odds ratio associated with exposing a student to one additional video call is [(3/7):(1/5)] = 15/7, which is a little over 2. This would mean that the odds of the student passing the course roughly double when they are exposed to one additional video call. Notice that this is not the same as saying they are twice as likely to pass the course. It says the odds got twice as good.

Significance: The “p-value” is used to quantify the significance of a relationship. The smaller the p-value, the stronger the evidence that some null hypothesis (stating that a relationship between two variables does not exist) can be rejected. Typical threshold p-values (to reject a null hypothesis) are 0.001, 0.01 and 0.05. The strength of the relationship can be determined based on which threshold the p-value falls under.

Acknowledgments

This research is based upon work supported by the National Science Foundation under Grant No. 1843391, SBIR Phase I: Positive Effects of Feedback and Intervention for Engagement in Online Learning.

About the Authors

Beth Porter is the CEO of Riff Analytics. Formerly, Beth was the VP of technical platform at Pearson Education and VP of product engineering at edX. Beth has led research and development of products that seek to transform online teaching and learning, including driving the Open edX initiative at edX and architecting the original Texas OnCourse program. Her engagements in higher education include lecturer at Boston University Questrom School of Business.

John A. Doucette is a research scientist. He was an employee of Riff Analytics at the time this research was conducted. Prior to working at Riff Analytics, John was an assistant professor at New College of Florida. He holds degrees from the University of Waterloo, and Dalhousie University.

Andrew Reilly is a data scientist at Riff Analytics, where he drives data-centric product development. He is currently a student at New College of Florida completing a Master’s degree in Data Science, and holds a Bachelor’s degree in Psychology from Central Washington University. Prior to working with Riff, he worked as a graduate intern providing business analysis to the Economic Development Corporation of Sarasota County.

Dan Calacci is a PhD student at the MIT Media Lab studying how data and algorithms impact community behavior and governance. Their recent work involves studying how new sources of data and its stewardship can help communities and worker collectives advocate for a more just future. Their artwork and research have been exhibited and presented around the globe.

Burcin Bozkaya is a professor and director of data science at New College of Florida. Prior to joining New College, he worked as a professor of business analytics and director of Behavioral Analytics and Visualization Lab at Sabanci University and a visiting professor at MIT Media Lab. His main research interests include behavioral analytics with big data, computational social science, spatio-temporal modeling and analysis and vehicle routing and logistics.

Alex Pentland, professor at MIT, directs the MIT Connection Science initiative, and previously co-created the MIT Media Lab. He is one of the most-cited computational scientists in the world, member of the National Academy of Engineering, and co-leads the IEEE Council for Extended Intelligence. He is a co-founder of over a dozen companies, including Riff Analytics.

Issue Nineteen

The Help Desk as a Community-Building Tool for Online Professional Development

Salome Apkhazishvili, Media Education Lab

Serene Arena, Independent Scholar

Renee Hobbs, University of Rhode Island

Abstract

COVID-19 safety measures have forced professional development programs to pivot to online environments, which affects how participants interact and collaborate. When the University of Rhode Island hosted their annual, week-long teacher professional development event as a fully-online program, the staff of the Summer Institute in Digital Literacy provided an online, real-time help desk service, knowing that some participants would benefit from targeted, individualized support. Using evidence from the help desk incident log and post-event qualitative interviews, this research deepens understanding of what teacher professional development can look like in online environments. Through the provision of personalized, real-time assistance that created a relationship between the participant and the staff member, those who used the Help Desk reduced their feelings of isolation, increased a sense of connectedness, and demonstrated agency as co-learners in a professional development learning experience. By providing intrapersonal, technical, and navigational support, the help desk deepened a sense of community connectedness in an online professional development program for educators who faced a dramatic pivot to online learning as a result of the COVID-19 pandemic.

Closures and physical distancing measures due to COVID-19 have shifted the way we interact, forcing many organizations to eliminate programs in teacher professional development (TPD) or move them to online platforms for the first time. In this shift, educators have faced some obstacles and adjustments. Although online learning is not a new model for digital literacy education, the COVID-19 pandemic has changed how and to what extent educators are expected to utilize online platforms for learning and community, bringing with it challenges to and opportunities for growth.

Given this backdrop, we look to understand how current research in TPD translates for fully-online experiences, exploring principles of community-building to understand the affordances of online learning. Importantly, our work seeks to understand the possibility of successfully applying known, effective in-person practices to online learning and professional development. This study documents a key feature of the 2020 Summer Institute in Digital Literacy (SIDL), a TPD program affected by COVID-19 restrictions. In its eighth year, SIDL was held completely online for the first time, gathering around 150 participants—mostly from the United States but including more than two dozen from 10 countries around the world. Educators, school leaders, researchers, librarians, and media literacy advocates come together annually for the week-long intensive program to learn about digital literacy, practicing skills and instructional techniques that support student learning via digital platforms (Hobbs and Coiro 2019; 2016). When pandemic restrictions emerged in March, program planners decided to use a combination of synchronous and asynchronous learning, using a learning management system plus video conference meeting rooms, along with flexible scheduling

Because of the intensive nature of the program, with its focus on hands-on media production activities and the activation of digital literacy competencies, they also decided to add an online help desk component to act as a support mechanism. The help desk would rely on a dedicated Zoom video conference room and text service (Google Voice) staffed continuously to offer hands-on, real-time support throughout the six-day, 42-hour event. By visiting the Lounge/Help Desk, participants could hang out and engage in informal dialogue but also get questions answered or receive individualized coaching.

In this study, we aim to better understand the value of the SIDL Lounge/Help Desk as a component of a teacher professional development program. Through the provision of personalized, real-time assistance that created a relationship between the participant and the staff member, we wondered if it could replace the “elbow-to-elbow” support that the program embodies when implemented in face-to-face learning contexts, where faculty and participants work side-by-side to create to learn (Hobbs and Coiro 2016).

Literature Review

The academic scholarship most relevant to this work focuses on the characteristics of professional learning environments that address the identity of teachers as learners and the role of help desks in community-building for both face-to-face and online learning contexts.

Teachers as co-learners

COVID-19 restrictions have required educators to adopt online teaching methods not as an option but as a necessity, and the suggestion that “what works in effective traditional learning environments may or may not work in online environments” has proven true in the forced remote learning of the 2020 pandemic (McCombs and Vakili 2005, 1582). In these unusual circumstances, teachers must “unlearn” traditional concepts in order to be receptive to new approaches that work better in online settings. While some teaching and learning habits are useful, they can also be detrimental, especially in unpredictable and unstable moments in time. Not only must educators learn new forms of social engagement, they must also “unlearn habits that have been useful in the past but may no longer be valuable to the future” (McWilliam 2008, 263).

One of the most dynamic settings where a teacher can embrace the identity of the learner is a TPD program. Ann Lieberman (1995, 592) argued for teachers to be actively involved in their own learning, noting that “the ways teachers learn may be more like the ways students learn than we have previously recognized.” When teachers actively learn from each other, they may create communities of practice where participants share, reflect on, and build new knowledge (Darling-Hammond et al. 2017; Desimone 2009).

During professional development, educators are placed in student roles, where they may enter into a “troubling zone” that can be also described as a discomfort, and it is this discomfort that helps to build a critical inner reflection leading to openness and empathy (Fasching-Varner et.al. 2019).

In online learning contexts, the ability to critically reflect on the identity of the learner is crucial for the design of effective TPD (Baran, Correia, and Thompson 2011). A profound learning opportunity can be created by the temporary disequilibrium caused by switching from “expert” to “learner” (O’Mahony et al. 2019). An aggregate review of how to improve TPD for online and blended learning confirms this, stating that teachers must have “the opportunity to reflect on the roles that they ascribe to themselves and their students in (online) environments” (Philipsen et al. 2019, 1157). This empathy for learners creates critical awareness that can be used during times of acute situational adjustments, such as with COVID-19.

Help desks as spaces for online community building

Online learning creates many opportunities for communities to form. Smith (2013) notes that community is variously developed by place, interest, and communion and is built through tolerance, reciprocity, and trust. But community doesn’t make itself: Connections are made through interaction, thus enabling people to build those communities (Smith 2013).

So what does one do when interaction becomes virtual, such as occurred during the COVID-19 pandemic? Coryell (2013) contextualizes collaborative and comparative inquiry in cross-cultural adult learning by framing learning as participation (partaking in knowledge), rather than learning as acquisition (possessing it). In referring to Sfrad’s (1998) work, she argues that in learning-as-participation mode, learners recognize knowledge as an interactional journey (Coryell 2013).

Community cannot exist without shared experience, and TPD programs must activate a sense of community if they are to be successful. A sense of community informs the formation of collective identity, which “is demonstrated when group members work interdependently with a shared purpose and responsibility for collective success” (Vrieling et al. 2018, 3).

Help desks may be spaces that support collaborative learning. In the field of information science, computer help desks located in universities have been studied to understand their organizational or technical functions, with focus on staffing, training and other issues. Some researchers have explored how help desk activity is used to create, manage, and share knowledge (Halverson et al. 2004). But there is limited research on collective identity, participation, or co-learning in help desk scenarios. Only one study is especially relevant to our work: it looked at what kind of learning takes place between those who need support and those who offer it. In this help desk research, a consistent sequence of four phases emerged to support communication, learning, and engagement in a face-to-face help desk. The phases included the processes of introduction, knowledge establishment, conceptual change, and agency. Findings showed that these interactions (consisting of two professionals of different expertise) activated metacognition, a type of reflection, leading to learner agency and personal fulfillment (O’Mahony et al. 2019).

With this understanding of community-building via help desks, we can consider the unique opportunities and challenges of online learning environments, including for TPD. As a result of the rise of social media, digital interaction has become normative for most people around the world. Yet for many educators, online learning has been thought to be inferior to face-to-face learning. For example, researchers who conducted a meta-analysis of various TPDs and how they affect student outcomes found that TPDs with online components yielded lower student achievement than programs that were entirely face-to-face. Yet, in that same study, several online learning practices were associated with gains, including having space to “troubleshoot and discuss implementation” of digital tools (Hill et al. 2020, 54).

To prepare teachers for online learning, online TPD may be a powerful treatment. But an understanding of the full potential of online TPD is still in development. Based on participant comments regarding collaborative and face-to-face engagement in Collins and Liang’s (2015) study of online TPD, little advancement in both the approach and implementation of these programs seems to have occurred. They report:

A number of individuals expressed they did not find OTPD as effective or meaningful as traditional face-to-face protocols…hardly anyone mentioned the online environment as engaging or encouraging participation through support or collaboration. A high number explicitly expressed that interaction was lacking … and many reported that even though they appreciated online delivery and its accessibility … they still missed the dialogue and collaboration of face-to-face PD. (Collins and Liang 2015, 28–29)

Online learning pedagogies are still primarily viewed through a prism of limitations when it comes to community-building. But scholars and practitioners are beginning to reimagine the use of technology and digital devices for collaborative learning. Bhati and Song (2019) conceptualize the creation of a dynamic learning space (DLS) in combination with mobile collaborative experiential learning (MCEL) as a means to encourage “high-level learning” and personalization. To our knowledge, approaches that level up these experiences by using the collaborative value of peer-to-peer synergy—proven instrumental to successful social learning—have not yet been studied.

Research Methods

The purpose of the paper was to understand the role of the help desk in online TPD as a form of informal learning and community building. Because this is a form of exploratory research, we asked: How did adult learners experience the value of an online help desk in the context of teacher professional development?

Participants and program context

The Lounge/Help Desk was fully-integrated into the Summer Institute in Digital Literacy (SIDL), the six-day, 42-hour TPD program, which included 135 participants and fifteen staff members. SIDL is an established program with a long history (Hobbs and Coiro 2019; 2016) but 2020 was the first time the TPD program was offered as a fully online program. Thus, many features of the program required adaptations that were new to the event organizers, faculty, staff, and returning participants.

The SIDL Lounge/Help Desk was conceptualized as an informal gathering space, where participants could go to get help—but also to interact with other participants and staff. Describing the Help Desk as a lounge was also intentionally designed as a means to reduce the stigma of asking for help. Participants were reminded of the Lounge/Help Desk every day. Each morning of the six-day program, participants received an email with the links to the learning management system, where links to video conference Zoom rooms and the Lounge/Help Desk were provided. The first and second authors were responsible for staffing the Zoom room Lounge/Help Desk, and the third author served as their supervisor.

The Lounge/Help Desk was both a synchronous and asynchronous communication channel for program participants and faculty, open to join at any time throughout event hours (9 AM–5 PM). Participants joined the Zoom Room or sent texts or emails, and these were handled throughout the day as the TPD program was in operation. Program faculty also participated in the Lounge/Help Desk, joining the online Zoom room for 1–3 hour shifts. In cases where the staff could not answer questions, one member would reach out to program organizers via a private Signal chat, which was used as a backchannel tool, in order to gain information needed to answer questions or solve problems.

As Lounge/Help Desk staff members, we gradually came to recognize that we were teachers in the TPD program and that our role was truly educational. We were not just providing a transactional service: Through our interaction, we were demonstrating the depth of community building that is at the heart of the SIDL program (Hobbs and Coiro 2019). People came to the Lounge/Help Desk needing different kinds of personalized support. Some were clearly beginners in their use of technology, while others had considerable expertise. But each of these individuals were people that we had a chance to interact with and learn from; during other components of the program we sometimes encountered them, particularly in small breakout groups and informal discussions. Indeed, it was the awareness of our own experience as co-learners with the participants that inspired our interest in this research project.

Data collection and analysis

Incident Log

During the program, we logged every visit to the Help Desk in an incident log to identify each time a participant visited the Zoom room or interacted via Google Voice text messages. During the real-time TPD program, this practice helped event organizers to understand participant pain points for particular learning activities that involved digital media and technology. It also functioned to help staff contact participants when reaching out to those whose questions could not be resolved in real time. The log documented: who contacted the help desk; who assisted them; what the question or problem was; and how the resolution occurred. The incident log was not initially designed for research, as we merely imagined its function as a tool for formative assessment during the program implementation.

During the program, the Help Desk Zoom room was accessed 76 unique times by 41 different participants. Fourteen text messages were sent to Help Desk staff. In our first phase of data analysis, the first and second authors used data from the incident log to categorize our encounters with participants. We worked independently to develop categories to account for the variety of interactions in order to increase divergent interpretations and reduce confirmation bias. By reviewing the categories created by each researcher using simple description, we identified emerging themes like: “emotional support needed after confusion caused by new platforms,” “tech glitches,” and “wanting to be told what to do.”

Interviews

After the event, we reached out to 41 participants who had used the Help Desk and eight agreed to participate in a research interview; one male and seven females. In terms of race and ethnicity, six participants were White, one Black, and one Latina. Seven participants were from the United States, while one participant was from Great Britain. The average age of the participants varied from 40 to 65. The demography of the research participants closely represents the SIDL demography, with the majority of participants white, female, and based in the United States.

The interview was conducted through Zoom and included ten scripted questions regarding the participant’s experience using the Help Desk. Participants were asked to describe what led up to their decision to access the Help Desk, the emotions they could recall at play before, during, and after its use, and how the experience compared to other help desk services they may have experienced in the past. Interviews were conducted three weeks after the event. The University’s institutional review board approved the research and participants gave permission for audio recording.

In the second phase of data analysis, we analyzed both the transcribed interview data from the individual interviews and the incident log data collected during the TPD program. The interview data helped to more deeply contextualize the documentation in the incident log. For example, interviews suggested that areas first coded as “tech glitches” may also relate to “confusion,” and that participants who we initially perceived to be “needing to be told what to do” were navigating the social loss of community interaction.

Findings

Three themes emerged from this work which give insight into how informal learning was experienced in the context of using a Help Desk during an online teacher professional development program. Participants came with a variety of very specific questions and problems during the week-long program. Of the 76 visits to the Help Desk, many were easy to answer, requiring only a few minutes. Examples of these include finding a link to a Zoom room, recalling a password, or noting the day’s agenda and schedule. These were often merely a matter of visiting a web page and clicking a link.

But some questions required some additional form of co-learning as Help Desk staff needed to answer a question by modeling a learning process with a participant. Some of the questions that participants asked could not be easily answered by Help Desk staff. For example, one participant needed help learning how to edit a post on Wakelet, a digital curation tool, while another wanted a tutorial on ThingLink, a visual annotation tool. Neither staff member was familiar with these digital tools but both were able to demonstrate co-learning with participants to answer their question or solve their problem. Another participant struggled to find a solution to the microphone on her laptop, which suddenly stopped working. In each case, the Help Desk staff demonstrated through inviting the participant to share their screen, using coaching that enabled participants to solve their own problem with scaffolded support from a member of the staff. For questions that Help Desk staff could not solve on their own, they explained and modeled how they reached out for help from the larger faculty team. In those cases, staff were able to find answers within an hour or two of the request being made. Considering the nature of the help provided in the context of the participant interviews, we found that many of the Help Desk encounters created a rich interpersonal relationship between participant and staff member that functioned to reduce isolation, deepen a sense of community, and increase learner agency.

Co-learning as a journey borne of isolation

The Lounge/Help Desk reinforced the perception that the TPD program was a co-learning journey that involved the participants and the staff as collaborators. Many participants (and program faculty) were experiencing online TPD for the first time; it was a new experience for everyone.

While describing initial feelings and the scenarios leading up to accessing the Lounge/Help Desk, participants mentioned experiencing “confusion,” “nervousness,” and “anxiety.”[1] For example:

“Before [coming to the help desk], it was confusion and a little bit of…I wouldn’t go as far as to say panic, but close.”
“I was a little lost a couple of times in terms of where I was supposed to be going.”

In the TPD program, the novelty of a fully online event was made even more intense by the expectation that participants would be practicing the use of new digital tools, including Pathwright LMS, Adobe Spark, Padlet, and many other platforms. This may have exacerbated concerns that participants naturally have in new learning scenarios, except that, instead of being able to organically turn to the person next to you and ask questions, participants were, in that moment, alone.

Interview data clearly reveals that awareness of a sense of isolation was a precipitating incident. Participants noted feeling confused about “where” to go and when, unsure of which “Zoom room” they belonged in. At various points during the week, there was uncertainty regarding task details and/or deadlines for completion. These are common in learning environments, and the accessibility of the Help Desk acted as a bridge in lieu of the missing opportunity to “turn to your neighbor,” thus helping participants keep involved and engaged.

Some veteran SIDL participants (attending for a second or third time) hesitated in reaching out to the help desk out of concern for others, downplaying their own need for support. Feelings of demoralization and inadequacy were also referenced in the moment of realizing help was needed.

“[Y]ou think, ‘should I know the answer to this—is this something I can figure out myself?’ … my hesitancy was that people might need [the Help Desk] more than I did.”
“Everybody sort of doesn’t want to take time away from other people or you don’t want to bother people. So there’s always that, but I felt more comfortable using it after I used it the first time…”
“The feeling before I joined the lounge was ‘I’m “supposed” to be doing this, but I can’t.’”

One participant said that she felt much more comfortable coming to the help desk when she realized she knew one of the staff members. Clearly, such relationships and bonds can support not only successful learning but also continued community development.

Co-learning as a journey to connect

The decision to share Google Voice numbers with participants offered additional options to connect with the Lounge/Help Desk staff through calling or texting. One participant noted this as particularly helpful; as a non-native English speaker, it was easier for her to write her question. Because the help desk was continuously available during the six days of the program, it created a sense of immediacy, efficiency, and effectiveness, as participants saw how the help desk embodied the empathy of the program’s tagline: “Everyone Learns from Everyone,” a phrase that made adult learners feel welcomed as peers (Hobbs and Coiro 2019). For example, participants noted:

“The people there were very helpful and compassionate … about leading me through where something was and actually, one time, the assistant was confused as well. They didn’t quite know where to go. So we were learning together—how to navigate the site. So it felt like a very welcoming place.”
“I was very reassured. I was helped immediately; I wasn’t kept waiting … and I felt as though my concerns were being dealt with.”

Many participants had experienced help desks at their workplace or school. There, they encountered a generally asynchronous system: submit query, wait for response, hope for solution. But the SIDL Lounge/Help Desk was different. Participants who reached out for help mentioned appreciating the immediacy and liveliness of the help desk interaction. The help desk was an online “place” for congregation; after all, it doubled as The Lounge. Participants noted:

“Having a real person to talk to is a bonus. It’s better than either a chatbot or talking with somebody online—having somebody to actually talk to and have working through it is definitely a good thing.”
“When you contact a regular help desk, you feel like you’re just lost—your request is out there; you may or may not hear from anybody. That wasn’t the case here.”

During the interaction, some participants realized their initial confusion was a result of inattention. In being able to focus and talk through a concern and visualize it on a shared screen with the help desk staff, participants gained awareness of what they had overlooked. As they worked together, the missing piece of information would often be noticed by participants themselves. The sense of pleasure in solving a problem transformed the sense of isolation into a shared experience.

Co-learning as a journey toward agency

Interview subjects described the calm and confident feelings they experienced upon resolving their questions or concerns through the help desk interaction. Important to supporting this sense of agency was the ability for both the staff and the participants to share their screens. Screen-sharing enabled help desk staff to model the iterative process of learning to use digital platforms and the shared experience of confronting and solving a problem together built trust and independence for the participants. For example, participants noted:

“I could see things that I needed to see and know that I wasn’t missing anything.”
“Afterward, I had very clearly seen where to go. So it was a sense of relief that now I could do that by myself.”
“I learned that it wasn’t as complicated as I thought it to be. And that there was more than one way to approach the issue we were having.”

Almost all interviewed noted how their own struggles aligned with what their students may experience with online learning. In fact, contrary to Collins and Liang’s (2015) suggestion that honoring the adult is part of effective PD (the idea that while learners, they are first and foremost experienced adults and professionals), we found that participants who could embrace the role of learner—complete with the requisite insecurities, needs, problems and questions—gave them the opportunity to deepen empathetic connections to their own learners. This is one way to understand how an online help desk can provide value to adult learners in the context of teacher professional development.

We found that three forms of support—intrapersonal, technical, and informational—all contributed to increased participant agency as co-learners. Intrapersonal support occurred as participants entered the Lounge/Help Desk with strong feelings, the full range of feelings that manifest when something does not work as expected or when obstacles occur. Emotions varied from frustration to panic. Sometimes, these feelings emerged from intra-actions related to self-imposed expectations; in other cases, external pressures like time constraints were activating strong emotion. Feelings often coincided with information structure and technical scenarios, as when one is distracted or flustered and forgets simple things like how to log in. The ability to acknowledge and validate participant concerns in real-time provided an immediate sense of relief to participants—even when a solution wasn’t immediate.

Technical support included both hardware, software, and online platform glitches, as well as password problems. During the week-long program, a variety of forms of basic IT support was provided, such as updating software, changing passwords, checking settings, and restarting computers. In one instance, the Help Desk assisted a participant who was experiencing prohibitive technical problems (e.g. a poor network connection) by emailing PDF copies of online content. Participants learned more about their digital devices from the transparent way in which these forms of support were modeled by staff.

Navigation support was provided to participants in helping them find what they needed using the learning management system, which was unfamiliar to them. Help desk staff demonstrated how to find specific information, and in the process, they recognized that some of the challenges that participants were experiencing was the result of errors made by program staff, including mislabeled or broken links or poorly expressed language or wording. The help desk participants enabled the TPD faculty to recognize weaknesses in their own explanations of program activities. For example, in one instance, a set of Zoom links were presented using a red font color, which led them to be easily overlooked on a page full of text, even as the red color was intended to make them stand out visually. Help desk staff thanked participants for calling attention to the problem—but participants were equally grateful, expressing feelings of relief as they realized the problem was not “their fault.”

By supporting participants emotionally, technically, and navigationally, feelings of community emerged, because despite the lack of face-to-face encounter in this fully online TPD program, participants felt taken care of. As one experienced participant put it:

all the things that I think made Summer Institute special for me (in-person in past years) … were present this year … And the Help Desk was part of that. So the Help Desk was an even bigger part because without it, SIDL couldn’t have flowed—somebody could get lost.

Through the provision of personalized, real-time assistance, those who used the Lounge/Help Desk reduced their feelings of isolation, increased a sense of connectedness, and demonstrated agency as co-learners in an online professional development learning experience.

Discussion

Our findings provide strong support for the ability of help desks to function as vital components of online teacher professional development programs. SIDL’s Lounge/Help Desk enabled participants to move through an arc of learning-as-participation that not just supports but enhances learning. Rather than conceptualizing the help desk as a merely transactional experience, at the 2020 Summer Institute in Digital Literacy, it functioned as a meaningful part of the overall learning experience.

Of course, this study has several limitations: the small sample size and potential respondent and research bias must be considered as limitations, given the researchers’ own roles as staff during the TPD. We aimed to minimize this limitation by developing the initial analysis of the incident log separately in order to increase divergent interpretations and minimize confirmation bias. We recognize that our ideas of community-building in TPD are framed through an American, Westernized cultural lens, though effort was made to review work from across the globe. The research reviewed for this study is gleaned mostly from abled/neurotypical interactions of spoken or auditory communication, potentially limiting outreach and input.

This research makes a unique contribution to new knowledge by re-framing the online help desk as a novel feature of teacher professional development. Because the online help desk was available throughout the TPD, it functioned to engage participants much like in face-to-face interactions, qualifying it as space to troubleshoot and discuss implementation, a category found to be successful in creating student learning gains from teachers’ TPD learning (Hill et al. 2020).

Key features of the Help Desk design were critical for its use as such an informal learning space: it was called the Help Desk/Lounge, and it was designated specifically as a hangout place online, thus reducing the stigma of being perceived as a place for “people who need help.” For those educators with insecurities about their digital competencies, there was no shame associated with visiting the Help Desk. Thus, it connected and strengthened the program’s core value of “Everyone Learns from Everyone” (Hobbs and Coiro 2018).

The potential to build personalized engagement is another feature needed for a help desk to be part of successful TPD. As designed and implemented, the Help Desk provided the situational context needed to question and solve problems immediately and in real time, running in parallel to the formal program. It also exposed pain points in the event and platform infrastructure, offering a form of continuous evaluation of the TPD experience and enabling event producers to make adjustments during the event itself, further enhancing the program’s overall quality. This tailored approach, so aligned with teacher needs and experiences during COVID-19, enhanced the TPD’s sense of relevance for participants, a requisite dimension of effective training (Stein et al. 2011). The Lounge/Help Desk contributed to this sense of relevance by engaging one-on-one with individuals on the emotional, technical, and navigation challenges they were likely to face as educators heading into an unparalleled 2020-21 school year. The process of engaging with a help desk that offered individualized support offered participants the opportunity to develop understanding of possible hiccups that may be encountered in their own classes and the confidence to troubleshoot these problems themselves. This finding aligns with research that demonstrates the value of helping educators critically reflect on how they approach their work and consider their roles in the educational dynamics of learning (Baran et al. 2011).

While some researchers claim that TPD support must come “from an educational technologist or an expert within the field” (Philipsen et al. 2019, 1155), we found that a help desk intentionally staffed as a peer-supported environment was effective in modeling how to investigate problems together. In such paradigms, trust helps to bridge the implied power dynamics between the helper and the “helped.” Because the help desk staff positioned themselves as participants and partners in the process, they offered the support for collaboration so valued as a critical ingredient for teacher learning (Bates and Morgan 2018; Darling-Hammond et al. 2017) As Bates and Morgan (2018, 623) point out, “a co-learner stance” ultimately contextualizes and personalizes support, guaranteeing “that actual problems are addressed.” The question moves from an individual, isolated/ing concern to a social learning opportunity, something Vygotsky (1978) addresses as essential to meaning-making.

By viewing an online help desk as a shared learning experience with value as a programmatic feature of TPD, we will need to consider how it could be adapted in post-pandemic times, as teacher professional development returns to be provided in face-to-face contexts. The help desk offers the value of providing that “in the moment” experience for individualized grappling and reflecting on problems, helping to meet the needs of every learner. Because the online format was new to everyone involved, including the help desk staff, the co-learning journey in finding answers offered value to faculty, staff and participants alike. Although it was intended to provide individualized support for those experiencing technology problems, the Lounge/Help Desk actually became a part of the overall TPD experience, enabling it to be a programmatic feature that extended the value of the Summer Institute in Digital Literacy as a genuinely collaborative learning experience.

Notes

[1] The quotations in this section come from research interviews with 2020 SIDL participants (names withheld) and were administered by Salome Apkhazishvili and Serene Arena in August, 2020.

Bibliography

Bates, Celeste C., and Denise N. Morgan. 2018. “Seven Elements of Effective Professional Development.” The Reading Teacher 71, no. 5. (Mar/Apr): 623–626. https://doi.org/10.1002/trtr.1674.

Baran, Evrim, Ana-Paula Correia, and Ann Thompson. 2011. “Transforming Online Teaching Practice: Critical Analysis of the Literature on the Roles and Competences of Online Teachers.” Distance Education 32, no. 3: 421-439.

Bhati, Abhishek, and Insu Song. 2019. “New Methods for Collaborative Experiential Learning to Provide Personalized Formative Assessment.” International Journal of Emerging Technologies in Learning (iJET) 14, no. 7. http://doi.org/10.3991/ijet.v14i07.9173.

Coryell, Joellen E. 2013. “Collaborative, Comparative Inquiry and Transformative Cross-Cultural Adult Learning and Teaching: A Western Educator Metanarrative and Inspiring a Global Vision.” Adult Education Quarterly 63, no. 4: 299–320.

Collins, Linda J., and Xin Liang. 2015.“Examining High Quality Online Teacher Professional Development: Teachers’ Voices.” International Journal of Teacher Leadership 6, no. 1 (Fall): 18–34.

Darling-Hammond, Linda, Maria E. Hyler, and Madelyn Gardner, with assistance from Danny Espinoza. 2017. Effective Teacher Professional Development. Palo Alto: Learning Policy Institute.

Desimone, Laura. 2009. “Improving Impact Studies of Teachers’ Professional Development: Toward Better Conceptualizations and Measures.” Educational Researcher 38, no. 3: 181–199.

Fasching-Varner, Kenneth J., Michaela P. Stone, Roberto M. Mella, Francisco O. Henriquez, and Macarena Y. Palma. 2019. “‘…4542 Miles from Home…’: Repositioning English Language Learners as Power Brokers and Teachers as Learners in the Study Abroad Context.” Education Sciences 9, no. 2 1–13. MDPI. http://dx.doi.org/10.3390/educsci9020146.

Halverson, Christine A., Thomas Erickson, and Mark S. Ackerman. 2004. “Behind the Help Desk: Evolution of a Knowledge Management System in a Large Organization.” In Proceedings of the 2004 ACM conference on Computer Supported Cooperative Work, 304–313.

Hill, Heather C., Kathleen Lynch, Kathryn E. Gonzalez, and Cynthia Pollard. 2020. “Professional Development that Improves STEM Outcomes.” Phi Delta Kappan 101: 50–56.

Hobbs, Renee, and Julie Coiro. 2019. “Design Features of a Professional Development Program in Digital Literacy.” Journal of Adolescent & Adult Literacy 62, no. 4: 401–409.

Hobbs, Renee, and Julie Coiro. 2016. “Everyone Learns from Everyone: Collaborative and Interdisciplinary Professional Development in Digital Literacy.” Journal of Adolescent & Adult Literacy 59, no. 6: 623–629.

Lieberman, Ann. 1995. “Practices that Support Teacher Development.” Phi Delta Kappan 76, no. 8: 591–596.

McCombs, Barbara, and Donna Vakili. 2005. “E-Learner-Centered Framework for E-Learning.” Teachers College Record 107, no. 8: 1582–1600.

McWilliam, Erica. 2008. “Unlearning how to Teach.” Innovations in Education and Teaching International 45, no. 3: 263–269.

O’Mahony, Timothy, Jason Petz, Jonathan Cook, Karen Cheng, and Marco Rolandi. 2019. “The Design Help Desk: A Collaborative Approach to Design Education for Scientists and Engineers.” PLoS ONE 14, no. 5: e0212501. https://doi.org/10.1371/journal.pone.0212501.

Philipsen, Brent, Jo Tondeur, Natalie Pareja Roblin, et al. 2019. “Improving Teacher Professional Development for Online and Blended Learning: a Systematic Meta-Aggregative Review.” Education Tech Research Development 67: 1145–1174. https://doi.org/10.1007/s11423-019-09645-8.

Smith, Mark K. 2001, 2002, 2013. “Community.” The Encyclopedia of Pedagogy and Informal Education. https://infed.org/mobi/community/.

Stein, Sarah J., Kerry Shephard, and Irene Harris. 2011. “Conceptions of E-Learning and Professional Development for E-Learning Held by Tertiary Educators in New Zealand.” British Journal of Educational Technology 42, no. 1: 145–165. https://doi.org/10.1111/j.1467-8535.2009.00997.x.

Vrieling, Emmy, Antoine van den Beemt, and Maarten de Laat. 2019. “Facilitating Social Learning in Teacher Education: A Case Study.” Studies in Continuing Education 41, no. 1: 76–93.

Vygotsky, Lev S. 1978. Mind in Society: The Development of Higher Psychological Processes. Cambridge, Massachusetts: Harvard University Press.

About the Authors

Salome Apkhazishvili is a media and communication researcher from the country of Georgia where she coordinates the media and digital literacy program for the conflict-affected youth in the South Caucasus. She is a Fulbright communication graduate from the University of Southern Indiana. Apkhazishvili is a communications officer at the European Communication Research and Education Association Children, Youth, and Media section and a staff member of the Media Education Lab.

Serene Arena is a communication design expert focused on language use and collaborative development in communication and social systems. She has a Masters in Civic Media from Columbia College Chicago, where she studied social power dynamics and informal social spaces as foundations for community and personal identity.

Renee Hobbs is a professor of communication studies and director of the Media Education Lab at the University of Rhode Island’s Harrington School of Communication and Media. She has offered professional development to educators on four continents and authored 12 books and more than 150 scholarly publications on digital and media literacy.

Screenshot of Mac OS computer interface, with an image of an Egon Schiele painting in a small window at the left, and a larger Gravit Designer window with a composition of rectangles based on the painting.

Issue Nineteen

Trauma-Informed Pedagogy in the Digital Media Pandemic Classroom

Michael Mandiberg, College of Staten Island and the Graduate Center, CUNY

Abstract

After CUNY suspended in-person instruction during the COVID-19 pandemic, I started teaching a half-semester long digital media production course. Rapidly migrating a digital media production course to remote learning creates problems specific to our software-based classrooms, as many of our students lack access to the fixed technology used in the course. As I negotiated these problems, I sought first and foremost to reduce the harm this course would cause my stressed students. To promote care, and minimize harm, I made several decisions that prioritized students’ needs and limits, without sacrificing the rigor that would prepare them for the subsequent courses in the program. These decisions included: delivering asynchronous lessons via Blackboard with flexible assignment deadlines and using two Adobe web app clones, Photopea, and Designer.io, rather than Adobe software itself. This essay articulates these decisions as a trauma-informed pedagogy of care. This theoretical framework builds on feminist ethics of care, public health principles of harm reduction, and social welfare’s trauma-informed practice. This approach allowed me to destigmatize illness and late assignments, and reduce the stress that this course would have on the already traumatized lives of my students, colleague, and myself.

As the COVID-19 pandemic took hold of New York City, the City University of New York moved all of its courses online. One of my courses was a half-semester-long digital media arts course which was scheduled to begin in the middle of March, after the transition to distance learning. As I converted the course, I made several key decisions that prioritized students’ needs and limits, and minimized complexity. Because I knew many of my students wouldn’t have access to fixed technology, I used two Adobe web app clones, Photopea and Designer.io, rather than Adobe software itself. Anticipating that COVID-19 would prevent students from participating consistently, I taught the course through asynchronous lessons, with a flexible timeline. Despite Blackboard’s many problems (Lapowsky 2015), I used it because I knew my students’ other classes would use this official CUNY platform and I didn’t want them to learn anything new or to remember any extra passwords.

In all of these decisions, I sought first and foremost to reduce the harm this course would cause my stressed students. In that mid-March moment, all signs indicated that this recently-designated pandemic would get really bad, and I didn’t want this class to make it worse. I balanced two competing pedagogical principles: the imperative to make this process as easy as possible and not produce unnecessary stress for the students, the other adjunct instructor who would be using my materials, and for myself; and the need to ensure that the students actually learned the material well enough that they would be able to succeed in the courses that follow this class. Or to put it more bluntly: I tried to prioritize care and reduce harm, without sacrificing rigor. The approach seemed to work, as more students successfully completed the class than during a regular semester.

In this essay I will articulate a theoretical framework for how I was thinking about these decisions as I made them and how I have come to understand these decisions in retrospect. At the time, I framed my pedagogical choices through a feminist lens as decisions about care and harm. Additionally, my familiarity with harm reduction principles gave me a loose framework to assess my decisions, destigmatizing illness and late assignments, and reducing the stress that this course would have on the already traumatized lives of my students, colleague, and myself. In retrospect, I will frame these decisions through trauma-informed pedagogy, a practice I only recently learned of.

Care and Harm

In recent years, many differently motivated organizations, movements, and individuals have deployed or theorized discourses of care, caregiving, and self-care. Just weeks before the pandemic took hold, both Social Text and The Sociological Review published special issues on radical care (Hobart and Kneese 2020; Silver and Hall 2020). Writing in the intro to Social Text, Hi‘ilei Julia Kawehipuaakahaopulani Hobart and Tamara Kneese juxtapose these competing claims:

On the one hand, self-care is both a solution to and a symptom of the social deficits of late capitalism, evident, for example, in the way that remedies for hyperproductivity and the inevitable burnout that follows are commoditized in the form of specialized diets, therapies, gym memberships, and schedule management. On the other hand, a recent surge of academic interest in care … considers how our current political and sociotechnical moment sits at the forefront of philosophical questions about who cares, how they do it, and for what reason.

Care means something very different for childcare worker advocates and Gwyneth Paltrow’s Goop brand. Of course, COVID-19 only further exacerbated these tensions between corporate carewashing and the strain on essential care workers (Chatzidakis et al. 2020).

Care has an extended relationship to pedagogy. Nel Noddings articulated the feminist ethics of care philosophy to argue that care is a core element and value in pedagogical relationships between teachers and students (Noddings 1984). Her pedagogy of care has been very influential, especially in early childhood education, and has also been critiqued for its gender essentialism (Monchinski 2010). Others have explored the ways in which the theory would need to be transformed to be applicable to online education, with its shifts in contexts and relationships (Rose and Adams 2014).

My own engagement with care comes out of my work on Art+Feminism, an international community that strives to close the information gap about gender, feminism, and the arts on Wikipedia. Taking inspiration from Audre Lorde’s statement that “Caring for myself is not self-indulgence, it is self-preservation, and that is an act of political warfare” (Lorde 1988), we design our events to support our participant’s minds, bodies, and psyches. We do this by constructing welcoming and accessible trainings, providing food and childcare at our events, and by maintaining a friendly spaces policy (Evans, Mabey, and Mandiberg 2015). We do this because we know that activism takes physical and emotional energy and is often met with resistance. We seek to care for the participants and to reduce the potential of any harm that may come to them (Tamani et al. 2020).

My work with Art+Feminism has caused me to think a lot about how to reduce the harm that our participants experience. It may be unconventional, but in that intense moment in March, I used my admittedly surface level understanding of harm reduction as a loose framework to assess my decisions. Originally articulated during the 1980’s HIV epidemic to describe needle exchanges, harm reduction is a public health theory that eschews an abstinence-only approach to risk and disease in favor of practices that minimize negative outcomes (Des Jarlais 2017). To be clear, I am in no way equating taking a digital media course in a pandemic with opioid addiction; the concept of harm reduction can be implemented in different circumstances. While harm reduction remains most frequently discussed in terms of drug and alcohol addiction or sex education, society has widely adopted many other harm reduction strategies: seat belt laws and rest stops reduce traffic deaths endemic to automotive travel; hard hats, bicycle, motorcycle, hockey and football helmets reduce serious brain injuries; life vests and fences around pools help prevent drowning; and sunscreen mitigates the danger of skin cancer inherent in being outdoors (UNAIDS 2017). During the pandemic, societies have encouraged social distancing, hand washing, and wearing masks to help reduce the likelihood of contracting COVID. These are forms of harm reduction: we accept that it is not possible for most people to completely abstain from interacting with other people and, for those that continue this inherently risky behavior, certain practices can help reduce the potential for physical harm (Kutscher and Greene 2020). Of course, these measures have not mitigated the pandemic’s significant mental health impact (Choi et al. 2020; Abbott 2021).

While it may be unconventional to apply harm reduction principles to the mental health impacts of the pandemic classroom, a small number of practitioners have discussed applying harm reduction principles to mental health (Krausz et al. 2014). While writing this essay I learned about social welfare’s use of trauma-informed practice to provide services that are sensitive to their clients’ traumatic histories. Trauma-informed practice is built around five principles: ensuring safety, establishing trustworthiness, maximizing choice, maximizing collaboration, and prioritizing empowerment (Fallot and Harris 2001). Educators have adapted these principles into a trauma-informed pedagogy in K–12 education (Thomas, Crosby, and Vanderhaar 2019), and more recently in post-secondary education—first in social welfare (Carello and Butler 2015), and to other disciplines in the wake of the pandemic (Imad 2020). At their core, these practices seek to minimize the potential for retraumatization and maximize students emotional and cognitive safety.

While I was motivated by discourses of care and harm, trauma is probably a more precise definition. I always enter the classroom with the knowledge that my students have already experienced trauma, as people living in an unequal society, whose divisions, imbalances and punishments are marked by the intersections of race, gender, and class. I knew that many of my students would be physically vulnerable, and the rest would be economically vulnerable. Most of my students hold jobs, many of which are full time. I knew that most of my students either work in parts of the service sector that would be deemed essential or in retail jobs that would be laid off or furloughed. They live at home with their parents, who are similarly vulnerable.

Thus, as I redesigned the course, I centered care by reducing the potential for harm and trauma that this course might cause my already traumatized students. I knew that the COVID-19 crisis would amplify and transform a task that would have previously been a productive challenge into a debilitating barrier to completing an assignment or the course. We had to get our way through the semester amidst a public health crisis, and I wanted to make sure I removed as many barriers as possible and reduced the stress that this course would have on my students, colleagues, and myself.

Digital Foundations Online

COM 115 Introduction to Media Environments is a one-credit, 7 ½-week course that introduces students to the basics of digital media production. It is required of all students in the Communications major at the College of Staten Island and is the prerequisite for all courses in the Design and Digital Media specialization. COM 115 is the only course in our program with a standardized syllabus used across all sections and instructors. I typically teach two to three sections a year, while the other six to eight sections are taught by adjunct faculty. During the second half of the Spring 2020 semester there was only one other section, taught by an adjunct instructor.

I developed the course alongside the Digital Foundations textbook I co-authored with xtine burrough (burrough and Mandiberg 2008) and maintain the wiki version of Digital Foundations, which is kept up to date with Adobe software releases. Like Digital Foundations, COM 115 integrates historical examples and the design principles of the Bauhaus into an introduction to digital media production. For example, in COM 115 students use the Josef Albers color theory exercises in order to understand the Color Picker tool, integrating history, aesthetics, and technique in the same lesson. While the course emphasizes design principles and techniques over software training, the class does function as the “Intro to Adobe” for our department. The Adobe Creative Cloud has developed a monopoly on design and digital imaging software in the creative industries and in the classrooms of students who aspire to enter those industries. Like all monopolies, Adobe extracts a hefty price—one that has become more unavoidable since they shifted to a subscription-only model.

In March, faculty from across CUNY converged on the usually quiet Media, Arts, and Technology Discipline Council group on the Academic Commons to participate in a thread titled “Moving production courses online for COVID // the Adobe problem.” Though media arts courses are well suited for online delivery, rapidly migrating a digital media production course to remote learning creates problems specific to the tools and techniques used in our software-based classrooms. This is not just a problem for this course, or for all digital media arts instructors, but for all courses that rely on fixed technology—especially those at underfunded public institutions like CUNY with students who have difficulty accessing fixed technology (Andre Becker, Bonadie-Joseph, and Cain 2013; Smale and Regalado 2017); as opposed to mobile technology like smartphones, fixed technology refers to desktop computers, printers, and other resources that are often only available to our students in computer labs.

My own experience piloting an online version of the class in 2012 confirmed this challenge. Because of a failure in the CUNY First registration software, very few of the students realized they were registering for an online course. Roughly half of the students dropped the class, and many of those that remained struggled to succeed because they were unable to access the necessary Adobe software. I made my decisions to ensure my students would not experience this kind of trauma as a result of my course.

Avoiding Adobe

Two side-by-side screenshots of the Photopea and Adobe Photoshop user interfaces, with the mouse cursor hovering over the Transform menu item in the Edit menu; the Transform sub menu is almost identical in both screenshots. — Figure 1. A comparison of the nearly identical Free Transform menu options in Photopea and Adobe Photoshop.

I decided to not use Adobe software, opting instead to use two Adobe web app clones: Photopea and Designer.io. I made this decision by following my goal to prioritize care and reduce harm, without sacrificing rigor. I can say with confidence that this was the right decision.

At the time that I made the decision to use web apps, we did not know if our department’s Mac lab or the library labs would stay open—they did not. Nor did we know whether or not students would be able to access the Adobe software on their own computers—they were, because Adobe granted a special license, but we only learned this a week after the course began. I knew that if Adobe didn’t make that special license available, most of my students would not be able to secure their own copies of the software because of its subscription model’s substantial cost. Once it was made available by the school several weeks into the course, only a small percentage of the students installed it—the majority used the web apps, including all the students who came to video office hours.

Many of our students do not have easy access to fixed technology, and those that do have access to a desktop or laptop may not be comfortable installing software, nor may the computer be powerful enough to effectively run the resource intensive Adobe software. Because so many of our students lacked access to computing, CUNY made an emergency purchase of 25,000 Chromebooks and 25,000 Android tablets to ensure our students would be able to access online learning. Many of my students used these Chromebooks for the course. I chose to avoid the open source Inkscape and GIMP because of the difficulty of installing software, because of the uncertainty about the capabilities of the Chromebooks, and because their interfaces diverge from the Adobe software more than the two web apps.

Two side-by-side screenshots of the Designer.io and Adobe Illustrator user interfaces, with the mouse cursor clicking on the Shape Tool, showing the tool options; the tool icon and options are similar, but not identical, and they are located in different areas of the interface. — Figure 2. A comparison of the Shape Tool in Designer.io and Adobe Illustrator.

The user interface for Photopea is almost identical to that of Photoshop, and Designer.io follows the principles of Illustrator though its interface diverges more. In Photopea, the menu item names are almost all exactly the same, in the same place, with almost identical iconography; in Designer.io the tools are very similar, if located in slightly different places. Photopea even exports the PSD format with layers. They aren’t exactly identical: for example, Designer.io a has a slightly different pathfinder tool, and intermediate tools such as unsharp mask are missing in Photopea. Neither has the kind of advanced features that the Adobe software has, but these are sufficient for an introductory class at the 100 or 200 level. Most importantly, these tools scaffold directly into using the Adobe software: the tools, menus and concepts are so similar.

Using these web apps was the right decision. I experienced very few difficulties with students getting set up to use the software, and it worked on macOS, Windows, and Chrome OS. While neither is set up for mobile use, both officially work on tablets, though they were a bit glitchy in my testing. Most importantly, I am confident that the students who will continue on to other 200-level courses in the program will be able to seamlessly move into the Adobe software.

A caveat: I cannot predict the longevity of these two websites. I couldn’t find much information about Gravit, the for-profit Canadian company that develops Designer.io. Confusingly, Photopea has a GitHub repository for “bug reports and general discussion” without any code, but “Photopea is not fully open-source.” I don’t know what their business models are nor if they have sufficient resources to continue keeping these tools up and running.

Another caveat: both Photopea and Designer.io are freemium web apps. They include advertisements, and up-sell pitches for their premium versions. Designer.io requires you to create a login. I’m conscious of the adage that “if you’re not paying for something, you’re not the customer; you’re the product being sold” (blue_beetle 2010). Working within my principles of harm and care, I felt confident that a few more ad targeting cookies were a lesser harm than not being able to access any software.

Asynchronous instructional design and learning experiences

Knowing that the pandemic was certain to destabilize my students’ lives, and their schedules, I chose to design an asynchronous course in order to prioritize care and minimize harm. I knew that some students’ lives would be completely derailed by the pandemic, but if I structured enough flexibility into the course so students could complete the assignments when their pandemic timelines allowed, I could reduce the potential that they would fail to complete the course. I used Blackboard as the platform for asynchronous videos, pairing these with video office hours during the regularly scheduled two-hour class period. As much as I detest Blackboard, I knew that my students would be using it in their other classes, and would be most comfortable there. In the best of times, I substantially but productively challenge my students when I use Wikipedia or the CUNY Academic Commons as the course platform (Davis 2012). In these worst of times, I feared a new interface would be harmful.

Figure 3. First video demonstration: Dynamic and Static Compositions.

Using the web apps, I made video demonstrations of each of the exercises we cover from Digital Foundations. I shared these videos with my students and the adjunct instructor via a public Dropbox folder, which also includes the course syllabus, and the text of each of the assignments; additionally I posted a Study Guide in preparation for the exam. In the principle of reducing strain for everyone, including my adjunct colleague, I shared all of my assessment and communication materials via a private folder. These materials included: quizzes and exam for Blackboard; exam text and individual images; and all emails and announcements. I decided to make the weekly quizzes for practice only, rather than grading them, in an effort to lower the stakes.

Course dynamics were starkly different from teaching this in-person, or as a synchronous online course. Essentially everyone was doing their own private effort. They had almost no interaction with each other. Trauma-informed pedagogy emphasizes communication between students, collaboration, and peer support, which were entirely absent from this model (Imad 2020). And yet, they were able to finish the course. Of the fifteen students in the course, one dropped, and one never completed any assignments, but the remaining thirteen completed the majority of the assignments and all of them passed. Of these, five came to my video office hours; one student came every week, two students came every other week, and the other two popped in once. Four of the students who never came to office hours were very self-directed and highly motivated, and maybe had some previous experience with digital imaging. The remaining four who struggled and never made it to office hours made clear that they were impacted by COVID-19.

In retrospect, I recognize that I fostered the trust and safety that trauma-informed pedagogy advocates by encouraging my students to keep their video off if they wanted, which they did. During video office hours I only saw one student’s face, briefly, when they pressed the wrong button when trying to share their screen. While some of my colleagues actively complained that they couldn’t see their students, I knew my students needed privacy. They have a right to not let their classmates and their professors see the inside of their messy bedroom, or the closet, bathroom where they retreated from their other family members to get quiet and privacy. I found I was able to build rapport with the five who came to office hours despite the absence of video; and maybe I succeeded precisely because I didn’t ask them for video.

Challenges

The course was not without challenges specific to the online format and the larger pandemic context. The main instructional challenge that I faced was in demonstrating resolution. We typically do this by scanning objects; we set the resolution on the scanner and analyze the image in Photoshop. Knowing they would not have access to scanners—I didn’t even have a scanner at home—I reframed the exercise on photographic composition, with the emphasis on printing the image in multiple resolutions so they could see the different print sizes. I should have seen this coming, but I falsely assumed that they would have access to printers, so I reframed the printing process with an emphasis on taking screenshots of the print preview interface, which shows how big the print is in relationship to the paper size. There were more hiccups: the “print actual size” option is not available in the default Windows tool, something that I didn’t know because I don’t have access to a Windows computer at home, so I worked with one of the students to figure out a workaround, and she made a video demonstrating it. Unfortunately, we did not find a workaround for Chromebooks.

The larger challenge, as expected, was that half of my students were mostly disengaged from the course. At the start of the course, I told the students that the course was self-paced, but they should try to do one chapter a week, and to have the first three chapters done by halfway through the course. At the halfway mark only half of the students had completed the three chapters, and I was worried. I spent a lot of time writing them to encourage them to complete the work and relied heavily on a College of Staten Island Student Affairs COVID specific “EDUCares” team that succeeded in reaching the students I could not get engaged. EDUcares’ mandate included checking in on unresponsive students, performing a hybrid wellness check/late homework reminder. I shared a list of students who had not responded to my emails with EDUcares, and they emailed the students on their non-CUNY email addresses and/or called them at home and in some cases on their cell phones. They were able to get responses from all but one of the students and all but that one student (who never responded throughout the course) completed the first three chapters shortly after. During these exchanges I learned what I suspected: many of them did not have internet access, were without a computer until they received a CUNY loaner Chromebook, or were sharing a computer with other members of their family.

Outcomes

Certain aspects of the course (and the knowledge they produce) were simply not possible in this format: when I teach color theory in person, we spend fifteen minutes of class looking at and describing the colors of the clothes that everyone in the class is wearing. By the end of those fifteen minutes, they understand that there really is no such thing as black or white and they start to see the blues and purples in the very dark grey they previously would have called black and the yellows and oranges in the 5 percent grey that they would have called white. That simply isn’t possible to do as a group, in this online format. It isn’t really possible to do it with colors on a screen, as these are so removed from their lived experience, and each person’s screen will have a different color profile. I tried to do it with the one student who came to office hours every week, and it took us thirty minutes of one-on-one discussion—it is very strange asking a student to describe the color of the computer they are working on and persuading them that it isn’t actually dark grey, as they claim, but rather is a very low saturation dark blue.

To speak more broadly, it seems so hard to do radical pedagogy online. The software is structured around the banking model of education (Freire [1970] 2000), except instead of the human instructor at the front of the classroom depositing knowledge into the students’ presumed empty minds, it is a video of the instructor. Paolo Freire would be sad to see this (Boyd 2016). When I teach this course again, I will try to work against that as much as possible. This is one place where I might have sacrificed too much of the rigor in favor of care and reducing harm.

On the other hand, I feel like more of the students demonstrated baseline competency in the techniques that we covered. More specifically: in a typical in-person class of fifteen students, two-to-four students fall behind and never catch up because they always came twenty minutes late, missed the first class, couldn’t complete assignments on time, etc. This format alleviated some of this problem. In a typical in-person section those two-to-four students per class sit for the final exam and still fail the course, but in this format everyone who took the final passed; the only person who did not pass the course, never completed a single assignment and earned a WU (Withdrew Unofficially) grade. In this online course, the students who struggled did show their limits: in the final exam they performed better than the typical students who sit for but fail the final exam in an in-person class, but worse than the students who had steadily completed assignments throughout the course. Overall, the cohort did as well or better than most in person classes on the hands-on section of the final exam.

This was an emergency effort. None of these students expected to take an online class. My adjunct colleague and I never expected to teach one. Despite this strained context, my decisions to prioritize care in order to minimize harm helped us all get through the semester, while preparing the students to succeed in future courses. Though our classrooms will “return to normal” some point—or whatever our new normal will be—our students will carry the trauma of this pandemic. I hope to continue this trauma-informed pedagogy of care, finding a new balance between reducing harm and maintaining rigor in what will hopefully be a less traumatic post-pandemic teaching environment.

Bibliography

Abbott, Alison. 2021. “COVID’s Mental-Health Toll: How Scientists Are Tracking a Surge in Depression.” Nature 590, no. 7845: 194–95. https://doi.org/10.1038/d41586-021-00175-z.

Andre Becker, Danielle, Ingrid Bonadie-Joseph, and Jonathan Cain. 2013. “Developing and Completing a Library Mobile Technology Survey to Create a User-Centered Mobile Presence.” Library Hi Tech 31, no. 4: 688–99. https://doi.org/10.1108/LHT-03-2013-0032.

blue_beetle. 2010. “User-Driven Discontent.” MetaFilter (blog). August 26, 2010. https://www.metafilter.com/95152/Userdriven-discontent#3256046.

Boyd, D. 2016. “What Would Paulo Freire Think of Blackboard: Critical Pedagogy in an Age of Online Learning.” The International Journal of Critical Pedagogy 7.

burrough, xtine, and Michael Mandiberg. 2008. Digital Foundations: Intro to Media Design with the Adobe Creative Suite. Berkeley: New Riders in association with AIGA Design Press.

Carello, Janice, and Lisa D. Butler. 2015. “Practicing What We Teach: Trauma-Informed Educational Practice.” Journal of Teaching in Social Work 35, no. 3: 262–78. https://doi.org/10.1080/08841233.2015.1030059.

Chatzidakis, Andreas, Jamie Hakim, Jo Littler, Catherine Rottenberg, and Lynne Segal. 2020. “From Carewashing to Radical Care: The Discursive Explosions of Care during Covid-19.” Feminist Media Studies 20, no. 6: 889–95. https://doi.org/10.1080/14680777.2020.1781435.

Choi, Kristen R., MarySue V. Heilemann, Alex Fauer, and Meredith Mead. 2020. “A Second Pandemic: Mental Health Spillover From the Novel Coronavirus (COVID-19).” Journal of the American Psychiatric Nurses Association 26, no. 4: 340–43. https://doi.org/10.1177/1078390320919803.

Davis, LiAnna. 2012. “Digital Media Professor Gives Students Real-World Experiences through Wikipedia Assignment.” Wikimedia Foundation Blog (blog). January 4, 2012. https://diff.wikimedia.org/2012/01/04/design-professor-gives-students-real-world-experiences-through-wikipedia-assignment/.

Des Jarlais, Don C. 2017. “Harm Reduction in the USA: The Research Perspective and an Archive to David Purchase.” Harm Reduction Journal 14, no. 1: 51. https://doi.org/10.1186/s12954-017-0178-6.

Evans, Siân, Jacqueline Mabey, and Michael Mandiberg. 2015. “Editing for Equality: The Outcomes of the Art+Feminism Wikipedia Edit-a-Thons.” Art Documentation: Journal of the Art Libraries Society of North America 34, no. 2: 194–203. https://doi.org/10.1086/683380.

Fallot, Roger D., and Maxine Harris. 2001. “Creating Cultures of Trauma-Informed Care: A Self-Assessment and Planning Protocol.” https://doi.org/10.13140/2.1.4843.6002.

Freire, Paulo. (1970) 2000. Pedagogy of the Oppressed. 30th anniversary ed. New York: Continuum.

Hobart, Hi‘ilei Julia Kawehipuaakahaopulani, and Tamara Kneese. 2020. “Radical Care: Survival Strategies for Uncertain Times.” Social Text 38, no. 1 (142): 1–16. https://doi.org/10.1215/01642472-7971067.

Imad, Mays. 2020. “Leveraging the Neuroscience of Now.” Inside Higher Ed (blog). June 3, 2020. https://www.insidehighered.com/advice/2020/06/03/seven-recommendations-helping-students-thrive-times-trauma.

Kutscher, Eric, and Richard E. Greene. 2020. “A Harm-Reduction Approach to Coronavirus Disease 2019 (COVID-19)—Safer Socializing.” JAMA Health Forum 1, no. 6: e200656. https://doi.org/10.1001/jamahealthforum.2020.0656.

Lapowsky, Issie. 2015. “The Reeducation of Blackboard, Everyone’s Classroom Pariah.” Wired, July 21, 2015. https://www.wired.com/2015/07/blackboard-reinvention/.

Lorde, Audre. 1988. A Burst of Light: Essays. Ithaca, New York: Firebrand Books.

M. Krausz, Reinhard, Gregory R. Werker, Verena Strehlau, and Kerry Jang. 2014. “Applying Addictions Harm Reduction Lessons to Mental Healthcare.” Advances in Dual Diagnosis 7, no. 2: 73–79. https://doi.org/10.1108/ADD-01-2014-0003.

Monchinski, Tony. 2010. Education in Hope: Critical Pedagogies and the Ethic of Care. Counterpoints, v. 382. New York: Peter Lang.

Noddings, Nel. 1984. Caring, a Feminine Approach to Ethics & Moral Education. Berkeley: University of California Press.

Rose, Ellen, and Catherine A. Adams. 2014. “‘Will I Ever Connect with the Students?’ Online Teaching and the Pedagogy of Care.” Phenomenology & Practice 7, no. 2: 5–16. https://doi.org/10.7939/R3CJ8803K.

Silver, Dan, and Sarah Marie Hall, eds. 2020. “Radical Care.” Special issue. The Sociological Review, March. https://www.thesociologicalreview.com/radical-care-as-the-foundation-for-a-better-world/.

Smale, Maura A., and Mariana Regalado. 2017. Digital Technology as Affordance and Barrier in Higher Education. 1st ed. 2017. Cham: Springer International Publishing : Imprint: Palgrave Macmillan. https://doi.org/10.1007/978-3-319-48908-7.

Tamani, Melissa, Michael Mandiberg, Jacqueline Mabey, and Siân Evans. 2020. “What We Talk About When We Talk About Community.” In Wikipedia @ 20: Stories of an Incomplete Revolution, edited by Joseph M. Reagle and Jackie L. Koerner. Cambridge, Massachusetts: The MIT Press.

Thomas, M. Shelley, Shantel Crosby, and Judi Vanderhaar. 2019. “Trauma-Informed Practices in Schools Across Two Decades: An Interdisciplinary Review of Research.” Review of Research in Education 43, no. 1: 422–52. https://doi.org/10.3102/0091732X18821123.

UNAIDS. 2017. “Explaining Harm Reduction with Hard Hats, Seatbelts and Sunscreen.” UNAIDS (blog). June 23, 2017. https://www.unaids.org/en/resources/presscentre/featurestories/2017/june/20179623_harm-reduction.

About the Author

Michael Mandiberg is an interdisciplinary artist who created Print Wikipedia, edited The Social Media Reader (NYU Press), and co-founded Art+Feminism. Their work has been exhibited at Los Angeles County Museum of Art, The Whitney Museum of American Art, and Musée d’Art Moderne de la Ville de Paris, amongst others. Mandiberg is Professor of Media Culture at the College of Staten Island, CUNY and Doctoral Faculty at The Graduate Center, CUNY.

Figure 2: An image of a student sitting in front of a camera with hands clasped together in front of her face.

Issue Nine

Reflecting on Reflections: Using Video in Learning Reflection to Enhance Authenticity

Emma J. Rose, University of Washington Tacoma
Jarek Sierschynski, University of Washington Tacoma
Elin A. Björling, University of Washington Tacoma

Abstract

Reflection is commonly used in the classroom to encourage students to think about and articulate what they have learned. However, when students produce reflections they typically create a written text for the instructor, outside of the classroom and as a summative retrospective account of learning. In this paper, we present the details of how we implemented Ecological Momentary Reflection (EMR), a video enabled reflection within the classroom environment to help students assess their perceptions of self and learning across time. In this paper, we recount how we implemented EMR in an informal learning environment and provide our own assessment of its effectiveness. We argue that using video makes the reflection experience more authentic and meaningful for both student and teacher.

Introduction

Reflection is commonly used in the classroom to encourage students to articulate what they have learned and to aid them in thinking about how they have learned. Traditionally, students reflect on their learning process through the act of writing. According to Yancey, written reflections benefit students by helping them remember details of how they completed an assignment, as a generative process to create meaning for future writing, and as a way to develop authority and expertise (Yancey 1998). While written reflection has its strengths, it also has some inherent limitations. Written reflection is typically geared toward oneself and is often produced as a text for the audience of the instructor — perhaps limiting the student’s authenticity.

Moreover, writing is a form of culturally constructed expression with its own peculiarities (see, Chafe 1991; Chafe & Tannen 1987) that simultaneously differentiate and distance written texts from more direct or immediate forms of communication such as speech, sign or gesture. Even though texts are a profound means of representing human thought and introspection, the process of writing a text can become an impediment to self expression. For example, when writing skills are underdeveloped, not available, or stymied by other factors, writing can be limiting rather than productive. Additionally, much of the writing process relies on drafting and revising, a reiterative process aimed at clarifying expression and distancing the writer from the initially captured “raw” and momentary expression. At the same time, it can be argued that the strength of writing as a reflective tool lies precisely in a symbolic and temporal chasm between the individual and experience that nurtures reflection.

Given the benefits of the reflection process, and the inherent downsides of written expression, we wanted to explore a mode of reflection that could be incorporated authentically into the context of science learning in an informal setting, in this case during a summer STEM (Science, Technology, Engineering, Mathematics) camp for teens. We ground our use of the term, authenticity, in Buxton’s (2006) framework for authenticity in science education. He conceptualizes a youth-centered model that focused on the lives and learning of underserved and marginalized youth and thus on equity and social justice (see Medin & Bang 2014; Barron, Mertl & Martin 2014; Barton 1998; Barton & Young 2000; Nasir & Cooks 2009). These youth-centered models view authentic science learning and knowledge as a sociocultural process situated in lived experiences, such as cultures, identities, communities, homes, and the wide range of informal environments where learning occurs. Implementing a reflection method that leverages the experience of the community and captures reflection in context was synergistic to the authentic, youth-centered model of learning at the heart of the summer STEM summer camp experience we were investigating.

To create an integrated and authentic notion of reflection in the learning environment, we introduced an exercise that asked students to record a series of videos. Adding video to the reflection process helped students see that their thoughts about themselves and the STEM subjects have changed. This activity also layered an additional element of a shared and community based reflection to the learning experience. Furthermore, the video reflections provided instructors and program directors with an authentic representation of the students’ struggles and triumphs throughout the duration of the camp. These factors helped students see their own learning and helped instructors in getting feedback on the course to inform future improvements of the camp.

In this article, we provide our own reflection on the process of introducing a new method of reflection into a learning environment. The aim of this article is to introduce the concept of Ecological Momentary Reflection (EMR) and to recount its effective use in an informal STEM learning environment. We propose that the addition of momentary, that is in the moment, video to capture real-time student reflections in the classroom provides an authentic reflective practice leading to valuable insights for both learner and instructor. First, we articulate the context of the learning environment where we implemented EMR. Second, we define reflection as a pedagogical practice and how it is used in writing and how video can support reflection on practice. Third, we provide details of how we implemented video reflection in the summer camp, a method we are calling Ecological Momentary Reflection (EMR), and invite educators and researchers to consider this method of reflection in their own teaching environments.

Context: Informal Learning in a STEM Summer Camp

Every summer at a Pacific Northwest University, middle and high school students come together for a summer camp that is focused on learning about STEM (science, technology, engineering and math). The STEM camp mission is to encourage and increase diversity in STEM fields by providing informal learning experiences for students (grades 7-12) that remain underrepresented in the sciences. Underrepresented groups included low-income, minority, female and potential first-generation college students, among others. The campers who attend the STEM Camp tell us they are drawn to it year after year because it is fun, and ‘you get to do things’. Participants build robots, design video games, wade out into the muddy banks of our local waterways to collect water samples, and more. It is a break from school and existing social pressures; it is a safe place. Students make new friends and deepen existing relationships as they interact with their peers, some who return each year.

Informal learning is a broad concept that refers to any learning that occurs outside of the formal realm of school (Dierking et al. 2003). Informal learning includes people engaging with their environment in a variety of contexts and settings. Learning experiences that are designed for broad audiences (i.e., museums, summer camps, etc.) are considered types of informal learning both inside and outside of the STEM disciplines. In these settings, informal learning tends to be momentary, unplanned, problem-based, learner-centered, driven by individual interests (National Research Council 2009). Many STEM summer camps can be categorized as informal learning environments in that they promote experiential learning and exist outside of the realm of formal schooling. The camp instructors include current college students or professionals such as educators or scientists from the local community. The authors of this article were involved with the STEM camp in the roles of faculty mentors to the instructors.

It is within this informal learning setting that we implemented EMR as both a pedagogical tool and a research method aimed to enable students to reflect on their changing identities as well as their relationship to STEM subjects. In the Summer of 2015, we conducted an IRB approved research study where we used EMR with 9th grade participants in the STEM camp. The students spent three weeks designing a video game using Kodu, a visual programming language. In this paper, we focus on the promise of EMR for use as a pedagogical tool in the classroom.

Reflection as a Pedagogical Practice

Reflection is a common pedagogical practice where students are asked to think about and articulate what they have learned. Reflection has long been viewed as synonymous with thinking and learning (Dewey 1933). Moreover, reflection is considered a core element of metacognition. Metacognition, a multifaceted term connected with reflection, refers to knowledge about, and the regulation of, cognitive processes such as self-regulated learning (Brown, Bransford, Ferrara, and Campione 1983; Flavell 1979; Zimmerman 2002). In other words, metacognition is a student’s awareness of how to learn and also an awareness of herself as a learner. Metacognition is also connected to students’ ability to transfer their learning across contexts (Bransford, Brown & Cocking 2000). In K-12 settings, both elements of metacognition, knowledge of strategies associated with specific academic tasks (such as reading, writing or math) and self-regulatory strategies (such as self-monitoring or self-evaluation) are commonly used in teaching and learning tasks. There is a rich variety of established pedagogical approaches that apply metacognitive strategies for learning. For instance, in writing, students use think-aloud and self-questioning strategies (Scardamalia, Bereiter & Steinbach 1984). In reading, students self-monitor to check for comprehension through questioning, summarizing or making predictions about a text (Palincsar & Brown 1984). In math, students can use self-assessment to evaluate their own mathematical capabilities (Schunk 1996). These strategies have been associated with increased achievement and also with higher self-efficacy (see Schunk 1996). Furthermore, reflection often occurs as an important process in the development of expertise. Looking at how expert practitioners engage in their work, Schön observes: “Reflection tends to focus interactively on the outcomes of action, the action itself, and the intuitive knowing implicit in the action.” (Schön 1982, 56).

According to Yauncey (1998), reflection is both a process and product and the product that is created is available to the world and is therefore a social act. She states, “because it works both inside and outside, reflection-in-presentation is personal, but it’s social as well” (Yancey 1998, 94). However, in the writing classroom, a reflection tends to be a written text, constructed by a student for the instructor and often disregards this social aspect referred to by Yauncey. When produced for the sole audience of the teacher, written reflections can pressure students to attempt to perform the type of writing expected by the teacher: demonstrating what they should have learned rather than reporting on what they actually learned (Jenson 2010).

Ecological Momentary Reflection

Our goal in implementing video enabled reflection in the classroom was twofold. We wanted to see how students’ ideas about and in relation to STEM were impacted by their experience in the STEM camp. But we also wanted students to see how their perceptions and attitudes may have changed over time. We wanted the reflections to be as natural, immediate and embedded as possible within the practices of the camp. We wanted the reflections to be as close to the learning experience as possible, both in terms of the timing of the reflections and where the reflection would take place. In other words, we wanted them to be momentary (i.e. quick and timely) and also ecologically valid (i.e. within the environment where learning is taking place). This rationale for this embedded aspect of the reflections was driven both by our research focus but also by past experience.

The design of our reflection method is drawn from an approach used in behavioral health, medicine and psychology known as Ecological Momentary Assessment or EMA (LaCaille et al. 2013). In EMA, research participants provide feedback on symptoms, feelings, or other measures in real time and these assessments are often repeated over time. This real time reporting is enabled by a variety of technologies, such as mobile phones. As proponents of EMA report, its strength is in the authentic context where the research takes place and the ability to capture data as it happens (Shiffman, Stone, and Hufford 2008). Additionally, EMA has been proven an effective method to capture change within individuals and avoids the “pitfalls and limitations of reliance on autobiographical memory” (Shiffman, et al. 2008, 7).

Based on our previous experience in the STEM camp, we had limited success with interview methods with students. Although we had seen the students’ progress in a variety of ways, their own assessment of their experience did not include an expression of awareness of these changes. We also felt that the interview environment seemed superficial and separate from the classroom activities, likely influencing the authenticity of the students’ responses.

As a result, we designed our methodology to be informed by the concept of reflection and also containing the ecological and momentary characteristics of EMA. Because of our use of video to capture student reflections in the moment we named this method Ecological Momentary Reflection (EMR).

Implementing EMR in the Classroom

In the summer of 2015, we worked closely with the 9^th grade cohort of the STEM camp program and their instructors to implement the EMR method. We explained to students that they were creating the videos for themselves but also for each other as a way to reflect on their learning and to capture their experiences at summer camp. Therefore, students were aware there was a larger audience for the reflections. Students were also told that highlights of the reflections would be compiled and they would watch this highlight compilation together on the last day of the camp. They were given digital copies of their personal reflections, their group videos and a copy of the final edited compilation to take home with them as a keepsake from their camp experience.

Students created three video reflections during the three-week summer camp: an introduction, mid-point, and final reflection. For the first reflection, students created an introductory video. They were asked to introduce themselves, talked about their hobbies or interests, and reflect on how they felt about STEM and about themselves. In the second video reflection, students were given two photographs of themselves from previous days at camp that captured them engaging in one of the main STEM camp practices, in this case coding a video game on a computer. Students were asked to reflect upon what they were doing in the photo, how they felt looking at themself and what the photographs reflected about them as individuals. In their final reflection, which took place at the beginning of the third week of the camp, the procedure was slightly different. Students watched the previous two reflection videos and were then asked to respond via video to the experience of watching themselves and how they changed over the course of the summer camp. The specific wording of the prompts is shown in Table 1 below.

Topic and timing	Prompts
Video reflection 1: Introduction (Day 1)	Introduction reflection 1. Who are you: What do you like to do? What makes you special? 2. You and technology: Do you think of yourself as a technical or computer person? Why or why not? Do you think other people in your life (friends or family) see you as a technical or computer person? Why or why not? 3. Complete this sentence: By the end of STEM camp this summer I expect….
Video reflection 2: Photo reflection (Day 8)	Photo reflection During one of the early days when students start coding they will photographed while they are working. The photograph will serve as part of the prompt: 1. How would you describe what you are doing in the photographs? How does this fit into the rest of your life? 2. Can you talk a little about what you feel and think when you look at these photographs? 3. What do these photos reflect about who you are? 4. What can somebody looking at these photos learn about you? 5. CHALLENGE—Come up with your own prompt (question for self) related to the photos and try to answer it.
3. Video reflection 3: Wrap up (Day 14_)	Final reflection: After watching the video of yourself from the start of the program, answer these questions: 1. What do you think after watching that video? 2. Do you see yourself any differently from when you started STEM camp? 3. Have you learned anything new about yourself? 4. What was the best and worst parts of STEM camp? 5. What surprised you about this experience? 6. Please complete the following sentence. “After participating in STEM camp this year, I feel that I… “

To create an appropriate space for the video reflections, we used a small, quiet, private room just outside of the main classroom where students were spending their days. The room was equipped with a GoPro Hero 4 camera and students could move or adjust the camera based on their comfort level (Figure 1 shows the room set up).

Figure 1: An image of a small room with two empty chairs and a table. On the table is a video camera on a tripod, and a list of questions that contain the prompts for the video.

Figure 1: Video reflection room set up with camera and prompts.

Our motivation for creating a private space adjacent to the classroom was to give students a place to be able to quietly reflect while still being close to the camp setting. Giving the students a private space, but one that is still connected in time and space to the learning environment maintained the ecological soundness of this method. Figure 2 shows a still photo from a student’s video reflection showing the setting where students made their recordings.

Figure 2: A still from a student’s video reflection.

In addition, we had anticipated, and hoped, that this mode of reflecting: speaking to a video camera, might emulate current, culturally appropriate and familiar practices. We took our inspiration from the many examples of young people posting reflections or product reviews on YouTube from their bedrooms. We often referred to the small room where the videos were being made as our “reality show confession booth.” This idea seemed to resonate with the students and they seemed very comfortable expressing themselves in front of the camera.

Assessing EMR

In order to retrospectively assess how EMR worked within this setting, our team applied thematic analysis (Guest et al. 2011) of the following qualitative data: (1) student video reflections (2) field notes, memos and reflections from the research team, and (3) data from personal interviews with the two instructors of the camp. The qualitative data was reviewed, coded and discussed by the research team to uncover common themes throughout the data. These data were discussed in relation to the researchers’ experiences of using standard textual reflections.

Theme 1: Initial Reticence, Overall Enthusiasm

During the creation of the first video reflection in week one, some students mentioned that they felt a little awkward creating the video diaries. In contrast, in the last video reflection, students commented that they looked awkward in the first video or remembered feeling awkward at the time. Although there was this initial reticence regarding the first video recording, students also described how much more comfortable they were recording their last reflection. Most students were overwhelmingly enthusiastic about the experience of having done the videos in retrospect. They mentioned how they enjoyed using the cameras, interviewing one another, and taking the cameras on their field trips. This enthusiasm was palpable when the students were showed the highlight compilation of their summer experience.

“Anyway, I really liked that video. I’m feeling good because it’s kinda like the whole three weeks packed into one little video and it kinda shows my progress, like what I thought before and what I think now. And it’s kinda different to think that like in the beginning, I didn’t think I could do it, and I know how to get it done.” – P6

As one instructor noted, “we even asked them if they liked [it]… and they all responded with an overwhelming “YESSSS!!” (Solis-Bruno 2015).

The instructors in the program while supportive of EMR, had a variety of questions concerning the feasibility of this technology. However, they helped to creatively embed the videos reflections into the environment and the curriculum. They were also helpful in communicating the purpose of the videos in a student-centered way by referring to it as “the documentary.” Similar to the students’ experiences, instructors grew to value this novel method. By the end of the program, they reflected that the felt EMR was highly valuable for the students and both instructors said they would incorporate video reflection in this way in future classes (Solis-Bruno 2015; Jordan 2015).

Theme 2: Seeing Themselves

In the previous year of the STEM summer camp, we had asked students to think about how they had changed over the three-week experience in a series of semi-structured interviews. Overall, students in previous years did not express seeing much change in themselves over the short experience in the camp. However, students who used EMR were able to visibly see themselves in retrospect and comment on the changes in their learning. They compared their feelings about technology, coding, and engineering over the three-week period and were able to see for themselves how their thoughts had changed. These “a-ha” moments were the most visible during the second and third video reflections when students were looking at pictures of themselves or looking back at their previous videos.

During the second video reflection, we gave students photographs of themselves at work during the camp. Figure 3 shows an example of a photograph that was used as a prompt. In this video reflection in particular, students expressed surprise and amazement at seeing themselves from this outside perspective.

Figure 3: An image of a student sitting in front of a computer, designing a video game.

Figure 3: A photograph of a student that was used as a prompt in the second video reflection.

Many said they had never seen themselves in this way before. They described seeing themselves as focused or that they looked like someone who was programming. Many mentioned that their families would be surprised to see the person they saw in this picture: someone who was focused and working hard.

“I think that I look determined. I feel- I feel pretty good with um the fact that I can do this … like knowing that I can do this kind of stuff, that’s cool.” – P2

This reflection in particular points out the strength of the Ecological Momentary Reflection (EMR) method, and when combined with photographs, gives students an external or third person view of themselves.

In the third and last video reflection, many students had revelations and moments of surprise as they looked back on their previous videos. Several of them provided very clear and impassioned reflections on how this experience had fundamentally changed the way they saw themselves in relations to STEM topics. One student said that before the camp, she had seen herself as an “artsy” person and now she saw she was equally strong in things like engineering. Given the goals of this STEM summer camp learning experience, this student’s shift is encouraging.

“After watching the video that I made felt really confident in myself and I felt like …[I’m] doing what I’m supposed to in MSL.” – P3

“I see myself as more of a techy person I guess I… I realized that I really like technology and I really enjoy programming these games that we’ve been doing.” – P1

Several students had noted during their reflections that they had been struggling with some aspects of the coding tasks they were doing in the camp. These struggles were temporary frustrations and only moments in time. All of the students successfully completed a working, playable video game during their time in the camp. Watching themselves talk about these struggles in the video reflections allowed them to see how they had been able to overcome them. Therefore, they were able to talk about their resilience in terms of overcoming these challenges. Being able to see how they overcame challenges and that they could overcome these challenges, enables students to see that with hard work and by asking for help they can succeed, which enables the development of a growth mindset (Dweck 2006) and grit (Duckworth et al. 2007).

Theme 3: Broadening the Notion of Audience

In contrast to the other types of reflection done in classroom settings, that tend to be solely focused on writing, we saw how the addition of the videos helps to broaden the notion of audience. The expectation and the format of the videos implied an external and broader audience than just the student and instructor. This expectation had been communicated as part of the video project and it was clear that students were thinking broadly about audience. Students had mentioned their family members in the reflections and also used the pronoun, “you”, in their reflections to invoke the audience of ‘us’ (the faculty advisors, instructors, and their peers in the course). One student’s final reflection seemed like a dedication to his peers, as he proclaimed “how cool you guys are.” According to the instructors, some students wanted an even broader audience, and were disappointed that the compilation video was not played at the end of camp celebration for their family and friends (Jordan 2015). Evidently, they were proud of not just the accomplishments of the products of the summer camp, but also the process in which they discussed their learning through the video diaries.

Theme 4: Logistics, Implementation and Technical Challenges

When introducing any new pedagogical method that incorporates technology in the classroom, there is much to learn for future iterations. We learned a great deal about implementing EMR and areas for improvement in the future, both for the STEM summer camp learning context and beyond. One of our concerns at the beginning of the study was that students entering and exiting the main classroom to record the video diaries would be disruptive to the learning environment. However, the instructors stated that they did not feel that the activity was disruptive. They stated that from their perspective, the process of making and viewing the videos was highly valuable for the students (Jordan 2015; Solis-Bruno 2015). The lack of disruption could be attributed to the nature of the informal learning environment, which can be less structured than a formal school-based learning situations. However, we assert that the EMR method would complement a learning environment that is project or inquiry based.

One technical challenge we encountered in the study was audio quality. We were using GoPro Hero 4 cameras and while the video is of very high quality, the audio was not. With that video camera in particular, an external microphone would greatly increase audio quality. In addition, the battery life and size of the video recordings were limiting factors. An additional technical challenge is the storing of video files. It is important to set up an appropriate technical infrastructure for the video files to be securely stored but still accessible to the students and instructors.

The Promise of EMR

As we reflect on our experience with EMR, we turn to its strengths and promise for use as an authentic reflection tool to augment and make visible learning that occurs in informal and formal settings. We assumed that EMR would be congruent with teens’ “selfie” culture. While students at first were reticent to film themselves on camera, they did grow more comfortable over time especially in the impromptu videos, like the ones on the field trips. Given its basis in Ecological Momentary Assessment, EMR creates a fitting and even attractive tool for student engagement. It helps to capture learning both in the environment it is taking place and also the moment it is happening. In this way, EMR captures a fleeting moment of the student’s experience and enables reflection on that otherwise inaccessible moment, allowing students to witness their thinking across time.

EMR appears to be an effective tool for student reflection. The strong theme of Seeing Themselves supports the use of EMR as an effective reflecting process in a learning environment as it allows students to see themselves from as an outside observer. EMR also overcomes some of the limitations of written reflection that can influence students to perform in an academic manner and conceptualize the teacher as the sole audience for the reflection.

One of EMR’s strengths is broadening the audience of reflection and going beyond the idea of the reflection being produced by one student for one teacher. Requiring students to produce reflections for themselves but also their peers strengthens the learning environment. The majority of students were interested in keeping their videos and also their photographs from the prompts as keepsakes. Consequently, using student-centered technology, methods, and artifacts as tools for thinking not only provides students with more meaningful learning experiences, but also promotes recurring and persistent practices of reflection.

The benefits of EMR as a technology far outweigh the drawbacks. It leverages a technology that is familiar, yet novel or unexpected in a classroom setting. The video camera engages students in a way that is low risk yet high reward. While there may be challenges to videos, there are too with written reflection, such as the varying literacy skills available to a student. We conclude that the technology is congruent with tools and technologies that many adolescents are already familiar and comfortable with.

In conclusion, the common themes that emerged from our data highlight how using EMR in the classroom can support authentic reflection that enhances students’ learning experience and educators’ assessment of student learning and the learning environment. EMR departs from static written reflections and instead provides students a way to see and reflect on their own thinking and learning as it is happening.

Thus, EMR is a promising method for reflection in any complex learning environment by capturing real-time learning, maintaining ecological validity, and allowing for authentic and powerful reflection. We highly encourage others to explore this technology in their classrooms.

Bibliography

Barron, Brigid, Véronique Mertl, and Caitlin K. Martin. “Appropriating the process: Creative production within informal interactions and across settings.” B. Barron, K. Gomez, N. Pinkard, & CK Martin, The Digital Youth Network: Cultivating new media citizenship in urban communities (2014): 167-190.

Barton, Angela Calabrese. “Teaching science with homeless children: Pedagogy, representation, and identity.” Journal of research in science teaching 35, no. 4 (1998): 379-394.

Barton, Angela Calabrese, and Kimberley Yang. “The culture of power and science education: Learning from Miguel.” Journal of Research in Science Teaching 37, no. 8 (2000): 871-889.

Bransford, John D., Ann L. Brown, and Rodney R. Cocking. How people learn: Brain, mind, experience, and school. National Academy Press, 1999.

Brown, A. L, Bransford, J. D., Ferrara, R. A., & Campione, J. C. “Learning, remembering, and understanding,” In J. H. Flavell & H. M. Markman (Eds.), Handbook of child psychology, Vol. 3: Cognitive development (1983): 77-166.

Buxton, Cory A. “Creating contextually authentic science in a “low‐performing” urban elementary school.” Journal of Research in Science Teaching 43, no. 7 (2006): 695-721.

Chafe, Wallace L. “Sources of Difficulty in the Processing of Written Language,” Center for the Learning and Teaching of Literature, University at Albany, State University of New York (1990).

Chafe, Wallace, and Deborah Tannen. “The relation between written and spoken language,” Annual Review of Anthropology (1987): 383-407.

Dewey, John. How we think: A restatement of the relation of reflective thinking to the educative process. Lexington, MA: Heath 1933.

Dierking, Lynn D., John H. Falk, Léonie Rennie, David Anderson, and Kirsten Ellenbogen. “Policy statement of the “informal science education” ad hoc committee,” Journal of research in science teaching 40, no. 2 (2003): 108-111.

Duckworth, Angela L., Christopher Peterson, Michael D. Matthews, and Dennis R. Kelly. “Grit: perseverance and passion for long-term goals,” Journal of personality and social psychology 92, no. 6 (2007): 1087.

Dweck, Carol. Mindset: The New Psychology of Success. New York: Ballantine Books 2006.

Guest, Greg, Kathleen M. MacQueen, and Emily E. Namey. Applied thematic analysis. Thousand Oaks, California: Sage, 2011.

Jensen, Kyle. “The Panoptic Portfolio: Reassessing Power in Process-Oriented Writing Instruction.” JAC (2010): 95-141.

Jordan, Stephanie. E-mail message to authors, October 8, 2015.

LaCaille, Lara, Anna Maria Patino-Fernandez, Jane Monaco, Ding Ding, C Renn Upchurch Sweeney, Colin D Butler, Colin L Soskolne, et al. “Ecological Momentary Assessment.” In Encyclopedia of Behavioral Medicine, New York, NY: Springer New York. (2013): 647–48.

Medin, Douglas L., and Megan Bang. Who’s asking?: Native science, western science, and science education. MIT Press, 2014.

Nasir, Na’ilah Suad, and Jamal Cooks. “Becoming a hurdler: How learning settings afford identities.” Anthropology & Education Quarterly 40, no. 1 (2009): 41-61.

National Research Council. Learning Science in Informal Environments: People, Places, and Pursuits. Committee on Learning Science in Informal Environments. Philip Bell, Bruce Lewenstein, Andrew W. Shouse, and Michael A. Feder, Editors. Board on Science Education, Center for Education. Division of Behavioral and Social Sciences and Education. Washington, DC: The National Academies Press. 2009.

Palinscar, Aannemarie Sullivan, and Ann L. Brown. “Reciprocal teaching of comprehension-fostering and comprehension-monitoring activities.” Cognition and instruction 1, no. 2 (1984): 117-175.

Scardamalia, Marlene, Carl Bereiter, and Rosanne Steinbach. “Teachability of reflective processes in written composition.” Cognitive science 8, no. 2 (1984): 173-190.

Schön, Donald A. The reflective practitioner: How professionals think in action. Vol. 5126. New York, Basic books, 1983.

Schunk, Dale H. Goal and self-evaluative influences during children’s cognitive skill learning. American educational research journal 33, no. 2 (1996): 359-382.

Shiffman, Saul, Arthur A. Stone, and Michael R. Hufford. “Ecological momentary assessment.” Annual Review of Clinical Psychology. 4 (2008): 1-32.

Solis-Bruno, Luis. E-mail message to authors, October 9, 2015.

Veenman, Marcel VJ, Bernadette HAM Van Hout-Wolters, and Peter Afflerbach. “Metacognition and learning: Conceptual and methodological considerations.” Metacognition and learning 1, no. 1 (2006): 3-14.

Yancey, Kathleen Blake, Reflection in the Writing Classroom. Utah State University Press. Book 120. 1998.

Acknowledgments

We would like to express our deep gratitude to Amanda Figueroa and DJ Crisostomo in the Student Transition Programs at the University of Washington Tacoma for their leadership and making the MSL program such a transformative learning experience for the students of our community. We also wish to thank Luis Solis-Bruno and Stephanie Jordan the instructors of the 9th grade cohort of MSL in 2015 who were so welcoming to us and embraced the idea of using videos. In addition, this work would not be possible without the amazing teens in the MSL program who shared their experience with us through their video reflections. Finally, we would like to thank special issue editors Tyler Fox & Carlos Hernandez for bringing this special issue to fruition.

About the Authors

Emma J. Rose, Ph.D. is an assistant professor in the School of Interdisciplinary Arts & Sciences at University of Washington Tacoma. Her research is motivated by a commitment to social justice and a belief that the way technologies are designed ultimately shapes our world. Her research interests include the practice of user experience, how people use expertise to overcome resource constraints, and the development of technical identity. She tweets @emmarosephd.

Jarek Sierschynski, Ph.D. is a learning scientist and assistant professor in Education at University of Washington Tacoma. His work examines definitions of STEM, scientific practices and technology integration by focusing on complexities inherent in cultural tools used by historically marginalized communities. Recently, he has been investigating how students think about their identities in relation to science and technology. His current project involves the design of an informal learning environment in which technology serves youths as an identity, cultural and scientific resource.

Elin A. Björling, Ph.D. holds both a professional research scientist position for the Office of Research and a clinical faculty position in the school of Nursing and Healthcare Leadership at University of Washington Tacoma. Over the past two decades, Elin has studied adolescent health utilizing mixed-methods in community based project designs. Her recent research has focused primarily on using an Ecological Momentary Assessment approach to study stress in adolescents. She tweets @elinbjorling.

Authoring an Open-Source Game for a Faculty Open Educational Resources Workshop: A Case Study

Katherine Foshko Tsan, Baruch College-CUNY Center for Teaching and Learning

Abstract

Introduction

The OER Faculty Development Workshop: Going Digital, Going Multimodal

Creating a New Mode for the OER Workshop at Baruch College

Twine and Digital Pedagogy

The Baruch Twine Game: Scenario-Based Learning in a “CYOA” Format

Reflections and Potential Applications

Conclusion

References

Acknowledgments

About the Author

Assessing the Effectiveness of Using Live Interactions and Feedback to Increase Engagement in Online Learning

Beth Porter, Riff Analytics

John Doucette, Riff Analytics

Andrew Reilly, New College of Florida

Dan Calacci, MIT Media Lab

Burcin Bozkaya, New College of Florida

Alex Pentland, MIT Media Lab

Abstract

Introduction

Current study

Materials and Methods

The Riff Platform and its user-facing applications

Video chat

Post-meeting metrics

Text chat

Learner data privacy

Supporting accessibility

Experiment design

Results

Results for questions 1–2 for students who completed the course

Results for questions 3–6 for all students

Summary, Discussion, and Conclusions

Summary

Discussion and Conclusions

Bibliography

Appendix A—Exit Survey Questions and Key Results

A1. Exit Survey Questions

A2. Key Results

General

Collaborative exercises

Appendix B—Definitions

Acknowledgments

About the Authors

The Help Desk as a Community-Building Tool for Online Professional Development

Salome Apkhazishvili, Media Education Lab

Serene Arena, Independent Scholar

Renee Hobbs, University of Rhode Island

Abstract

Literature Review

Teachers as co-learners

Help desks as spaces for online community building

Research Methods

Participants and program context

Data collection and analysis

Incident Log

Interviews

Findings

Co-learning as a journey borne of isolation

Co-learning as a journey to connect

Co-learning as a journey toward agency

Discussion

Notes

Bibliography

About the Authors

Trauma-Informed Pedagogy in the Digital Media Pandemic Classroom

Michael Mandiberg, College of Staten Island and the Graduate Center, CUNY

Abstract

Care and Harm

Digital Foundations Online

Avoiding Adobe

Asynchronous instructional design and learning experiences

Challenges

Outcomes

Bibliography

About the Author

Reflecting on Reflections: Using Video in Learning Reflection to Enhance Authenticity

Emma J. Rose, University of Washington Tacoma Jarek Sierschynski, University of Washington Tacoma Elin A. Björling, University of Washington Tacoma

Emma J. Rose, University of Washington Tacoma
Jarek Sierschynski, University of Washington Tacoma
Elin A. Björling, University of Washington Tacoma