Tagged augmented reality

Three students stand at the front of a classroom. They are pointing at a computer and consult a map. A student stands behind them wearing virtual reality equipment.
22
Assignments / Short Form Pieces

Immersive Education: Virtual Reality and Project-Based Learning in the History Classroom

Andrew Wilson

This article details a student-centered assignment that integrates primary source analysis and the immersive medium of virtual reality. The goal of the activity was to increase student spatial familiarity and geographic knowledge of historic spaces, as well as expand their interest in primary sources.

Read more… Immersive Education: Virtual Reality and Project-Based Learning in the History Classroom

Read More
Left, User testing the system with HoloLens headset in the historic home. Right, What the user sees through the HoloLens.
1
Issue Seventeen

Blending Disciplines for a Blended Reality: Virtual Guides for a Living History Museum

Juilee Decker, Rochester Institute of Technology

Amanda Doherty, Rochester Institute of Technology

Joe Geigel, Rochester Institute of Technology

Gary D. Jacobs, Rochester Institute of Technology

Abstract

This article describes the early stages of a virtual guide for onsite museum experiences, a project undertaken at Rochester Institute of Technology (RIT) involving students and faculty in computer science, museum studies, art and design, and theatre in conjunction with Genesee Country Village & Museum, the third-largest living history museum in the US and the largest in NY state. Our work focuses on the use of augmented reality, where technology and devices are used to superimpose digital assets over real elements in physical spaces, to demonstrate potential for enhancing storytelling within a historic village context. We outline our process—involving students and faculty from three colleges within our university, and staff from the museum partner—from exploration, research, and design to capture, delivery, and testing. With four faculty leading a cross-disciplinary collaboration among more than eighty students, three additional faculty from RIT (theatre and music), and six museum staff members thus far, our interest lies in facilitating opportunities for incidental learning (Crawford and Machemer 2008). We are keenly interested in pushing the boundaries of Pomerantz’s “spirit of experimentation” (2019) among the students and instructors where the former learn about the technology and subject matter, while the faculty forfeit prescriptive outcomes in an effort to foster experimentation within the context of courses and assignments where this project is facilitated. Ultimately, we see this application of XR as a mode for the conception, creation, and dissemination of storytelling within the classroom experience that simultaneously shares attributes of constructivist learning proffered in education and museums.

Overview

This project emerged from an existing partnership between Rochester Institute of Technology (RIT) and Genesee Country Village & Museum (GCV&M). This perhaps unlikely pairing between a research university with more than 19,000 students and the largest living history museum in New York provides opportunities for faculty, staff, and students from virtually every college within RIT to foster collaborations from a variety of disciplines. Many projects and research areas have multi-disciplinary or cross-disciplinary foci. This project, “Blending Disciplines for a Blended Reality: Virtual Guides for a Living History Museum,” is one such example where an interdisciplinary, research-inspired question forges connection among multiple constituencies within the university with the museum as the site for developing tangible skills and undertaking projects that have scholarly reach and long-term, mutual benefit. Because of this partnership, and the trust and history of association between the two organizations, we, as faculty researchers, have the freedom and flexibility to foster interdisciplinary collaboration in a meaningful way and to engage our students in developing skills in storytelling, digital composition, and multimodal literacy. The museum contributes to, and benefits from, the research and output of this collaboration, thereby serving as a site where our research can thrive.

Process

While this concept at present involves faculty and students from several disciplines with the production geared toward realizing work around one historical person that we have developed and provided with historically accurate contextual narrative, our project began with a much broader framing that our students helped to refine. The collaboration began in 2018 between computer science and museum studies faculty who wanted to set a research problem at the museum, employ technology as a possible solution, and engage our students in the research and scholarship around this project. Inspired by ongoing research with intelligent virtual agents (IVA) (Norouzi et al. 2018), we pondered as to what role an IVA might play in the context of a living museum. We posed the question: “Could a stylized avatar, serving as a historical guide, be used to augment visitors’ physical experiences at Genesee Country Village & Museum?” Over two semesters, the faculty and students from museum studies and computer science, with the help of faculty and students from theatre, developed 38 historical narratives which were recorded via audio only or audio and motion capture. This earliest phase of exploration was evaluated by team members, Decker and Geigel, in November 2018 and in April 2019, which in turn enabled them to pivot the project in five ways over the past several months. The project team expanded to include collaborators from among art and design faculty. In turn, we began to focus on researching and developing one character initially; to create historically accurate clothing, props, and environment for the character; and to refine our workflow—all with the end goal of stacking historical narratives into a six-minute story, delivered in monologue form as a vignette to engage with the visitor. This article outlines the project over this entire span of 18 months, with primary focus on the past several months, which is the period of robust project development and testing by students and faculty.

Timeline Between Spring 2018 and Spring 2021 showing collaboration between 4 University programs and museum
Figure 1. Roles and tasks of AR storytelling team. Informal feedback will continue, and formalized user testing will be developed, through spring of 2021.

Exploration

The earliest iteration of this project was exploratory in order to see the viability of our launching a long-term project. Led by two of our four-person faculty team (Decker and Geigel) from spring 2018 through spring 2019, we tested the technology and research/script-writing as well as recording. In terms of technology, we chose a Microsoft HoloLens as a delivery platform as it provides an intuitive, hands-free interface as well as built-in voice recognition. Furthermore, the HoloLens has been shown to be an effective platform in other museum contexts (Hammady et al. 2019). Development was done using Unity, a 3D software platform for rapid prototyping of VR and AR applications. We developed three short sprints using prescribed, pre-loaded character types in Unity, writing an application in Unity to allow for placement of the virtual storyteller at specific, appropriate spots on our campus (in lieu of the museum). As the app was running on a HoloLens, users had the opportunity to interact with the application by asking pointed questions of the avatar, to which the storyteller would respond, making users feel like they were having a conversation with a real person.

Research

Initially, neither the text nor the visuals used in this exploratory phase were keyed to our historical site. However, simultaneous to the technology testing, we asked museum studies students to research the buildings situated at the museum and to develop “character types” who might be viable suggestions for developing an AR character for this project. Over two semesters of increasingly focused, exploratory research, the students created 38 one-to-three-minute monologues situated at 12 of the 68 historic structures at the museum. Each of these monologues was historically based and researched using resources from the museum as well as contextual sources (Bolger 1985). While only a small portion of this overall work was, in turn, used as part of our refined prototype (explored fully in this article), the initial research phase informed our workflow, as well as the decision to develop one character more fully to focus our team’s concept development and execution.

Over the summer of 2019, in consultation with the Genesee Country Village & Museum staff,[1] the team selected Dr. Frederick F. Backus (1794–1858) as the inspiration for our first fully developed character. First and foremost, Backus had myriad interests and connections to Rochester history, making his story rich with intersections that could, in turn, be amplified through research-informed narrative writing. Second, we chose this individual in order to tether our virtual character with his actual home, one of the first grand mansions in Rochester which Backus purchased in 1838. Third, and perhaps most interestingly in terms of creative output, no images exist of his appearance, thereby making him an opportunity to blend historical reality and interpretation.

After deciding upon a character, primary and secondary research guided the script-writing, with the immediate need to develop one narrative for this phase of testing. Immediately, the decision was made to situate the character a bit later in his life, so as to draw upon a wealth of experiences documented by Backus in letters. As the museum interprets the home to the year 1850, writing a script that would be situated at around the same time of the museum’s interpretation bolstered our ability to render a seamless integration between the AR experience and the museum environment.

Historically accurate assets were gathered as part of the research. These include professional and domestic contexts, including newspaper accounts from the years that Backus served in the New York State Senate and background information on Backus’s neighborhood gleaned from property records and maps of Rochester’s wealthy Third Ward. This portfolio of research was passed on to the museum studies students in the spring of 2020 to guide their development of academically and historically rigorous narratives for five characters (Backus and four additional characters).

The current student cohort (spring 2020) developed 15 monologues focused on individuals who lived in the region over the years that the museum interprets (Pioneer Settlement era of 1780s through the 1920s), with particular attention to the 1820s–1860s. These individuals included the aforementioned Frederick Fanning Backus (1794–1858); Candace Beach (1790–1850), a teacher at a one-room schoolhouse who lived through the historic “year without summer” that occurred in 1816, over a three-year period of climate change and uncertainty as a result of the eruption of Indonesia’s Mt. Tambora in the spring of 1815; John Carlin (1813–1891), a poet and painter who graduated in 1825 from Pennsylvania Institute for the Deaf and Dumb before traveling to England and France for a Grand Tour and returning to New York and picking up clients across the state; Austin Steward (1793–1869), who was born to enslaved parents in Virginia before moving to New York and becoming engaged in antislavery and temperance as well as the black convention movement, all the while being engaged as a merchant, publisher, and orator years before Frederick Douglass settled in this region; and Lavinia Fanning Watson (1818–1900), a Philadelphia socialite, with ties to the region, who was the first woman to commission a naval ship, the USS Germantown (1846). The monologues were sited at three of the buildings on the museum campus.[2]

Design: character, model, and rigging

Throughout the research phase, the faculty team had discussed how to proceed with the digital design phase. The decision process for creating the first 3D character (described below) would also inform projects and workflow for continued production, including the development of additional characters in spring 2020 and beyond.

The design process began with the choice to build a stylized avatar, rather than a realistic 3D animation, so as to avoid the “uncanny valley”: a feeling of unease and disconnect experienced when humans encounter robotic or audio/visual simulations that are too realistic. This key decision was informed by the work of Masahiro Mori who presented the theory of the uncanny valley five decades ago (Mori 2012). Mori posited that an individual’s feeling about a human-like robot would go from empathy to revulsion the closer the representation grew to reality, because the representation would naturally not achieve true realism. Mori’s premise has been applied to the development of digital characters as well, as the uncanny valley is often referenced vis-à-vis the film Polar Express (Noe 2012) and CGI characters that fail to achieve true realism and therefore alienate the viewer (Weschler 2011). For Weschler in particular, the “vacant” quality of the eyes and unrealistic movement are cited as features that foster the eeriness associated with the uncanny valley.

While some scholars are now exploring the ability of digital artists to create avatars realistic enough to foster trust and empathy, such production is at a level of digital artistry that requires mastery and extensive experience. Students would not have the expertise to overcome this valley and therefore we chose to pursue a stylized character. The choice meant the final agent would be distinctly unrealistic in an authentic historical environment. We accepted this anachronism as a way of attuning to the museum’s approach to onsite interpretation. GCV&M does not presume visitors are transported to 1850; it sets out to interpret and demonstrate the era authentically while acknowledging that the museum staff, chiefly the costumed interpreters and the guests, are inhabitants of the present. Additionally, we intended to utilize modern technology (HoloLens) to immerse the viewer in the experience of interacting with the character, further removing them from the idea of being transported to the past. Our digital agent, viewed through the HoloLens, would clearly be an AR animation and not an actual human interpreter, so the decision to opt for a stylized avatar meant students could design all aspects of the character with the burden of bridging the uncanny valley relieved.

The avatar needed to be approachable and warm in order to appeal to older adults and children alike. To avoid a sense of unease, certain attributes are exaggerated in digital human representations—most often the size of the head, hands, and feet. For continuity of design, the style developed would be carried through into additional avatars, to be executed by 3D digital design students.

As a character, Backus presented the unusual but fortunate position of having an actual historical figure for whom there is no visual record, only written references. With no extant images, the team was left to interpret his appearance through the use of his father’s portrait from Hamilton College and his own writings of his life experiences. The character design incorporates the physical input of Azel Backus as the subject’s father, with historic, social, and economic aspects of the 1851 time period. As we move forward with students to develop further avatars and agents for the museum, classes will follow the same pattern of character analysis in the design, regardless of any visual references we may have of subjects. Thorough research of the fashion of the period was balanced with the knowledge that Rochester, NY in 1851 was both rural and remote and therefore not on trend with the latest styles. It was also clear from the writings of our historic subject that he had traveled the area and experienced the hardships of practicing medicine in such a time and place.

Left, Artist engraving of Azel Backus from 1813. Right, Costume Design Sketch of Azel Backus
Figure 2. Portrait of Azel Backus and preliminary costume design.

It was also important to have knowledge of the character’s setting in the actual house and take color scheme into account. The AR device through which the character will be viewed will superimpose the image on the surroundings, so it was important to make sure the agent would stand out from the environment. The buildings at the Genesee Country Village & Museum are from several different decades and span a wide range of architectural styles. Everything from the number of windows in a building to the color trends and financial status of its residents will impact how well the avatar is seen in the setting. For Backus, this meant opting for cool, darker colors so he would be better distinguished amongst the tans, browns, and reds of the well-lit entryway.

We began with a rough sketch to outline the physical properties of the character before moving into 3D development. The 3D digital design program utilizes software from a variety of companies in order for students to experience the full range of programs in use throughout the professional industry. For this project we chose to use software from well-established and reliable companies with the idea that we will be able to upgrade and improve the designs as the software advances. In order to achieve the stylized character we had determined would best suit our needs, we utilized Character Creator by Reallusion, a 3D software that would allow us to morph realistic human proportions. This software utilizes an interface and key strokes that are common in several 3D programs, making it approachable and intuitive for students of 3D art. Facial and body features were exaggerated; the nasal, cheek, and chin areas were expanded to match historic drawings of Azel Backus, along with digitally sculpted hair and sideburns matching historic styles. The head and eyes were enlarged, as seen in many animated characters, to make them less realistic and more childlike. The avatar’s physique and appearance were also altered to better reflect that of an older gentleman of 1851.

Left, 3D model of exxagerated body propotions. Right, 3D model next to 2D patterns of costume garments.
Figure 3. Body and garment modeling.

We then used Marvelous Designer 8, a digital patterning and simulation software to build period-appropriate clothing for Backus. This software in particular is not only widely embraced by the 3D industry, but is advancing rapidly in its effectiveness and efficiency. As we develop further historic digital avatars for the museum, students will be utilizing this software to create historically accurate garments that are uncommon in the 3D world.

In selecting garments for Backus, as well as any future characters for the museum, it was important to keep in mind that clothing production was not yet industrialized, meaning it was not mass produced nor readily available (Holkeboer 1993; Gorsline 1994; Armstrong 1995; Tortora and Marcketti 2015). Most, if not all, of the garments worn by Backus would have been home or locally produced. Men’s shirts in particular were traditionally made by a wife or mother, but a man’s tailored waistcoat and frock coat would have been made by a skilled, male tailor. Additionally, the materials used would have been relatively expensive, so tailored menswear tended to be an investment that was worn for several years. Considering the remoteness of Rochester to any major metropolitan hub of 1851, it’s likely his garments could have been 5–10 years old at the time. To that end, we opted to dress Backus in a slightly dated frock coat with the soft, sloping shoulders and high back collar of the mid-1840s, and a waistcoat with a wide lapel and only slightly rounded hem of the 1840s. Here, we opted for a deep navy blue melton wool that would be a strong contrast to the wood staircase and tan wallpaper of the home’s entry. His trousers also bear the marks of the 1840s, with the relatively new center front fly closure, as opposed to the earlier fall front. Men’s trousers of this early Victorian era were tapered and narrow at the hem and tended towards large-scale patterns, especially plaids. We opted for a somewhat subdued gray wool plaid flannel as Backus was more an elder statesman than fashionable dandy. Students’ detailed character analysis informs these design decisions, and the design choices inform how the patterning software is used. These decisions, coupled with the research prepared by museum studies students in their development of monologues, inform the 3D students’ design choice, right down to the type of fabric used in a waistcoat and whether or not a collar is top stitched.

Capture

Transitioning from research and design to capture and render meant involving actors from performing arts faculty who, based on their vocal style, could offer a viable presentation of Dr. Frederick F. Backus. In order to preserve the legibility of the narrative in performance, the actor’s voice-over track was recorded in advance, which enabled the actors to adjust inflection and vocal emphasis of segments of the script in a sound-isolated recording booth. The performers then recreated the character’s movements in front of a motion capture system, using the audio playback as reference. Using Character Creator, the motions were then transferred to the avatar and any jitter was removed. Additionally, a digital face rig was created and lip-synched to the narrated audio track recorded earlier, and additional gestures were added as required.

Left, Actor and audio technician reviewing script outside audio recroding booth. Right, Actor performing for motion capture system
Figure 4. Audio and motion capture recording.

Delivery

The rigged and animated model was exported as an FBX file from Character Creator into the Unity game engine for use in the HoloLens, which would simultaneously display the character in the museum space and create the user experience of interaction with the virtual guide. (A user interacting with Backus in the museum setting and the view seen through the HoloLens are shown in Figures 8 and 9.) Using the speech-recognition capabilities of the device, the application can recognize key spoken phrases to which the character will respond with a predefined and prerecorded monologue.

Left, User testing the system with HoloLens headset in the historic home. Right, What the user sees through the HoloLens.
Figure 5. User interacting with the character in the museum setting.

In this way, the narrative is designed as contextually rich, narrative-driven storytelling delivered by a historical character, set in the home that the historical Backus did, at one time, inhabit. The AR character, house, and site are woven together in a storytelling construct that engenders historical information, situates a conversation between an agent and a visitor at a site that the character may have once visited, and presents an opportunity for incidental learning—the learning along the edges that Falk and Dierking proclaim as critical to the museum visitor experience (Falk and Dierking 2012). In short, the monologue as written, performed in audio and motion capture, and associated with the digital asset, which includes the character creation as well as historical treatment, are tethered and presented through the HoloLens. These facets come together to create an experience that provides an opportunity for visitors to engage with a person from the past that is only possible through this medium.

Informal feedback

Our ultimate goal is to deploy at GCV&M with visitors, wearing HoloLens, who have entered the threshold of the doorway of the Livingston-Backus house at the museum. While we have not yet deployed at the museum, as of November 2019, our team has reached a significant milestone of creating the Backus who can deliver a monologue and respond to voice commands from the user. As of this writing (March 2020), our research and design has continued with four new characters that will be captured in the coming months.
Our progress thus far has been informed by preliminary informal feedback in two phases in 2019, both working toward the goal of user testing this system and content onsite at Genesee Country Village & Museum.[3] These two iterations of informal feedback, while very different in design and nature, have offered us the opportunity to see an increase in ease of use and interest, as well as fulfillment. These facets will be measured again as we move into our third iteration of informal feedback, involving students from across the collaboration team, as well as museum staff. We plan to develop and conduct formalized user testing at the museum in the summer of 2020. Each of these feedback scenarios has enabled us to reflect on our work as faculty, and, in coordination with our students, to assess our pedagogical goals and structure our next advancement.

Left, Student learning navigation on HoloLens. Right, Student adjusting HoloLens head straps.
Figure 6. Students Lizzy (left) and Brie (right) learned how to use the HoloLens in order to facilitate informal feedback as part of our university’s annual AR/VR/XR symposium. November 22, 2019.

Authenticity and living history museums

The creation of a virtual museum guide may seem at odds with the history and context of our museum partner and our intended location for delivering the XR experience, Genesee Country Village & Museum, which belongs to the classification of living history museums. This genre grew out of world’s fairs and international displays in the 19th century that offered exhibitions arranged in village-like settings to provide viewers with an engaging sense of culture and history simultaneously (Alexander, Alexander, and Decker, 118). Founded in 1966 and open to the public a decade later, GCV&M has sought, from its earliest days, “an endeavor to visualize and interpret this bygone era…[and] has assembled authentic examples—functional buildings and artifacts of the period—from a score of area towns. It has not endeavored to recreate any specific village but to recapture and portray the character and atmosphere of the village era” (McKelvey in Bolger 1985, 2).

Walking through the gates of the toll house and entering into the historic village, visitors are treated to a vision of the past before their eyes. Such a treatment of living history museums affirms folklorist Jay Anderson’s (1985) definition of living history as “the simulation of life in another time.” Museum interpretation at living history museums is often mediated through the costumed interpreters who may take on a particular role, often with the premise that they are conveying what it was like to live in the past, and the modern visitor has encountered them in their daily life (Reid 2001; Roth 2005; Thierer 2010). Because they are the primary communicators with museum visitors, costumed interpreters are essential to the interpretation function of living history museums, which are entirely re-contextualized environments. Interpreters serve as the key factor of on-site engagement for visitors. They communicate with visitors through demonstration and conversation. As theorists Handler and Saxton (1988) argue, living history practitioners are keenly concerned with authenticity and that the role of the interpreter is to bridge past and present.

This connection between past and present while simultaneously seeking authenticity is key to our project which utilizes extended reality as a medium for the dissemination of a first-person narrative keyed to the identity of a known, historical person. These choices were made by the project team so as to distance the digital work and its outlay from the onsite, face-to-face, interpreter-to-visitor experience. In addition, we wanted to push the limits of this medium to see the extent to which our virtual tour guide can convey authenticity even while avoiding the aforementioned uncanny valley.

Whereas traditionally, onsite at the museum, visitors come into contact with costumed interpreters who staff approximately a dozen buildings and engage in third-person dialogue, meaning that they are dressed in historically accurate costume yet use contemporary language and are fully aware of the present, our virtual tour guide offers the opportunity to hear from a character speaking in first-person, performing a role for visitors, and speaking in paraphrases or direct quotes from diaries, notes, and primary sources. Both methods of interpretation—the third-person, interpreter-based and first-person, avatar-based—seek to serve as relevant, authentic, and historically accurate bridges between past and present for visitors.[4]

Pedagogy

Our project design has been informed by pedagogy, as this project was conceived from the outset as a collaboration among faculty and student researchers across several disciplines. Over the eighteen months of this project, students and faculty have been involved at every phase of our project (see figure 1, Roles and Tasks). Some aspects have been developed within the framework of a course assignment for museum studies students, including research, monologue development, participation in audio and motion capture, and collection of feedback. The early iteration of the virtual museum guide was developed by computer science students enrolled in Applications in Virtual Reality, a course focusing on the use of VR/AR technologies in creating unique mixed reality experiences. And, enhancements of the application have been taken on by several master’s students in computer science as part of their capstone projects. Other facets took place outside of the classroom assignment or context; students self-selected to become involved in that phase of the work. For instance, theatre students were involved as actors for motion and audio capture, a 3D design student facilitated the motion capture as part of advanced study toward her thesis project, and museum studies students facilitated informal feedback in November 2019.

The application of XR as a mode for the conception, creation, and dissemination of storytelling within the classroom experience shares attributes of constructivist learning of educational systems in general (Dewey 1998). Specifically, our project—involving students and faculty from three colleges at a research university, along with a museum partner—encourages discourse during knowledge construction. For instance, the collaboration necessary for success of this project provides a unique learning opportunity for computer science students. Though the focus of the work of the computer science students may be technical in nature, the design, implementation, and approach of the application development are shaped by the continual interaction with the creative team. Back and forth communication regarding the assets, both visual and aural, guides the development activities of the application, and, at the same time, directs the work of the design team creating the assets as they must assure proper formatting, timing, and synchronization of the models and animations to work on the HoloLens device.

Faculty have served as mentors to one another and students, but also have let go of prescriptive outcomes for classroom assignments or milestones of our project in an effort to foster experimentation within the context of our collaboration. We have embraced key facets of Pomerantz’s “spirit of experimentation” (2019) which contends that success can be measured by virtue of experimentation rather than meeting criteria on a traditional rubric. As Pomerantz notes, “Sometimes experimentation is the point.” As faculty, too, our learning experiences as collaborators and facilitators guiding our students’ work throughout this project have embraced this facet of experimentation.

Our blending of disciplines to create a blended-reality experience realizes constructivist pedagogy and further mirrors attributes of visitor experiences at museums, where knowledge is actively produced by the learner. For instance, throughout this project, students engage in incidental learning, which may be defined as “unplanned or tacit learning, stemming from the learner’s actions” which is “an often hidden aspect of higher education” (Crawford and Machemer 2008, 106, 109). These attributes are hallmarks of a “learner-centered environment” (104) and are key to understanding the pedagogical outcomes of our project.

Our conception of a virtual museum guide to be developed among a cohort of interdisciplinary researchers and their students intended, from the outset, for incidental learning to occur as a means of individual student work (as an assignment or other framework for involvement in this project). In fact, we found, in review of Crawford and Machemer’s characterization of 19 incidental learning skills associated with project-based learning, that students across the project were developing (and continue to develop) each of these skills at various points throughout the project.[5] In addition to particular facets skills gained by particular cohorts of students involved in our project, all students and faculty gained “teamwork skills,” “time management skills,” and “potential to apply what is learned here to other situations” (variables 2, 4, and 19 of Crawford and Machemer). Each of these attributes described above enabled the students to develop skills that were not part of the initial project requirements, indeed, but they also fostered a sense of real-world experience. That is, the workflow and processes defined above—with collaborators having domain knowledge and expertise entering into a project for a particular purpose and then exiting until called upon again—mirrors the work world of industry where various aspects of a large-scale project are completed independently in contribution of a larger whole. Importantly, the undergraduate students across all disciplines expressed an interest in continuing to be updated on the project’s progress, long after their semester or other engagement had come to an end, thereby affirming the pedagogical impact of this project.

While much of our decision-making was informed by pedagogical aims and aspirations for cross-disciplinary learning, we were collectively interested in how XR can inform storytelling practices. Our conception of storytelling based at a living history museum was informed by Bedford’s proclamation of storytelling as a key attribute of museum work (2001) and Lowe’s articulation of who defines stories as the “interpretive tales we craft” and narrative as “the way that we consciously and unconsciously shape those stories” (Lowe 2015, 45). Such a framing of the past impacts the process of meaning making. As David Allison notes, “The way that museums present the history and the prejudices and biases they bring to the design process [of living history interpretation] will affect the meaning that individuals construct for themselves” (2016, 29). Allison thus affirms Lowe’s assertion that particular institutions do a “much better job explaining the complexity of history making—the craft, the methods, and the narrative construction” and sees such places as sites of innovation where leveraging “the old, bad history” (Lowe 2015, 47, 52) can—through storytelling—foster multivocality and inclusive interpretations of the past. Such museum-focused outcomes cross over to our pedagogical aims of storytelling and our project’s framing by affirming the value, relevance, and importance of storytelling as a form of historical communication, bridging between past and present as well as opportunities for authenticity, empathy, and inclusion.

Notes

[1] Museum staff involved in this discussion included the museum director and curator of collections. In developing further characters, we also consulted the senior director of interpretation and interpretation office manager. Two costumed interpreters will be involved in motion capture in the spring of 2020.

[2] The buildings include: Livingston-Backus House, the Land Office, and the Schoolhouse. The Livingston-Backus House is a plausible location for the Backus monologues as well as those involving his niece, Lavinia Fanning Watson, and the painter John Carlin, who befriended the Backuses. The Land Office is a reasonable location for Steward, who worked for Henry Towar when the structure was onsite in Lyons, New York. The Schoolhouse (built in 1822) is a reasonable site for the monologues by Candace Beach who, although employed as teacher in the region before this structure was built, is positioned much later in life, as she reminisces on her years teaching. Our process of monologue creation has involved the expertise of the museum staff, taking cues from Maria Roussou et al.’s understanding and assessment of the importance of collaborative participatory creation (2015) while also being mindful of the developments, research, and outcomes of storytelling on mobile devices in the cultural heritage sector (Lombardo 2012).

[3] Our first testing took place in April 2019 within the context of a museum studies course where students were familiar with the project because each student had contributed to it by writing monologues for characters. The feedback at that time indicated that only 31.6% of the users felt that the experience fulfilled their desire of a museum experience (Decker 2019). Because the results were promising in terms of desirability of use and potential for engagement, we expanded the team to include collaborators from among art and design faculty; to focus on researching and developing one character initially; to create historically accurate clothing, props, and environment for the character; and to refine our workflow. Each of these facets was accomplished in the intervening months, leading to a second phase of informal feedback in November 2019, when we deployed the HoloLens with the Backus content as part of a demonstration at our university’s annual AR/VR/XR symposium (Carr and Johnson-Morris, 2019).

[4] Such a bridging of past and present is part of the living history tradition, as defined by Scott Magelssen who reads living history interpretation through the lens of performance practices and argues that living history has fallen into a comfort zone of merely “undoing history” and tracing time back to a past moment directly, and effortlessly, from today. Such homogeneity, Magelssen argues, is native to the work of museum professionals who may aspire to a linear format rather than addressing the ebbs and flows of history on the margins (2007, xiii, 59). Beyond the scope of this study is David Allison’s examination of museum staff who use costumed interpretation in museums that are not entirely living history museums, such as the Children’s Museum of Indianapolis which employed live, first-person accounts in the gallery for the program The Power of Children to tell the stories of Anne Frank, Ruby Bridges, and Ryan White. See David B. Allison, Living History: Effective Costumed Interpretation and Enactment at Museums and Historic Sites, Lanham: AASLH, 2016, 41–61.

[5] For instance, “communication skills” and “leadership skills” (variables 1 and 3 of Crawford and Machemer) were developed in particular by 3D student Hannah Chase who guided the theatre actors in the fall 2019 and communicated what the software needed from them in order to correctly/effectively acquire what we needed. Her directives not to cross hands over the body and how to gesture properly put her in the position of domain knowledge (motion capture) that would supersede domain knowledge from theatre by asking actors to act unlike actors in order to yield the results that we needed for the motion capture.

In addition, “understanding through social interaction” and “flexibility in day-to-day project management skills” (variables 13 and 5 of Crawford and Machemer) were gained by computer science student Kunal Shitut as he received the rigged and animated model from the design team which was used to create the application for the HoloLens. The navigation back-and-forth between computer science and 3D digital design guided the way the application was created and achieved an outcome that would not have been otherwise achieved if working on one’s own, without conversation and input from the art and design faculty.

Further, museum studies students gained the “ability to direct [their] own learning” and “ability to identify needs and tasks” (variables 15 and 16 of Crawford and Machemer), particularly through their research and writing of monologues. Finally, it is anticipated that 3D digital design costume students, in the spring 2020, will also gain incidental learning skills as they develop costumes for our virtual museum guide and additional characters that we will develop over the next several months.

Bibliography

Alexander, Edward P., Mary Alexander, Juilee Decker. 2017. Museums in Motion: An Introduction to the History and Functions of Museums. Lanham, MD: Rowman & Littlefield for the AASLH.

Allison, David B. 2016. Living History: Effective Costumed Interpretation and Enactment at Museums and Historic Sites. Lanham: AASLH.

Anderson, Jay. 1982. “Living History: Simulating Everyday Life in Living Museums.” American Quarterly 343: 290–306. https://doi.org/10.2307/2712780.

Armstrong, Helen Joseph. 1995. Patternmaking for Fashion Design. New York: HarperCollins Publishers.
Bedford, Leslie. 2001. “Storytelling: The Real Work of Museums.” Curator: The Museum Journal 441: 27–34. https://onlinelibrary.wiley.com/doi/abs/10.1111/j.2151-6952.2001.tb00027.x.

Bolger, Stuart. 1985. Genesee Country Museum: Scenes of Town & Country in the Nineteenth Century. Rochester, New York: Flower City Printing.

Carr, Lizzy, and Brienna Johnson-Morris. 2019. “User Testing at Frameless Labs Symposium, November 22.” Unpublished, anecdotal evidence.

Crawford, Pat, and Patricia Machemer. 2008. “Measuring Incidental Learning in a PBL Environment.” Journal of Faculty Development 22, no. 2 (May): 104–111.

Decker, Juilee. 2019. “MUSE 360 User Testing #4: Virtual Museum Assistant.” 19 respondents, April 9. Unpublished data set.

Dewey, John. 1998. The Essential Dewey, edited by Larry Hickman and Thomas M. Alexander. Bloomington, Indiana: Indiana University Press.

Falk, John H., and Lynn D. Dierking. 2012. The Museum Experience Revisited. London: Routledge.
Gorsline, Douglas. 1994. What People Wore: 1,800 Illustrations from Ancient Times to the Early Twentieth Century. New York: Dover Publications.

Hammady, Ramy, Minhua Ma, and Carl Strathearn. 2019. “User Experience Design for Mixed Reality: A Case Study of HoloLens in Museum.” International Journal of Technology Marketing 13, no. 3/4. http://www.inderscience.com/offer.php?id=104600.

Handler, Richard and William Saxton. 1988. “Dyssimulation: Reflexivity, Narrative, and the Quest for Authenticity in ‘Living History.’” Cultural Anthropology 33: 242–260.

Holkeboer, Katherine Strand. 1993. Patterns for Theatrical Costumes: Garments, Trims, and Accessories from Ancient Egypt to 1915. New York: Drama Book Publishers.

Lombardo, Vincenzo, and Rossana Damiano. 2012. “Storytelling on Mobile Devices for Cultural Heritage.” New Review of Hypermedia and Multimedia 18, no. 1/2: 11–35. doi:10.1080/13614568.2012.617846.

Lowe, Hilary Iris. 2015. “Dwelling in Possibility: Revisiting Narrative in the Historic House Museum,” The Public Historian 37, no. 2 (May): 42-60.

Magelssen, Scott. 2007. Living History Museums: Undoing History through Performance
Lanham, Scarecrow.

Mori, Masahiro. 2012. “The Uncanny Valley: The Original Essay by Masahiro Mori,” translated by Karl F. MacDorman and Norri Kageki. IEEE Spectrum. https://spectrum.ieee.org/automaton/robotics/humanoids/the-uncanny-valley. Previously published as Masahiro Mori, “The Uncanny Valley,” Energy 7, no. 4 (1970): 33–35.

Noe, A. 2012. “Storytelling and the Uncanny Valley.” NPR, January 20. https://www.npr.org/sections/13.7/2012/01/20/145504032/story-telling-and-the-uncanny-valley.

Norouzi, Nahal, Kangsoo Kim, Jason Hochreiter, Myungho Lee, Salam Daher, Gerd Bruder, and Greg Welch. 2018. “A Systematic Survey of 15 Years of User Studies Published in the Intelligent Virtual Agents Conference.” In Proceedings of the 18th International Conference on Intelligent Virtual Agents IVA ’18. Association for Computing Machinery, Sydney, NSW, Australia, 17–22. https://dl.acm.org/doi/10.1145/3267851.3267901.

Pomerantz, Jeff. 2019. “XR for Teaching and Learning: Year 2 of the EDUCAUSE/HP Campus of the Future Project,” Educause October 10. https://www.educause.edu/ecar/research-publications/xr-for-teaching-and-learning-year-2-of-the-educause-hp-campus-of-the-future-project/factors-that-influence-learning.

Reid, Debra Ann. 2001. Living History-Social History or Post-Modernism: Toward a Historiography of Open-Air Museum Interpretation in the United States. Charleston, IL: Eastern Illinois University.

Roth, Stacy Flora. 2005. Past into Present: Effective Techniques for First-Person Historical Interpretation. Chapel Hill: University of North Carolina Press.

Roussou, Maria, Laia Pujol, Akrivi Katifori, Angeliki Chrysanthi, Sara Perry, and Maria Vayanou, 2015. “The Museum as Digital Storyteller: Collaborative Participatory Creation of Interactive Digital Experiences.” MW2015: Museums and the Web 2015. Published January 31. https://mw2015.museumsandtheweb.com/paper/the-museum-as-digital-storyteller-collaborative-participatory-creation-of-interactive-digital-experiences/.

Thierer, Joyce M. 2010. Telling History: A Manual for Performers and Presenters of First-Person Narratives. Lanham, MD: AltaMira for AASLH.

Tortora, Phyllis G., and Sara B. Marcketti 2015. Survey of Historic Costume. New York: Fairchild Books.

Weschler, Lawrence. 2011. Uncanny Valley and Other Adventures in the Narrative. Berkeley: Counterpoint.

Acknowledgments

The authors are grateful to the staff of Genesee Country Village & Museum, in particular Becky Wehle and Peter Wisbey. At RIT, we thank students Hao Su, Kunal Shitut, Hannah Chase, Lizzy Carr, Brienna Johnson-Morris as well as a number of students from the following courses: MUSE 360/Visitor Engagement and Museum Technologies; MUSE 225/Museums and the Digital Age; FNRT 231/Fundamentals of Acting; DDDD 517/Costume Hair and Makeup; DDDD 521/Character Design and Rigging; and CSCI 715/Applications in Virtual Reality. In addition, we are grateful to faculty colleagues in theatre at RIT, Andy Head and David Munnell, and Katherine Collett, Archivist, Hamilton College Archives, for providing historical images.

About the Authors

Juilee Decker is an Associate Professor of Museum Studies at Rochester Institute of Technology (RIT). She has served as Editor of Collections: A Journal for Museums and Archives Professionals since 2008. She earned her PhD in 2003 from the joint program in Art History and Museum Studies at Case Western Reserve University and the Cleveland Museum of Art.

Amanda Doherty is an Adjunct Professor in the department of 3D Digital Design at Rochester Institute of Technology (RIT). She is a costume designer and historian who has been working principally in the entertainment industry and is now teaching character development and costume design for digital characters. She received her MFA in Design from Penn State University.

Joe Geigel is a Professor of Computer Science at Rochester Institute of Technology (RIT) and co-director of the CS Graphics and Applied Perception Lab there. He earned his DSc. in Computer Science from George Washington University in 2000. His research interests focus on mixed reality multimedia projects that combine computer science, real-time graphics, art, music, and theatre to create interactive, live experiences.

Gary D. Jacobs is an Assistant Professor of 3D Digital Design at Rochester Institute of Technology (RIT). He has designed public spaces, stage productions, and themed environments for over 15 years. He is a certified LEGO® Serious Play facilitator and leads Design Thinking workshops for creative teams. Gary received his MFA in Entertainment Design from Pennsylvania State University.

Read More
A hand holds a digital tablet over a page of text with a decorative border, while the tablet's screen displays a 3D model of a cathedral.
0
Issue Seventeen

Truly Immersive Worlds? The Pedagogical Implications of Extended Reality

Tamara F. O’Callaghan, Northern Kentucky University

Andrea R. Harbin, State University of New York, Cortland

Abstract

This article provides an overview of the extended reality applications virtual reality (VR) and augmented reality (AR) and examines the affordances and constraints of each with regards to their application in the humanities. The interactive nature of these extended realities engages their audiences in new and compelling ways. VR and AR applications have moved beyond gaming and are proving particularly effective and engaging for historic recreations. However, these technologies also present new challenges, precisely because they create immersive worlds so captivating that these environments may be perceived as “real” rather than as simulacra, especially by students and the general public. Using both VR and AR projects based in medieval Europe (Bologna 3D Open Repository and 3D Paris Saga) as case histories, we discuss some of the issues that these technologies pose to their creators and to their consumers—from how they might be used to make a heritage site more meaningful, to how they pose dangers of an excess of verisimilitude. As these technologies become more ubiquitous in academic settings, these early ventures into extended realities highlight some perhaps hitherto unconsidered pitfalls as well as demonstrate the promise that these new technologies offer in terms of pedagogy and community outreach.

Introduction

In the summer of 2016, the world was introduced to the emerging technology of augmented reality (AR) in the form of Pokémon Go, a location-based, AR-enhanced game that became one of the most popular mobile apps of the year. Many people were already familiar with virtual reality (VR), “a medium composed of interactive computer simulations that sense the participant’s position and actions and replace or augment the feedback to one or more senses, giving the feeling of being mentally immersed in the simulation (a virtual world)” (Sherman and Craig 2003, 13). As a popular gaming environment, VR has four key elements: it is a virtual space for the participant; it is immersive on both a physical and mental level for the participant; it provides sensory feedback directly to the participant; and it is interactive, responding to the participant’s actions (Sherman and Craig 2003, 6–11).[1] VR, in its most effective form, requires the user to be isolated from a conscious awareness of the real world by some sort of head-mounted display, such as Oculus Rift, Microsoft HoloLens, or HTC Vive. Alternatively, the user can experience VR in an enclosed, projection-based or flat-monitor-based environment, such as a CAVE.[2] Typically, the experience must be held in a static, controlled space; otherwise, the user might collide with real-world objects in the effort to participate fully in the virtual world. And, for many individuals, the VR experience results in motion sickness, sometimes known as VR sickness or cybersickness.[3] In contrast, AR is a medium in which digital information is overlaid on the physical world that is in both spatial and temporal registration (i.e., alignment) with the physical world and that is interactive in real time (Craig 2013, 36). Consequently, AR is much more accessible because the required equipment, usually a smart device (iPad, iPhone, Android tablet, or Android phone), is minimal. The fact the user remains cognizant of the real world around them while using the technology reduces the possibility of motion sickness and does not typically limit the user to a static, controlled space for the experience.

Both technologies have applications beyond gaming and are proving particularly effective and engaging for historic recreations. Such recreations can have a significant impact on learning, for they engage viewers—both the general public and students—in an educational immersive experience. Many of these viewers may never visit the actual historic site in their lifetime, so accuracy is important. Consequently, we need to keep in mind that a 3D digital model is a re-creation and not the real place. And as we move forward with VR and AR, we must give serious consideration to the goals we need and/or wish the technologies to meet, particularly with respect to pedagogy. At this point in time, VR and AR are very successful in engaging audiences for both entertainment and educational purposes:

The increasing development of VR technologies, interfaces, interaction techniques and devices has greatly improved the efficacy and usability of VR, providing more natural and obvious modes of interaction and motivational elements. This has helped institutions of informal education, such as museums, media research, and cultural centers to embrace virtual technologies and support their transition from the research laboratory to the public realm. (Rousso 2002, 93)

For the user visiting a virtual heritage site, the experience can be highly engaging and educational as long as expert guidance is provided. VR and AR cannot substitute for pedagogical instruction. It is not so much that the user must be reminded that the virtualization is not real; rather, supporting documentation must be easily accessible within the virtual world to help the participant understand the meaning and significance of the 3D models they encounter. And content builders must take an interdisciplinary, if not transdisciplinary, approach to the creation of the 3D models and their VR- or AR-enhanced worlds if the learning experience of the participant is to be as significant and valuable.

These technologies have the promise not only of engaging students in the history itself, but also of inviting them to consider how the work of history is done. As scholars and experts, we require the 3D models and their environments to be historically accurate, but that accuracy is necessarily limited. All models are inevitably interpretations of available evidence, and making that process more transparent to the student leads not only to a better understanding of the subject matter but of the process as well. As Willard McCarty has noted,

The best model [e.g., digital humanities tool] of something, that is, comes as close as possible to what we think we know about the thing in question yet fails to duplicate perfectly that knowledge. Failure of the model in an engineering sense is its success as an epistemological instrument of research, because skillfully engineered failure shows us where we are ignorant. (McCarty 2003, 1232)

Failing to create the perfect 3D model of an object in terms of historical authenticity is to be expected and appreciated for what it can teach us not just about the technology but about the 3D model itself in terms of our understanding of its historic accuracy. As teaching tools, VR and AR force the historical experts, as content creators, and their students, as content consumers, to think very carefully and intentionally about the recreation. For example, precise verisimilitude of a medieval English village could only be achieved by travelling back in time to the Middle Ages to conduct the kind of fieldwork envisioned by Connie Willis in her 1992 science fiction novel The Doomsday Book—an unlikely prospect by anyone’s standards.[4] However, it is important that we think beyond what VR and AR can do today. Even if we fail to achieve what we want the technology to do, we will learn from our mistakes and, in so doing, improve both the technology and our students’ understanding of the historical method.

Historical Accuracy: A Theoretical Approach

Virtual constructions of historical objects and architecture raise very real concerns about verisimilitude. To what extent are such 3D models accurate representations of the original? In many ways, VR serves to validate Jean Baudrillard’s understanding of simulacra and concerns about the hyperreal. In Simulacra and Simulation, he argues that the loss of distinction between reality and its representation results in the hyperreal—a world “without origin or reality” (Baudrillard 1994, 1–7). It is pure simulation and, as a result, creates an anxiety of origin and authenticity. Virtual worlds, including those associated with VR, can evoke an apprehension about the hyperreal, especially if the 3D model is used to substitute for the original. The current interest by computer graphic experts and enthusiasts in the creation and redistribution of virtual historic sites illustrates the problem. “Archaeological illustration and reconstruction is not new,” as Clifford L. Ogleby notes,

but the advent of high-speed affordable computers and the associated graphics capability gives people the opportunity to create better looking imagery. The imagery, however, is often the result of the technology, not archaeological or historical research. When this imagery is distributed without the accompanying research that explains the decisions made in the reconstruction, it is open to a variety [of] interpretations. This problem is compounded when the imagery is posted on the [world wide web], as the image can be extracted from the surrounding text and interpreted as an artifact rather than as a diagram. (Ogleby 2007)

Ogleby demonstrates this issue using easily obtainable images from the web that purport to portray accurate reconstructions (some computer generated) of the mausoleum at the ancient Greek city of Halicarnassus.[5] The images are imprecise and even erroneous, yet accepted by the general public as real: “Many people will tend to ‘see’ a photo-like image to be more like a photograph, and therefore a record of a real place in time” (Ogleby 2007). Not surprisingly, these online images almost always fail to include provenance, authorship, and veracity—information that would help the viewer to determine the authenticity of each 3D model and would serve as a reminder that the image being viewed is just that, an image, and not the original. The problem is only exacerbated when these models are incorporated into a virtual environment such as Google Earth or Second Life (Ogleby 2007).[6] These immersive and interactive worlds can encourage the non-expert user, such as a student, to accept the computer-generated model as an overly realistic recreation of the original.

Nevertheless, we should not be dissuaded from using the technology for pedagogical purposes both in the classroom and the community at large. Pierre Lévy argues convincingly against viewing the virtual as simply unreal: “The virtual, strictly defined, has little relationship to that which is false, illusory, or imaginary. The virtual is by no means the opposite of the real. On the contrary, it is a fecund and powerful mode of being that expands the process of creation, opens up the future, injects a core of meaning beneath the platitude of immediate physical presence” (Lévy 1998, 16). It is an actualization rather than a realization, one that involves “the production of new qualities, a transformation of ideas, a true becoming which nourishes the virtual in a feedback process” (Lévy 1998, 15).[7] The virtual and the real are not binary opposites. Rather, they exist on a continuum that supports a complete range of realness from the fully real to the fully virtual. Such a reality-virtuality continuum was first proposed by Paul Milgram and his colleagues. They suggest that everything in between is a mixture of reality and virtuality, including AR in which the real world is augmented by virtual enhancements and AV (augmented virtuality) in which the virtual world is augmented by the real (Milgram et al. 1995, 282–92).[8] The more obviously artificial nature of AR/VR visualizations may be used in a classroom setting to illustrate the sorts of choices that historians make in any evaluation/representation of historical data. What becomes important is not the degree of artificiality but rather the transparency of the method. Just as the creator of the virtual representation must make choices about how “real” to make their visualization (what to include and exclude), so the historian makes choices regarding what data to include and how that data is represented. The artificiality of extended reality technologies thus opens the door to conversations about not only the material being studied, but also the means by which it is studied.

The appeal of VR and AR is not new. Humanity has long held a fascination for trying to create a virtual experience of reality. In the nineteenth and early twentieth centuries, panoramic paintings became particularly popular, including the development of 360º murals that were intended to fill the entire field of vision and make the viewer feel as if he or she were in the virtual world depicted by the paintings (Thompson 2015).[9] The nineteenth century also saw the development of the stereoscopic[10] viewer and images, precursors to the View-Master and, more recently, Google Cardboard (Virtual Reality Society 2016). Experimentation in film also contributed to the development of the technology, particularly the widescreen camera lens. French filmmaker Abel Gance introduced “polyvision,” a specialized widescreen film format that involved the simultaneous projection of three reels of film in a lateral montage, in his 1927 silent epic Napoléon (Cuff 2015, 24). Polyvision, as well as the later development of CinemaScope and Panavision using widescreen lenses, gave the audience a panoramic and, subsequently, more immersive film experience. It was not until 1929 and the development of the flight simulator (Virtual Reality Society 2016) that a virtual environment was designed for teaching rather than for entertainment purposes. This focus on the pedagogical potential of virtual environments has become even more important today as VR and AR evolve from game platforms to teaching tools.

Both technologies exemplify the concerns faced by experts building virtual heritage sites.[11] For historians, archaeologists, and other scholars, the photorealism of the 3D models is the primary goal. In general, there are ten principles of 3D photorealism: clutter and chaos; personality and expectations; believability; surface texture; specularity; aging dirt, rust, and rot; flaws, tears, and cracks; rounded edges; object material depth; and radiosity (light reflections off diffused surfaces) (Fleming 1998, 3). To achieve photorealism, the computer-generated object should demonstrate at least seven of these ten principles (Fleming 1998, 3–4). The virtual world should not be pristine and unblemished because reality is messy and dirty. This concern for photorealism does not, however, apply in the same way to human 3D models. In fact, few virtual heritage reconstructions include human figures and for good reason. Firstly, creating realistic human models is time consuming and expensive since it requires a digital artist with considerable skill in drawing and modelling figures from life. Architectural and cultural artifacts are usually less difficult to build as 3D models. Secondly, living models, unlike objects, are expected to move in some way. Animation adds a complex layer of technology that is usually not the primary focus of the recreated physical environment. Thirdly, and most importantly from a pedagogical point of view, human 3D models can complicate the virtual experience by encouraging the user to try and interact with them rather than focus on the physical reconstruction of the heritage site. Finally, there is the consideration of how exactly “real” such human figures should be. The more realistic the 3D model of the living figure, the more likely that it will become an example of the uncanny valley phenomenon described in social robotics: that is, the 3D model will be almost too real so that the minor imperfections of the recreation become disturbing and even repulsive.[12] Thus a caricature of a human figure may be more appealing and effective than a truly realistic and complex representation in VR or AR.

Two Historic Recreations: Modelling Challenges

Bologna 3D Open Repository is the result of a collaborative project between the municipality of the city of Bologna and CINECA Interuniversity Consortium, an academic supercomputing group that offers technological support to education, business, and the community. The project’s primary goal was to build 3D models for the creation of a virtual Bologna that the municipality could use to promote the candidacy of the city’s historic porticoes, or arcades, as a UNESCO World Heritage Site. The repository is now maintained as a site dedicated to the collection and sharing of the 3D models for didactic purposes—namely teaching students about the city and its history. Figures 1 through 3 show some of the 3D models created by the consortium:

View of 3D model of the Portico of San Luca in Bologna

Figure 1. Portico of San Luca.

Aerial view of 3D model of the hilly landscape south of Bologna

Figure 2. Hilly landscape south of the city.

3D model of the medieval character, Apa, leaning against a desk with an open book on it

Figure 3. Scene of a medieval university lecture.

Through these visualizations, students can learn about the architectural history of Bologna from the medieval period through to the 18th century. The computer graphics are high quality and demonstrate a number of the principles of digital photorealism. In particular, the architecture and landscapes exhibit great attention to detail and authenticity. The project includes human figures, not typical of most historic recreations, and these figures are generally caricatures rather than realistic representations of people. Certainly, such a use of humor in a virtual historic re-creation emphasizes the project’s desire to appeal to a broad, public audience (Guidazzoli, Liguori, and Felicori 2013, 58–65).[13] And the less-than-realistic style of the human figures avoids the potential issue of the uncanny valley.

Like the Bologna 3D Open Repository, the 3D Paris Saga project uses AR and VR to tell the narrative of the architectural history of Paris. Their approach, however, differed considerably. Dassault Systèmes, a European software company that specializes in 3D design, built a complex virtual world that traces the history of the city through almost 2,000 years with a special focus on a 3D reconstruction and interactive experience of the fourteenth-century Palais de la Cité and the Sainte-Chapelle (“Voici” 2015). The project originally included a 90-minute television documentary, a CAVE experience of the virtual world using 3D glasses (Vitaliev 2013), a PC-compatible interactive 3D website, and an AR-enhanced print book (Dassault Systèmes 2012). The visual accuracy and detail of the 3D architecture, topography, and atmosphere enrich the photorealism of the virtual world (see Figure 4). The fact that familiar monuments are shown in various stages of construction transforms the virtual experience into a deeper educational one. Considerable attention is also given to the appearance of the skies, reflecting typical Parisian weather rather than an idealized and eternal perfect sunny day (see Figure 5). Again, 3D human models that inhabit the virtual city are not a common feature of such historic recreations. They are merely shadowy figures and remind the viewer that Paris was always inhabited; however, because the figures are so ethereal, they avoid the uncanny valley phenomenon and encourage the viewer to explore the historic constructions rather than try to interact with the animated models themselves.

Aerial view of 3D model of the Grande Cour and Trésor de Chartres in Paris

Figure 4. View of the Grande Cour and Trésor de Chartres with shadowed human figures in the courtyard (Dassault Systèmes).

Despite its initial success, the VR element of the project is no longer easily accessible: the CAVE environment is only available at Dassault’s Paris headquarters by appointment to select visitors.

View of 3D model of the rose window on the west façade of Notre Dame Cathedral in Paris

Figure 5. View of the rose window on the west facade (Dassault Systèmes).

Virtual reconstructions such as these help students understand cultures, histories, and artifacts that are physically, temporally, or culturally distant. While it may be difficult for American students to visit Notre Dame, extended realities can help them experience it in a way that more traditional media cannot.[14]

The AR-Enhanced Text

The most successful component of the 3D Paris Saga has been the AR-enhanced companion print book published by Flammarion. Whereas current AR technology uses a mobile application on a smart device to trigger the digital enhancements embedded in the printed page, Dassault requires the user to hold select pages from the print volume up to the web camera on a PC.[15] Like a virtual pop-up book, the 3D models appear on the page as viewed through the computer screen (see Figure 6).

AR-enhanced book opened to show the 3D model of Paris emerging from the printed page

Figure 6. AR-enhanced print text (Dassault Systèmes).

The user may turn the book in order to see all sides of the 3D model, thereby gaining a greater appreciation of Parisian architecture throughout history, including the Middle Ages. However, interacting with the book and the technology is awkward and lacks the mobility that a smart device offers. It is also counterintuitive to the standard reading process since the user holds the book but looks away from it at the computer screen.

AR-enhanced texts are not new. Mark Billinghurst and his team at HitlabNZ (the Human Interface Technology Lab at the University of Canterbury, New Zealand) created some of the first examples in the early 2000s. Called “MagicBooks,” the texts are designed to encourage children to read:

The computer interface has become invisible and the user can interact with graphical content as easily as reading a book. This is because the MagicBook interface metaphors are consistent with the form of the physical objects used. Turning a book page to change virtual scenes is as natural as rotating the page to see a different side of the virtual models. Holding up the AR display to the face to see an enhanced view is similar to using reading glasses or a magnifying lens. Rather than using a mouse and keyboard based interface users manipulate virtual models using real physical objects and natural motions. Although the graphical content is not real, it looks and behaves like a real object, increasing ease of use. (Billinghurst, Kato, and Poupyrev 2001, 747)

Although early forms of AR used abstract, specifically designed images (often QR codes) to trigger enhancements, the technology has advanced to the point that any complex, informationally dense image may serve as a fiducial marker. The use of mobile apps and smart devices makes interaction with the text easy and intuitive.

A new wave of AR technology seems to be driven by the increased capability and ubiquity of our mobile devices. Jordan Frith notes that early theories about the internet hypothesized that humanity (or at least that bit of it that could afford computers) would become more isolated and private—living their lives at home—we assume spending their time (and money) ordering from Amazon (Frith 2002, 136). Mobile computing has diverted us from this possible future. Instead, we are bringing our private lives into public spaces, attempting to control these spaces through our AirPods or earbuds, our Google maps, and Four Square—all the while curating our experience of the urban environment on social media.

It is to this mobile landscape that AR brings such promise. AR’s ability to overlay the physical world with digital information offers a new kind of experience and understanding of our world. Victoria Szabo argues that AR may be used to make the site of cultural history more meaningful to their visitors through the layering of digital information over the physical space. As she explains, “Mobile AR systems have the potential to help users create situated knowledge by bringing scholarly interpretation and archival resources in dialog with the lived experience of a space or object” (Szabo 2018, 373). In so doing, she argues, the visitors move from comprehension of the site which entails historical distance and critical interpretation—in other words traditional educational materials that might guide visitors through the site—to apprehension. Apprehension is more experiential learning and “relies on the tangible and felt qualities of the immediate experiences” (Martin 2017, 837; quoted in Szabo 2018, 374). The ability of AR to merge the “real” physical world of the historical site with digital material such as reconstructions, interpretive data, etc. facilitates both apprehension and comprehension.

When we consider an AR publication, however, we are moving away from Szabo’s paradigm to its inverse. With the book form, we are beginning not with the physical space—which already brings with it the tangible learning central to apprehension—but with the more traditional way of making meaning within education: the book. AR is still in its infancy in the publishing industry, but interest in its possibilities is growing. According to one 2017 poll, only 9% of Americans have experienced an AR application (Martin 2017, 20). Yet in this same year, five major tech companies, including Apple, launched AR frameworks or apps following the surprising success of the AR game Pokémon Go in 2016 (Tan 2018, 22). According to Digital Capital, an investment group, AR and VR are poised to become major players in technology. They estimate an AR/VR market of $108 billion with AR as the primary force and with predicted revenues of $90 billion by 2022 (Tan 2018, 22). This market data may seem irrelevant to academia, but what it means is that publishers are beginning to move into AR as well, creating new opportunities for academic AR publications. Major news media such as The New York Times, The Guardian, The Wall Street Journal, BBC, CNN, Hulu, and Huffington Post have all experimented with some form of Virtual, Augmented, or Mixed Reality (VAMR) media (Martin 2017, 21). Deniz Ergurel, technology journalist and founder of the media start-up Haptical, asserts that VAMR marks the next major technological shift. According to Ergurel, “Every 10–15 years, the technology landscape is reshaped by a major new cycle. In 1980s, it was the PC. In 1994, it was the Internet. And in 2007, it was the smartphone. By 2020, the next big computing platform will be virtual reality” (Martin 2017, 20).

AR text, because it is multisensory, can bring some of the features of experiential learning to its readers including the visual features of the text, historical contextualization, images, audio, video, data visualizations, supplementary text, and most importantly, 3D AR augmentations. The multimodal possibilities of AR texts make them particularly useful to teachers of literature that is culturally or historically distant because, through such reading environments, students may be more easily introduced to the material culture that surrounds and creates the texts they are studying. Furthermore, this approach allows the students to engage with the material in a multimodal fashion, appealing not only to the language centers of the brain, but to the visual and aural centers as well. The digital environment encourages the reader (and even the author) to “play” with the text in terms of design and interactive engagement (Douglas 2000, 65). The brain’s ability to play is something we, like many animals, are hardwired to do for survival; consequently, the process of reading text, especially digital text, has neurological value precisely because it encourages the brain’s playfulness (Armstrong 2013, 26–53).

Conclusion: The Future of VR and AR

The argument can be made that neither VR nor AR offers a truly immersive experience because not all five primary senses of the participant are engaged. Certainly, computer technology can generate both visual and aural enhancements in the form of 3D models and recorded sound. However, touch, smell, and taste are more challenging. Haptic tools, such as gloves or a stylus device, are becoming more popular and offer both the VR and AR user the ability to touch and sense physical contact with virtual objects. AR actually has the advantage of offering much more real-world haptic information by default than VR can. With AR, the user can feel the actual book because it can be a real-world object, but, in VR, the technology must do something to allow the participant to feel such an object because the entire environment is computer created. Demand has been less so far for smell and taste, although there have been some experiments, largely unsuccessful, in adding odors to virtual worlds. Recent developments in the creation of technological tools to trigger the sensation of taste in an individual, such as the “digital lollipop” (Ramasinghe and Do 2016) and Electronic Food Texture System (Niijima and Ogawa 2016, 48–9), show promise for the eventual incorporation of this primary sense into the VR experience.

If full sensory engagement is required for a virtual world to be completely realized, then perhaps the most immersive and interactive experience of the Middle Ages may be one that is not computer-generated at all: Jorvik Viking Centre. Located in York, England, the museum and tourist attraction was created in 1984 and has long been famous for its appeal to the senses of its visitors, most significantly the sense of smell. A quick glance at such online review sites as Trip Advisor, Virtualtourist.com, etc. makes it clear that the intentional smells associated with the exhibit are not just memorable but also a significant factor in recommending the Jorvik Viking Centre. The exhibit’s use of scents to enhance the Viking experience has even generated scholarship exploring the effectiveness of odor in retrieving the memory of the tourist experience. Apparently, it is very effective (Aggleton and Waskett 1999, 1–7).[16] The Centre, in fact, intentionally engages all the senses of its visitors in order to make the historic re-creation a memorable and educational experience. In 2015, it actively promoted its non-digital exhibit in the language of virtual and augmented technologies, inviting guests to have a 4D Viking encounter rather than a mere 3D one. In this campaign, the Centre emphasized that all five primary senses of its visitors will be fully engaged (Jorkvik Viking Centre 2015):

  • “Touch: Handling collection of Viking Age artefacts, including bone, antler and pottery, on offer to visitors in the queue—participants will be blindfolded and asked to identify the object/material.”
  • “Sight: Binoculars are available in the ‘Time Capsules’ that take visitors around the recreated Viking city. These are to be used to spot the various animals that inhabit the scenes of the ride experience. A ‘spotter’s guide’ will be issued, allowing visitors to score themselves against their finds.”
  • “Taste: A Viking Host will be on hand to explain the Viking diet and offer up tasters of unsalted, dried cod (a Norse delicacy) and for visitors over 18, Mead, a beverage made of fermented honey, will be available.”
  • “Smell: JORVIK is already famed for its re-creation of the smells of the 10th century York but this will be taken a step further with the introduction of ‘smell boxes’ in the ‘Artefacts Alive’ gallery. A new aroma will be located next to a display of object, with the smell paired to match the contents. [Four] smells will be available: Iron (for the Iron working display), Leather (next to the leather and shoemaking), Beef (for the general living display), and wood (for our wood finds).”
  • “Sound: A Viking will entertain visitors with period-specific musical instruments (including a recreation of the panpipes found at Coppergate) and retellings of some favourite Viking sagas.”

But as entertaining as the Jorvik Viking Centre clearly is, do we really want, or even need, a fully immersive and interactive experience? From the perspective of pedagogical effectiveness and student engagement, perhaps not. AR may, in fact, be the technology that has greater potential as a pedagogical tool precisely because it allows the user to learn in a digital environment while always keeping a strong foothold in the physical world—a reminder that the 3D world is not, ultimately, a real place.

Notes

[1] For further discussion of these key elements, see Søraker 2011, 44–72.

[2] Cave Automatic Virtual Environment: an immersive video theatre experience in which a participant wearing shuttering glasses views stereoscopic images as they are projected on the walls of a self-contained space in response to the participant’s position and actions.

[3] Such motion sickness may be caused by display and technology issues, sensory conflict, or postural instability; see LaViola, Jr. 2000, 47–56.

[4] Curiously, in 1935, a version of what we consider to be VR glasses was, in fact, envisioned by science fiction writer Stanley Grauman Weinbaum in his short story “Pygmalion’s Spectacles”; see Project Gutenberg http://www.gutenberg.org/files/22893/22893-h/22893-h.htm.

[5] Given the current interest in VR and AR, it is tempting to turn to 3D model sites, such as TurboSquid, to purchase ready-to-use models; however, evaluating these models for historical accuracy is essential. For example, searching on “medieval castle” brings up a wide selection of 3D models from fairly realistic structures to fantasy, fairytale confections that should be avoided for virtual historic sites; searching on “medieval woman” is even more problematic in terms of the results.

[6] Rousso expresses similar concerns about virtual heritage representation: “First, the issue of validity of information, commonly referred as authenticity. Second, the importance of accuracy in the representation of this information. Authenticity and accuracy are characteristics that archeologists, historians, and museum people strive to achieve and that the general public comes to expect from them. On the other hand, technologists dealing with the visualization of certain content are more concerned with the technical issues that pertain to implementation of the visualization and less concerned with authenticity and accuracy of the content itself” (Rousso 2002, 93).

[7] For a fuller analysis of Lévy’s understanding of actualization, see Ryan 1999, 78–107.

[8] For a detailed analysis of Milgram’s concept, see Craig 2013, 28–35.

[9] For an example of a 360º mural, see the Mural Room of the Santa Barbara County Courthouse which depicts the history of Santa Barbara, California, painted by Daniel Sayre Groesbeck in the early twentieth century: https://www.billheller.com/vr/Santa-Barbara-County-Courthouse-Mural-Room-360/.

[10] Stereoscopic imaging is the technique of creating an illusion of depth by using two offset images, one for the left eye and the other for the right, so that the brain processes both as a single, 3D image.

[11] We are making a distinction here between virtual heritage sites, which are 3D reconstructions of archaeological sites, architecture, or any other type of object, and 3D “real virtual worlds,” which combine 3D with “community, creation, and commerce,” such as World of Warcraft and Second Life; see Sivan 2008, 1–32.

[12] The phenomenon was first described by Masahiro Mori in 1970 and translated as “uncanny valley” by Jasia Reichard (Mori 1978).

[13] The project team has, in fact, used the 3D models to produce an award-winning stereoscopic short film, APA Etruscan (2012), for the Museum of the History of Bologna in which APA, an Etruscan character (see Figure 3), takes the viewer through a virtual history of the city.

[14] It is perhaps worth noting that, even though such virtual reconstructions are typically informed by the real world, the 3D digital exterior model of Notre-Dame de Paris created by Dassault Systèmes for the 3D Paris Saga as well as the 3D interior model created by Unisoft for the game Assassin’s Creed Unity may prove to be valuable resources for the rebuilding of the Cathedral after it was severely damaged by fire on April 15, 2019. Ironically, the real may now be informed by the virtual; see Wong 2019.

[15] For an example of how the book works, please see the following video: https://www.youtube.com/watch?v=sbZuQcXchkM.

[16] Capitalizing on the Centre’s success with odor and its notoriety, York’s tourism board published Britain’s first scented tourist guidebook in 2014 (Gordon 2014).

Bibliography

Aggleton, J.P., and L. Waskett. 1999. “The Ability of Odors to Serve as State-Dependent Cues for Real-World Memories: Can Viking Smells Aid the Recall of Viking Experiences?” British Journal of Psychology 90.

Armstrong, Paul B. 2013. How Literature Plays with the Brain: The Neuroscience of Reading and Art. Baltimore, MD: The Johns Hopkins University Press.

Baudrillard, Jean. 1994. Simulacra and Simulation. Trans. Sheila Faria Glaser. Ann Arbor: University of Michigan Press.

Billinghurst, Mark, Hirokazu Kato, and Ivan Poupyrev. 2001. “The MagicBook: A Transitional AR Interface.” Computers & Graphics 25, no. 5.

Craig, Alan B. 2013. Understanding Augmented Reality: Concepts and Applications. Waltham, MA: Elsevier.

Cuff, Paul. 2015. A Revolution for the Screen: Abel Gance’s Napoleon, Film Culture in Transition. Amsterdam: Amsterdam University Press.

Dassault Systèmes. 2012. “Revivez Paris en 3D! Dassault Systèmes nous raconte l’histoire fascinante de la Ville Lumière.” EXALEAD Blog. September 27, 2012. Accessed December 17, 2019, https://blogs.3ds.com/exalead/fr/2012/09/27/revivez-paris-en-3d-dassault-systemes-nous-raconte-lhistoire-fascinante-de-la-ville-lumiere/.

Dieleman, Hans, and Donald Huisingh. 2006. “Games by which to Learn and Teach about Sustainable Development: Exploring the Relevance of Games and Experiential Learning in Sustainability,” Journal of Cleaner Production 14.

Douglas, J. Yellowlees. 2000. The End of Books-Or Books without End: Reading Interactive Narratives. Ann Arbor, MI: The University of Michigan Press.

Fleming, Bill. 1998. 3D Photorealism Toolkit. New York: John Wiley & Sons.

Frith, Jordan. 2012. “Splintered Space: Hybrid Spaces and Differential Mobility.” Mobilities 7, no. 1.

Gordon, Sarah. 2014. “The scratch ‘n sniff tourist guide! York issues the UK’s FIRST scented guidebook to tempt the noses of tourists.” DailyMail.com, March 7, 2014. Accessed December 17, 2019. http://www.dailymail.co.uk/travel/article-2575496/York-releases-UKs-scented-guidebook-tempt-noses-tourists.html.

Guidazzoli, Antonella, Maria Chiara Liguori, and Mauro Felicori. 2013. “Open Creative Framework for a Smart Cultural City: Bologna Porticoes and the Involvement of Citizens for a UNESCO Candidacy.” Information Technologies for Performing Arts, Media Access, and Entertainment Lecture Notes in Computer Science, Vol. 7990: 58–65.

Jorvik Viking Centre. 2015. “Forget 3D – Discover Vikings in 4D this Summer.” Jorvik Viking Centre. Accessed December 17, 2019. http://www.thejorvikgroup.com/press/press-releases/forget-3d-discover-vikings-in-4d-this-summer/.

LaViola, Jr., Jospeh L. 2000. “A Discussion of Cybersickness in Virtual Environments.” SIGCHI Bulletin 32, no. 1: 47–56.

Lévy, Pierre. 1998. Becoming Virtual: Reality In The Digital Age. Trans. Robert Bononno. New York: Plenum Press.

Martin, Erik J. 2017. EContent Magazine. May/June 2017.

Masahiro, Mori. Robotics: Fact, Fiction, and Prediction. Trans, Jasia Reichard. Viking Press, 1978.

McCarty, Willard. 2003. “Humanities Computing.” Encyclopedia of Library and Information Science. New York: Marcel Dekker.

Milgram, Paul, et al. 1995. “Augmented Reality: A Class of Displays on the Reality-Virtuality Continuum.” Proceedings of the SPIE Conference on Telemanipulator and Telepresent Technologies 2351: 282–92.

Niijima, Arinobu and Takefumi Ogawa. 2016. “Virtual Food Texture by Electrical Muscle Stimulation.” Proceedings of the 2016 ACM International Symposium on Wearable Computers. New York: ACM.

Ogleby, Clifford. 2007. “The ‘Truthlikeness’ of Virtual Reality Reconstructions of Architectural Heritage: Concepts and Metadata.” 3D-ARCH 2007: 3D Virtual Reconstruction and Visualization of Complex Architectures. International Society for Photogrammetry and Remote Sensing (ISPRS). 36, no. 5: n.p. Accessed December 17, 2019, https://www.semanticscholar.org/paper/THE-%22-TRUTHLIKENESS-%22-OF-VIRTUAL-REALITY-OF-%3A-AND-Ogleby/87533316e22a172b015d10e67cd69f758f18c887.

Ramasinghe, Nimesha, and Ellen Li-Yuen Do. 2016. “Digital Lollipop: Studying Electrical Stimulation on the Human Tongue to Simulate Taste Sensations.” ACM Transactions on Multimedia Computing, Communications, and Applications 13, no. 1.

Rousso, Maria. 2002. “Virtual Heritage: From the Research Lab to the Broad Public.” Virtual Archaeology: Proceedings of the VAST Euroconference. Ed. Franco Niccolucci (BAR International Series 1075.

Ryan, Marie-Laure. “Cyberspace, Virtuality, and the Text.” Cyberspace Textuality: Computer Technology and Literary Theory. Ed. Marie-Laure Ryan. Bloomington: Indiana University Press. 78–107.

Sherman, William R., and Alan B. Craig. 2003. Understanding Virtual Reality: Interface, Application, and Design. The Morgan Kaufmann Series in Computer Graphics. San Francisco: Morgan Kaufmann Publishing.

Sivan, Yesha. 2008. “3D3C Real Virtual Worlds Defined: The Immense Potential of Merging 3D, Community, Creation, and Commerce.” Journal of Virtual Worlds Research 1, no. 1: 1–32.

Søraker, Johnny Hartz. 2011. “Virtual Entities, Environments, Worlds and Reality: Suggested Definitions and Taxonomy.” Trust and Virtual Worlds: Contemporary Perspectives. Eds. Charles Ess and May Thorseth. New York: Peter Lang Publishing. 44–72.

Szabo, Victoria. 2018. “Apprehending the Past: Augmented Reality, Archives, and Cultural Memory.” The Routledge Companion to Media Studies and Digital Humanities. Ed. Jentery Sayers. New York: Routledge.

Tan, Teri. 2018 “Making AR and VR Work in Publishing.” Publishers Weekly. July 2, 2018.

Thompson, Seth. 2015. “VR Panoramic Photography and Hypermedia: Drawing from the Panorama’s Past.” ISEA 2015: Proceedings from the 2015 International Symposium on Electronic Art. n.p. Accessed December 19, 2019. http://isea2015.org/proceeding/submissions/ISEA2015_submission_46.pdf.

Vitaliev, Vitali. 2013. “AR and 3D in Travel and History Applications, “ Engineering and Technology Magazine 8, no. 4. Accessed December17, 2019, https://eandt.theiet.org/content/articles/2013/04/ar-and-3d-in-travel-and-history-applications/.

Virtual Reality Society. 2016. “The History of Virtual Reality.” Virtual Reality Society. Accessed December 17, 2019. http://www.vrs.org.uk/virtual-reality/history.html.

“Voici à quoi ressemblait le Sainte-Chapelle au 14e siècle.” 2015. Sciences et Avenir. June 25, 2015. Accessed December 17, 2019, http://www.sciencesetavenir.fr/voyage/20150622.OBS1313/voici-a-quoi-ressemblait-la-sainte-chapelle-au-14eme-siecle.html.

Wong, Kenneth. 2019. “Dassault Systèmes, Ubisoft Pledge to Help Rebuild Notre-Dame.” Digital Engineering 247. April 17, 2019.
https://www.digitalengineering247.com/article/dassault-systemes-unisoft-pledge-to-help-rebuild-notre-dame/simulate.

Acknowledgments

We are extremely grateful to Alan B. Craig for reading and commenting on an earlier version of this article and for drawing our attention to the Bologna 3D Open Repository.

About the Authors

Tamara F. O’Callaghan is a Professor of English at Northern Kentucky University where she teaches medieval literature, history of the English language, and introductory linguistics as well as digital humanities approaches to literature. She received a Ph.D. in medieval studies from the Centre for Medieval Studies, University of Toronto, with a specialization in Middle English and Old French literature and medieval manuscript studies. She is the co-author of the textbook Introducing English Studies (Bloomsbury, 2020) and has published on medieval literature and manuscript studies as well as on the digital humanities and teaching. She also co-directs The Augmented Palimpsest Project, a digital humanities tool which explores how the medium of AR can be used in teaching medieval literature.

Andrea R. Harbin is an Associate Professor of English at the State University of New York, Cortland where she teaches medieval literature, the history of English, and Shakespeare and serves as department chair.  She has worked as a digital humanist since 1998 as curator/editor of NetSERF: an Internet Database of Medieval Studies. She received a Ph.D. in Medieval English Literature with a specialization in medieval drama from The Catholic University of America, and has published articles on digital humanities, pedagogy in medieval studies, and medieval drama. She is likewise a co-director of The Augmented Palimpsest Project.

Read More
Students sitting on the ground at a museum smile and discuss the 3D printed antiquity replicas they're discussing.
0
Issue Sixteen

Animating Antiquity: Student-Generated Approaches to Recontextualizing Ancient Artworks using Digital Technologies

Karen Rose Mathews, University of Miami

Gemma Henderson, University of Miami

With contributions by Jinqi Li, Lorena Lopez, Laura Miller, MacKenzie Miller, Aishwarya Navale, and Monica Travis

Abstract

Animating Antiquity was a student-generated curatorial project undertaken at the University of Miami in the Spring of 2019. The project consisted of multifaceted approaches to recontextualizing the ancient artworks in the Lowe Art Museum at the University of Miami. Ancient objects were functional at their core, but their display in a museum setting makes it difficult to recreate and understand their original significance and context. Through the use of digital tools—3D modeling and printing, and extended reality technologies—this project aimed to reinsert these objects into their original settings, reanimate their tactility and functionality, and form new modes of interaction with artworks in the space of the museum and the virtual realm. Students engaged in hands-on, museum-based learning through the compilation of art historical research contextualizing the objects; the creation of 3D digital models and prints; and the design of interactive strategies in real world and virtual environments. The Animating Antiquity Project combined multiple innovative technologies, pedagogical strategies, and community outreach to provide students with transferable professional skills and expertise while expanding the boundaries of the museum and connecting people and objects in innovative ways. This paper discusses the pedagogical and technical strategies employed during the project, foregrounding the approaches generated by undergraduate and graduate students.

Introduction

The Animating Antiquity Project was funded by a CREATE grant from the Mellon Foundation, whose aim is to foster the connection between students and cultural resources on the University of Miami campus (University of Miami Libraries and Lowe Art Museum 2019). The project was implemented in an interdisciplinary course, Greek and Roman Art (ARH 333P/CLA226P), co-taught by Professors Karen Mathews (Art and Art History) and Han Tran (Classics) in the Spring Semester of 2019. Twenty-one undergraduate students in the course undertook a curatorial project to recontextualize eight ancient objects in the Lowe Art Museum at the University of Miami, using various digital technologies to recreate and understand their original function and context (Lowe Art Museum 2019). Students devised multifaceted ways of reanimating antiques for visitors to the Lowe; research dossiers provided information about the function, context, and historical background of the objects, 3D digital models allowed viewers to manipulate the artwork and experience it in the round, 3D prints incorporated the elements of tactility and interactivity, while augmented reality (AR) and virtual reality (VR) experiences inserted the ancient artworks into new, virtual contexts. The innovation of this project resides in the student use of digital technologies to facilitate the staging of interactions between viewers and objects, creating a complex interplay between original, digital model, and printed replica in various spaces and modes—the museum gallery, a new space viewed through a smartphone, or an immersive virtual reconstruction.

The educational practices embedded for this project were guided by multiple components including: 1) program and course student-learning outcomes within the art history and classics BA programs 2) previous research undertaken by the authors in their respective fields of art history and digital education 3) emerging research and literature involving digitization within education 4) theoretical frameworks of historical preservation 5) and previous implementation of a similar project led by the authors (see footnote 1). The collaborative, pluralistic, and hands-on approach to the study of ancient art provided the most profound outcomes for students involved in the Animating Antiquity Project, as undergraduate and graduate students engaged in traditional and emerging methodologies associated with museum work. The project therefore informed the course workflow, including the weaving of content-specific lectures and classroom discussions, technical workshops with project partners, visits to the museum, student conferences, and several project assignments. The incremental design of coursework allowed students to create and present what they learned in different ways while facilitating instructor feedback and assessment. Through the creation of an art historical dossier, undergraduate student teams gained knowledge in interpretative analysis and research of primary and secondary sources, demonstrated their understanding of art historical terminology, and effectively synthesized their findings through documentation, in-class discussions, a project website, and a final presentation. The digitization and fabrication of antiquities gave students the rare opportunity to interact closely with precious historical artifacts often overlooked within the museum. Students engaged in a variety of tasks including photographing objects and hands-on technical workshops to produce materials for museum patrons, including digital and tangible facsimiles of each artifact. The creation of interactive strategies for museum visitors (activities that encourage interaction with the 3D print of the digital model and create a dialogue between the 3D print and the real object) allowed students to become instrumental in providing solutions to remove visual or tactile restrictions in a visitor’s engagement with objects. The final element of the project involved cross-program partnerships, where graduate students leveraged the dossier, 3D digital models, and prints to design augmented and virtual reality applications and employ the technical knowledge from their coursework within a practice-based museum context.

Many of these digital technologies and pedagogical approaches have been deployed in museum and higher education institutions over the past few years (Balletti, Ballarin, and Guerra 2017; Flynn 2018; Grayburn et al. 2019; Jeffs et al. 2017; Saunders 2017; Schofield et al. 2018; Younan and Treadaway 2015). Similar pedagogical approaches that align with this project address: the handling of objects, transformations in interpretation, the recontextualization of heritage, and approaches to digitizing, editing, and fabricating 3D digital heritage. At the Victoria University of Wellington, New Zealand, an initial partnership between the schools of Industrial Design and Classical Studies used 3D scans and prints of antiquities to lessen students’ fear of handling objects by having them recreate the original function of the object (Guy, Burton, and Challies 2018; Victoria University of Wellington 2017). At the University of South Florida, undergraduate students participated in a crowdsourcing project to digitize heritage artifacts at the Archaeological Museum of Syracuse and create digital storytelling guides (Bonacini, Tanasi, and Trapani 2018). For almost a decade, Duke University’s Wired! Lab for Digital Art History & Visual Culture has transformed its program offerings to respond to digital technologies within the fields of art and architectural history, involving students in the digital reconstruction of heritage in a variety of contexts (Lanzoni, Olson, and Szabo 2015; Wired! Lab 2019). Recent research (Di Franco et al. 2015; Pollalis et al. 2018) has also addressed hands-on engagement with digital and 3D printed objects from the perspective of undergraduate students. As such, digital technologies within art history and classics curricula have spurred the transformation of higher education practices.

Finally, this project was guided by conceptual frameworks that address the complex relationship between humans and objects. Material culture studies have highlighted the central role of objects in human thought and action (Hicks and Beaudry 2010; Tilley et al. 2006). Specifically, thing theory argues for the active role of objects in defining human actions, giving something inanimate a rich biography and social life (Appadurai 1986; Brown 2001; Kopytoff 1986). Within the project, the artifacts were central actors in the design of the course assessments, the project components, and the student interaction with the antiquities. As student teams advanced throughout the project, the artifacts adopted new meanings through content generated by the students. Students were encouraged to make historical inferences about these objects to develop an original narrative about their history. As a result, the relationship between students and the objects was multifaceted and symbiotic, as objects interact with people and other objects to produce unexpected effects, strengthening or redefining pre-existing social or cultural relationships, or forging completely new connections (Mathews 2015). Furthermore, a significant outcome of this project was the incorporation of graduate students, who were encouraged to present their perspective on the role of the museum, its objects, and visitors. The interaction between people and objects therefore locates artworks in a constant state of redefinition as they engage with human actors in different spatial and temporal contexts (Figure 1).

Diagram connecting the center project component, art historical research and 3d modeling, with three resulting project components, 3D printing, augmented reality application and virtual reality installation.
Figure 1. Visual diagram of the Animating Antiquity Project components.

This paper will outline the implementation and outcomes of the Animating Antiquity Project, addressing the key components of the project—the creation of an art historical research dossier, 3D digital models of ancient artworks, prints of 3D models, interactive strategies for the printed models, and a website presenting student research—in chronological order. A discussion of the creation of an art historical research dossier and digital 3D models for eight artworks in the Lowe Art Museum will be followed by a description of the varied applications of these core materials for interactive strategies that forged creative connections between ancient objects and modern viewers. This paper therefore seeks to demonstrate the rich and multifaceted relationships between people and objects that were forged through the use of digital technologies in the museum and classroom and share the pedagogical methodologies and outcomes associated with this project with the wider academic community.[1]

Art Historical Dossier and 3D Digital Models

Art historical dossier

The research conducted for the art historical dossier served as the first reanimation of ancient artworks in the Lowe, as the students began to understand the role that objects like glass vessels, ceramic wares, and portrait sculpture played in the ancient world. In their present configuration, the antiquities at the Lowe Art Museum reside in glass display cases with minimal background information and limited opportunities for visitor interaction. In order to foster a deeper understanding of these antiquities, students conducted art historical research on the eight objects chosen for 3D modeling and compiled that research in digital dossiers. The research focused particularly on the function and context of the artworks, as all ancient art served a purpose, be it political, religious, economic, or social. The research materials consulted by the students were eclectic in nature, given the lack of documentation on these specific objects. Students consulted dossiers on file at the Lowe Art Museum, art historical materials and practical guides from other museums and cultural organizations, scholarly books and articles, and web-based resources to understand and contextualize these ancient objects. The analysis of the antiquities began with a basic physical description, extracting visual information from the object itself. Then the students delved deeper into the history of ancient Greece and Rome to understand how these objects functioned, where they would have been placed and used, and how they would have been perceived by ancient audiences.

The subdividing of the written elements into thematic units facilitated the writing process. Students could concentrate on one topic at a time—function, iconography, context—and also review peer feedback that they could revisit and revise (Carless and Boud 2018). In the final stages of the dossier’s creation, the students worked on the text as a whole, making it flow as a seamless narrative, omitting redundancies and focusing the text on a particular theme related to the object itself. The preparation of the art historical dossier served a number of purposes: familiarizing the students with objects and their meaning in original contexts; creating the foundation for student-generated interactive strategies; and providing background information on the objects for visitors to the Lowe Art Museum. This original research conducted by the students provided a key outcome of the project, as most of these objects have not been studied in a systematic manner. The Canosan vase, for example, is a fairly common object, but this type of pottery has not been addressed extensively in the scholarly literature (Figure 2). In the absence of basic data, students researched comparable objects to make scholarly inferences about the artworks in the Lowe, collecting comparative materials with which to support their conclusions. The completed art historical dossier consisted of the following elements: a document presenting a visual description of the object and discussing its form, function, and original context; a photograph of the object; the 3D digital model; and images of comparable artworks. This research dossier was shared on Google Drive so that all the students devising interactive strategies would have access to this important contextual information.

Portrait and side profile of Greek Canosan funerary vessel, decorated with two female figures, next to portrait and side profile of a comparative work.
Figure 2. Canosan funerary vessel, Lowe Art Museum, 98.0009, and a comparative work.

In the final phase of the project, the dossier was uploaded onto a website that serves as the public portal to the research project [https://www.animatingantiquity.net]. Students were encouraged to review the Spring 2018 digital dossier (see footnote 1) as an ‘exemplar’ to illustrate assessment expectations (Carless and Boud 2018). Using WordPress, students created posts for each of the eight objects featured in the class project (Figure 3). Once uploaded to Sketchfab (a web platform for hosting and viewing immersive and interactive 3D files), the digital model was embedded within the post so that visitors could review still photographs and interactive 3D models while reading the text. The content of the website is accessible in the immediate context of the museum itself through a tablet mounted on a pedestal next to the artworks on display, so visitors can gain knowledge about the object while viewing it firsthand and performing the interactive strategies described on the website.

Project homepage including the project overview next to the calyx krater page depicting a gallery, interactive 3D model, and text.
Figure 3. Animating Antiquity website homepage and calyx krater page.

3D digital models of ancient artworks

In addition to the dossier, the creation of the 3D digital models profoundly influenced new interpretations and re-presentations of the objects (Kalay 2008, 8). Harrison (2015) proposes that heritage is inseparable from the interconnected relationships between social, political, and environmental issues. The 3D digitization process and the digital and fabricated objects invited conversations with students about conservation, management, and ethical implications of heritage artworks. As the fragility and antiquity of the artworks limited engagement with them, students were tasked with the design of digital and physical reproductions for the public to touch. The process of photographing the objects introduced students to the careful protocols established by museum staff for handling antiquities in order to ensure their proper conservation. Indeed, students contributed to the conservation of these artworks by creating 3D digital versions of the originals and compiling a set of photographs that documented the current state of the object. The ethical implications connected to fabricating artifacts were also discussed with students, encouraging them to consider the implications of creating digital copies of original artworks and the ways in which they could alter the meaning of the ancient artworks (Colley 2015).

Students created the 3D digital models in a multi-step, group-based, collaborative process, in which tasks related to the various components rotated through each student group. Everyone contributed to all the interrelated parts and the students themselves were responsible for the completion of the digital model. Objects were selected based on their applicability for photogrammetry, and students created the 3D models of these artworks in three 75-minute hands-on workshops. First, the photogrammetry session at the Lowe Art Museum allowed students to capture multiple photographic viewpoints of the objects outside the display case, transforming how they usually interact with artworks. Eight students (two from each group) served as “digital preservation experts,” engaging with museum staff during the session as collaborators in the photography process. Museum staff installed, handled, and rotated the pieces while students determined the most effective ways to capture each object (Figure 4).

Two images depicting students taking photos of artefacts within the museum, while museum staff observe.
Figure 4. Students conducting photography using DSLR cameras and lighting soft boxes.

Students were provided some initial guidance and allocated an hour to capture the artworks. They checked the assembled equipment for any discrepancies, decided which object they would photograph, communicated with the museum staff to place objects, prepared and tested the camera settings, and completed four 360° rotations to ensure adequate overlap of focused photos at multiple camera angles to capture complex sculptural contours.[2] They then uploaded the files to a shared Google Drive folder.

Post session, two different students per group took the lead in editing raw photos, exporting them for photogrammetric processing using Autodesk ReCap Pro Photo. Once processed and downloaded, the files were reduced in size and exported for students to edit with Autodesk Meshmixer. Two modeling sessions completed the process of creating the 3D digital models. In the first session, two students per group edited mesh files, learning how to repair holes in the mesh, add surface texture to the model, and create a clean base. Students were encouraged to interact with the artworks in the digital realm, dismantling the object, creating new shapes, sculpting and adding textures to the reconstructed artwork. In this part of the process, emphasis was not placed on absolute accuracy, but rather on an authentic presentation of the digital object. With help from the authors and student assistants, timely, personalized feedback (Carless and Boud 2018) was shared with each student to prepare for the second modeling session in which further editing tasks were performed to prepare the files for uploading to Sketchfab and for 3D printing.[3] Once the 3D models were completed, the authors uploaded the .OBJ files onto Sketchfab. Students then created annotations for the digital models, noting aspects of the object’s iconography, material, and technique in short texts that can be read while manipulating the model (Figure 5). The publishing of the annotated models on a well-known website platform constituted another reanimation of these antiquities, as the digital model and its descriptive annotations bridged the distance between visitors, students, and the wider public audience.

3D digital models of the bearded roman and calyx krater objects captured with numbered annotations.
Figure 5. 3D digital models with annotations on Sketchfab.

Student-Generated Applications of Art Historical Research and 3D Digital Models

Reconstructing artworks through 3D printing

The 3D prints actualized the premise of the Animating Antiquity Project, bringing alive the culture of ancient Greece and Rome through a recontextualization of ancient artworks while creating new, contemporary connections between people and objects. The digital models themselves served an important didactic function, providing museum visitors with a more comprehensive understanding of an artwork through the manipulation of the digital model (Jeffs et al. 2017), but they also served as the basis for creating prints of the models that could be handled in the museum gallery. Complementing photographs and the digital model, the 3D print provided a third visual manifestation and reanimation of the object, replicating the art object in an accurate manner through its size and material. The printed replicas were painted to recreate the decoration on the artworks and provided viewers with tactile access to the objects. In museum settings, 3D prints of artworks allow visitors to engage with art in a more intimate way, complementing the original works in the display case and providing opportunities for blind and visually impaired visitors to experience art objects (Di Franco et al. 2015; Henderson 2018; Nancarrow 2017; Sportun 2014; Williams 2017; Wilson et al. 2018).

Two images depicting multiple students and faculty sitting in the museum, talking and handling 3D prints of antiquities.
Figure 6. Students handling draft 3D prints at the Lowe Art Museum.

The 3D printing of digital models was essential for the interactive activities described below. The printing component was ambitious and interdisciplinary in scope, with a number of individuals, spaces, and institutions contributing to its successful completion. The digital models were prepared for 3D printing by students in Meshmixer with additional refinements made by the authors. Digital versions of vessels, for example, required thickened surfaces for structural integrity or were hollowed out so that they could be printed as vessels. After the first modeling session, ‘draft’ 3D models were printed at a reduced scale in polylactic acid (PLA) filament for students to handle, evaluate, and employ in devising interactive strategies (Figure 6). The remaining modeling edits and student-generated strategies informed the number, scale, and structure of the 3D prints.

The printing workflow entailed dividing the objects between makerspaces in the College of Engineering, the Department of Art and Art History, and an Ultimaker 3 managed by Academic Technologies.[4] As the Engineering printers could also accommodate larger-scale objects, most models were printed to emulate the scale of the original artwork.[5] Prints were made using various PLA filament types, with a marble-like filament emulating the crystalline structure of stone sculpture, while wood-fiber and terracotta-colored plastic reproduced the original materials of other ancient objects. Printing times could vary from three to thirty-six hours per object, so the completion of these prints was scheduled over a month time frame, anticipating printing errors and print queue issues while accommodating other users on campus. Once the prints were ready, students prepared models for painting. Some models remained unpainted, as the PLA imitated the original material of the artwork, but others had their original polychromy “restored” or were painted to more closely emulate the original decoration (Figure 7). Students in this phase of the project could interact freely with the models in ways that would be impossible with the original artwork, emulating the original use of the objects in their handling and manipulation.

Landscape view of a table display presenting multiple 3D printed versions of antiquities within the Lowe Art Museum.
Figure 7. Display of completed prints in the Lowe Art Museum.

Undergraduate student strategy: Restoration of the Canosan funerary vessel

The 3D prints provided the opportunity for an imaginative reconstruction of the object. Instead of copying the current state of an artwork, prints can recreate the original, often vibrant, decoration of ancient objects. The Canosan vessel print, for example, displays the bright primary colors that would have characterized this funerary offering in its original state (Figure 8). The comparison between the modern copy and the artwork itself presents significant historical information about material culture in the ancient world.

Two images depicting the process of students and faculty painting the Canosan vase using primary colors. One final image presents the finished object painted in blue, red and yellow.
Figure 8. “Restoring” the color of the Canosan funerary vessel print.

Undergraduate student strategy: Recreating Theseus and the labyrinth

Student-generated strategies strove to combine three versions of the same object—original, digital model, and 3D print—in order to animate the artwork for museum visitors. These interactions could highlight the form and iconography of the object or shed light on its function and original context. The interactive strategy devised for the double-headed sculpture of Theseus and Ariadne focused on the duality and complementarity of the two figures portrayed (Figure 9).

Three images depicting a two headed Theseus and Ariadne sculpture, including two portrait views and a side profile view.
Figure 9. Theseus and Ariadne sculpture, Lowe Art Museum, 2005.7.2.

For the 3D print, the students separated the heads to emphasize the distinct characteristics of each figure but also the complementary nature of the pair. Added to the Theseus print was a maze representing the labyrinth that he had to negotiate with the help of Ariadne to kill the Minotaur. Ariadne gave Theseus a ball of thread so that he could trace his way out of the maze. In the gallery activity, a stylus attached to a string on the Ariadne side traces the path of Theseus through the maze, demonstrating the cooperation between the couple that helped them destroy their monstrous adversary (Figure 10). Color-coded pins inserted into the print provided information about the myth in stages so that viewers can follow the path of the protagonist and discover the intertwined nature of these two mythological figures.

Three side-profile images depicting the 3D printed two headed Theseus and Ariadne sculpture, cut in half with a maze visible.
Figure 10. Theseus and Ariadne 3D prints with maze.

Undergraduate student strategy: Simulating the calyx krater’s function at the symposion

A third interactive strategy recontextualizes the calyx krater and the role it played in the ancient Greek symposion, a gathering of Greek men who engaged in conversation, listened to music, and enjoyed entertainment while drinking wine mixed with water. The mixing of the two liquids was essential for ensuring the longevity of the symposion and displayed the civilization of the Greeks, as only “barbarians” drank unmixed wine. The krater was the vessel used to combine water and wine and was thus an indispensable component of the symposion itself. The imagery on this krater alludes to its function, representing the god of wine, Dionysus, and his followers in a procession (Figure 11).

Three images depicting the Calyx krater displayed at three rotated angles
Figure 11. Calyx krater, Lowe Art Museum, 2011.5.

The interactive activity recreated the symposion environment by distributing a number of smaller drinking vessels to the “participants” of the gathering while demonstrating the dilution of the wine central to this ritualized activity. The interactive strategy highlighted the function of the krater while elucidating the iconography of the vessel (Figure 12).

Three images depicting a full-scale 3D printed calyx krater decorated, multiple small-scale 3D printed calyx kraters in red and black, and students demonstrating the pouring of water in the krater.
Figure 12. Student-generated 3D prints of the calyx krater and reconstruction of wine mixing.

Interdisciplinary work by MFA students

Though the content of the Animating Antiquity Project was devised in the context of an undergraduate course, MFA students from two different schools designed and implemented their own creative engagements with 3D models and prints using the art historical dossiers and digital models created by the undergraduates. One project emerged through a graduate student’s exploration of makerspaces across the UM campus and the potential of 3D printing as a creative sculptural medium. Two additional projects developed through partnerships with students in the MFA program of Interactive Media at the School of Communication. Enrolled in courses dedicated to the study and implementation of AR and VR technologies, these graduate students used the content created by undergraduate students to design interactive approaches with the Lowe antiquities. The undergraduates, then, served as the experts in terms of knowledge of the art objects themselves and in terms of the creation of raw digital data. Through shared Google Drive folders, graduate students could access all the work folders and employ the raw materials compiled by the undergraduates in innovative technological formats.

MFA student project: Experimentation with 3D modeling and printing (Monica Travis)

The greatest experimentation with printing materials took place with the reproduction of a Hellenistic theater mask by Monica Travis, MFA student in sculpture in the Department of Art and Art History. The theater mask is a bronze object, and Monica wished to print it in metal to emulate the original material. The Johnson & Johnson Lab in the College of Engineering possesses a metal 3D printer that produces parts with titanium powder, and these advanced facilities provided the opportunity to print the bronze theater mask in a metallic medium, that of titanium. This printer is often used for the prototyping of parts, so the printing of an art object constituted an innovative application of its capabilities. Monica undertook a meticulous preparation of the file for printing, editing photographs in Adobe Photoshop and processing them in Agisoft Metashape, an advanced photogrammetry tool. The modeling application Rhinoceros was used to clean up the model and prepare it for printing. A test print was performed using a Lulzbot Taz 6 in the Department of Art and Art History before queuing it on the titanium printer (Figure 13). This project provided an opportunity for Monica to innovate in both the methods and materials used; she experimented with photogrammetry, photographic editing, and modeling tools to produce a print using materials not often employed in 3D printed artworks.

Three images depicting multiple versions of the Theatre mask in yellow PLA, titanium and wood PLA.
Figure 13. Theatre mask (left to right) in PLA filament; Titanium print; all 3D prints in PLA, titanium, and wood-PLA filament.

MFA student project: Antiquities in real world contexts using augmented reality (Jinqi Li, MacKenzie Miller, Laura Miller, Aishwarya Navale)

In the context of a course offered in the School of Communication at the University of Miami, graduate students from the MFA Interactive Media program created two AR applications using the student-generated digital models of the Lowe antiquities.[6] The concept behind both apps was to enhance visitor interaction with the objects using a smart device (Marques and Costello 2018). The first AR application allows museum-goers to observe the antiquities up close in 3D. Users employ a brochure that serves as the platform for the experience, allowing them to use the app anywhere. The brochure possesses four recognizable image targets that trigger the AR program when a device’s camera is directed at them (Figure 14). When the image target is detected, users view the rotating digital model on their mobile screen, supplemented with audio narration addressing significant information about the artwork. To create the first application, the students used Vuforia Augmented Reality SDK in Unity 3D to enable the image target detection of the artifacts. Dr. Mathews recorded the audio that was used in the application, and a rotation effect was added so that the viewer could see the artifact from all angles. Lastly, the application was exported to a mobile phone using Xcode.

A landscape brochure detailing four models used for image targets in the augmented reality experience
Figure 14. Brochure for AR experience.

The second application enables museum visitors to take the artworks home with them virtually, observe the objects in new contexts, and share their experiences on social media platforms. Users of this application scan a room and then project the 3D model of the object onto that space. The object can be resized and moved around the room to the ideal virtual setting, and the user can then photograph it and share it. The students used ARKit SDK in Unity 3D to make an application where a virtual object can be placed in the real world. A menu allows the user to choose the object they would like to place in a new real-world context (Figure 15). Both of these AR applications used digital models and art historical research to connect people and objects through interactive experiences. The users of these AR applications learn more about the objects in a personalized manner, employing a smart device as a tool to display and manipulate the 3D models, place the antiquities in novel contexts, and share their interactions with others.

Three panes representing the user perspective of using the augmented reality application including instructions, digital model placed on ground and annotations.
Figure 15. AR application with photo “souvenir” capabilities.

MFA student project: Feeling Antiquity in virtual reality (Lorena Lopez)

This virtual reality (VR) experience attempts to show the myriad ways in which immersive environments can enhance a visitor’s understanding of artworks in a museum setting by emphasizing the power of touch. Both museum and VR experiences generally lack a tactile component, that is, the ability to touch objects in real and virtual spaces (Candlin 2010; Candlin 2017). In the Feeling Antiquity experience, visitors can interact with a 3D print in a virtual realm. The object selected for the VR experience was the calyx krater, a vessel that was used to mix wine and water in the context of the symposion (Figures 11 and 12). As part of a summative project to demonstrate her knowledge and construction of virtual worlds, Lorena Lopez used the art historical dossier created by undergraduates to study the physical setting of the symposion, a space called an andron that was used exclusively by males. Using the Unity Game engine and its asset store, she constructed the 3D scene where men would gather for the symposion and hold the krater (Totten 2014). Through an HTC Vive Pro VR headset, the viewer was immersed into the ancient Greek space of the andron. Once there, the user of the headset could interact with a full-scale print of the calyx krater, touching and picking up the printed model of the vessel using the handheld controllers (Figure 16). The virtual visitor was not only transported to the ancient past, but they could reach out and actually touch an object within the space of the andron. In a museum setting, the sense of touch is generally subordinated to the visual, as visitors are discouraged and prohibited from touching artworks (Di Franco et al. 2015). VR can break down these barriers and allow museumgoers to engage art with more than just vision, handling ancient artworks in the same way that they would have been used in the past. There are great, but still untapped, possibilities in the realm of haptic gloves and suits, though handheld controllers were used in this experience in the interest of time (Needleman 2018; Hall 2016). These VR technologies can animate places and objects that are distant geographically or no longer extant, integrating touch into an interactive and immersive environment that complements and enhances a museum experience.

Graduate student wearing a virtual reality headset in front of a projector displaying a room, and standing next to a table displaying a black 3D printed version of the calyx krater.
Figure 16. VR experience with the calyx krater.

Conclusion: Outcomes and Reflections

In the context of the spring semester course, students successfully completed all the stated objectives: the creation of an art historical dossier, a digital model, a 3D print, strategies for visitor interaction with the 3D print, and a website. What was actually gained from this experience, however, was far richer. The interdisciplinarity and interactivity of the work conducted established meaningful connections between the students and the objects they studied. Students were exposed to numerous ways in which research and digital content could reanimate ancient objects. They also gained invaluable, hands-on practice with various digital technologies and processes, helping them determine where their personal interests and talents lay, be it in painting, 3D modeling, printing, or photography. The implementation of this project did pose some challenges, as the production schedule for the multiple components had limited flexibility, and a less-condensed time frame would have allowed for more exploration and mastery of research and technical skills. The display of 3D prints and implementation of XR technologies also raise ethical and logistical issues concerning the use of digital technologies in traditional museum spaces (Colley 2015, 17). Where do you stage such interactive experiences, and who provides oversight and monitoring? AR applications can be integrated easily into museum galleries through the use of smart devices, but VR often requires expensive equipment, space for movement, and timed sessions (Meier 2017). Furthermore, while the 3D/XR digital assets created in this project aimed to align with existing imaging and digital preservation practices (Alliez et al. 2017; Bedford 2017), limited control of metadata is a topic being addressed by open communities and experts (Moore et al. 2019; Rossi, Blundell, and Wiedemeier 2019). Once these practical challenges are addressed, however, myriad possibilities exist for the use of digital products and technologies in museum and undergraduate education, as 3D prints and their interactive applications can play a central role in the pedagogical mission of museums, encouraging visitors to devise their own strategies for connecting to people and objects from the ancient past.

Notes

[1] Henderson and Mathews employed photogrammetry techniques in a Spring 2018 course at the University of Miami addressing Spanish colonial art objects in the Lowe Art Museum; see their website. Work on this course facilitated the creation of the lesson plans, digital technologies, and partnerships that informed the Animating Antiquity Project.

[2] The equipment assembled to photograph the models included four DSLR Canon Rebel cameras, four tripods, three portable photo studio kits, four rotating platforms, one photography tent, and one larger lighting soft box.

[3] The file of the theater mask required extensive editing, and a graduate assistant, Monica Travis, became a co-creator of the digital version, using her expertise in Rhino to separate the shells and extra data to create a 3D printable file.

[4] The authors organized the print queues and Monica Travis managed most of the printing.

[5] The College of Engineering has several Makerbot printers, with two in particular (Makerbot Replicator Z18) that boast an 18” Z (vertical build space) ideal for larger print projects. These printers are available to all students and the college provides filament for the printing projects.

[6] The course was CIM 624—Augmented Reality, taught by Dr. Ching-Hua Chuan. Monica Travis was instrumental in forging this collaboration.

Bibliography

Alliez, Pierre, Laurent Bergerot, Jean-François Bernard, Clotilde Boust, George Bruseker, Nicola Carboni, Mehdi Chayani, et al. 2017. Digital 3D Objects in Art and Humanities: Challenges of Creation, Interoperability and Preservation. White Paper. https://hal.inria.fr/hal-01526713.

Appadurai, Arjun. 1986. “Introduction: Commodities and the Politics of Value.” In The Social Life of Things: Commodities in Cultural Perspective, edited by Arjun Appadurai, 3–63. Cambridge: Cambridge University Press.

Balletti, Caterina, Martina Ballarin, and Francesco Guerra. 2017. “3D Printing: State of the Art and Future Perspectives.” Journal of Cultural Heritage 26 (July): 172–82. https://doi.org/10.1016/j.culher.2017.02.010.

Bedford, Jon, ed. 2017. Photogrammetric Applications for Cultural Heritage: Guidance for Good Practice. Swindon: Historic England. https://historicengland.org.uk/images-books/publications/photogrammetric-applications-for-cultural-heritage/.

Bonacini, Elisa, Davide Tanasi, and Paolo Trapani. 2018. “Participatory Storytelling, 3D Digital Imaging and Museum Studies: A Case Study from Sicily.” In 2018 3rd Digital Heritage International Congress (DigitalHERITAGE) Held Jointly with 2018 24th International Conference on Virtual Systems & Multimedia (VSMM 2018), 1–4. San Francisco, CA, USA: IEEE. http://doi.org/10.1109/DigitalHeritage.2018.8810058.

Brown, Bill. 2001. “Thing Theory.” Critical Inquiry 28, no. 1: 1–22.

Candlin, Fiona. 2010. Art, Museums and Touch. Manchester: Manchester University Press.

———. 2017. “Rehabilitating Unauthorised Touch or Why Museum Visitors Touch the Exhibits.” The Senses and Society 12, no. 3: 251–66. https://doi.org/10.1080/17458927.2017.1367485.

Carless, David, and David Boud. 2018. “The Development of Student Feedback Literacy: Enabling Uptake of Feedback.” Assessment & Evaluation in Higher Education 43, no. 8: 1315–25. https://doi.org/10.1080/02602938.2018.1463354.

Colley, Sarah. 2015. “Ethics and Digital Heritage.” In The Ethics of Cultural Heritage, edited by Tracy Ireland and John Schofield, 13-32. New York: Springer-Verlag. https://www.springer.com/us/book/9781493916481.

Di Franco, Paola Di Giuseppantonio, Carlo Camporesi, Fabrizio Galeazzi, and Marcelo Kallmann. 2015. “3D Printing and Immersive Visualization for Improved Perception of Ancient Artifacts.” Presence: Teleoperators and Virtual Environments 24, no. 3: 243–64. https://doi.org/10.1162/PRES_a_00229.

Flynn, Thomas. 2018. “Explorable, Shareable 3D.” presented at the DigiDoc 2018, The Engine Shed, Edinburgh, October 15. https://web.archive.org/web/20190527204229/.

Grayburn, Jennifer, Zack Lischer-Katz, Kristina Golubiewski-Davis, and Veronica Ikeshoji-Orlati, eds. 2019. 3D/VR in the Academic Library: Emerging Practices and Trends. Council on Library and Information Resources Publication 176. https://www.clir.org/wp-content/uploads/sites/6/2019/02/Pub-176.pdf.

Guy, Bernard, Diana Burton, and Zach Challies. 2018. “Creative Expression through Tangible Narrative: How 3D Printing May Complement Our Pedagogical Investigation of Heritage.” In 2018 3rd Digital Heritage International Congress (DigitalHERITAGE) Held Jointly with 2018 24th International Conference on Virtual Systems & Multimedia (VSMM 2018), 1–4. San Francisco, CA, USA: IEEE. http://doi.org/10.1109/DigitalHeritage.2018.8810121.

Hall, Charlie. 2016. “Welcome to The Void.” Polygon. May 5. https://www.polygon.com/features/2016/5/5/11597482/the-void-virtual-reality-magician-tracy-hickman.

Harrison, Rodney. 2015. “Beyond ‘Natural’ and ‘Cultural’ Heritage: Toward an Ontological Politics of Heritage in the Age of Anthropocene.” Heritage & Society 8, no. 1: 24–42. https://doi.org/10.1179/2159032X15Z.00000000036.

Henderson, Gemma. 2018. “Exploring the Meanings of 3D Digital Heritage Objects in Undergraduate Education.” MSc Dissertation, University of Edinburgh.

Hicks, Dan, and Mary C. Beaudry. 2010. The Oxford Handbook of Material Culture Studies. Oxford: Oxford University Press.

Jeffs, Amy, Fern Insh, Erich Hatala Matthes, Tom Flynn, Daniel Pett, Tim Ayers, et al. 2017. “Disciplining the Digital: Virtual 3D Reproduction, Pilgrim Badges, and the Stuff of Art History.” British Art Studies, no. 6 (June). https://doi.org/10.17658/issn.2058-5462/issue-06/conversation.

Kalay, Yehuda. 2008. “Introduction: Preserving Cultural Heritage through Digital Media.” In New Heritage: New Media and Cultural Heritage, edited by Yehuda E. Kalay, Thomas Kvan, and Janice Affleck, 1-10. London: Routledge.

Kopytoff, Igor. 1986. “The Cultural Biography of Things: Commoditization as Process.” In The Social Life of Things: Commodities in Cultural Perspective, edited by Arjun Appadurai, 64–91. Cambridge: Cambridge University Press.

Lanzoni, Kristin, Mark Olson, and Victoria Szabo. 2015. “Wired! And Visualizing Venice: Scaling up Digital Art History.” Artl@s Bulletin 4 (1). https://docs.lib.purdue.edu/artlas/vol4/iss1/3.

Lowe Art Museum. 2019. “Lowe Art Museum | University of Miami.” Lowe Art Museum | University of Miami. https://www.lowe.miami.edu/.

Marques, Diana, and Robert Costello. 2018. “Concerns and Challenges Developing Mobile Augmented Reality Experiences for Museum Exhibitions.” Curator: The Museum Journal 61, no. 4: 541–58. https://doi.org/10.1111/cura.12279.

Mathews, Karen Rose. 2015. “Decorating with Things: Spolia as Material Culture in the Italian Maritime Republics, 1100–1300.” bfo-Journal 1 (2015): 4–13. http://bauforschungonline.ch/sites/default/files/publikationen/mathews.pdf.

Meier, Allison. 2017. “Walk Inside a Gothic Prayer Bead in a VR Experience at the Cloisters.” Hyperallergic. February. https://hyperallergic.com/361000/walk-inside-a-gothic-prayer-bead-at-the-cloisters/.

Moore, Jennifer, Adam Rountrey, Hannah Scates Kettler, and Community Standards for 3D Data Preservation. 2019. “Community Standards for 3D Data Preservation (CS3DP).” Community Standards for 3D Data Preservation (CS3DP). http://gis.wustl.edu/dgs/cs3dp/.

Nancarrow, Jane-Heloise. 2016. “Democratizing the Digital Collection: New Players and New Pedagogies in Three-Dimensional Cultural Heritage.” Museum Worlds 4, no. 1: 63–77. https://doi.org/10.3167/armw.2016.040106.

———. 2017. “Emotions3D: Remediating the Digital Museum.” In The Routledge Research Companion to Digital Medieval Literature, edited by Jennifer Boyle and Helen J. Burgess, 199-211. London: Routledge. https://doi.org/10.4324/9781315696041-14.

Needleman, Sarah E. 2018. “Virtual Reality, Now With the Sense of Touch.” Wall Street Journal, April 3, sec. Life. https://www.wsj.com/articles/virtual-reality-now-with-the-sense-of-touch-1522764377.

Outhwaite, William, and Stephen Turner, eds. 2007. The SAGE Handbook of Social Science Methodology. London: Sage Publications.

Pollalis, Christina, Elizabeth Joanna Minor, Lauren Westendorf, Whitney Fahnbulleh, Isabella Virgilio, Andrew L. Kun, and Orit Shaer. 2018. “Evaluating Learning with Tangible and Virtual Representations of Archaeological Artifacts.” In Proceedings of the Twelfth International Conference on Tangible, Embedded, and Embodied Interaction, 626–637. TEI ’18. New York, NY, USA: ACM. https://doi.org/10.1145/3173225.3173260.

Rossi, Vince, Jonathan Blundell, and Ralph Wiedemeier. 2019. “The Smithsonian Open Source 3D Pipeline – from Preservation and Processing to Authoring and Delivery.” Presented at 2+3D Photography-Practice and Prophecies 2019, Rijksmuseum, Amsterdam, May 10. https://docs.google.com/presentation/d/15dfHl-Cij89nZvSy47_FOmFFWD8OIc7TvOCr_DZ8yLc/edit?usp=sharing.

Saunders, Sarah. 2017. “Students Use 3D Printing Technology to Interact with Ancient Greek Artifacts at Victoria University of Wellington’s Classics Museum.” 3DPrint.Com / The Voice of 3D Printing / Additive Manufacturing. July 31. https://3dprint.com/182487/3d-printed-greek-amphora/.

Schofield, Guy, Gareth Beale, Nicole Beale, Martin Fell, Dawn Hadley, Jonathan Hook, Damian Murphy, Julian Richards, and Lewis Thresh. 2018. “Viking VR: Designing a Virtual Reality Experience for a Museum.” In Proceedings of the 2018 Designing Interactive Systems Conference 2018 – DIS ’18, 805–15. Hong Kong: ACM Press. https://doi.org/10.1145/3196709.3196714.

Sportun, Samantha. 2014. “The Future Landscape of 3D in Museums.” In The Multisensory Museum: Cross-Disciplinary Perspectives on Touch, Sound, Smell, Memory, and Space, edited by Nina Levent and Alvaro Pascual-Leone, 273–79. Lanham, MD: Rowman & Littlefield Publishers. http://ebookcentral.proquest.com/lib/miami/detail.action?docID=1648843.

Tilley, Christopher, Webb Keane, Susanne Küchler, and Michael Rowlands. 2006. Handbook of Material Culture. London: Sage Publications. https://doi.org/10.4135/9781848607972.

Totten, Christopher W. 2014. An Architectural Approach to Level Design. Boca Raton: CRC Press.

University of Miami Libraries and Lowe Art Museum. 2019. “The Andrew W. Mellon CREATE Grants Program | University of Miami Libraries and Lowe Art Museum.” https://www.library.miami.edu/create/.

Victoria University of Wellington. 2017. “3D Printing Technologies Transport Students to Ancient Greece.” Victoria University of Wellington. July 31. https://www.victoria.ac.nz/news/2017/07/3d-printing-technologies-transport-students-to-ancient-greece.

Williams, Tasia L. 2017. “More than Just a Novelty? Museum Visitor Interactions with 3D Printed Artifacts.” MA Thesis, University of Washington. https://digital.lib.washington.edu:443/researchworks/handle/1773/39772.

Wilson, Paul F., Janet Stott, Jason M. Warnett, Alex Attridge, M. Paul Smith, and Mark A. Williams. 2018. “Museum Visitor Preference for the Physical Properties of 3D Printed Replicas.” Journal of Cultural Heritage 32 (July): 176–85. https://doi.org/10.1016/j.culher.2018.02.002.

Wired! Lab. 2019. “Courses.” Wired! Lab | Digital Art History & Visual Culture. Accessed October 7. http://www.dukewired.org/courses/.

Younan, Sarah, and Cathy Treadaway. 2015. “Digital 3D Models of Heritage Artefacts: Towards a Digital Dream Space.” Digital Applications in Archaeology and Cultural Heritage 2, no. 4: 240–47. https://doi.org/10.1016/j.daach.2015.11.001.

Acknowledgments

The authors would like to acknowledge the generous support of the CREATE grant from the Mellon Foundation for the implementation of this project. They would like to thank the Lowe Art Museum, the University of Miami College of Arts and Sciences, School of Communication, the College of Engineering, the University of Miami Libraries, and the Academic Technologies unit. In addition to these institutional partners, the authors would like to recognize a number of individuals who contributed to the success of this project: Diana Arboleda, Ching-Hua Chuan, Christina Larson, Paige Morgan, Kojo Opuni, Michaela Senior, and Han Tran. Finally, this project would not have come to fruition without the creativity and hard work of the University of Miami students, undergraduate and graduate, who participated in ARH 333P/CLA 226P, CIM 616, and CIM 624 in the Spring Semester of 2019. The photos of antiquities used in this paper were captured by the undergraduate students; photos of 3D printed reconstructions were taken by T.J. Lievonen.

About the Authors

Karen Mathews is an Associate Professor in the Department of Art and Art History at the University of Miami. She specializes in ancient, medieval, and Islamic art and has taught a number of courses that integrate photogrammetry and 3D modeling into the art history curriculum. She is currently working on several class-based projects that explore the presentation of art historical content in AR and immersive VR platforms.

Gemma Henderson is a Senior Instructional Designer on the Learning Innovation and Faculty Engagement Team at the University of Miami. Gemma partners and consults with faculty, academic units, and other university stakeholders on curriculum development and digital pedagogies. On behalf of the Academic Technologies unit, she primarily engages in institution-wide educational outreach to support innovation in undergraduate and graduate courses, including initiatives such as faculty learning communities, educational scholarship, and the university’s annual teaching and learning conference.

Read More

Skip to toolbar