A roadmap for implementing VR-based teaching courses in archaeology

​​​​• Title and abstract

o Does the title clearly reflect the content of the article? Yes
o Does the abstract present the main findings of the study? Yes

• Introduction
o Are the research questions/hypotheses/predictions clearly presented? Yes
o Does the introduction build on relevant research in the field? Yes

• Materials and methods
o Are the methods and analyses sufficiently detailed to allow replication by other researchers? Yes
o Are the methods and statistical analyses appropriate and well described? Yes

• Results
o In the case of negative results, is there a statistical power analysis (or an adequate Bayesian analysis or equivalence testing)? n/a
o Are the results described and interpreted correctly? Yes

• Discussion
o Have the authors appropriately emphasized the strengths and limitations of their study/theory/methods/argument? No (see recommendations below)
o Are the conclusions adequately supported by the results (without overstating the implications of the findings)? n/a

This article presents an overview of a pilot study that will provide an opportunity to test the relative merit of implementing VR immersive experiences in college classrooms. The authors plan to use 360-video, rather than synthetic experiences built in VR, to simulate experiences in international study abroad settings, thus combatting problems of accessibility for economically differentiated students and non-traditional learners with financial obligations that do not facilitate enrollment in expensive study abroad programs. The article is well written and conceived.

Additionally, the authors touch upon several important benefits and limitations of implementing VR in the college classroom, positing that the experience outlined in the article will be nearly if not equally as engaging in terms of content and delivery, despite absence of tactility.
Some points that warrant further consideration and clarification are as follows:
1. The authors delve into the pedagogical rationale behind VR implementation in the college classroom, but the course has not yet been implemented. Will headsets be provided to online users taking the course remotely? Will they be made available only in a university setting?

2. How will the program be made a learning tool? Are their learning objectives associated with the experience so that it can be deployed in other university classrooms? Will learning be assessed in a way that parallels classroom instruction? How will leaders and/or instructors address questions from students? These characteristics of the study abroad program should be clarified in the text, particularly when one of the contemporary critiques of VR implementation in in-person classrooms is how such technologies can be transformed into substantive learning tools. This could be included in the form of a table.

3. How many students will benefit from the experience? Although high-enrollment courses are mentioned in the introductory section, it is unclear if the experience will be deployed in high-enrollment classes or if this is only the pool from which data will be drawn. This should be clarified early in the text

4. Given that one of the aims of this project is to make the study abroad program more accessible, it is important to recognize that students who cannot normally afford traditional study abroad experiences may have financial challenges like limited or poor internet connectivity and inability to purchase personal headsets. How will the authors address problems of bandwidth and equipment when deploying the experience? This should be clarified.

In general, I find the article a strong contribution to our work as archaeologists to bring the past to life in new and exciting ways.

Summary
An interesting paper that addresses a gap in the literature on the longevity of benefits resulting from VR displays, and justifies the project on the basis of diversity and opportunity. The literature review does an excellent job covering the existing research, identifying key ideas and themes in the use of VR in cultural heritage and education. A good amount of detail is provided on the nature of the VR tour and how it was created, which will be helpful for readers to contextualise how similar projects can be developed.

As the paper discusses a project that is still in progress and has yet to be fully implemented and evaluated, there is little in the way of tangible data analysis or hypotheses. This isn’t necessarily a problem, but I think the paper would be much improved through a clearer discussion of the intended hypotheses for the eventual evaluation studies, and how they relate to the research questions. Several frameworks and metrics are discussed in the literature review and discussion, which will be the focus of the authors’ evaluation study for this project? The only clearly defined planned metric is ‘the degree to which students opt for the full-immersion headset version of the course’, which I think could be influenced by a number of external factors and is not, by itself, a clear indicator of student engagement. Learning outcomes are mentioned several times, but I’m uncertain how these are planned to be evaluated and would like to know - even accepting that this plan may change as the evaluation draws nearer. It would also help to acknowledge in the abstract that the tour implementation and evaluation is only planned, and has yet to be undertaken.

Alternatively, if the focus for this article is on the VR tour itself, rather than the future evaluation and its impacts on student learning and engagement, then it would help to have additional details available. How, specifically, is the tour supporting intended learning outcomes? More examples of the viewpoints, or perhaps some snippets from the narration, or examples of interactive elements could help.

The article is clear and well-written, but there are a few structural changes I think could help. The discussion of diversity seems to be an important part of the project, but is not mentioned in either the introduction or abstract, and so feels out of place. I think signposting early that this is a significant motivation of the project will help the paper feel more cohesive. The title of the paper makes it seem as though this will be a broad discussion of a variety of ‘immersive’ approaches to archaeological education, but really the paper seems exclusively focused on VR, which is a bit misleading. Something like ‘Virtual Reality Tours as an Immersive Approach to Archaeology in Higher Education’ (although a bit wordy) would I think be more representative of the content

Questions answered no:

• Does the title clearly reflect the content of the article?
  • Would be improved with more specificity
• Does the abstract present the supported findings of the study concerned and no other?
  • Would be improved by noting that it is a yet to be implemented pilot study, and include a discussion of diversity,.
• Is the research question/hypothesis/prediction clearly presented?
  • ​It would be good to know more clearly and specifically what metrics are considered for success. How are engagement and enhanced learning experience to be determined? It would also help to express more clearly upfront that the VR experience is yet to be trialed, and that this article is discussion one that is still in development. 
• ​Are the methods and analysis described in sufficient detail to allow replication by other researchers?
  • ​The article provides excellent detail on the production method for the VR tours and how they will be used in the classroom, but little detail on how the success of its implementation will be determined. It would improve the article to provide more tangible discussion of what kinds of quantitative and qualitative analysis is planned.​
• Are the statistical analyses appropriate?
  • ​The discussion of diversity is interesting and valuable, but I think should have been introduced in the abstract and introduction as a focal point.​

This is a robust, up-to-date, well-written and pleasantly short paper on integration of immersive technologies, specifically Virtual Reality (VR), in higher education archaeology courses. I only spotted few points that can be improved:
1. Add call of figures (Fig. 1, Fig. 2).
2. In the Fig. 1 caption, mention explicitly that the sequence of numbers (1, 2, 3, etc.) refers to the location, or stations, of the mini-lectures
3. In the Fig. 2, mention where the photo cames from (Athens Agora?, station 1?)
4. The DOI of the "Nobles, G., Çakirlar, C., Svetachov.." reference is not properly written. Identically, the Nobles et al. (2019) statement on VR and AR as "pretty pictures" could be discussed a little bit further.