Three levels of reproducible workflow remove barriers for archaeologists and increase accessibility

This is a very useful article setting out accessible routes to reproducible research in archaeology. The key merits of this article are the setting out of definitions of, barriers to, and solutions for  reproducible research in archaeology in an accessible, richly resourced and easily understandable format. The Appendix and FAQs are a rich resource for those looking to adopt open science practices in their research, and the profusion of hyperlinks are welcome. The article text itself is at times a touch brief in alluding to much bigger debates. The high-level review style aids accessibility, but there is perhaps a lack of engagement in longer term debates about the structure and philosophy of archaeological research, and initiatives around databases, data-sharing and LOD.  I have suggested some small additions of archaeological examples to keep the archaeological reader engaged, and I consider that the richness of information and clarity of the article make it a beneficial and useful article for many archaeologists.  The article is well cited, and links to a range of research products. The figures and tables, language and structure are all clear. My comments below relate to minor rewording to ensure clarity, and the addition of more archaeological examples to aid the goal of the article.

Abstract

-          Minor language point in sentence 3 – are these barriers or requirements? If using barriers, as in the title, then I would suggest rewording this list as the barrier ‘skill level of researchers’ or ‘software and infrastructure availability’.

Introduction

-          This section clearly outlines the timeliness of the article.

-          Some more content on the motivation for this paper would be useful.

-          I would suggest adding a ‘hook’ on why reproducibility for archaeology matters – why should an archaeologist invest the time to read this article – what will making their research more reproducible do for them? Some of these aspects are raised later in the article E.g. archaeology as unique observations, but I would suggest including a few here to keep the readers interest. I would suggest considering whether the profusion of micro-specialisms with distinct methodologies and data standards, plus the blurred disciplinary position may be other reasons why reproducible practices would be of benefit to archaeologists.

-          I am supportive of this small step approach outlined in this article – could the authors give an example of another discipline or field where such small step individual learning approaches have worked? It would be useful to briefly mention other pathways towards reproducibility, such as stringent peer reviewing of study reproducibility by journals, or inclusion of reproducibility within under- and post-graduate training.

Why is reproducible archaeology important?

-          Paragraph 4 – I would question why rescue or commercial work alone are highlighted for their time limited nature – Research excavations are also time-limited and destructive.

-          Paragraph 5 – The authors could also consider money limitations for scientific analysis such as machine time, consumables, technician time etc.

-          Line 177 sentence beginning “Consequently…” – I suggest a clarification of this sentence– make clear that you are calling for reproducible research to enable reassessment from the point of the dataset, rather than the point of the material assemblage.

-          Para beginning line 180 – I’m unclear what Transparent recording is referring to – is this the recording of the original data,  or of the analysis undertaken? An example here would be useful.

-          Spelling out the CARE principles would be useful here – FAIR principles now appear fairly widely known and used in archaeology, less so with the CARE principles. I see you do this in more detail below, I would briefly allude to them in this section.

What does reproducible research look like?

-          Line 216 , the sentence beginning “Large meta-analysis” – I suggest rewording to improve clarity. Is it that both the large meta-analysis studies and studies that want to reuse the same methods need computational reproducibility?

-          Para beginning Line 224- perhaps introduce the concept that the data code and methods are files in the previous paragraph.

-          Line 258 – can you give examples of proprietary software – SPSS, Excel etc.

-          Table 1 is useful – could you add some more examples dealing with different forms of archaeological data? I think they would be very useful to readers.

-          Line 296 – perhaps highlight that the Analysis output file is different to a graphical depiction of these analysis results, which is the much more ubiquitous feature of archaeological publications.

-          Line 329 – Again, some archaeological examples of research compendium would be really useful here.

-          Line 447 – with regards to sensitive location data, the PAS in the UK provides a good example of dealing with sensitive find spot location (https://finds.org.uk/help/database/topic/id/10

-          Line 462 – Another scenario which you might want to consider is when researchers are using analysis code shared within a lab group – it may be written in R, but developed by someone else, and passed on between researchers, so not the researchers to share.

-          Line 472 – I would suggest mentioning citation of data sets as an encouragement to data sharing.

5. Join a community or association 

-          I would suggest the addition of the   UK Reproducibility Network, which has local nodes across the UK https://www.ukrn.org/

Conclusion

Currently, this reads like the end of a help document – I would suggest reiterating points from the first half of the article on the importance of, and barriers to, the current adoption of reproducible research in archaeology.

The FAQ is very useful, especially the section on preregistration.

This article is a fantastic piece which explains how and why reproducible and open research
in archaeology is attainable and important for moving the discipline forward. I hope there
are other follow-up pieces in the works to provide more detailed information for
researchers on specific aspects of mentioned within. I really enjoyed reading this piece and
its detailed appendices, it is a long overdue article for our field, and will have a lasting
resonance across all the archaeological sub-disciplines. I will certainly be adding it to my
student resource lists and taking on board many of the suggestions and resources suggested
in here for my own research.

Overall, I thoroughly recommend this paper for publication, it’s extremely well written,
timely, covers important ground about the future of open workflows in our discipline and
will serve and a foundational guide moving forward. It has lots of great resources and
examples which are extremely helpful to researchers at various stages of their
reproducibility and open workflow journeys, and answers many frequently asked questions
from colleagues I’ve seen time and again. Below are my more detailed comments and
suggestions, all of which are minor.

The extensive appendices are to be particularly commended as they detail commonly used
terms in Open Science and reproducibility literature which are not accessible to the
uninitiated. The glossary and Q&A sections are especially helpful, however as a someone
who is familiar with many of these terms I’m not sure I am best placed to suggest other
areas which other colleagues might want answered or definitions of, so I hope people do
interact with this Open format and leave queries for the authors to keep the conversation
going.

There are a few suggestions I have for additions to the glossary – binders and containers I
think could use a little further explanation for those who aren’t familiar with coding and that
side of software. Along with R and Python, Git/Github should also warrant an entry as
they’re mentioned in text for version control. ‘Protocol’ as a term should also be added,
again I think those in more wet lab based archaeological science will be familiar, but it is a
term less familiar to others who likely use their own protocols but might not label them as
such. I would also recommend that the glossary be more heavily signposted from the start
of the article as it is a great key resource you’ve created and needs to be flagged early on for
reference throughout.

Figure 1 whilst nice I think isn’t strictly necessary and doesn’t add substantially to the text so
could be left out.

The differences between replication, reproducibility, robustness, and generalisability are
explained well in text but could do with further clarity in Figure 2, especially generalisable as
in reality if you got completely different results with different data and different analysis (as
shown in Figure 2), it is unlikely you would come up with the same generalizations about a
phenomenon. Similarly robust could do with clarification in Fig 2 for the same reasons. But
again I stress this is made clear in text, but to limit misinterpretation on reuse of Figure 2 I’d
edit it slightly.

Figure 3 needs an explicit citation or source link.

I suggest adding a little more detail around lines 131-133 about the differences described,
and the Marwick 2020b reference does not align with how that paper is listed in the
bibliography, please cross-check all references before publication as I think a few others
with “a” and “b” in text have also slipped through.

The definition of the discipline on line 136, particularly as a scientific study is hotly debated,
and I worry that defining it as only that may not remove barriers to all archaeologists
working more openly but could put some back up, especially for those who see working
reproducibly and openly as something for those on the more laboratory-based side of
things. Problematising this more or offering up a broader definition of the field is advised.

The point about validation in line 199-200 is such an important and poignant one, that I feel
warrants further explanation or signposting to a future article, you left me wanting more
detail here, and I definitely agree with you that it extremely important and underrated.

Where you discuss large meta-analyses in lines c. 215-220 it would be great to have
examples of these kinds of studies like you do for other parts of reproducible research later
in the article to showcase how data can be collated and re-used.

Table 1 and the steps from line 280 onwards with the examples are fantastic! I particularly
like the inclusion of various freely available and easy to use websites and software such as
Google Docs alongside the more advanced tools. With the raw data file formats, csv is the
only option mentioned in method 1 – perhaps include recommendations for non-tabulated
data such as image files, and best practice/formats for other data types. Another suggestion
to perhaps improve the uptake of your steps for reproducibility would be to expand Table 1
and parts of the text into a flowchart, checklist, or template(s) of steps for researchers to
follow as a workflow as an accessible route for helping to set up either a project from the
beginning or ensure your work meets as many steps as possible while preparing for
publication.

The “confronting your barriers” section is a great idea and I really commend the authors for
this section, especially the inclusion and consideration of CARE principles alongside the
more widely touted FAIR principles. Whilst I love the idea of using synthetic data suggested
in lines 434-441 I think this is a huge conceptual and training hurdle for many, and so might
not be very accessible or executable in a lot of cases. Therefore, any additional resources
the authors can point people to here would be a great help for how to go about creating
synthetic data that meets replicability criteria.

I would also suggest adding “RLadies” to the list of communities and associations in lines
518-528.

In the appendix under the metadata standards, I would also include those widely used in the
journal “Ecology” which have been widely applied in aspects of environmental archaeology,
especially isotopic data collation - Michener, William K., James W. Brunt, John J. Helly, Thomas B. Kirchner, and Susan G. Stafford. ‘Nongeospatial Metadata for the Ecological
Sciences’. Ecological Applications 7, no. 1 (1997): 330–42. https://doi.org/10.1890/1051-
0761(1997)007[0330:NMFTES]2.0.CO;2. Also here: https://www.esa.org/wpcontent/
uploads/2022/05/ESA-Data-Paper-Guidelines.pdf

The pre-registration section is such a welcome addition – thank you! I think this is
something we really do not take advantage of enough in archaeology.
In the section “my supervisor won’t let me work reproducibly…” I’d also add in links to the
Wellcome Trust funding guidelines on working openly here as they are a fantastic resource
with lots of signposting to further resources. To make this more globally inclusive perhaps
also link to other research councils like those in Australia (https://www.arc.gov.au/aboutarc/
program-policies/open-access-policy), the NSF in the United States
(https://www.nsf.gov/pubs/2016/nsf16009/nsf16009.jsp#q1https://www.nsf.gov/pubs/2016/nsf16009/nsf16009.jsp#q1) and this policy draft for India
(https://openaccessindia.org/national-open-access-policy-of-india-draft-ver-3/https://openaccessindia.org/national-open-access-policy-of-india-draft-ver-3/).

My final suggestion would be that your last sentence referring readers to the Kansa et al.
paper could be duplicated earlier in the main text and more frequently – I know you cite it
regularly as it’s such a key paper but hammering it home as you do here earlier would be
beneficial as some people may not look at the very end of the appendices.

Thank you for all of your hard work and dedication compiling this fantastic paper and its
thorough appendices!

​​​Karoune and Plomp present an insightful, functional, and significant discussion of reproducible research in archaeology within this manuscript. Their stated goal, to “introduce reproducible research in an understandable manner so that archaeologists can learn where and how to start improving the reproducibility of their research,” is certainly achieved with examples, workflows, clear definitions, and more. A caveat to my review: I do not consider myself an expert in the philosophy of reproducibility in archaeology, I see myself instead as a practitioner of my own niche understanding of how best to “do” reproducible research (e.g., https://github.com/cylerc). I share this because the strength of this manuscript is that archaeologists who are new, or experienced, practitioners in reproducibility will undoubtedly find helpful guidance within Karoune’s and Plomp’s manuscript – I certainly have, and I thank the authors for their exhaustive efforts in making these concepts accessible to us all within archaeology (and perhaps elsewhere!). Readers will take away an appreciation for why reproducibility matters and how to accomplish reproducible research.

The manuscript includes an introduction with key background (i.e., what is reproducibility, why is it important, etc.) and excellent figures/illustrations. In fact, the illustrations throughout this entire manuscript are impressive. The authors clearly understand the necessary intersection between text and visual aids for the greatest dissemination of concepts (reproducibility of ideas within a reproducibility manuscript). Karoune and Plomp then provide three examples of reproducible workflows, and discussions on barriers in reproducibility, training, resources, definitions, and an appendix of frequently asked questions and additional resources.

I have virtual no substantive constructive comments on the manuscript in its current form. As I previously mentioned, I gained new insights into the process (and capabilities) of reproducibility in archaeology while conducting this review. There are only a few minor thoughts that the authors may be interested in considering:

Line 180-189: I certainly agree that we need to move away from the sole ownership of research kept on our local computers that only benefit ourselves and a few other researchers. That seems to be a critical aspect of reproducibility, or really the success of reproducibility in archaeology. This also brought up an issue that I think about often which is the ability to share and practice reproducibility within non-academic (not a great term for this distinction) settings. I’m thinking particularly about State/Federal/Private agency and company archaeological records. For example, how do archaeologists practice reproducibility in their work when their research occurs within an agency that is consulting with Indigenous Nations, States, Federal agencies, or some combination of the above? Their research – regardless of the scale – still contributes to the archaeological body of knowledge, but there may be regulatory or proprietary reasons that data, concepts, results, even ideas, cannot be shared. There does not seem to be an easy way to currently manage this “grey” literature and reproducibility framework within archaeology except on a case-by-case basis. tDAR is a logical location where these types of data are currently curated, but I suspect a struggle is still the ability to have transparent reproducibility (e.g., tDAR might curate a record without easy access to the record itself). This is all just something to consider. It is an ongoing challenge.

Line 224-235: This is a key section, and I would recommend adding in a brief mention of the ability to use university repositories in addition to Zenodo, OSF, Figshare, and more. I recognize
that in some cases university repositories have limitations, and that these limitations may also be present in non-university repositories. For example, long-term preservation of digital data, curation of servers, etc., but I suspect that in some cases archaeologists beginning their reproducibility journey might find helpful resources and support within their university system. Arizona State University has a non-exhaustive list of some university open access repositories, here: https://libguides.asu.edu/openaccessresources/repositorieshttps://libguides.asu.edu/openaccessresources/repositories

A final note relates to the concept of “full reproducibility” (Line ~212, Figure 4) in archaeology. Something that I have found disconcerting in our modern world of archaeology is exactly how research projects are created. I think this is as critical to research as how research projects occur in reproducible systems. This perhaps relates to “ideas”, “discussions”, “consultation”, “questions” under the research iceberg. I wish there were mechanisms in place to exhaustively document how research begins – or a sense of self-reporting responsibility to document how research began. Did a published paper spur a new idea? A tweet? A conference presentation? A conversation at a conference? An overheard conversation at a conference? You can see where I’m going with this…there are so many possible sources of inspiration for research in archaeology, and an equally large number of routes to accomplish that research (e.g., fully funded, and transparent research proposals vs. behind-the-scenes lab analyses through colleagues, etc.) but in very few instances are those processes made entirely clear in final products. I hope that as reproducibility in archaeology continues to evolve and take prominence that there will be a shift towards making the formation of research projects transparent and open as well. To me, this would create the ultimate cycle of reproducibility by identifying the underground roots of projects, not just the tree that grows once the roots are established (using a similar metaphor to the research iceberg). However, I’m also aware that once open, transparent, and reproducible archaeology begins to approach this aspect of the research process, there will likely be a needed evaluation of how archaeological ethics facilitate (or not) those specific research projects.

This well-written manuscript provides a convincing narrative of the importance of reproducible archaeological research and a brief how-to guide for beginners in the appendix. In the meantime, it also presents a balanced and fair view of the ethical concerns behind data sharing. In short, I highly recommend this paper to be published in a peer-reviewed venue in the future and hope it can persuade more archaeologists to value and practice open and reproducible research. Below are just a few minor comments to be addressed.

Minor Comments:

line 400: Mukurtu (https://mukurtu.org/about/) is a possible data management and sharing platform when working with indigenous communities. Their code is fully available on GitHub but setting up a server might involve some cost. One example using the Murkurtu platform is the Rowasu’u project (https://rowasuu.org/about).

line 472: Related to the scooping issue, it is very common for archaeologists to claim “data available on request” at the end of the manuscript to get away because of the mandatory data availability statement of journals. However, according to a recent study conducted by Gabelica et al. (2022), only the authors of 122 (6.8%) out of 1792 manuscripts actually responded to their request for data sharing and provided the corresponding data. I think this is a strong piece of evidence of why “data available on request” is not enough.

line 483: “more easy” should be “easier”

line 495: My personal recommendation here is a Coursera course called Reproducible Templates for Analysis and Dissemination (https://www.coursera.org/learn/reproducible-templates-analysis), covering the basics of Git and Rmarkdown. Everyone can enroll in the course for free, but the exercise, which is rather unnecessary, requires payment.

line 541: One more thing I would recommend here is R style guides like Google’s R Style Guide (https://google.github.io/styleguide/Rguide.html) and ISAAK's R Style Guide (https://gitlab.com/ISAAKiel/StyleGuide). Style guide can be useful, particularly for beginners,
because one psychological barrier to reproducible research is that people are worrying if their codes are too ad-hoc, messy, and inconsistent to be shared with anyone. These style guides can help beginners foster good habits of programming. Some other archaeology-related resources include general archaeological sciences using R (https://benmarwick.github.io/How-To-Do-Archaeological-Science-Using-R/), archaeological network analysis (https://book.archnetworks.net/index.html), and Marwick’s compiled list of archaeological papers including R codes (https://github.com/benmarwick/ctv-archaeology).

line 810: Maybe also consider adding the diamond open access in the glossary as this concept is mentioned in the article.

Line 960: Within the question “How do I clean up the data and code before sharing this publicly”, I would suggest that the author can use one paragraph to first address the psychological barrier here that people are too ashamed to share their messy codes or afraid of potential criticism (https://www.computerworld.com/article/2833340/4-reasons-developers-are-scared-of-making-their-code-public.html). It is also important to emphasize that even researchers who have several years of programming experience will constantly seek help on Stack Overflow or similar platforms.

Line 1012: Language and package version should also be explicitly described. This is becoming increasingly important in R. The curse of the great ecology of R (many very specialized and ready-to-use packages) is that heavy package dependence of new packages makes them highly unstable. Maybe one tiny update of a dependent package will cause the dysfunction of the new package. Also, individual researchers who developed those small packages tend not to maintain them in the long term. For this reason, several researchers I knew started to use base R as much as possible or move to new languages like Julia. Perkel (2020) covered the irreproducibility of codes written years ago in a recent news piece.

line 1142: Another example of the diamond open access journal in archaeology would be PaleoAnthropology (https://paleoanthropology.org/ojs/index.php/paleo/index).

line 1225: Perhaps the authors can mention that there are specialized venues for publishing data like the Journal of Open Archaeology Data and Scientific Data. The former is designed for archaeologists, while the latter is a Nature portfolio journal that also accepts archaeological
datasets like p3k14c (Bird et al. 2022) and SignBase (Dutkiewicz et al. 2020). As a side note, although impact factor is known as a highly problematic metric for research evaluation if exercised without caution, particularly within the open science framework, for those who do care about impact factor because of their university policy, Scientific Data actually has a higher impact factor (6.444) than Scientific Reports (4.379) or any archaeology journals. This number to some extent shows that publishing data is a behavior appreciated by the research community and can have a direct benefit to the authors.

Figure 1 and Figure S2: these two figures are not particularly informative.

References:
Bird, D., Miranda, L., Vander Linden, M., Robinson, E., Bocinsky, R. K., Nicholson, C., ... & Freeman, J. (2022). p3k14c, a synthetic global database of archaeological radiocarbon dates. Scientific Data, 9(1), 1-19.

Dutkiewicz, E., Russo, G., Lee, S., & Bentz, C. (2020). SignBase, a collection of geometric signs on mobile objects in the Paleolithic. Scientific data, 7(1), 1-14.

Gabelica, M., Bojčić, R., & Puljak, L. (2022). Many researchers were not compliant with their published data sharing statement: mixed-methods study. Journal of Clinical Epidemiology.

Perkel, J. M. (2020). Challenge to scientists: does your ten-year-old code still run?. Nature, 584(7822), 656-659.