Open source code enhances the reproducibility and impact of academic papers.

Bio-medical Big Data Operating System (Bio-OS): An Integrated Data Mining Environment for Data Intensive Scientific Research. 2024. https://doi.org/10.1101/2024.10.17.612997

Abstract The advent of high throughput sequencing has ushered life science and clinical research into the era of big data, posing significant challenges for reproducibility due to the complexity of data integration and analysis. Although the FAIR principles advocate for the transparent and reliable sharing of scientific data, their implementation remains hampered by technical barriers. The Global Alliance for Genomics and Health (GA4GH) has made strides in standardizing data and tools, yet a comprehensive solution for reproducibility is lacking. In response, we present BioOS, an open source, cloud native Biomedical big data Operating System. This system encapsulates study components data, code, tools, and environments into workspaces, enhancing reproducibility and validation. BioOS employs JSON Schema for machine readability and includes a Hierarchy Hash Mechanism to ensure data integrity. Adhering to GA4GH protocols, BioOS simplifies complex technological implementations, making advanced research tools accessible. Demonstrated through representative workspaces, BioOS fosters seamless research replication, peer review, and editorial evaluation. Its cloud native infrastructure supports dynamic resource allocation, enabling efficient handling of large scale analyses. By integrating AI driven Large Language Models, BioOS enhances user interaction and operational flexibility. As an evolving open source platform, BioOS exemplifies a transformative approach to biomedical research,

On the open-source landscape of PLOS Computational Biology. 2021. https://doi.org/10.1371/journal.pcbi.1008725

Over the past year, I (M.B.) have been investigating the landscape of code-sharing in academic journals across different research fields. At the end of my PhD, I made the choice to share code that reproduces figures from one of my papers [1], and since then, I’ve been involved in several open-source projects (qMRLab and AxonDeepSeg) and initiatives dealing with open science in publishing (NeuroLibre and Canadian Open Neuroscience Platform). Recently, following an editorial by N.S. on reproducibility and the future of MRI research [2], we wrote a blog post presenting an analysis of the open-source landscape for the journal Magnetic Resonance in Medicine (MRM), which broadly focuses on MRI research for medical applications. These findings provided a snapshot of the current state of the open-source landscape for that journal (e.g., most used coding language is still MATLAB) and some insights into new trends (12% of the articles shared code that reproduced figures). In this editorial, we examine the open-source landscape of PLOS Computational Biology. PLOS Computational Biology is inherently different from MRM not only because of the difference in research topics, but also because it’s an openaccess journal that focuses primarily on computational studies. The broad questions that were of interest are the following:

Nine quick tips for open meta-analyses. 2024. https://doi.org/10.1371/journal.pcbi.1012252

Open science principles are revolutionizing the transparency, reproducibility, and accessibility of research. Meta-analysis has become a key technique for synthesizing data across studies in a principled way; however, its impact is contingent on adherence to open science practices. Here, we outline 9 quick tips for open meta-analyses, aimed at guiding researchers to maximize the reach and utility of their findings. We advocate for outlining preregistering clear protocols, opting for open tools and software, and the use of version control systems to ensure transparency and facilitate collaborat

CyberballOS: An Updated, Easily Implemented, Open-Source Virtual Ball-Toss Game to Study Social Interactions. 2026. https://doi.org/10.1177/25152459261437994

Researchers continually face challenges finding ways to test the fundamentals of psychology: affect, behavior, and cognition. Since 2000, researchers have used a universal tool called Cyberball to investigate these outcomes. Cyberball is a virtual ball-toss game played with computer-controlled players. We updated the Cyberball paradigm, creating CyberballOS (Open Source), to make it require no special resources, have easy-to-set-up games, and record behavioral data. In addition, we developed CyberballOS as open-source software, making its code transparent, accessible, and extendable, which enhances reproducibility of findings and enables limitless program modifications. In this tutorial, we provide an overview of CyberballOS and instructions on how to create a new game, how to load a previous game, and where to find detailed help. We illustrate the process, step by step, for configuring a game (e.g., characteristics of the players, who the computer-controlled players throw to) for both typical use cases (with ready-to-go presets) and newer, more advanced CyberballOS features (e.g., the participant and other players being able to leave). To implement CyberballOS easily, we incorporated a critical feature: the ability to integrate CyberballOS into the popular online survey platform, Qualtrics (including collecting gameplay data). To demonstrate CyberballOS’s utility and how to use its features, we highlight three example studies based on research from developmental, social, and

An Analysis of Retrieval-Augmented Generation: A Systematic Review Addressing Architectures, Components, and Evaluation. 2026. https://doi.org/10.34808/tq2025/29.3/a

Retrieval-Augmented Generation (RAG) enhances Large Language Models (LLMs) by integrating external retrieval mechanisms to improve factuality and currency. This systematic literature review characterizes current RAG architectures, components, and evaluation practices in peer-reviewed studies published between 2021 and 2025 across IEEE Xplore, Scopus, and Web of Science. Conducted in accordance with the PRISMA guidelines, this review analyzes 41 studies that met the predefined inclusion criteria. Most research addresses Question Answering (QA) and dialogue systems, employing diverse encoders and retrieval optimization methods. Key findings reveal a strong trend toward integrating OpenAI’s GPT models, alongside growing adoption of open-source alternatives. Persistent challenges include hallucination control, computational efficiency, and inconsistent evaluation metrics. Despite the potential of RAG, the evidence base is limited by a focus on English-language, high-resource domains. Furthermore, reproducibility is constrained by heterogeneous evaluation standards and a lack of open-access code or datasets. This review maps the RAG research landscape and identifies gaps in standardization, scalability, and application to low-resource languages. The protocol was not prospectively registered, and no funding was received for this work.

Revisiting Vul-RAG: Reproducibility and Replicability of RAG-based Vulnerability Detection with Open-Weight Models. 2026. https://doi.org/10.48550/arxiv.2606.04739

Large language models (LLMs) have shown strong potential for automated software vulnerability detection, particularly in retrieval-augmented generation (RAG) settings. However, for approaches relying on proprietary models and APIs, reproducibility and replicability remain largely unexplored, raising the question of whether reported results generalize or depend primarily on specific model choices. In this work, we present a reproducibility study of Vul-RAG, a RAG-based framework for source code vulnerability detection that enhances LLMs with high-level vulnerability knowledge. We first replicate the results in a fully local and open-weights setting using the reported open-weight baseline models. We then extend the evaluation to a diverse set of recent open-weight LLMs, including code-specialized, general-purpose, and reasoning models of varying parameter sizes. The results confirm that the findings of Vul-RAG are reproducible under local deployment, but with minor deviations. Across all evaluated models, we observe a performance plateau at approximately 0.30 pairwise accuracy (code pairs for which both the vulnerable and the patched function are correctly classified). Notably, this plateau persists even for more recent and advanced models, indicating that improvements in model capacity alone do not substantially enhance performance. Finally, we discuss practical implications and trade-offs between detection effectiveness, model capabilities, and model scale. Implementation and

Reproducibility Study of "XRec: Large Language Models for Explainable Recommendation". 2025. https://doi.org/10.48550/arxiv.2510.06275

In this study, we reproduced the work done in the paper "XRec: Large Language Models for Explainable Recommendation" by Ma et al. (2024). The original authors introduced XRec, a model-agnostic collaborative instruction-tuning framework that enables large language models (LLMs) to provide users with comprehensive explanations of generated recommendations. Our objective was to replicate the results of the original paper, albeit using Llama 3 as the LLM for evaluation instead of GPT-3.5-turbo. We built on the source code provided by Ma et al. (2024) to achieve our goal. Our work extends the original paper by modifying the input embeddings or deleting the output embeddings of XRec's Mixture of Experts module. Based on our results, XRec effectively generates personalized explanations and its stability is improved by incorporating collaborative information. However, XRec did not consistently outperform all baseline models in every metric. Our extended analysis further highlights the importance of the Mixture of Experts embeddings in shaping the explanation structures, showcasing how collaborative signals interact with language modeling. Through our work, we provide an open-source evaluation implementation that enhances accessibility for researchers and practitioners alike. Our complete code repository can be found at https://github.com/julianbibo/xrec-reproducibility.