What impact is generative AI having on research software engineering and scientific computing, and how can we identify where the real benefits could lie? ICCS Co-director (Computer science) Professor Dominic Orchard outlines the current situation and how GenAI might be explored in future.
The rise of generative AI via pre-trained transformers presents exciting possibilities for increasing productivity in research software engineering (RSE) and scientific computing.
Our experiences at ICCS have shown us that GenAI techniques have the potential for use in test generation, debugging, code review, code generation, documentation, code explanation, optimisation and more. However, by their nature, all such outputs have the potential to be incorrect, undermining the final scientific endeavour.
It is clear that validation and verification continue to be important to guard against error, especially in the case of probabilistically-generated machine outputs. Furthermore, there is a need to separate corporate interest from meaningful data on the effective use of these tools. Our experiences suggest that there are cases where the tools are of considerable help, but other cases in which there is a reduction in productivity (increase in completion time) or possible long-term negative effects on understanding and skills.
Over-reliance may lead to skill atrophy, and self-reported productivity gains may mask a drop in productivity
Early independent studies are mixed: METR’s randomised control trial in 2025 found that developers self-reported an expected reduction in completion time of 24% (on average) using GenAI tools when in fact GenAI increased completion time by 19% [Becker et al. 2025]; some studies show an increase in productivity (decrease in completion time of 50%) for implementing certain kinds of software development tasks from scratch [Peng et al. 2023], or an 12-22% increase in number of PRs completed [Cui et al. 2024]; others found only minor increases in productivity (measuring commit activity) [Stray et al. 2025].
Even Anthropic’s own study in early 2026 showed that study participants using an AI assistant scored 17% lower on a follow-up comprehension test and the productivity gains of this group were not statistically significant: “we find that AI use impairs conceptual understanding, code reading, and debugging abilities, without delivering significant efficiency gains on average” [Shen and Tamkin 2026]. In summary, over-reliance may lead to skill atrophy, and self-reported productivity gains may actually mask an overall drop in productivity.
Yet, generative AI tools are already being adopted in scientific software development, often by students and those with less software experience already [O’Brien 2026]. This increases the need for training and software engineering skills to be taught more widely given that “programmers are at risk of over-reliance on these tools, leading them to accept undetected errors in generated code” [O’Brien 2025].
Possible productivity gains need to be balanced with long-term development and understanding
Within our network of collaborators, some have been making use of LLMs and agent-based development whilst others have tried but struggled to find a benefit. Use of GenAI without scrutiny, validation or verification therefore poses a threat to scientific validity, and even has the potential to erode trust. Possible productivity gains need to be balanced with long-term development and understanding, especially when the science may ultimately be used to inform policy. There is a clear need for user-training and for ‘human-in-the-loop’ workflows combined with analytical tools integrated within agentic workflows to provide validation and verification and, ultimately, preserve scientific integrity.
Our survey says...
As part of a recent internal project to investigate generative AI use in RSE at ICCS (conducted by ICCS Principal RSE Tom Meltzer and Senior RSE Matt Archer), we conducted a survey (following the same rubric as the Stack Overflow 2025 Developer Survey on AI use) to gauge existing internal usage and sentiments towards generative AI. Our RSEs are currently cautious/apprehensive about GenAI integration into our existing workflows. Yet the survey indicates that 80% of ICCS (the whole team – researchers, Directors, operations team and RSEs) are currently using GenAI daily or weekly.
Concerns were raised about the correctness or accuracy of GenAI tools, with a majority distrusting the output of GenAI tools. Our analysis is that this is the primary factor inhibiting adoption. RSEs are rightfully concerned that the accuracy and correctness of climate and weather codes supersedes the potential benefits of GenAI-assisted development. There were also concerns about environmental impact and intellectual property rights issues around the training of some models.
Furthermore, not unexpectedly, the idea of 'vibe' coding was essentially ruled out as viable by almost everyone surveyed, again, due to concerns around correctness but also maintainability. There is potential room to improve here in future, but it relies on better testing and accuracy frameworks being developed around GenAI systems to provide guarantees that then enable greater trust between developers and the LLMs.
On the cautious side of optimism
ICCS RSEs are well-positioned, with a foot in both camps of software development and scientific rigour. Whilst the overall sentiment remains on the cautious side of optimism, we have the technical knowledge and scientific experience in the team to evaluate the impacts of generative AI in research software. Our focus on maintainability, correctness and scientific accuracy will drive our exploration of GenAI tools and workflows. This may take more time than jumping in head-first, but it will allow us to keep our commitments to reproducible and sustainable scientific software.
To investigate these ideas further we have just advertised a new 2-year postdoc position at ICCS on Trustworthy Generative AI for Computational Science, closing on 6 September 2026.
In summary, as a scientific software institute, part of a university, we are seeking to both support the inevitable use of GenAI tooling through training as well as to better understand its effectiveness and effective modes of operation. We are excited by the possibilities and the unique position afforded to us by being embedded in the working practices of many different scientific research groups as well as being part of Schmidt Sciences’ Virtual Institute for Scientific Software (VISS).
Our aim is to meaningfully contribute to the shared vision within VISS, bringing our unique domain focus combined with expertise in software engineering and computer science research, to continue in developing best practice with sufficient evidence, and to continue training scientists and the next generation of climate modellers.
References
Becker, J. et al., (2025), Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity, https://arxiv.org/pdf/2507.09089 (see also the summary article https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/)
Cui, K. Z. et al. (2024), “The Productivity Effects of Generative AI: Evidence from a Field Experiment with GitHub Copilot.” An MIT Exploration of Generative AI, March. https://doi.org/10.21428/e4baedd9.3ad85f1c https://mit-genai.pubpub.org/pub/v5iixksv/release/2
O’Brien, G. (2026), More code, less validation: Risk factors for over-reliance on AI coding tools among scientists https://arxiv.org/abs/2512.19644
O’Brien, G. (2025), Threats to scientific software from over-reliance on AI code assistants. Nat Comput Sci 5, 701–703 (2025). https://doi.org/10.1038/s43588-025-00845-2
Peng, S. et al. (2023), "The impact of AI on developer productivity: Evidence from GitHub Copilot." arXiv preprint arXiv:2302.06590
Stray, V., et al. (2025), Developer Productivity With and Without GitHub Copilot: A Longitudinal Mixed-Methods Case Study. arXiv preprint arXiv:2509.20353
Shen, J. and Tamkin, A. (2026), How AI Impacts Skill Formation, https://arxiv.org/abs/2601.20245 (https://www.anthropic.com/research/AI-assistance-coding-skills).