Workflow for requesting the support of ICCS for VESRI work (Resource Allocation Process)

Aims

The aim of the resource allocation process is to ensure that the ICCS meets its funder’s obligations to drive forward advances in ground breaking climate research through continuous collaboration between scientists and software engineers. The aims of ICCS are to provide:

1. Operational support and optimisation of the software engineering and data science capabilities of the VESRI science teams.
2. Long term software quality assurance and open-source dissemination of the codes/data created/used by the Science teams.
3. Research breakthroughs in key areas of climate science, data science, and computer science, arising from close and continuous collaboration between scientists and software engineers.
4. Capacity building of tripartite expertise in climate science, data science, and computer science both within the software engineering community and the scientific research community through the development and delivery of world-class training/personal development.

The provision of embedded research software engineers (RSEs) to collaborate on projects is expected to align directly to items 1 and 2 and indirectly contribute to items 3 and 4 alongside the broader ICCS initiative which couples support for the international science teams with a long-term research agenda to improve the development, implementation, maintenance, and dissemination of models for scientific computing, data assimilation and analysis.

The directors, co-directors and engineering lead will convene a quarterly Resource Allocation Board meeting with a delegated representative of the VESRI Advisory Board. The meetings are supported administratively by the Programme Training and Communications Manager, and chaired by the Executive Programme Director. This group will lead an assessment process for formally submitted requests for collaboration from the ICCS RSE team. The group will further determine the allocation of resources to requests for the following quarter using the process described in this document.

The ICCS RSE team relies on transparency and common purpose in selecting projects to support, looking for those that offer to achieve excellent scientific outcomes whilst seeking to provide a balance of resources and technical skills. Our assessment criteria have been designed to reflect this and our outcomes will be monitored using a process of impact assessment. The decisions of the Resource Allocation Board are strongly informed by the Engineering Lead but the Academic Director will have ultimate decision-making authority. The implementation of the decisions concerning resource allocation is supported by the Executive Programme Director, who is also responsible for communication and liaison with the various Science Teams.

Request for Collaboration Process Overview

Research Software Engineering Collaborations are defined on a per-project basis and RSEs are not assigned to specific VESRI teams on a per-person basis. Projects have a defined duration and expected output(s) much like traditional research projects. The resource allocation process aims to balance project needs with availability of RSE skills. In order to formally request resources for your project, a primary investigator (PI), Co-Investigator, Work Package lead or appropriate deputy associated with a VESRI project must submit a Request for Collaboration via the online form (with approval of a VESRI PI). Prior to submission it is encouraged to discuss the project with members of the ICCS team (either the Directors, Engineering Lead or one of our RSEs) to ensure that the request is within scope and so that prior feedback can be provided. There is an opportunity to seek feedback and develop ideas within the monthly VESRI PI meetings or by arranging a one-off meeting via iccs@maths.cam.ac.uk.

Requests for collaboration can be submitted through the ICCS “regular call” which is scheduled every three months corresponding with the resource allocation panel. The deadlines are expected to be 1st Jan, 1st April, 1st July, 1st October unless communicated otherwise. Submissions will be assessed according to the criteria described later in this document. Outcomes will be one of the following which will be communicated to the project PI after the assessment and resource allocation process have completed (see also Figure 1):

1. Accepted for Immediate Support - A member of RSE staff has been allocated to your project and the collaboration will be supported as soon as practically possible. There may be some feedback or guidance suggested from the resource panel and this will be communicated to the PI.
2. Accepted for Deferred Support - The project can be supported but is unable to have resources allocated immediately. This may be due to availability of RSEs (or a requirement for a specific set of RSE skills) or a result of resources being allocated to high ranking proposals (see Assessment Criteria and Assessment). Resource allocation will be considered at the next panel. There may be some feedback or guidance suggested from the resource panel and this will be communicated to the PI. If there is guidance that may improve the assessment and ultimate outcome of the proposal then it can optionally be resubmitted at the next request for collaboration deadline.
3. Deferred (with option for resubmission) - The project cannot be supported in its current form and will need to be resubmitted to the next call with additional information following feedback from the review. Feedback or guidance from the resource panel will be communicated to the PI.
4. Rejected (without option for resubmission) - The project cannot be supported or resubmitted. There may be some feedback from the resource panel and this will be communicated to the PI.

In addition to the “regular call” for collaboration requests there is an alternative “high priority” track in which urgent requests can be considered at any time. This route for submission will require explanation as to why it is high priority and unsuitable for regular submission. High priority track applications will be assessed by the same review criteria and will have the same potential set of outcomes, but will only be accepted for immediate support in exceptional circumstances and where staff resources are available. Within the high priority track a shorter minimum one week project request can be made.

It is anticipated that requests for projects will exceed the capacity of the ICCS team and as such our assessment process and resource allocation acts to ensure that the highest quality applications are supported whilst also balancing the need to provide equitable opportunity for all VESRIs centres.

Assessment Criteria and Assessment

The purpose of the assessment process is first to determine if a project collaboration request should be supported. Furthermore, the assessment will be used in prioritisation of projects both within the VESRI centres as well as between centres where there are competing requests for resources or a limited specific technical skill. In assessing a request for collaboration, the following items are used as assessment criteria:

• Scientific and Technical quality - This criteria considers the quality and excellence of the project and specifically how the RSE collaboration will have an impact on facilitation of the scientific or technical goal. The focus of this assessment criteria is not on the overall quality of the VESRI centre’s research but on the specific scientific or technical question being investigated during the project duration. Projects which clearly define a technical challenge which will be transformative on a scientific outcome will score highly. The project proposal should clearly indicate the novelty of any technical or scientific approach highlighting the ambition and/or timeliness.
• Outcomes Including Software Quality - This criteria will focus on the outcomes of the project. This may consider a mixture of traditional research outputs (i.e. papers) and alternative outputs, as defined by the San Francisco DORA initiative, such as software and datasets. Where publications are planned as outputs of the project, emphasis will be given to reproducibility of the research. The proposal should describe how effort will be made to ensure that software and data attributed to papers is FAIR. An excellent mechanism to understand software sustainability is a code review which will consider software quality and sustainability. For existing software, the ICCS team can provide software sustainability and reproducibility reviews prior to your application so that specific plans can be proposed.
• Urgency and Importance - Importance will be considered with regards to the significance of the project to the overall VESRI project deliverable plan. Urgency will consider the resource request and any flexibility indicated regarding the potential start date. Within any single VESRI centre, feedback from the VESRI PI may be sought in prioritising urgency of projects in assessing the urgency component.
• Work Plan and Schedule - The work plan and schedule should define any final or interim deliverables or milestones that will be used to assess progress. An assessment will be made on the technical feasibility given the scale of resources requested. This will consider the broader team in which the RSE will collaborate. Details of the scale of the collaboration should be provided including names of key collaborators and the time commitments of any VESRI staff who will contribute to the project. The RSE component of the project should not be seen as simply a coding exercise but as a collaboration where team members are guided and mentored by the embedded RSE to improve their own research software practice. As part of the project proposal any additional request for resources such as code review, training, delivery or workshops, or mentoring of staff related to the project should be included with justification provided. Resources that require multiple RSE input can be considered as additional effort to any fixed resource request.

Each application must meet a minimum threshold for each of these criteria. E.g. If a Scientifically excellent proposal is received it will not be supported if the work plan for the RSE collaboration is not feasible.

Please see the flowchart of the Resource Allocation Process here

Allocation of Resources

The resource allocation process will consider proposals that have met the threshold for support following the assessment process. The pool model of the ICCS RSE team ensures that there is a broad range of skills but where these need to be prioritised it will be done based on the assessment outcomes. Resources will be allocated based on ranking until a VESRI meets its allocation of 1 FTE (Full Time Employee) of RSE effort. Additional allocations beyond 1 FTE per VESRI team are possible and may be made depending on resource allocation within the other VESRI teams and also by considering any previous under or over commitment a VESRI has received.

RSE staff from the ICCS are always committed in half FTE chunks over the duration of the project. A request for collaboration can be over a duration of 3, 6 or 12 months (with an option for a shorter period of 1-2 weeks via the high priority track). Each request can be for either half or a full RSE (a full RSE is recommended). In the case of a full RSE this will be delivered by two members of staff at half capacity. Although staff are assigned in half chunks of time, the ICCS RSE pool operates under a model of 40:40:20 in which staff allocated to any particular project are expected to commit two days per week (the actual days are flexible) with a day a week reserved for activity which indirectly benefit all VESRI centres. Activities within the reserved day include the development and delivery of training, code review and technical assessment of VESRI projects and personal development.

Over the course of the project each collaboration will be monitored by the Executive Programme Director and relevant Co-Directors via the VESRI PI monthly meetings to identify areas to improve and areas where further training, or even research, is required. These reports will also feed into the meetings of the Advisory Board and Resource Allocation Board. After staff have been assigned to your project there is a small possibility that staff may need to be recalled. This would be in exceptional circumstances to meet short-term, high-priority needs or where there has been a determination from the VESRI PI or ICCS executive team that either a project should be ended early. There are also a number of specific compulsory events which ICCS staff are expected to engage with such as the ICCS summer school which may result in time being diverted from a particular project. If RSEs become unavailable to the project due illness or resignation from the team the project will re-enter the resource allocation process as an “urgent track” application.

Summary of Project Duration and Commitment Options

Project Duration Options

• 1-2 weeks (high priority track only)
• 3 months
• 6 months
• 12 months

RSE Commitment Options

• 1/2 RSE = 1 person 2 days a week
• 1 RSE = 2 people at 2 days per week each

Project End and Renewal

As the majority of development is expected to take place on GitHub (or another version control system) the final project handover is expected to be lightweight. The RSE team strives to reduce the “bus factor” of software development expertise and therefore it is anticipated that at any stage of the project it should be clear what development progress has been made and the progress of any particular task or deliverable should be clear. Although a project has a defined end, it is expected that the RSE involvement will have embedded changes to software development practice of science teams which will benefit further development activities and we expect to report on this.

The ICCS RSE team utilises an agile development process in which prioritisation of work will have been communicated throughout the lifecycle of the project. Extensions to projects are not possible and it is the responsibility of the PI to work with the development team to set and review regular priorities to ensure that deliverables can be met within the time frame that resources have been allocated. Support can be provided from the RSE Engineering Lead in using agile processes for software project management. If there is scope for further work this can be submitted at any stage as a new request for collaboration. It is possible for work on a particular technical or scientific problem to receive long term support but this must always be delivered via the assessment and resource allocation process.

As part of the assessment of the impact of the ICCS the Resource Allocation Boardwill request information from project PIs. This will includes:

1. Qualitative feedback from the science teams collected at delivery milestones.
2. Measures of uptake or interest in software beyond the originating science team.
3. Publication records and citation counts.
4. Team sizes, development, and career tracking of those engaged by our activities.

Submit Resource Request via the Form

A link to the form is availble here: Resource Allocation Request Form