Table 4 Key pre-implementation recommendations for developing data-driven algorithms for LRS with illustrations from the present study

1. Inquire about the type of algorithm to be developed

Get to know the context and potential end-users. Determine if data-driven algorithms could answer the needs of potential users. Consider that algorithms can have different aims, such as risk prediction, diagnosis, medication, or discharge. Involve the potential users early in the design process to investigate potential acceptance in the local context and improve the system’s design and alignment with the clinical workflow. Consider user’s specific requirements, preferences, and challenges on the ground

Analysis of the clinical context from clinical, design, and social sciences perspectives identified that an algorithm could assist in alleviating staff shortages and improving care provision. Interviews confirmed the users’ perceived added value of algorithms for risk prediction in critical care

2. Foster trust in the data-driven algorithm

Fostering trust in data-driven algorithms depends on factors specific to the population. Researchers must, therefore, invest in identifying which factors enable or undermine trust. Different aspects such as actionability (ease of use, transparency, explainability), context specificity, and performance will probably be constant, but their interpretation and how to balance these aspects may vary depending on the users

Nurses working in critical care in LRS advocate for the primacy of their clinical expertise over a potential algorithm, independently of how good it is. In order to trust it, nurses found it necessary to be easy to use and capable of interacting to some basic degree with guardians. Nurses require outputs to be transparent and explainable, interpretable and actionable

3. Design actionable outputs

Design the outputs of the data-driven algorithm for specific purposes and tailor them to the context. Ensure that the output is actionable (understood as being able to act in response to the output). For example, the output facilitates clinical decision-making according to the accepted medical practices of the context. Hardware such as tablets and alarms can help address shortages in healthcare staff and suboptimal physical infrastructures

Analysing the workflow and working practices in a context with high patient-nurse ratios and where physical infrastructures do not provide an easy visual overview of patients allowed us to design the algorithm’s output in a clinically meaningful way for nurses. It also provided us with insights into how technology can shorten distances between nurses and patients and provide a constant overview of the patients, for example, using tablets to display all the patients being constantly monitored and the output of the algorithm

4. Prefer minimal complexity and clarity

Simplify system complexity to accommodate educational variations among users. In doing so, avoid information overload. Providing different information layers may help simplify the system while making in-depth information available. Ensure clarity in distinguishing various aspects of your algorithm output by employing techniques like colour coding or visual cues

During the co-design sessions, nurses emphasised the importance of presenting the algorithm’s output intuitively and simplistically, suggesting using color-coded scores, such as red to green, combined with numeric scales provided more detailed clinical information which is considered necessary for clinical decision making. This method enhanced understanding of which factors demand attention and facilitates interpretation of the algorithm’s output, improving its applicability across diverse scenarios

5. Identify the most important aspects for designing tailored training programmes

Consider that providing users with comprehensive and customised training will define the adoption and success of a data-driven algorithm in clinical practice. Understanding aspects such as the users’ responsibilities while using the algorithm and the changes in the clinical decision making that it can cause is necessary. Identify what is necessary for users for a correct interpretation of the algorithm’s output and accompanying information, and how to communicate to other health professionals and guardians. Examine the role of guardians and determine if this needs to be part of the training provided to users. Consider, for example, if nurses need to be provided with information tools to be shared with guardians

Nurses expressed their need to have the right skills to interpret the algorithm, the capacity to critically interpret the results. They were already familiar with possible reasons why an algorithm could make mistakes (quality of input data). Training was seen as the means to achieve a coherent interpretation of the algorithm by nurses with different levels of education and training. Nurses confirmed that they would involve guardians in care while using such a data-driven algorithm (e.g. one that predicts the risk of deterioration) in clinical practice, aspect that we consider necessary to include in the plans for training nurses