Optimising the machine and the human

The role of human factors in the safe design and use of AI in healthcare

  • Holistic care — caring for patients is more than giving medication for a specific condition. The nurse usually interacts with the patient, building up an understanding of their physical and emotional needs. They pick up subtle signs (e.g. the patient looking paler than normal) that might indicate a different dose of insulin is required than that predicted by the AI. These signs are not picked up by the AI infusion pump; it does not have the bigger picture. This is particularly relevant where the patient has multiple illnesses and might receive as many as ten infusions concurrently.
  • Trust — the clinician looking after the ward usually makes a dynamic trade-off when they consider who’s working on the ward: they trust the nurse they’ve worked with for ten years, but if it’s a new starter they may decide to double-check on the dosage being given and provide some additional teaching at the same time. Do they now trust the AI or do a double-check? Can they build trust in the AI without having that reassurance that the AI will be aware of the same things clinicians would be aware of?
  • Clinical management — while in theory taking insulin management off clinicians’ list of duties should give them more time to look after their patients, in reality, economic pressures may mean they are assigned other tasks instead. This may take them away from the bedside, leaving them to supervise the AI system remotely: there’s a danger that instead of giving clinicians back the gift of time, we turn them into carers for AI rather than carers for people.
  • Teaming — what happens if the patient doesn’t respond as anticipated? Do the clinical team now know enough about the situation to be able to safely step in and take over from the machine? In an autonomous vehicle, a safety driver must remain vigilant to take over from the autopilot in an emergency — but we have seen this kind of human-machine setup fail catastrophically in practice. In this hospital ward scenario, the clinicians can’t take back control meaningfully unless the infusion pump has a way of communicating to them what it’s doing in a timely and understandable fashion. Clinicians will have to remain active in the loop, and the AI needs to be designed to be part of the clinical team.