Hint

To view additional information and to make edit suggestions, click the individual items.

Safety

Objective

To (a) prevent Model-based physical and/or irreparable harms; and (b) identify and mitigate risks due to Product failures, including Model failures, IT failures, and process failures.

18.1. Product Definition(s)

Objective

To establish the appropriate safety-oriented attitudes based on first principles and Product Definitions.

Item nr.	Item Name and Page	Control	Aim
18.1.1.	Physical and Irreparable Harm Risk	Document and assess whether any likely failure modes can cause physical and/or irreparable harm, based on the Product Definitions. If such is the case, warrant increased oversight and attention throughout the Product Lifecycle to risks and controls in general and from this section in particular.	To warrant the necessary amount of control and resources throughout the Product Lifecycle with regard to preventing and mitigating significant threats to individuals' physical, financial, social, and psychological well being.
18.1.2.	Domain-first Humble Culture	Document and establish tenents for Product Team culture to promote risk awareness and prevent blind spots, inclusive of (a) put Domain expertise central; (b) never assume only positive effects; (c) admit uncertainty when assessing impacts.	To promote risk awareness and prevent blindspots in analysing failure modes and other safety related controls and (b) highlight associated risks that might occur in the Product Lifecycle.

18.2. Exploration

Objective

To start the process of identifying,specifying and controlling (potential) risks and failures modes of the Model(s) and Product based on research and exploration, and sustain this process throughout the Product Lifecycle.

Item nr.	Item Name and Page	Control	Aim
18.2.1.	Forecast Failure Modes	(a) Document and assess continuously throughout the Product Lifecycle all potential failure modes that can be identified through - (i) researching past failures; and/or (ii) interrogating all components or product and context with an open mind; (b) rank identified failure modes according to likelihood and severity; (c) prepare for and mitigate these risks as far as is reasonably practical in order of risk throughout the Product Lifecycle.	To (a) reduce harmful consequences of failures through anticipation and preparation; and (b) highlight associated risks that might occur in the Product Lifecycle.
18.2.2.	Prediction Limits	Document and assess with a diversity of Stakeholders the real limitations on the Product and Model Outcomes that ought be strictly enforced in order to prevent physical and/or irreparable harm and/or other Failure Modes.	To prevent Model and Product Outcomes from violating clear, fixed and safe operating bounds.
18.2.3.	Surprise Diary	Document continuously throughout the Product Lifecycle all surprising findings and occurrences.	To discover and subsequently control for previously unknown or unanticipated failures modes.

18.3. Development

Objective

To control Safety risks and failure modes based on testing and controlling Model and Product technical components.

Item nr.	Item Name and Page	Control	Aim
18.3.1.	General IT Testing Practices	Adhere to all traditional IT/Software Testing best practices.	To warrant and control the risk of failures due to code, software and other IT mistakes in general.
18.3.2.	Testing by Domain Experts	Document and perform testing of the Model(s) and Product by Domain experts.	To warrant that Product and Product Safety are tested against the most relevant requirements and prevent blind spots caused by lack of multidisciplinarity.
18.3.3.	Algorithm Benchmarking	Document and perform benchmark testing of Models and Model code against well-known, trusted and/or simpler Models/code.	To warrant the correct implementation of Models and code, and safeguard reproducibility in general.

18.4. Production

Objective

To control for Safety risks and failure modes and prevent physical and/or irreparable harm by performing assessments and implementing measures at the systemic and organisational level.

Item nr.	Item Name and Page	Control	Aim
18.4.1.	System Failure Propagation	Document and assess how failures in Models and Product components propagate to other components and other systems, and what damage they may cause there. Incorporate such information in Failure Mode risk assessments and implementation of Graceful Failures and Kill Switches.	To (a) prevent blind spots and cascading failures and (b) provide essential input for creating mitigation measures with a minimum of uncontrolled side-effects.
18.4.2.	Graceful Failure	Document and assess whether (i) Model errors, (ii) Model failures, (iii) Product failures, (iv) IT failures, and/or (v) process and implementation failures - whether caused by attack or not - can result in physical or irreparable harm to humans, society and/or the environment. If present, mitigate these risks by implementing technological and/or process measures that make these failures graceful.	To identify risks and mitigating measures throughout the Product Lifecycle with regard to significant threats to individuals' physical, financial, social and psychological wellbeing.
18.4.3.	Kill Switch	Document and implement Kill Switches according to the findings of all previous controls, taking into account (a) instructions and procedures for engaging the Kill Switch; (b) who is/are responsible for engaging the Kill Switch; (c) what impacts the engagement of the Kill Switch has on users, other parts of the Product and other systems.	To (a) guarantee that the Product can be stopped immediately if significant harms are imminent or already occurring; (b) do so in a controllable and reliable manner with minimum collateral harm.
18.4.4.	Safety Stress Testing	Document and perform scenario-based stress testing of Product in Domain, Society and Environmental contexts, for realistic but rare high-impact scenarios, recording the Product's reaction to and influence on the Domain, Society and Environment.	To control and prepare for worst-case scenarios in the context of rapid and/or large changes in Domain, in Society or in Environment.
18.4.5.	Product Incident Response	Document and prepare Product-specific implementation of the Organisation Incident Response Plan insofar as that does not cover the Product's specific risks, if appropriate.	To (a) control for and contain Product Incidents; (b) minimize harms stemming from Product Incidents; (c) repair harms caused by Product Incidents; and (d) incorporate lessons learned.