View source for Monitoring & Maintenance

{{#isin: Monitoring & Maintenance }}
{{ #isin: Technical Best Practices }}

{{Section overview hint}}

= Monitoring & Maintenance =

{{Objective
| content = To ensure that Products and Models remain within acceptable operational bounds. 
}}

<div class="noautonum">__TOC__</div>



== 15.1. Product Definition(s)  == 

{{Objective
| content = To (a) track Model performance in production; and (b) ensure desired Model performance.}}

{| class="wikitable"
! Item nr.
! Item Name and Page
! Control
! Aim
|-
|15.1.1.
| [[Monitoring Objectives ]]
| {{#lst:Monitoring Objectives |control}}
| {{#lst:Monitoring Objectives |aim}}
|-
|15.1.2.
| [[Monitoring Risks ]]
| {{#lst:Monitoring Risks |control}}
| {{#lst:Monitoring Risks |aim}}
|-
|}

== 15.2. Exploration  == 

{{Objective
| content = To (a) define robust Product and/or Model monitoring requirements, inclusive of concerns related to Features and skews of the data; and (b) ensure the continued monitoring of Products and/or Models throughout their lifecycles. }}

{| class="wikitable"
! Item nr.
! Item Name and Page
! Control
! Aim
|-
|15.2.1.
| [[Data Source Mismatch: Training & Production Data ]]
| {{#lst:Data Source Mismatch: Training & Production Data |control}}
| {{#lst:Data Source Mismatch: Training & Production Data |aim}}
|-
|15.2.2.
| [[Data Definitions and Measurements: Training & Production Data]]
| {{#lst:Data Definitions and Measurements: Training & Production Data|control}}
| {{#lst:Data Definitions and Measurements: Training & Production Data|aim}}
|-
|15.2.3.
| [[Data Dependencies and Upstream Changes]]
| {{#lst:Data Dependencies and Upstream Changes|control}}
| {{#lst:Data Dependencies and Upstream Changes|aim}}
|-
|15.2.4.
| [[Data Drift Detection]]
| {{#lst:Data Drift Detection|control}}
| {{#lst:Data Drift Detection|aim}}
|-
|15.2.5.
| [[Product and/or Product Domain Changes: Trends and Preferences]]
| {{#lst:Product and/or Product Domain Changes: Trends and Preferences|control}}
| {{#lst:Product and/or Product Domain Changes: Trends and Preferences|aim}}
|-
|}

== 15.3. Development  == 

{{Objective
| content = To (a) create metrics for (i) Model performance and (ii) Model performance deterioration; and (b) ensure the continued monitoring of Products and/or Models throughout their lifecycles. }}

{| class="wikitable"
! Item nr.
! Item Name and Page
! Control
! Aim
|-
|15.3.1.
| [[Model Performance Deterioration Thresholds]]
| {{#lst:Model Performance Deterioration Thresholds|control}}
| {{#lst:Model Performance Deterioration Thresholds|aim}}
|-
|15.3.2.
| [[Product Contextual Indicators: Model Performance Deterioration]]
| {{#lst:Product Contextual Indicators: Model Performance Deterioration|control}}
| {{#lst:Product Contextual Indicators: Model Performance Deterioration|aim}}
|-
|15.3.3.
| [[Reactive Model Maintenance Indicators]]
| {{#lst:Reactive Model Maintenance Indicators|control}}
| {{#lst:Reactive Model Maintenance Indicators|aim}}
|-
|15.3.4.
| [[Awareness of feedback loops]]
| {{#lst:Awareness of feedback loops|control}}
| {{#lst:Awareness of feedback loops|aim}}
|-
|}

== 15.4. Production  == 

{{Objective
| content = To (a) identify operational maintenance metrics; and (b) ensure timely update, re-train and re-deployment of Model(s).}}

{| class="wikitable"
! Item nr.
! Item Name and Page
! Control
! Aim
|-
|15.4.1.
| [[Operational Performance Thresholds]]
| {{#lst:Operational Performance Thresholds|control}}
| {{#lst:Operational Performance Thresholds|aim}}
|-
|15.4.2.
| [[Continuous Delivery of Metrics: Model Performance ]]
| {{#lst:Continuous Delivery of Metrics: Model Performance |control}}
| {{#lst:Continuous Delivery of Metrics: Model Performance |aim}}
|-
|15.4.3.
| [[Model Decay & Data  Updates]]
| {{#lst:Model Decay & Data  Updates|control}}
| {{#lst:Model Decay & Data  Updates|aim}}
|-
|15.4.4.
| [[Model Re-training]]
| {{#lst:Model Re-training|control}}
| {{#lst:Model Re-training|aim}}
|-
|15.4.5.
| [[Create Contingency Plans]]
| {{#lst:Create Contingency Plans|control}}
| {{#lst:Create Contingency Plans|aim}}
|-
|}