Management & Monitoring

From The Foundation for Best Practices in Machine Learning
Technical Best Practice Guideline > Management & Monitoring
Jump to navigation Jump to search


Hint
To view additional information and to make edit suggestions, click the individual items.

Management & Monitoring

Objective
To ensure an effective and auditable Product Lifecycle.
Item nr. Item Name and Page Control Aim
6.1. Product Requirements

Draft and document clear Product requirements. Review Model Type - Metrics and Acceptance Criteria, as discussed in Section 4 - Problem Mapping, to ensure alignment. Regularly review Product & Outcome Definitions Data & Model Metrics and User Experience Mapping, as discussed in Section 4 - Problem Mapping, and update as necessary

To (a) ensure current, clear, and actionable Product requirements; and (b) highlight associated risks that might occur in the Product Lifecycle.

6.2. Product Roadmap and Pipeline

Develop and document a Product roadmap and pipeline that enable the experience envisioned in the User Experience Mapping, as discussed in Section 4 - Problem Mapping, and include the following sections: Schedule and milestones, tasks and deliverables, limitations and exclusions (scope), initial prioritization, and methods for determining future priority.

To (a) ensure a clear, actionable, and prioritized Product roadmap and pipeline; and (b) highlight associated risks that might occur in the Product Lifecycle.

6.3. Experimentation Constraints

Develop and document a method for evaluating the quality of predictions. Develop and document criteria for determining when to stop the experimentation process.

To (a) develop processes to ensure a balance between effectiveness and efficiency in the experimentation cycle; and (b) highlight associated risks that might occur in the Product Lifecycle.

6.4. Behavioral Change Analysis - Process Changes

Research and assess the business processes that will be affected by the new Product and/or the infrastructure changes that enable the Product. Review the User Experience Mapping, Data Capacity Analysis, and Organisation Capacity Analysis, as discussed in Section 4 - Problem Mapping, and reformulate as necessary. Develop and document a plan to retrain affected parties as necessary and mitigate business disruptions as much as feasible.

To (a) determine business processes that may be affected by the project and create a plan to retrain or mitigate impacts as necessary; and (b) highlight associated risks that might occur in the Product Lifecycle.

6.5. Behavioral Change Analysis - Social (Off-Platform)

Research and document ways in which the Product can be abused or negatively impact customers, end users, or the broader society. Develop and document a plan to mitigate negative impacts as much as feasible. Develop and document counter metrics to assess whether users or the model are 'gaming' the system.

To (a) determine (i) negative product uses, (ii) negative product impacts (iii) negative user or model behaviors and create a plan to counter behaviors or mitigate impacts as necessary; and (b) highlight associated risks that might occur in the Product Lifecycle.

6.6. Resource Assessment

Document the processes, tools, and staffing that are required for every phase of the project, including the Data Capacity Analysis, Organisation Capacity Analysis, Product Scaling Analysis, and Product Cost Analysis, as discussed in Section 4 - Problem Mapping, before starting each phase of the project and update as necessary.

To (a) ensure adequate resources and funding during every phase of the Product Lifecycle; and (b) highlight associated risks that might occur in the Product Lifecycle.

6.7. POC-to-Production Checklist

Document and define a POC-to-Production Checklist that details the existing system modifications, and new system builds, required for integrating the Product into Organisation infrastructure and incorporating additional data sources. If gaps in organisational capacity are detected, reiterate formulations of Organisation Capacity Analysis, as discussed in Section 4 - Problem Mapping, as necessary.

To (a) ensure sufficient planning for Product development and production; and (b) highlight associated risks that might occur in the Product Lifecycle.

6.8. Update Schedule

Develop and document a schedule for updating and upgrading Product and pipeline-related software. Coordinate with IT to determine whether it is optimal to align with the Product roadmap or system maintenance schedules.

To (a) ensure the Product and its related software are updated and upgraded regularly and that the schedule for said updates are coordinated with information technology department(s); and (b) highlight associated risks that might occur in the Product Lifecycle.

6.9. Project Records

Develop and document a process for preserving data of the information considered when making significant product decisions. Include any methods for standardized experiment tracking and artifact capturing that are developed by Data Science and Engineering. Develop a continuously maintained and consistently available repository for Product Requirements and any data related to their updates.

To (a) maintain a historical record of Product and data and ensure that all iterations of Product Requirements are continuously available to Stakeholders; and (b) highlight associated risks that might occur in the Product Lifecycle.

6.10. Project Records - Stakeholder Sign-offs

Develop a standard Stakeholder Sign-off Document to be utilized (i) after the finalization of the following documents and analyses: Problem Statement & Solution Mapping, Outcomes Definition, Product & Outcome Definitions, Product Integration Strategy, Model Type - Best Fit Analysis, Acceptance Criteria - Key Business Metrics/Targeted Metrics, Testing Design and Scheduling Framework, Resource Assessment, as discussed in Section 4 - Problem Mapping, Section 5 - Model Decision-Making; and (ii) at Project Checkpoints, as discussed in Section 10 - Project Checkpoints.

To (a) ensure stakeholder buy-in; and (b) provide an auditable record of project and stakeholder expectations at every major project decision-point.

6.11. Custody

Develop a system for documenting the chain of custody for Product(s) and the data, microservices, and applications that it is built on and with, that indicates: i) provenance ii) control iii) transfer, iv) analysis, and v) transformation.

To (a) ensure that the building blocks of the Product can be traced back to their origins; (b) allow for undesirable changes to be reverted; and (c) highlight associated risks that might occur in the Product Lifecycle.