Model Buy v. Build Analysis

From The Foundation for Best Practices in Machine Learning

Model Buy v. Build Analysis

Control

The Product Team should work cross-functionally with relevant Stakeholders to define and document a Buy v. Build checklist that considers the following areas: (a) Is the scope of the Product manageable, given the results of the Organisation Capacity Analysis; (b) Can bought Models be used for other Products (eg. transfer learning); (c) Does the Organisation have the in-house expertise required to acquire and label the training data, given the Product Team Composition; (d) How much would it cost to acquire a properly labeled training dataset; (e) Given the Product Team Composition, does the Organisation have the in-house expertise required to retrain Models, if necessary; (f) How important is Model customization and, if so, can bought Models be customised; (g) Are the Acceptance Criteria - Accuracy, Bias, and Fairness requirements for bought Models feasible given the timeline, Product Team Composition, and Organisation Capacity Analysis; and/or (h) What are the usage limits and costs for pre-trained Models.


Aim

To (a) ensure that the Organisation's decision to either purchase or build the Models is appropriate based on Organisation capacity and/or constraints; and (b) highlight associated risks that might occur in the Product Lifecycle.


Additional Information