Module 4: Designing effective ML monitoring

This module reviews how to set up an ML monitoring system, considering the model risks, criticality, and deployment scenario.

In previous modules, we reviewed possible metrics and approaches to tracking the performance of ML models in production.

In this module, we will put it all together and review specific questions you might have when setting up an ML monitoring system for a particular model. We’ll do a deeper dive and cover:

• How to select and prioritize ML monitoring metrics.

• How and when to retrain ML models.

• How to choose a reference dataset.

• How to implement custom metrics in ML monitoring.

• How to choose an appropriate ML monitoring architecture.

At the end of this module, you will understand how to design an optimal approach to ML monitoring considering the model risks, criticality, and deployment scenario.