LogoLogo
DiscordGitHub
  • Welcome!
  • ML OBSERVABILITY COURSE
    • Module 1: Introduction
      • 1.1. ML lifecycle. What can go wrong with ML in production?
      • 1.2. What is ML monitoring and observability?
      • 1.3. ML monitoring metrics. What exactly can you monitor?
      • 1.4. Key considerations for ML monitoring setup
      • 1.5. ML monitoring architectures
    • Module 2: ML monitoring metrics
      • 2.1. How to evaluate ML model quality
      • 2.2. Overview of ML quality metrics. Classification, regression, ranking
      • 2.3. Evaluating ML model quality [CODE PRACTICE]
      • 2.4. Data quality in machine learning
      • 2.5. Data quality in ML [CODE PRACTICE]
      • 2.6. Data and prediction drift in ML
      • 2.7. Deep dive into data drift detection [OPTIONAL]
      • 2.8. Data and prediction drift in ML [CODE PRACTICE]
    • Module 3: ML monitoring for unstructured data
      • 3.1. Introduction to NLP and LLM monitoring
      • 3.2. Monitoring data drift on raw text data
      • 3.3. Monitoring text data quality and data drift with descriptors
      • 3.4. Monitoring embeddings drift
      • 3.5. Monitoring text data [CODE PRACTICE]
      • 3.6. Monitoring multimodal datasets
    • Module 4: Designing effective ML monitoring
      • 4.1. Logging for ML monitoring
      • 4.2. How to prioritize ML monitoring metrics
      • 4.3. When to retrain machine learning models
      • 4.4. How to choose a reference dataset in ML monitoring
      • 4.5. Custom metrics in ML monitoring
      • 4.6. Implementing custom metrics in Evidently [OPTIONAL]
      • 4.7. How to choose the ML monitoring deployment architecture
    • Module 5: ML pipelines validation and testing
      • 5.1. Introduction to data and ML pipeline testing
      • 5.2. Train and evaluate an ML model [OPTIONAL CODE PRACTICE]
      • 5.3. Test input data quality, stability and drift [CODE PRACTICE]
      • 5.4. Test ML model outputs and quality [CODE PRACTICE]
      • 5.5. Design a custom test suite with Evidently [CODE PRACTICE]
      • 5.6. Run data drift and model quality checks in an Airflow pipeline [OPTIONAL CODE PRACTICE]
      • 5.7. Run data drift and model quality checks in a Prefect pipeline [OPTIONAL CODE PRACTICE]
      • 5.8. Log data drift test results to MLflow [CODE PRACTICE]
    • Module 6: Deploying an ML monitoring dashboard
      • 6.1. How to deploy a live ML monitoring dashboard
      • 6.2. ML model monitoring dashboard with Evidently. Batch architecture [CODE PRACTICE]
      • 6.3. ML model monitoring dashboard with Evidently. Online architecture [CODE PRACTICE]
      • 6.4. ML monitoring with Evidently and Grafana [OPTIONAL CODE PRACTICE]
      • 6.5. Connecting the dots: full-stack ML observability
Powered by GitBook
On this page
  • What is a good ML monitoring system?
  • Logging and instrumentation
  • Summing up
  1. ML OBSERVABILITY COURSE
  2. Module 4: Designing effective ML monitoring

4.1. Logging for ML monitoring

What a good ML monitoring system is, and how to set up the logging architecture to capture metrics for further analysis.

PreviousModule 4: Designing effective ML monitoringNext4.2. How to prioritize ML monitoring metrics

Last updated 1 year ago

Video 1. , by Emeli Dral

What is a good ML monitoring system?

A good ML monitoring system consists of three key components:

  • Instrumentation to ensure collection and computation of useful metrics for analyzing model behavior and resolving issues.

  • Alerting to define unexpected model behavior through metrics and thresholds and design action policy.

  • Debugging to provide engineers with context to understand model issues for faster resolution.

When it comes to ML monitoring setup, there is no “one size fits all.” Here are some factors that affect the ML monitoring architecture and choice of metrics:

  • ML service implementation (online service vs. batch model).

  • Environment stability.

  • Feedback loop (immediate or delayed feedback).

  • Team resources (capacity to implement and operate the ML monitoring system).

  • Use case criticality.

  • Scale and complexity of the ML system.

Logging and instrumentation

ML monitoring starts with logging. Before talking about metrics, you need to implement a way to collect the data for analysis.

Step 1. Capture service (event) logs

Capturing service logs is a must-have for any production service, as it helps to monitor and debug service health. You may record different types of events that happen in your service. One will be the prediction event when the service gets the input data and returns the output.

Step 2. Capture prediction logs

When you record the prediction event, make sure to log all prediction-related information, including model input data, model output, and ground truth, if available.

These prediction logs are the key input for ML model quality monitoring. You also need them for model retraining, debugging, and audits.

Step 3. Log ML monitoring metrics

Logging architecture heavily depends on how you deploy your models.

Typically, it involves a prediction store where prediction data is recorded. It requires long-term secure storage with features like backups.

Once you set up this prediction logging, here are two main approaches to ML monitoring implementation:

  • ML monitoring service can pull data from the prediction store – or data can be pushed directly from the ML service – to compute monitoring metrics.

  • Monitoring jobs can be operated with the help of a pipeline manager and load data from the prediction store to compute monitoring metrics.

The next element is a monitoring dashboard:

  • A metric store is created to store computed metrics. It needs quick querying capabilities for efficient dashboard interactions.

  • The monitoring dashboard uses this metric store as the data source to visualize calculated metrics.

For smaller datasets, connecting the monitoring dashboard directly to the prediction store can suffice. It also works well if you run ad-hoc or scheduled reports.

Summing up

We discussed setting up the logging architecture to capture useful metrics for further analysis. Next, we will cover what exactly to log.

Logging for ML monitoring