LogoLogo
DiscordGitHub
  • Welcome!
  • ML OBSERVABILITY COURSE
    • Module 1: Introduction
      • 1.1. ML lifecycle. What can go wrong with ML in production?
      • 1.2. What is ML monitoring and observability?
      • 1.3. ML monitoring metrics. What exactly can you monitor?
      • 1.4. Key considerations for ML monitoring setup
      • 1.5. ML monitoring architectures
    • Module 2: ML monitoring metrics
      • 2.1. How to evaluate ML model quality
      • 2.2. Overview of ML quality metrics. Classification, regression, ranking
      • 2.3. Evaluating ML model quality [CODE PRACTICE]
      • 2.4. Data quality in machine learning
      • 2.5. Data quality in ML [CODE PRACTICE]
      • 2.6. Data and prediction drift in ML
      • 2.7. Deep dive into data drift detection [OPTIONAL]
      • 2.8. Data and prediction drift in ML [CODE PRACTICE]
    • Module 3: ML monitoring for unstructured data
      • 3.1. Introduction to NLP and LLM monitoring
      • 3.2. Monitoring data drift on raw text data
      • 3.3. Monitoring text data quality and data drift with descriptors
      • 3.4. Monitoring embeddings drift
      • 3.5. Monitoring text data [CODE PRACTICE]
      • 3.6. Monitoring multimodal datasets
    • Module 4: Designing effective ML monitoring
      • 4.1. Logging for ML monitoring
      • 4.2. How to prioritize ML monitoring metrics
      • 4.3. When to retrain machine learning models
      • 4.4. How to choose a reference dataset in ML monitoring
      • 4.5. Custom metrics in ML monitoring
      • 4.6. Implementing custom metrics in Evidently [OPTIONAL]
      • 4.7. How to choose the ML monitoring deployment architecture
    • Module 5: ML pipelines validation and testing
      • 5.1. Introduction to data and ML pipeline testing
      • 5.2. Train and evaluate an ML model [OPTIONAL CODE PRACTICE]
      • 5.3. Test input data quality, stability and drift [CODE PRACTICE]
      • 5.4. Test ML model outputs and quality [CODE PRACTICE]
      • 5.5. Design a custom test suite with Evidently [CODE PRACTICE]
      • 5.6. Run data drift and model quality checks in an Airflow pipeline [OPTIONAL CODE PRACTICE]
      • 5.7. Run data drift and model quality checks in a Prefect pipeline [OPTIONAL CODE PRACTICE]
      • 5.8. Log data drift test results to MLflow [CODE PRACTICE]
    • Module 6: Deploying an ML monitoring dashboard
      • 6.1. How to deploy a live ML monitoring dashboard
      • 6.2. ML model monitoring dashboard with Evidently. Batch architecture [CODE PRACTICE]
      • 6.3. ML model monitoring dashboard with Evidently. Online architecture [CODE PRACTICE]
      • 6.4. ML monitoring with Evidently and Grafana [OPTIONAL CODE PRACTICE]
      • 6.5. Connecting the dots: full-stack ML observability
Powered by GitBook
On this page
  • Why build a monitoring dashboard?
  • ML monitoring architectures recap
  • Code practice overview
  1. ML OBSERVABILITY COURSE
  2. Module 6: Deploying an ML monitoring dashboard

6.1. How to deploy a live ML monitoring dashboard

ML monitoring architectures recap and how to deploy a live ML monitoring dashboard.

PreviousModule 6: Deploying an ML monitoring dashboardNext6.2. ML model monitoring dashboard with Evidently. Batch architecture [CODE PRACTICE]

Last updated 1 year ago

Video 1. , by Emeli Dral

Why build a monitoring dashboard?

Tests and reports are great for running structured checks and exploring and debugging your data and ML models. However, when you run tests or build an ad-hoc report, you evaluate only a specific batch of data. This makes your monitoring “static”: you can get alerts but have no visibility into metric evolution and trends.

That is where the ML monitoring dashboard comes into play, as it:

  • Tracks metrics over time,

  • Aids in trend analysis and provides insights,

  • Provides a shared UI to enhance visibility for stakeholders, e.g., data scientists, model users, product managers, business stakeholders, etc.

ML monitoring architectures recap

Before proceeding to the code practice of building a monitoring dashboard, let’s quickly recap ML monitoring architectures and how they differ.

ML monitoring starts with logging. To set up an effective ML monitoring backend for your service, you need to log input data, model outputs, and labeled data (if available) – they can be recorded in a prediction store.

You can send logged data to an ML monitoring service or run monitoring jobs to calculate monitoring metrics. Both methods allow recording calculated metrics in a metric store, which you can use as a data source to build a monitoring dashboard.

Code practice overview

We will show how to design and deploy an ML monitoring dashboard for batch and near real-time model monitoring architectures.

For batch monitoring, we will use Evidently open-source ML monitoring architecture. We will:

  • Create snapshots that are individual Reports or Test Suites computed for a specific period (e.g., hour, day, week) in a rich JSON format.

  • Log them to a file system and run an ML monitoring service that reads data from snapshots.

  • Design dashboard panels and visualize data and model metrics over time. You can easily switch between time-series data and individual Reports/Test Suites for further analysis and debugging.

For near real-time monitoring, Evidently provides a collector service:

  • It collects incoming data and computes Reports / Test Suites continuously.

  • You can configure the service to calculate snapshots over various time intervals (as opposed to writing batch jobs on your own – you can define the frequency of batches using a config file).

  • The service outputs the same snapshots as in the batch monitoring scheme: the front-end is the same.

We will also cover an alternative architecture using Grafana, a popular open-source visualization tool. We will walk you through the Evidently and Grafana integration example – you can parse data computed by Evidently, store it in a chosen database, and visualize it in Grafana. In this case, Evidently serves as an evaluation layer.

This can be a good alternative if you already use Grafana for monitoring and alerting for other services.

And now, to practice!

How to deploy a live ML monitoring dashboard