{"version":1,"pages":[{"id":"orDAs1W6XbMpB8wHoS6F","title":"Welcome!","pathname":"/","siteSpaceId":"sitesp_CP4Ci","description":"Free Open-source ML observability course for data scientists and ML engineers by Evidently AI."},{"id":"lcl7lXIR2Da2OWmaragk","title":"Module 1: Introduction","pathname":"/ml-observability-course/module-1-introduction","siteSpaceId":"sitesp_CP4Ci","description":"Key concepts of machine learning monitoring and observability and how they fit in the ML lifecycle.","breadcrumbs":[{"label":"ML OBSERVABILITY COURSE"}]},{"id":"g5R5aU6t96PZvluigooU","title":"1.1. ML lifecycle. What can go wrong with ML in production?","pathname":"/ml-observability-course/module-1-introduction/ml-lifecycle","siteSpaceId":"sitesp_CP4Ci","description":"What can go wrong with data and machine learning services in production. Data quality issues, data drift, and concept drift.","breadcrumbs":[{"label":"ML OBSERVABILITY COURSE"},{"label":"Module 1: Introduction"}]},{"id":"eyMK0mwLgZEaGn8Qo8xO","title":"1.2. What is ML monitoring and observability?","pathname":"/ml-observability-course/module-1-introduction/ml-monitoring-observability","siteSpaceId":"sitesp_CP4Ci","description":"What ML monitoring is, the challenges of production ML monitoring, and how it differs from ML observability.","breadcrumbs":[{"label":"ML OBSERVABILITY COURSE"},{"label":"Module 1: Introduction"}]},{"id":"vUUEmBph2IBwsG2grEpJ","title":"1.3. ML monitoring metrics. What exactly can you monitor?","pathname":"/ml-observability-course/module-1-introduction/ml-monitoring-metrics","siteSpaceId":"sitesp_CP4Ci","description":"A framework to organize ML monitoring metrics. Software system health, data quality, ML model quality, and business KPIs.","breadcrumbs":[{"label":"ML OBSERVABILITY COURSE"},{"label":"Module 1: Introduction"}]},{"id":"18JzFL3igQFJG4jaHCtl","title":"1.4. Key considerations for ML monitoring setup","pathname":"/ml-observability-course/module-1-introduction/ml-monitoring-setup","siteSpaceId":"sitesp_CP4Ci","description":"Key considerations for ML monitoring setup. Service criticality, retraining cadence, reference dataset, and ML monitoring architecture.","breadcrumbs":[{"label":"ML OBSERVABILITY COURSE"},{"label":"Module 1: Introduction"}]},{"id":"JhB0v6bECQbb9vwtG3Xi","title":"1.5. ML monitoring architectures","pathname":"/ml-observability-course/module-1-introduction/ml-monitoring-architectures","siteSpaceId":"sitesp_CP4Ci","description":"ML monitoring architectures for backend and frontend and what to consider when choosing between them.","breadcrumbs":[{"label":"ML OBSERVABILITY COURSE"},{"label":"Module 1: Introduction"}]},{"id":"jcVt55pbTm4BvLCat0SV","title":"Module 2: ML monitoring metrics","pathname":"/ml-observability-course/module-2-ml-monitoring-metrics","siteSpaceId":"sitesp_CP4Ci","description":"This module covers different aspects of the production ML model performance. You will learn how to apply data quality, model quality, and data drift metrics for structured data.","breadcrumbs":[{"label":"ML OBSERVABILITY COURSE"}]},{"id":"50eky4lkNlAMGmqrDk01","title":"2.1. How to evaluate ML model quality","pathname":"/ml-observability-course/module-2-ml-monitoring-metrics/evaluate-ml-model-quality","siteSpaceId":"sitesp_CP4Ci","description":"How to evaluate ML model quality directly and use early monitoring to detect potential ML model issues.","breadcrumbs":[{"label":"ML OBSERVABILITY COURSE"},{"label":"Module 2: ML monitoring metrics"}]},{"id":"cjsvW38Bi0hcTo85UYru","title":"2.2. Overview of ML quality metrics. Classification, regression, ranking","pathname":"/ml-observability-course/module-2-ml-monitoring-metrics/ml-quality-metrics-classification-regression-ranking","siteSpaceId":"sitesp_CP4Ci","description":"Commonly used ML quality metrics for classification, regression, and ranking problems.","breadcrumbs":[{"label":"ML OBSERVABILITY COURSE"},{"label":"Module 2: ML monitoring metrics"}]},{"id":"6gkMLFreqposje6zbh0t","title":"2.3. Evaluating ML model quality [CODE PRACTICE]","pathname":"/ml-observability-course/module-2-ml-monitoring-metrics/ml-model-quality-code-practice","siteSpaceId":"sitesp_CP4Ci","description":"A code example walkthrough of ML model quality evaluation using Python and the open-source Evidently library.","breadcrumbs":[{"label":"ML OBSERVABILITY COURSE"},{"label":"Module 2: ML monitoring metrics"}]},{"id":"XWP9EnfyI6ADbtNkM8ED","title":"2.4. Data quality in machine learning","pathname":"/ml-observability-course/module-2-ml-monitoring-metrics/data-quality-in-ml","siteSpaceId":"sitesp_CP4Ci","description":"Types of production data quality issues, how to evaluate data quality, and interpret data quality metrics.","breadcrumbs":[{"label":"ML OBSERVABILITY COURSE"},{"label":"Module 2: ML monitoring metrics"}]},{"id":"fNXzZfPZh5q4GoYDaGID","title":"2.5. Data quality in ML [CODE PRACTICE]","pathname":"/ml-observability-course/module-2-ml-monitoring-metrics/data-quality-code-practice","siteSpaceId":"sitesp_CP4Ci","description":"A code example walkthrough of data quality evaluation using Evidently Reports and Test Suites.","breadcrumbs":[{"label":"ML OBSERVABILITY COURSE"},{"label":"Module 2: ML monitoring metrics"}]},{"id":"t9TUheGSTfrLXStiybM4","title":"2.6. Data and prediction drift in ML","pathname":"/ml-observability-course/module-2-ml-monitoring-metrics/data-prediction-drift-in-ml","siteSpaceId":"sitesp_CP4Ci","description":"What data and prediction drift is, and how to detect distribution drift using statistical methods and rule-based checks.","breadcrumbs":[{"label":"ML OBSERVABILITY COURSE"},{"label":"Module 2: ML monitoring metrics"}]},{"id":"ZAVtlD807YU92kPjTQ57","title":"2.7. Deep dive into data drift detection [OPTIONAL]","pathname":"/ml-observability-course/module-2-ml-monitoring-metrics/data-drift-deep-dive","siteSpaceId":"sitesp_CP4Ci","description":"A deep dive into data drift detection methods, how to choose the right approach for your use case, and what to do when the drift is detected.","breadcrumbs":[{"label":"ML OBSERVABILITY COURSE"},{"label":"Module 2: ML monitoring metrics"}]},{"id":"GzpDLf3UF9NlQ8uJ05kC","title":"2.8. Data and prediction drift in ML [CODE PRACTICE]","pathname":"/ml-observability-course/module-2-ml-monitoring-metrics/data-prediction-drift-code-practice","siteSpaceId":"sitesp_CP4Ci","description":"A code example walkthrough of detecting data drift and creating a custom method for drift detection using Evidently.","breadcrumbs":[{"label":"ML OBSERVABILITY COURSE"},{"label":"Module 2: ML monitoring metrics"}]},{"id":"ifyhDjTltvJfNvenGnHI","title":"Module 3: ML monitoring for unstructured data","pathname":"/ml-observability-course/module-3-ml-monitoring-for-unstructured-data","siteSpaceId":"sitesp_CP4Ci","description":"This module covers evaluating and monitoring the production performance for models that use unstructured data, including NLP, LLMs, and embeddings.","breadcrumbs":[{"label":"ML OBSERVABILITY COURSE"}]},{"id":"Zg3zwUQSkY1o65C6Ydqd","title":"3.1. Introduction to NLP and LLM monitoring","pathname":"/ml-observability-course/module-3-ml-monitoring-for-unstructured-data/introduction-nlp-llm-monitoring","siteSpaceId":"sitesp_CP4Ci","description":"How to evaluate model quality for NLP and LLMs and monitor text data without labels using raw data, embeddings, and descriptors.","breadcrumbs":[{"label":"ML OBSERVABILITY COURSE"},{"label":"Module 3: ML monitoring for unstructured data"}]},{"id":"pfPVxwoqWlAyTS85vEDm","title":"3.2. Monitoring data drift on raw text data","pathname":"/ml-observability-course/module-3-ml-monitoring-for-unstructured-data/monitoring-data-drift-on-raw-text-data","siteSpaceId":"sitesp_CP4Ci","description":"How to detect and evaluate raw text data drift using domain classifier and topic modeling.","breadcrumbs":[{"label":"ML OBSERVABILITY COURSE"},{"label":"Module 3: ML monitoring for unstructured data"}]},{"id":"fMJp57etVc7q2eoouTkx","title":"3.3. Monitoring text data quality and data drift with descriptors","pathname":"/ml-observability-course/module-3-ml-monitoring-for-unstructured-data/monitoring-data-drift-with-descriptors","siteSpaceId":"sitesp_CP4Ci","description":"What text descriptors are and how to use them to monitor text data quality and data drift.","breadcrumbs":[{"label":"ML OBSERVABILITY COURSE"},{"label":"Module 3: ML monitoring for unstructured data"}]},{"id":"a0VHLpA52yvsl6OPy7s9","title":"3.4. Monitoring embeddings drift","pathname":"/ml-observability-course/module-3-ml-monitoring-for-unstructured-data/monitoring-embeddings-drift","siteSpaceId":"sitesp_CP4Ci","description":"Strategies for monitoring embedding drift using distance metrics, model-based drift detection, and share of drifted components.","breadcrumbs":[{"label":"ML OBSERVABILITY COURSE"},{"label":"Module 3: ML monitoring for unstructured data"}]},{"id":"SVurD0SG7nSqID2jOoKY","title":"3.5. Monitoring text data [CODE PRACTICE]","pathname":"/ml-observability-course/module-3-ml-monitoring-for-unstructured-data/monitoring-text-data-code-practice","siteSpaceId":"sitesp_CP4Ci","description":"A code example walkthrough of unstructured data evaluations using the open-source Evidently Python library.","breadcrumbs":[{"label":"ML OBSERVABILITY COURSE"},{"label":"Module 3: ML monitoring for unstructured data"}]},{"id":"OPjdmAdOjSxvUGkcEl4f","title":"3.6. Monitoring multimodal datasets","pathname":"/ml-observability-course/module-3-ml-monitoring-for-unstructured-data/monitoring-multimodal-datasets","siteSpaceId":"sitesp_CP4Ci","description":"Strategies for monitoring data quality and data drift in multimodal datasets.","breadcrumbs":[{"label":"ML OBSERVABILITY COURSE"},{"label":"Module 3: ML monitoring for unstructured data"}]},{"id":"Ttu1VHvKy9zXWaXNX4ss","title":"Module 4: Designing effective ML monitoring","pathname":"/ml-observability-course/module-4-designing-effective-ml-monitoring","siteSpaceId":"sitesp_CP4Ci","description":"This module reviews how to set up an ML monitoring system, considering the model risks, criticality, and deployment scenario.","breadcrumbs":[{"label":"ML OBSERVABILITY COURSE"}]},{"id":"NG2tUMjjxse6Gok6szVL","title":"4.1. Logging for ML monitoring","pathname":"/ml-observability-course/module-4-designing-effective-ml-monitoring/logging-ml-monitoring","siteSpaceId":"sitesp_CP4Ci","description":"What a good ML monitoring system is, and how to set up the logging architecture to capture metrics for further analysis.","breadcrumbs":[{"label":"ML OBSERVABILITY COURSE"},{"label":"Module 4: Designing effective ML monitoring"}]},{"id":"PSLKcqGyStIW58AqxXSS","title":"4.2. How to prioritize ML monitoring metrics","pathname":"/ml-observability-course/module-4-designing-effective-ml-monitoring/how-to-prioritize-monitoring-metrics","siteSpaceId":"sitesp_CP4Ci","description":"ML monitoring depth, metrics to collect when building a monitoring system, and how to prioritize them.","breadcrumbs":[{"label":"ML OBSERVABILITY COURSE"},{"label":"Module 4: Designing effective ML monitoring"}]},{"id":"1VjMrOzTOzpl3YizVDUd","title":"4.3. When to retrain machine learning models","pathname":"/ml-observability-course/module-4-designing-effective-ml-monitoring/when-to-retrain-ml-models","siteSpaceId":"sitesp_CP4Ci","description":"Scheduled and trigger-based retraining and what to consider when making the retaining decision.","breadcrumbs":[{"label":"ML OBSERVABILITY COURSE"},{"label":"Module 4: Designing effective ML monitoring"}]},{"id":"wWTinEyw5mmYVqfpawJ2","title":"4.4. How to choose a reference dataset in ML monitoring","pathname":"/ml-observability-course/module-4-designing-effective-ml-monitoring/how-to-choose-reference-dataset-ml-monitoring","siteSpaceId":"sitesp_CP4Ci","description":"What a reference dataset is in ML monitoring, how to choose one for drift detection, and when to use multiple references.","breadcrumbs":[{"label":"ML OBSERVABILITY COURSE"},{"label":"Module 4: Designing effective ML monitoring"}]},{"id":"N9uq7xfAviQbNKCAxIoC","title":"4.5. Custom metrics in ML monitoring","pathname":"/ml-observability-course/module-4-designing-effective-ml-monitoring/custom-metrics-ml-monitoring","siteSpaceId":"sitesp_CP4Ci","description":"Types of custom metrics. Business or product metrics, domain-specific metrics, and weighted metrics.","breadcrumbs":[{"label":"ML OBSERVABILITY COURSE"},{"label":"Module 4: Designing effective ML monitoring"}]},{"id":"O2I0Utx39yzgesgC7ScA","title":"4.6. Implementing custom metrics in Evidently [OPTIONAL]","pathname":"/ml-observability-course/module-4-designing-effective-ml-monitoring/custom-metrics-evidently-code-practice","siteSpaceId":"sitesp_CP4Ci","description":"A code example walkthrough of creating a custom metric using the Evidently Python library.","breadcrumbs":[{"label":"ML OBSERVABILITY COURSE"},{"label":"Module 4: Designing effective ML monitoring"}]},{"id":"AOCgZUkHQwJ3A9I0LsrH","title":"4.7. How to choose the ML monitoring deployment architecture","pathname":"/ml-observability-course/module-4-designing-effective-ml-monitoring/choosing-ml-monitoring-deployment-architecture","siteSpaceId":"sitesp_CP4Ci","description":"ML monitoring architectures and choosing the right architecture for your use case.","breadcrumbs":[{"label":"ML OBSERVABILITY COURSE"},{"label":"Module 4: Designing effective ML monitoring"}]},{"id":"mF1AJg1HRPBJxoOqM9zK","title":"Module 5: ML pipelines validation and testing","pathname":"/ml-observability-course/module-5-ml-pipelines-validation-and-testing","siteSpaceId":"sitesp_CP4Ci","description":"This code-focused module demonstrates how to deploy an end-to-end pipeline for data and ML model quality checks.","breadcrumbs":[{"label":"ML OBSERVABILITY COURSE"}]},{"id":"CrIOfS5jfhgAeIuh3vRA","title":"5.1. Introduction to data and ML pipeline testing","pathname":"/ml-observability-course/module-5-ml-pipelines-validation-and-testing/introduction-data-ml-pipeline-testing","siteSpaceId":"sitesp_CP4Ci","description":"A brief introduction to different types of tests and testing conditions and how to incorporate them in data and ML pipelines.","breadcrumbs":[{"label":"ML OBSERVABILITY COURSE"},{"label":"Module 5: ML pipelines validation and testing"}]},{"id":"JL0AAMYQkdkAv5yGMF4y","title":"5.2. Train and evaluate an ML model [OPTIONAL CODE PRACTICE]","pathname":"/ml-observability-course/module-5-ml-pipelines-validation-and-testing/train-evaluate-ml-model-code-practice","siteSpaceId":"sitesp_CP4Ci","description":"A code example walkthrough of preparing the data, training, evaluating, and saving an ML model using the Evidently Python library.","breadcrumbs":[{"label":"ML OBSERVABILITY COURSE"},{"label":"Module 5: ML pipelines validation and testing"}]},{"id":"iOOZSKhL1UCgP990MJsU","title":"5.3. Test input data quality, stability and drift [CODE PRACTICE]","pathname":"/ml-observability-course/module-5-ml-pipelines-validation-and-testing/test-input-data-quality-stability-drift-code-practice","siteSpaceId":"sitesp_CP4Ci","description":"A code example walkthrough of running test suites for data quality, data stability, and data drift on raw and pre-processed data.","breadcrumbs":[{"label":"ML OBSERVABILITY COURSE"},{"label":"Module 5: ML pipelines validation and testing"}]},{"id":"BZ2Cr3zo0rmphM79BSAF","title":"5.4. Test ML model outputs and quality [CODE PRACTICE]","pathname":"/ml-observability-course/module-5-ml-pipelines-validation-and-testing/test-ml-model-outputs-quality-code-practice","siteSpaceId":"sitesp_CP4Ci","description":"A code example walkthrough of testing the quality of the model outputs after generating predictions and getting the new labeled data.","breadcrumbs":[{"label":"ML OBSERVABILITY COURSE"},{"label":"Module 5: ML pipelines validation and testing"}]},{"id":"wEKc2eVfss1JBHPkjMxC","title":"5.5. Design a custom test suite with Evidently [CODE PRACTICE]","pathname":"/ml-observability-course/module-5-ml-pipelines-validation-and-testing/design-custom-test-suite-evidently-code-practice","siteSpaceId":"sitesp_CP4Ci","description":"A code example walkthrough of creating a custom Test Suite from individual tests and passing custom test conditions in the Evidently Python library.","breadcrumbs":[{"label":"ML OBSERVABILITY COURSE"},{"label":"Module 5: ML pipelines validation and testing"}]},{"id":"AsI1GFHDkoO6aJtnhJgI","title":"5.6. Run data drift and model quality checks in an Airflow pipeline [OPTIONAL CODE PRACTICE]","pathname":"/ml-observability-course/module-5-ml-pipelines-validation-and-testing/data-drift-model-quality-checks-airflow-code-practice","siteSpaceId":"sitesp_CP4Ci","description":"A code example walkthrough of automating data and model quality checks implemented with the Evidently Python library using Airflow.","breadcrumbs":[{"label":"ML OBSERVABILITY COURSE"},{"label":"Module 5: ML pipelines validation and testing"}]},{"id":"36PBBkSfPp6mq8XcaKey","title":"5.7. Run data drift and model quality checks in a Prefect pipeline [OPTIONAL CODE PRACTICE]","pathname":"/ml-observability-course/module-5-ml-pipelines-validation-and-testing/data-drift-model-quality-checks-prefect-code-practice","siteSpaceId":"sitesp_CP4Ci","description":"A code example walkthrough of automating data and model quality checks implemented with the Evidently Python library using Prefect.","breadcrumbs":[{"label":"ML OBSERVABILITY COURSE"},{"label":"Module 5: ML pipelines validation and testing"}]},{"id":"oWYEU2Rkpt4KBZWLEgOc","title":"5.8. Log data drift test results to MLflow [CODE PRACTICE]","pathname":"/ml-observability-course/module-5-ml-pipelines-validation-and-testing/log-data-drift-test-results-mlflow-code-practice","siteSpaceId":"sitesp_CP4Ci","description":"A code example walkthrough of logging the results of data drift tests implemented with Evidently to Mlflow and viewing the results in the MLflow interface.","breadcrumbs":[{"label":"ML OBSERVABILITY COURSE"},{"label":"Module 5: ML pipelines validation and testing"}]},{"id":"Q7Ga4uWHB7Oh63wCEEEQ","title":"Module 6: Deploying an ML monitoring dashboard","pathname":"/ml-observability-course/module-6-deploying-an-ml-monitoring-dashboard","siteSpaceId":"sitesp_CP4Ci","description":"This module shows an end-to-end code example of designing and deploying an ML monitoring dashboard for batch and near real-time ML monitoring architectures.","breadcrumbs":[{"label":"ML OBSERVABILITY COURSE"}]},{"id":"BiFaam0rrfVaABzhvdBs","title":"6.1. How to deploy a live ML monitoring dashboard","pathname":"/ml-observability-course/module-6-deploying-an-ml-monitoring-dashboard/how-deploy-ml-monitoring-dashboard","siteSpaceId":"sitesp_CP4Ci","description":"ML monitoring architectures recap and how to deploy a live ML monitoring dashboard.","breadcrumbs":[{"label":"ML OBSERVABILITY COURSE"},{"label":"Module 6: Deploying an ML monitoring dashboard"}]},{"id":"TY4OdAPwDlVDgw9yJMIe","title":"6.2. ML model monitoring dashboard with Evidently. Batch architecture [CODE PRACTICE]","pathname":"/ml-observability-course/module-6-deploying-an-ml-monitoring-dashboard/batch-ml-monitoring-dashboard-code-practice","siteSpaceId":"sitesp_CP4Ci","description":"A code example walkthrough of creating a live ML monitoring dashboard for batch architecture using Evidently.","breadcrumbs":[{"label":"ML OBSERVABILITY COURSE"},{"label":"Module 6: Deploying an ML monitoring dashboard"}]},{"id":"1OjUgUJEyhtlskCKeqws","title":"6.3. ML model monitoring dashboard with Evidently. Online architecture [CODE PRACTICE]","pathname":"/ml-observability-course/module-6-deploying-an-ml-monitoring-dashboard/online-ml-monitoring-dashboard-code-practice","siteSpaceId":"sitesp_CP4Ci","description":"A code example walkthrough of creating a live ML monitoring dashboard for online architecture using Evidently.","breadcrumbs":[{"label":"ML OBSERVABILITY COURSE"},{"label":"Module 6: Deploying an ML monitoring dashboard"}]},{"id":"ZrvIDzovWCCJPUP89qMf","title":"6.4. ML monitoring with Evidently and Grafana [OPTIONAL CODE PRACTICE]","pathname":"/ml-observability-course/module-6-deploying-an-ml-monitoring-dashboard/ml-monitoring-evidently-grafana-code-practice","siteSpaceId":"sitesp_CP4Ci","description":"A code example walkthrough of adding ML monitoring metrics to Grafana using Evidently as an evaluation layer.","breadcrumbs":[{"label":"ML OBSERVABILITY COURSE"},{"label":"Module 6: Deploying an ML monitoring dashboard"}]},{"id":"S0zb50FuZ4yo0ZeDEOrH","title":"6.5. Connecting the dots: full-stack ML observability","pathname":"/ml-observability-course/module-6-deploying-an-ml-monitoring-dashboard/full-stack-ml-observability","siteSpaceId":"sitesp_CP4Ci","description":"A brief summary of the Open-source ML observability course learnings.","breadcrumbs":[{"label":"ML OBSERVABILITY COURSE"},{"label":"Module 6: Deploying an ML monitoring dashboard"}]}]}