# ML OBSERVABILITY COURSE - [Module 1: Introduction](https://learn.evidentlyai.com/ml-observability-course/module-1-introduction.md): Key concepts of machine learning monitoring and observability and how they fit in the ML lifecycle. - [1.1. ML lifecycle. What can go wrong with ML in production?](https://learn.evidentlyai.com/ml-observability-course/module-1-introduction/ml-lifecycle.md): What can go wrong with data and machine learning services in production. Data quality issues, data drift, and concept drift. - [1.2. What is ML monitoring and observability?](https://learn.evidentlyai.com/ml-observability-course/module-1-introduction/ml-monitoring-observability.md): What ML monitoring is, the challenges of production ML monitoring, and how it differs from ML observability. - [1.3. ML monitoring metrics. What exactly can you monitor?](https://learn.evidentlyai.com/ml-observability-course/module-1-introduction/ml-monitoring-metrics.md): A framework to organize ML monitoring metrics. Software system health, data quality, ML model quality, and business KPIs. - [1.4. Key considerations for ML monitoring setup](https://learn.evidentlyai.com/ml-observability-course/module-1-introduction/ml-monitoring-setup.md): Key considerations for ML monitoring setup. Service criticality, retraining cadence, reference dataset, and ML monitoring architecture. - [1.5. ML monitoring architectures](https://learn.evidentlyai.com/ml-observability-course/module-1-introduction/ml-monitoring-architectures.md): ML monitoring architectures for backend and frontend and what to consider when choosing between them. - [Module 2: ML monitoring metrics](https://learn.evidentlyai.com/ml-observability-course/module-2-ml-monitoring-metrics.md): This module covers different aspects of the production ML model performance. You will learn how to apply data quality, model quality, and data drift metrics for structured data. - [2.1. How to evaluate ML model quality](https://learn.evidentlyai.com/ml-observability-course/module-2-ml-monitoring-metrics/evaluate-ml-model-quality.md): How to evaluate ML model quality directly and use early monitoring to detect potential ML model issues. - [2.2. Overview of ML quality metrics. Classification, regression, ranking](https://learn.evidentlyai.com/ml-observability-course/module-2-ml-monitoring-metrics/ml-quality-metrics-classification-regression-ranking.md): Commonly used ML quality metrics for classification, regression, and ranking problems. - [2.3. Evaluating ML model quality \[CODE PRACTICE\]](https://learn.evidentlyai.com/ml-observability-course/module-2-ml-monitoring-metrics/ml-model-quality-code-practice.md): A code example walkthrough of ML model quality evaluation using Python and the open-source Evidently library. - [2.4. Data quality in machine learning](https://learn.evidentlyai.com/ml-observability-course/module-2-ml-monitoring-metrics/data-quality-in-ml.md): Types of production data quality issues, how to evaluate data quality, and interpret data quality metrics. - [2.5. Data quality in ML \[CODE PRACTICE\]](https://learn.evidentlyai.com/ml-observability-course/module-2-ml-monitoring-metrics/data-quality-code-practice.md): A code example walkthrough of data quality evaluation using Evidently Reports and Test Suites. - [2.6. Data and prediction drift in ML](https://learn.evidentlyai.com/ml-observability-course/module-2-ml-monitoring-metrics/data-prediction-drift-in-ml.md): What data and prediction drift is, and how to detect distribution drift using statistical methods and rule-based checks. - [2.7. Deep dive into data drift detection \[OPTIONAL\]](https://learn.evidentlyai.com/ml-observability-course/module-2-ml-monitoring-metrics/data-drift-deep-dive.md): A deep dive into data drift detection methods, how to choose the right approach for your use case, and what to do when the drift is detected. - [2.8. Data and prediction drift in ML \[CODE PRACTICE\]](https://learn.evidentlyai.com/ml-observability-course/module-2-ml-monitoring-metrics/data-prediction-drift-code-practice.md): A code example walkthrough of detecting data drift and creating a custom method for drift detection using Evidently. - [Module 3: ML monitoring for unstructured data](https://learn.evidentlyai.com/ml-observability-course/module-3-ml-monitoring-for-unstructured-data.md): This module covers evaluating and monitoring the production performance for models that use unstructured data, including NLP, LLMs, and embeddings. - [3.1. Introduction to NLP and LLM monitoring](https://learn.evidentlyai.com/ml-observability-course/module-3-ml-monitoring-for-unstructured-data/introduction-nlp-llm-monitoring.md): How to evaluate model quality for NLP and LLMs and monitor text data without labels using raw data, embeddings, and descriptors. - [3.2. Monitoring data drift on raw text data](https://learn.evidentlyai.com/ml-observability-course/module-3-ml-monitoring-for-unstructured-data/monitoring-data-drift-on-raw-text-data.md): How to detect and evaluate raw text data drift using domain classifier and topic modeling. - [3.3. Monitoring text data quality and data drift with descriptors](https://learn.evidentlyai.com/ml-observability-course/module-3-ml-monitoring-for-unstructured-data/monitoring-data-drift-with-descriptors.md): What text descriptors are and how to use them to monitor text data quality and data drift. - [3.4. Monitoring embeddings drift](https://learn.evidentlyai.com/ml-observability-course/module-3-ml-monitoring-for-unstructured-data/monitoring-embeddings-drift.md): Strategies for monitoring embedding drift using distance metrics, model-based drift detection, and share of drifted components. - [3.5. Monitoring text data \[CODE PRACTICE\]](https://learn.evidentlyai.com/ml-observability-course/module-3-ml-monitoring-for-unstructured-data/monitoring-text-data-code-practice.md): A code example walkthrough of unstructured data evaluations using the open-source Evidently Python library. - [3.6. Monitoring multimodal datasets](https://learn.evidentlyai.com/ml-observability-course/module-3-ml-monitoring-for-unstructured-data/monitoring-multimodal-datasets.md): Strategies for monitoring data quality and data drift in multimodal datasets. - [Module 4: Designing effective ML monitoring](https://learn.evidentlyai.com/ml-observability-course/module-4-designing-effective-ml-monitoring.md): This module reviews how to set up an ML monitoring system, considering the model risks, criticality, and deployment scenario. - [4.1. Logging for ML monitoring](https://learn.evidentlyai.com/ml-observability-course/module-4-designing-effective-ml-monitoring/logging-ml-monitoring.md): What a good ML monitoring system is, and how to set up the logging architecture to capture metrics for further analysis. - [4.2. How to prioritize ML monitoring metrics](https://learn.evidentlyai.com/ml-observability-course/module-4-designing-effective-ml-monitoring/how-to-prioritize-monitoring-metrics.md): ML monitoring depth, metrics to collect when building a monitoring system, and how to prioritize them. - [4.3. When to retrain machine learning models](https://learn.evidentlyai.com/ml-observability-course/module-4-designing-effective-ml-monitoring/when-to-retrain-ml-models.md): Scheduled and trigger-based retraining and what to consider when making the retaining decision. - [4.4. How to choose a reference dataset in ML monitoring](https://learn.evidentlyai.com/ml-observability-course/module-4-designing-effective-ml-monitoring/how-to-choose-reference-dataset-ml-monitoring.md): What a reference dataset is in ML monitoring, how to choose one for drift detection, and when to use multiple references. - [4.5. Custom metrics in ML monitoring](https://learn.evidentlyai.com/ml-observability-course/module-4-designing-effective-ml-monitoring/custom-metrics-ml-monitoring.md): Types of custom metrics. Business or product metrics, domain-specific metrics, and weighted metrics. - [4.6. Implementing custom metrics in Evidently \[OPTIONAL\]](https://learn.evidentlyai.com/ml-observability-course/module-4-designing-effective-ml-monitoring/custom-metrics-evidently-code-practice.md): A code example walkthrough of creating a custom metric using the Evidently Python library. - [4.7. How to choose the ML monitoring deployment architecture](https://learn.evidentlyai.com/ml-observability-course/module-4-designing-effective-ml-monitoring/choosing-ml-monitoring-deployment-architecture.md): ML monitoring architectures and choosing the right architecture for your use case. - [Module 5: ML pipelines validation and testing](https://learn.evidentlyai.com/ml-observability-course/module-5-ml-pipelines-validation-and-testing.md): This code-focused module demonstrates how to deploy an end-to-end pipeline for data and ML model quality checks. - [5.1. Introduction to data and ML pipeline testing](https://learn.evidentlyai.com/ml-observability-course/module-5-ml-pipelines-validation-and-testing/introduction-data-ml-pipeline-testing.md): A brief introduction to different types of tests and testing conditions and how to incorporate them in data and ML pipelines. - [5.2. Train and evaluate an ML model \[OPTIONAL CODE PRACTICE\]](https://learn.evidentlyai.com/ml-observability-course/module-5-ml-pipelines-validation-and-testing/train-evaluate-ml-model-code-practice.md): A code example walkthrough of preparing the data, training, evaluating, and saving an ML model using the Evidently Python library. - [5.3. Test input data quality, stability and drift \[CODE PRACTICE\]](https://learn.evidentlyai.com/ml-observability-course/module-5-ml-pipelines-validation-and-testing/test-input-data-quality-stability-drift-code-practice.md): A code example walkthrough of running test suites for data quality, data stability, and data drift on raw and pre-processed data. - [5.4. Test ML model outputs and quality \[CODE PRACTICE\]](https://learn.evidentlyai.com/ml-observability-course/module-5-ml-pipelines-validation-and-testing/test-ml-model-outputs-quality-code-practice.md): A code example walkthrough of testing the quality of the model outputs after generating predictions and getting the new labeled data. - [5.5. Design a custom test suite with Evidently \[CODE PRACTICE\]](https://learn.evidentlyai.com/ml-observability-course/module-5-ml-pipelines-validation-and-testing/design-custom-test-suite-evidently-code-practice.md): A code example walkthrough of creating a custom Test Suite from individual tests and passing custom test conditions in the Evidently Python library. - [5.6. Run data drift and model quality checks in an Airflow pipeline \[OPTIONAL CODE PRACTICE\]](https://learn.evidentlyai.com/ml-observability-course/module-5-ml-pipelines-validation-and-testing/data-drift-model-quality-checks-airflow-code-practice.md): A code example walkthrough of automating data and model quality checks implemented with the Evidently Python library using Airflow. - [5.7. Run data drift and model quality checks in a Prefect pipeline \[OPTIONAL CODE PRACTICE\]](https://learn.evidentlyai.com/ml-observability-course/module-5-ml-pipelines-validation-and-testing/data-drift-model-quality-checks-prefect-code-practice.md): A code example walkthrough of automating data and model quality checks implemented with the Evidently Python library using Prefect. - [5.8. Log data drift test results to MLflow \[CODE PRACTICE\]](https://learn.evidentlyai.com/ml-observability-course/module-5-ml-pipelines-validation-and-testing/log-data-drift-test-results-mlflow-code-practice.md): A code example walkthrough of logging the results of data drift tests implemented with Evidently to Mlflow and viewing the results in the MLflow interface. - [Module 6: Deploying an ML monitoring dashboard](https://learn.evidentlyai.com/ml-observability-course/module-6-deploying-an-ml-monitoring-dashboard.md): This module shows an end-to-end code example of designing and deploying an ML monitoring dashboard for batch and near real-time ML monitoring architectures. - [6.1. How to deploy a live ML monitoring dashboard](https://learn.evidentlyai.com/ml-observability-course/module-6-deploying-an-ml-monitoring-dashboard/how-deploy-ml-monitoring-dashboard.md): ML monitoring architectures recap and how to deploy a live ML monitoring dashboard. - [6.2. ML model monitoring dashboard with Evidently. Batch architecture \[CODE PRACTICE\]](https://learn.evidentlyai.com/ml-observability-course/module-6-deploying-an-ml-monitoring-dashboard/batch-ml-monitoring-dashboard-code-practice.md): A code example walkthrough of creating a live ML monitoring dashboard for batch architecture using Evidently. - [6.3. ML model monitoring dashboard with Evidently. Online architecture \[CODE PRACTICE\]](https://learn.evidentlyai.com/ml-observability-course/module-6-deploying-an-ml-monitoring-dashboard/online-ml-monitoring-dashboard-code-practice.md): A code example walkthrough of creating a live ML monitoring dashboard for online architecture using Evidently. - [6.4. ML monitoring with Evidently and Grafana \[OPTIONAL CODE PRACTICE\]](https://learn.evidentlyai.com/ml-observability-course/module-6-deploying-an-ml-monitoring-dashboard/ml-monitoring-evidently-grafana-code-practice.md): A code example walkthrough of adding ML monitoring metrics to Grafana using Evidently as an evaluation layer. - [6.5. Connecting the dots: full-stack ML observability](https://learn.evidentlyai.com/ml-observability-course/module-6-deploying-an-ml-monitoring-dashboard/full-stack-ml-observability.md): A brief summary of the Open-source ML observability course learnings. --- # Agent Instructions: Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://learn.evidentlyai.com/ml-observability-course.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.