# ML OBSERVABILITY COURSE

- [Module 1: Introduction](/ml-observability-course/module-1-introduction.md): Key concepts of machine learning monitoring and observability and how they fit in the ML lifecycle.
- [1.1. ML lifecycle. What can go wrong with ML in production?](/ml-observability-course/module-1-introduction/ml-lifecycle.md): What can go wrong with data and machine learning services in production. Data quality issues, data drift, and concept drift.
- [1.2. What is ML monitoring and observability?](/ml-observability-course/module-1-introduction/ml-monitoring-observability.md): What ML monitoring is, the challenges of production ML monitoring, and how it differs from ML observability.
- [1.3. ML monitoring metrics. What exactly can you monitor?](/ml-observability-course/module-1-introduction/ml-monitoring-metrics.md): A framework to organize ML monitoring metrics. Software system health, data quality, ML model quality, and business KPIs.
- [1.4. Key considerations for ML monitoring setup](/ml-observability-course/module-1-introduction/ml-monitoring-setup.md): Key considerations for ML monitoring setup. Service criticality, retraining cadence, reference dataset, and ML monitoring architecture.
- [1.5. ML monitoring architectures](/ml-observability-course/module-1-introduction/ml-monitoring-architectures.md): ML monitoring architectures for backend and frontend and what to consider when choosing between them.
- [Module 2: ML monitoring metrics](/ml-observability-course/module-2-ml-monitoring-metrics.md): This module covers different aspects of the production ML model performance. You will learn how to apply data quality, model quality, and data drift metrics for structured data.
- [2.1. How to evaluate ML model quality](/ml-observability-course/module-2-ml-monitoring-metrics/evaluate-ml-model-quality.md): How to evaluate ML model quality directly and use early monitoring to detect potential ML model issues.
- [2.2. Overview of ML quality metrics. Classification, regression, ranking](/ml-observability-course/module-2-ml-monitoring-metrics/ml-quality-metrics-classification-regression-ranking.md): Commonly used ML quality metrics for classification, regression, and ranking problems.
- [2.3. Evaluating ML model quality \[CODE PRACTICE\]](/ml-observability-course/module-2-ml-monitoring-metrics/ml-model-quality-code-practice.md): A code example walkthrough of ML model quality evaluation using Python and the open-source Evidently library.
- [2.4. Data quality in machine learning](/ml-observability-course/module-2-ml-monitoring-metrics/data-quality-in-ml.md): Types of production data quality issues, how to evaluate data quality, and interpret data quality metrics.
- [2.5. Data quality in ML \[CODE PRACTICE\]](/ml-observability-course/module-2-ml-monitoring-metrics/data-quality-code-practice.md): A code example walkthrough of data quality evaluation using Evidently Reports and Test Suites.
- [2.6. Data and prediction drift in ML](/ml-observability-course/module-2-ml-monitoring-metrics/data-prediction-drift-in-ml.md): What data and prediction drift is, and how to detect distribution drift using statistical methods and rule-based checks.
- [2.7. Deep dive into data drift detection \[OPTIONAL\]](/ml-observability-course/module-2-ml-monitoring-metrics/data-drift-deep-dive.md): A deep dive into data drift detection methods, how to choose the right approach for your use case, and what to do when the drift is detected.
- [2.8. Data and prediction drift in ML \[CODE PRACTICE\]](/ml-observability-course/module-2-ml-monitoring-metrics/data-prediction-drift-code-practice.md): A code example walkthrough of detecting data drift and creating a custom method for drift detection using Evidently.
- [Module 3: ML monitoring for unstructured data](/ml-observability-course/module-3-ml-monitoring-for-unstructured-data.md): This module covers evaluating and monitoring the production performance for models that use unstructured data, including NLP, LLMs, and embeddings.
- [3.1. Introduction to NLP and LLM monitoring](/ml-observability-course/module-3-ml-monitoring-for-unstructured-data/introduction-nlp-llm-monitoring.md): How to evaluate model quality for NLP and LLMs and monitor text data without labels using raw data, embeddings, and descriptors.
- [3.2. Monitoring data drift on raw text data](/ml-observability-course/module-3-ml-monitoring-for-unstructured-data/monitoring-data-drift-on-raw-text-data.md): How to detect and evaluate raw text data drift using domain classifier and topic modeling.
- [3.3. Monitoring text data quality and data drift with descriptors](/ml-observability-course/module-3-ml-monitoring-for-unstructured-data/monitoring-data-drift-with-descriptors.md): What text descriptors are and how to use them to monitor text data quality and data drift.
- [3.4. Monitoring embeddings drift](/ml-observability-course/module-3-ml-monitoring-for-unstructured-data/monitoring-embeddings-drift.md): Strategies for monitoring embedding drift using distance metrics, model-based drift detection, and share of drifted components.
- [3.5. Monitoring text data \[CODE PRACTICE\]](/ml-observability-course/module-3-ml-monitoring-for-unstructured-data/monitoring-text-data-code-practice.md): A code example walkthrough of unstructured data evaluations using the open-source Evidently Python library.
- [3.6. Monitoring multimodal datasets](/ml-observability-course/module-3-ml-monitoring-for-unstructured-data/monitoring-multimodal-datasets.md): Strategies for monitoring data quality and data drift in multimodal datasets.
- [Module 4: Designing effective ML monitoring](/ml-observability-course/module-4-designing-effective-ml-monitoring.md): This module reviews how to set up an ML monitoring system, considering the model risks, criticality, and deployment scenario.
- [4.1. Logging for ML monitoring](/ml-observability-course/module-4-designing-effective-ml-monitoring/logging-ml-monitoring.md): What a good ML monitoring system is, and how to set up the logging architecture to capture metrics for further analysis.
- [4.2. How to prioritize ML monitoring metrics](/ml-observability-course/module-4-designing-effective-ml-monitoring/how-to-prioritize-monitoring-metrics.md): ML monitoring depth, metrics to collect when building a monitoring system, and how to prioritize them.
- [4.3. When to retrain machine learning models](/ml-observability-course/module-4-designing-effective-ml-monitoring/when-to-retrain-ml-models.md): Scheduled and trigger-based retraining and what to consider when making the retaining decision.
- [4.4. How to choose a reference dataset in ML monitoring](/ml-observability-course/module-4-designing-effective-ml-monitoring/how-to-choose-reference-dataset-ml-monitoring.md): What a reference dataset is in ML monitoring, how to choose one for drift detection, and when to use multiple references.
- [4.5. Custom metrics in ML monitoring](/ml-observability-course/module-4-designing-effective-ml-monitoring/custom-metrics-ml-monitoring.md): Types of custom metrics. Business or product metrics, domain-specific metrics, and weighted metrics.
- [4.6. Implementing custom metrics in Evidently \[OPTIONAL\]](/ml-observability-course/module-4-designing-effective-ml-monitoring/custom-metrics-evidently-code-practice.md): A code example walkthrough of creating a custom metric using the Evidently Python library.
- [4.7. How to choose the ML monitoring deployment architecture](/ml-observability-course/module-4-designing-effective-ml-monitoring/choosing-ml-monitoring-deployment-architecture.md): ML monitoring architectures and choosing the right architecture for your use case.
- [Module 5: ML pipelines validation and testing](/ml-observability-course/module-5-ml-pipelines-validation-and-testing.md): This code-focused module demonstrates how to deploy an end-to-end pipeline for data and ML model quality checks.
- [5.1. Introduction to data and ML pipeline testing](/ml-observability-course/module-5-ml-pipelines-validation-and-testing/introduction-data-ml-pipeline-testing.md): A brief introduction to different types of tests and testing conditions and how to incorporate them in data and ML pipelines.
- [5.2. Train and evaluate an ML model \[OPTIONAL CODE PRACTICE\]](/ml-observability-course/module-5-ml-pipelines-validation-and-testing/train-evaluate-ml-model-code-practice.md): A code example walkthrough of preparing the data, training, evaluating, and saving an ML model using the Evidently Python library.
- [5.3. Test input data quality, stability and drift \[CODE PRACTICE\]](/ml-observability-course/module-5-ml-pipelines-validation-and-testing/test-input-data-quality-stability-drift-code-practice.md): A code example walkthrough of running test suites for data quality, data stability, and data drift on raw and pre-processed data.
- [5.4. Test ML model outputs and quality \[CODE PRACTICE\]](/ml-observability-course/module-5-ml-pipelines-validation-and-testing/test-ml-model-outputs-quality-code-practice.md): A code example walkthrough of testing the quality of the model outputs after generating predictions and getting the new labeled data.
- [5.5. Design a custom test suite with Evidently \[CODE PRACTICE\]](/ml-observability-course/module-5-ml-pipelines-validation-and-testing/design-custom-test-suite-evidently-code-practice.md): A code example walkthrough of creating a custom Test Suite from individual tests and passing custom test conditions in the Evidently Python library.
- [5.6. Run data drift and model quality checks in an Airflow pipeline \[OPTIONAL CODE PRACTICE\]](/ml-observability-course/module-5-ml-pipelines-validation-and-testing/data-drift-model-quality-checks-airflow-code-practice.md): A code example walkthrough of automating data and model quality checks implemented with the Evidently Python library using Airflow.
- [5.7. Run data drift and model quality checks in a Prefect pipeline \[OPTIONAL CODE PRACTICE\]](/ml-observability-course/module-5-ml-pipelines-validation-and-testing/data-drift-model-quality-checks-prefect-code-practice.md): A code example walkthrough of automating data and model quality checks implemented with the Evidently Python library using Prefect.
- [5.8. Log data drift test results to MLflow \[CODE PRACTICE\]](/ml-observability-course/module-5-ml-pipelines-validation-and-testing/log-data-drift-test-results-mlflow-code-practice.md): A code example walkthrough of logging the results of data drift tests implemented with Evidently to Mlflow and viewing the results in the MLflow interface.
- [Module 6: Deploying an ML monitoring dashboard](/ml-observability-course/module-6-deploying-an-ml-monitoring-dashboard.md): This module shows an end-to-end code example of designing and deploying an ML monitoring dashboard for batch and near real-time ML monitoring architectures.
- [6.1. How to deploy a live ML monitoring dashboard](/ml-observability-course/module-6-deploying-an-ml-monitoring-dashboard/how-deploy-ml-monitoring-dashboard.md): ML monitoring architectures recap and how to deploy a live ML monitoring dashboard.
- [6.2. ML model monitoring dashboard with Evidently. Batch architecture \[CODE PRACTICE\]](/ml-observability-course/module-6-deploying-an-ml-monitoring-dashboard/batch-ml-monitoring-dashboard-code-practice.md): A code example walkthrough of creating a live ML monitoring dashboard for batch architecture using Evidently.
- [6.3. ML model monitoring dashboard with Evidently. Online architecture \[CODE PRACTICE\]](/ml-observability-course/module-6-deploying-an-ml-monitoring-dashboard/online-ml-monitoring-dashboard-code-practice.md): A code example walkthrough of creating a live ML monitoring dashboard for online architecture using Evidently.
- [6.4. ML monitoring with Evidently and Grafana \[OPTIONAL CODE PRACTICE\]](/ml-observability-course/module-6-deploying-an-ml-monitoring-dashboard/ml-monitoring-evidently-grafana-code-practice.md): A code example walkthrough of adding ML monitoring metrics to Grafana using Evidently as an evaluation layer.
- [6.5. Connecting the dots: full-stack ML observability](/ml-observability-course/module-6-deploying-an-ml-monitoring-dashboard/full-stack-ml-observability.md): A brief summary of the Open-source ML observability course learnings.