AI AI Engineer: MLOps and Model Deployment 2 — Questions and Answers
Question 1: What is the role of an A/B test in an MLOps deployment workflow?
- To compress model size
- To statistically compare two model versions using live traffic (Correct answer)
- To retrain a model on new data
- To validate training data quality
Correct answer: To statistically compare two model versions using live traffic
A/B testing splits live traffic between two model versions and uses statistical analysis to determine which performs better.
Question 2: Which metric is most useful for detecting concept drift in a classification model?
- Training loss
- Inference latency
- Prediction distribution shift over time (Correct answer)
- GPU utilization
Correct answer: Prediction distribution shift over time
Monitoring the distribution of model predictions over time reveals concept drift because the model's output distribution changes as real-world patterns evolve.
Question 3: What is 'shadow mode' deployment in MLOps?
- Running the model only at night to save costs
- Sending production traffic to a new model without using its outputs to serve users (Correct answer)
- Encrypting model weights before deployment
- Deploying a model without logging
Correct answer: Sending production traffic to a new model without using its outputs to serve users
Shadow mode runs a new model on real traffic in parallel with the live model, comparing outputs without impacting users.
Question 4: Which tool is commonly used for experiment tracking in ML, allowing teams to log parameters, metrics, and artifacts?
- Jenkins
- MLflow (Correct answer)
- Terraform
- Prometheus
Correct answer: MLflow
MLflow is an open-source platform for tracking experiments, packaging code, and managing model lifecycle.
Question 5: What is 'online serving' in ML deployment as opposed to 'batch inference'?
- Training models using online learning algorithms
- Generating predictions in real time for individual requests (Correct answer)
- Serving models over the internet
- Running inference on streaming data lakes
Correct answer: Generating predictions in real time for individual requests
Online serving generates low-latency predictions on demand for individual requests, unlike batch inference which processes large datasets periodically.
Question 6: Which practice helps prevent training-serving skew in production ML systems?
- Using different programming languages for training and serving
- Sharing the same feature pipeline code between training and serving (Correct answer)
- Retraining models weekly
- Normalizing outputs post-inference
Correct answer: Sharing the same feature pipeline code between training and serving
Sharing the same feature transformation code between training and serving ensures the model sees identical feature representations in both contexts.
AI AI Engineer: MLOps and Model Deployment 2
What is the role of an A/B test in an MLOps deployment workflow?