Apache Spark MLlib and Machine Learning 1 — Questions and Answers
Question 1: What is the primary high-level API for machine learning in Apache Spark?
- spark.ml (DataFrame-based API) (Correct answer)
- spark.mllib (RDD-based API)
- spark.learn
- spark.ai
Correct answer: spark.ml (DataFrame-based API)
spark.ml is the primary, DataFrame-based API for machine learning in Spark, while spark.mllib is the older RDD-based library.
Question 2: What is a Pipeline in Spark MLlib?
- A data ingestion tool for streaming ML features
- A sequence of stages (Transformers and Estimators) that form an ML workflow (Correct answer)
- A distributed model serving layer
- A feature store for machine learning datasets
Correct answer: A sequence of stages (Transformers and Estimators) that form an ML workflow
A Pipeline chains multiple Transformers and Estimators into a single ML workflow, simplifying training and prediction.
Question 3: What is the difference between a Transformer and an Estimator in Spark ML?
- A Transformer trains on data; an Estimator applies a transformation
- A Transformer applies a transformation to a DataFrame; an Estimator fits data to produce a Transformer (Correct answer)
- Both are identical; the names are interchangeable
- A Transformer is for feature engineering; an Estimator is for evaluation
Correct answer: A Transformer applies a transformation to a DataFrame; an Estimator fits data to produce a Transformer
An Estimator has a fit() method that trains on data and returns a Transformer, which has a transform() method.
Question 4: Which Spark MLlib class is used to convert categorical string labels to numeric indices?
- OneHotEncoder
- StringIndexer (Correct answer)
- LabelEncoder
- CategoryMapper
Correct answer: StringIndexer
StringIndexer encodes a string column of labels to a column of label indices, ordered by label frequencies.
Question 5: Which Spark MLlib algorithm is used for collaborative filtering-based recommendations?
- LinearRegression
- ALS (Alternating Least Squares) (Correct answer)
- KMeans
- RandomForest
Correct answer: ALS (Alternating Least Squares)
ALS (Alternating Least Squares) is Spark's collaborative filtering algorithm for building recommendation systems.
Question 6: What is the purpose of the VectorAssembler in Spark MLlib?
- Scales feature values to a standard range
- Combines multiple feature columns into a single vector column (Correct answer)
- Reduces the dimensionality of features
- Encodes categorical features as binary vectors
Correct answer: Combines multiple feature columns into a single vector column
VectorAssembler merges multiple numeric columns into a single feature vector required by Spark ML algorithms.
Apache Spark MLlib and Machine Learning 1
What is the primary high-level API for machine learning in Apache Spark?