Apache Spark MLlib and Machine Learning

Question 1

What is the primary high-level API for machine learning in Apache Spark?

Accepted Answer

spark.ml (DataFrame-based API)

Answer

spark.mllib (RDD-based API)

Answer

spark.learn

Answer

spark.ai

Question 2

What is a Pipeline in Spark MLlib?

Accepted Answer

A sequence of stages (Transformers and Estimators) that form an ML workflow

Answer

A data ingestion tool for streaming ML features

Answer

A distributed model serving layer

Answer

A feature store for machine learning datasets

Question 3

What is the difference between a Transformer and an Estimator in Spark ML?

Accepted Answer

A Transformer applies a transformation to a DataFrame; an Estimator fits data to produce a Transformer

Answer

A Transformer trains on data; an Estimator applies a transformation

Answer

Both are identical; the names are interchangeable

Answer

A Transformer is for feature engineering; an Estimator is for evaluation

Question 4

Which Spark MLlib class is used to convert categorical string labels to numeric indices?

Accepted Answer

StringIndexer

Answer

OneHotEncoder

Answer

LabelEncoder

Answer

CategoryMapper

Question 5

Which Spark MLlib algorithm is used for collaborative filtering-based recommendations?

Accepted Answer

ALS (Alternating Least Squares)

Answer

LinearRegression

Answer

KMeans

Answer

RandomForest

Question 6

What is the purpose of the VectorAssembler in Spark MLlib?

Accepted Answer

Combines multiple feature columns into a single vector column

Answer

Scales feature values to a standard range

Answer

Reduces the dimensionality of features

Answer

Encodes categorical features as binary vectors