Apache Spark Spark Architecture and Cluster Management

Question 1

What is the role of the Spark Driver in a Spark application?

Accepted Answer

Hosts the SparkContext, maintains the DAG, and coordinates task scheduling across executors

Answer

Executes the actual task computations on data partitions

Answer

Manages cluster resources and allocates containers

Answer

Stores intermediate shuffle data between stages

Question 2

What are Spark Executors?

Accepted Answer

JVM processes launched by the cluster manager that execute tasks and store RDD data

Answer

Worker nodes in the cluster that manage data replication

Answer

Processes on the driver node that compile the DAG into tasks

Answer

Memory managers that handle executor JVM garbage collection

Question 3

Which cluster manager is natively built into Apache Spark for standalone deployments?

Accepted Answer

Spark Standalone Mode

Answer

YARN

Answer

Mesos

Answer

Kubernetes

Question 4

What is a Spark Stage?

Accepted Answer

A phase of the DAG separated by wide transformations (shuffle boundaries)

Answer

A group of tasks that can be executed in parallel without a shuffle

Answer

A single task running on one partition

Answer

A container for multiple Spark jobs

Question 5

What is a Spark Task?

Accepted Answer

A unit of work sent to one executor to process one data partition

Answer

A complete Spark job submitted by the driver

Answer

A scheduled operation that runs on a timer

Answer

A sub-DAG of transformations within a single stage

Question 6

In Spark's DAG (Directed Acyclic Graph), what does each node represent?

Accepted Answer

An RDD or DataFrame with transformations applied

Answer

An executor JVM process

Answer

An RDD or DataFrame partition

Answer

A shuffle exchange operation