Spark Architecture and Cluster Management Flashcards
6 cards from real Apache Spark practice questions. Tap to flip, then mark Knew It or Still Learning — missed cards come back until you master them.
Read the first 6 Spark Architecture and Cluster Management flashcards as text
What does deploy mode 'cluster' mean in Spark?
Answer: The Driver process runs on one of the worker nodes managed by the cluster manager
In cluster deploy mode, the Driver runs on one of the worker nodes in the cluster, which is preferred for production jobs.
What is dynamic resource allocation in Spark?
Answer: Scaling the number of executors up and down based on workload
Dynamic resource allocation allows Spark to add executors when tasks are queued and remove idle executors to free cluster resources.
What is the purpose of the external shuffle service in Spark?
Answer: Maintains shuffle files on worker nodes so executors can be released during dynamic allocation
The external shuffle service runs on worker nodes and holds shuffle files, allowing executors to be released while preserving shuffle data.
What does spark-submit --master yarn --deploy-mode cluster do?
Answer: Submits a Spark application to a YARN cluster with the driver running on a YARN container
This command submits a Spark application to YARN and runs the Driver in a YARN ApplicationMaster container on the cluster.
What is YARN in the context of Apache Spark?
Answer: A cluster resource manager that allocates containers for Spark executors and the driver
YARN (Yet Another Resource Negotiator) is a Hadoop resource manager that Spark uses to allocate containers for executors and the application master.
How does Spark handle executor failures during a job?
Answer: Spark re-schedules the failed tasks on other available executors using RDD lineage
When an executor fails, Spark reschedules its tasks on other executors and recomputes any lost RDD partitions using lineage.