Big Data Technologies Flashcards
7 cards from real MS-DS Master of Data science practice questions. Tap to flip, then mark Knew It or Still Learning — missed cards come back until you master them.
Read the first 7 Big Data Technologies flashcards as text
Which Hadoop component is responsible for resource management and job scheduling across a cluster?
Answer: YARN
YARN (Yet Another Resource Negotiator) manages cluster resources and schedules jobs in Hadoop 2.x and later.
In Apache Kafka, what is the role of a 'consumer group'?
Answer: A collection of consumers that jointly consume messages from a topic without duplication
A consumer group allows multiple consumers to divide topic partitions among themselves, ensuring each message is processed by only one member.
What distinguishes Apache Flink's 'event time' processing from 'processing time' processing?
Answer: Event time uses the timestamp embedded in the data record; processing time uses the wall-clock time at the operator
Event time relies on timestamps within events, enabling correct results even for out-of-order or late-arriving data.
Which storage format is columnar, supports predicate pushdown, and is commonly used in the Hadoop ecosystem for analytics?
Answer: Parquet
Parquet is a columnar storage format that enables efficient predicate pushdown and compression, making it ideal for analytical queries.
In Apache Spark, what is the primary difference between a 'transformation' and an 'action'?
Answer: Transformations are lazy and return a new RDD; actions trigger computation and return a result
Transformations are lazy operations that define a new RDD lineage, while actions trigger actual computation and return values to the driver.
Which CAP theorem property does Apache Cassandra prioritize by default?
Answer: Availability and Partition tolerance (AP)
Cassandra is designed as an AP system, favoring availability and partition tolerance over strong consistency, though tunable consistency levels are available.
What is the purpose of 'watermarks' in Apache Flink streaming applications?
Answer: To signal the progress of event time and trigger window computations despite late data
Watermarks indicate that no event with a timestamp earlier than the watermark value is expected, allowing Flink to close windows and emit results.