DSE Big Data Technologies 2 — Questions and Answers
Question 1: Which component of the Hadoop ecosystem is responsible for distributed storage?
- YARN
- HDFS (Correct answer)
- MapReduce
- HBase
Correct answer: HDFS
HDFS (Hadoop Distributed File System) is the storage layer that splits files into blocks and distributes them across cluster nodes.
Question 2: In Apache Kafka, what is a 'consumer group' used for?
- Grouping producers by topic
- Allowing multiple consumers to read different partitions in parallel (Correct answer)
- Batching messages before writing to disk
- Encrypting messages at rest
Correct answer: Allowing multiple consumers to read different partitions in parallel
A consumer group lets multiple consumers divide the partitions of a topic among themselves, enabling parallel and scalable consumption.
Question 3: What does the CAP theorem state about distributed systems?
- You can guarantee Consistency, Availability, and Partition tolerance simultaneously
- You can guarantee at most two of Consistency, Availability, and Partition tolerance (Correct answer)
- Partition tolerance is optional in modern cloud systems
- Consistency is always sacrificed for availability
Correct answer: You can guarantee at most two of Consistency, Availability, and Partition tolerance
The CAP theorem states that a distributed system can guarantee only two of the three properties: Consistency, Availability, and Partition tolerance.
Question 4: Which Apache project provides a SQL-like interface for querying data stored in HDFS?
- Pig
- Hive (Correct answer)
- Sqoop
- Flume
Correct answer: Hive
Apache Hive provides HiveQL, a SQL-like language that translates queries into MapReduce or Tez jobs over HDFS data.
Question 5: In Spark, what is the primary advantage of using DataFrames over RDDs?
- DataFrames support more programming languages
- DataFrames enable Catalyst optimizer and Tungsten execution engine optimizations (Correct answer)
- DataFrames allow in-place mutation of data
- DataFrames store data in HDFS by default
Correct answer: DataFrames enable Catalyst optimizer and Tungsten execution engine optimizations
DataFrames expose schema information that allows Spark's Catalyst optimizer and Tungsten execution engine to apply significant performance optimizations.
Question 6: What is 'data skew' in the context of distributed big data processing?
- Data that fails schema validation
- Uneven distribution of data across partitions causing some tasks to run much longer (Correct answer)
- Corrupted records in a dataset
- Data that arrives out of chronological order
Correct answer: Uneven distribution of data across partitions causing some tasks to run much longer
Data skew occurs when some partitions hold significantly more data than others, creating bottleneck tasks that slow down the entire job.
Question 7: Which storage format is columnar and commonly used in the Hadoop ecosystem for analytical workloads?
- Avro
- JSON
- Parquet (Correct answer)
- CSV
Correct answer: Parquet
Parquet is a columnar storage format that provides efficient compression and encoding, making it well-suited for analytical queries on large datasets.
DSE Big Data Technologies 2
Which component of the Hadoop ecosystem is responsible for distributed storage?