DSE Big Data Technologies 3 — Questions and Answers
Question 1: What is the role of Apache ZooKeeper in a big data cluster?
- Executing distributed SQL queries
- Providing distributed coordination and configuration management services (Correct answer)
- Compressing data blocks in HDFS
- Scheduling Spark jobs across nodes
Correct answer: Providing distributed coordination and configuration management services
ZooKeeper provides distributed coordination services such as leader election, configuration management, and distributed locking for cluster components.
Question 2: In Apache Flink, what distinguishes it from Apache Spark Streaming?
- Flink supports only batch processing
- Flink is a true streaming engine where batch is a special case, whereas Spark Streaming uses micro-batches (Correct answer)
- Flink cannot process unbounded data streams
- Flink requires HDFS as its only storage backend
Correct answer: Flink is a true streaming engine where batch is a special case, whereas Spark Streaming uses micro-batches
Flink treats streaming as the first-class primitive and handles batch as a finite stream, while Spark Streaming originally processed streams as a series of small batches.
Question 3: What does 'replication factor' mean in HDFS?
- The compression ratio applied to each block
- The number of copies of each data block stored across DataNodes (Correct answer)
- The number of NameNodes in the cluster
- The number of MapReduce jobs running concurrently
Correct answer: The number of copies of each data block stored across DataNodes
The replication factor (default 3) defines how many copies of each HDFS block are written to different DataNodes for fault tolerance.
Question 4: Which pattern best describes the Lambda Architecture for big data systems?
- Processing all data exclusively with stream processing
- Combining batch and speed layers to serve both historical and real-time queries (Correct answer)
- Using only a serving layer backed by a key-value store
- Processing data in scheduled nightly batch jobs only
Correct answer: Combining batch and speed layers to serve both historical and real-time queries
Lambda Architecture uses a batch layer for comprehensive historical processing and a speed layer for real-time updates, with a serving layer merging both views.
Question 5: What is Apache Sqoop primarily used for?
- Real-time stream ingestion from IoT devices
- Bulk transfer of data between relational databases and Hadoop (Correct answer)
- Monitoring and alerting on HDFS health
- Scheduling workflow dependencies between Hadoop jobs
Correct answer: Bulk transfer of data between relational databases and Hadoop
Sqoop is designed to efficiently transfer bulk data between relational databases (via JDBC) and Hadoop storage systems like HDFS or HBase.
Question 6: In the context of big data, what is 'schema-on-read'?
- Enforcing a schema when data is written to storage
- Applying the schema interpretation only when data is queried or read (Correct answer)
- Auto-generating schemas from machine learning models
- Storing schema definitions in a separate metadata catalog
Correct answer: Applying the schema interpretation only when data is queried or read
Schema-on-read defers schema enforcement to query time, allowing raw data to be stored in any format and interpreted flexibly when accessed.
Question 7: What is the purpose of a Kafka offset?
- The byte position of a message within an Avro file
- A sequential identifier that tracks a consumer's read position within a partition (Correct answer)
- The lag between producer and broker in milliseconds
- The number of replicas for a Kafka topic
Correct answer: A sequential identifier that tracks a consumer's read position within a partition
An offset is a unique sequential number assigned to each message in a partition, allowing consumers to track and resume their read position.
DSE Big Data Technologies 3
What is the role of Apache ZooKeeper in a big data cluster?