Databricks Certified Data Engineer Associate Auto Loader and Data Ingestion

Question 1

What is the primary advantage of using Auto Loader over `spark.read` for cloud storage ingestion?

Accepted Answer

It automatically identifies and processes only new files incrementally

Answer

It supports more file formats than spark.read

Answer

It provides faster read speeds for all existing files

Answer

It automatically compresses ingested data before writing

Question 2

Which Spark source format string is used to configure Auto Loader in a readStream call?

Accepted Answer

cloudFiles

Answer

autoloader

Answer

deltaStream

Answer

incrementalFiles

Question 3

What is the purpose of the `cloudFiles.schemaLocation` option in Auto Loader?

Accepted Answer

Stores the inferred schema so it persists and evolves across stream restarts

Answer

Specifies where to write output schema documentation

Answer

Defines the location of manually provided input schema files

Answer

Sets the schema registry endpoint for external validation

Question 4

What is the key difference between Auto Loader's 'directory listing' mode and 'file notification' mode?

Accepted Answer

File notification uses cloud services (e.g., AWS SNS/SQS) for real-time updates; directory listing polls the directory periodically

Answer

Directory listing is faster; file notification supports more file formats

Answer

Directory listing works with Delta tables only; file notification works with raw files

Answer

File notification is the default mode and directory listing must be explicitly enabled

Question 5

When Auto Loader detects new columns in incoming files with schema evolution enabled, which option controls this behavior?

Accepted Answer

cloudFiles.schemaEvolutionMode

Answer

cloudFiles.allowNewColumns

Answer

cloudFiles.evolveSchema

Answer

cloudFiles.inferColumnTypes

Question 6

Which of the following is a REQUIRED option when writing an Auto Loader stream output?

Accepted Answer

checkpointLocation

Answer

cloudFiles.format

Answer

cloudFiles.schemaLocation

Answer

cloudFiles.maxFilesPerTrigger

Question 7

What happens to files already present in a directory when Auto Loader is started for the very first time?

Accepted Answer

They are processed along with all subsequent new files that arrive

Answer

They are skipped; only files arriving after startup are processed

Answer

Auto Loader fails with an error if existing files are detected

Answer

They are automatically moved to an archive folder before processing begins