Prepare and process data

Use DESCRIBE HISTORY on the landing Delta table after each load to reconstruct row-level employee history.

Use AUTO CDC FROM SNAPSHOT in a Python pipeline, and provide a source function that returns the next snapshot DataFrame and its snapshot version.

Load each file with COPY INTO, enable Change Data Feed on the landing table, and treat the landing table's CDF as employee history.

Use AUTO CDC in SQL on the snapshot files and sequence records by pipeline ingestion time.

Use AUTO CDC FROM SNAPSHOT in SQL with STORED AS SCD TYPE 2.