Outcome: Run reliable upserts on object storage: the transaction log, MERGE, time travel, and the OPTIMIZE/VACUUM maintenance pair.
1. The transaction log (0:00–2:30)
A Delta table is Parquet files plus a _delta_log/ of JSON commits. Every write is atomic: readers see a consistent snapshot, concurrent writers serialize through the log. No log, no ACID — that is the whole trick.
2. MERGE upserts (2:30–5:00)
MERGE INTO target USING source ON target.id = source.id WHEN MATCHED THEN UPDATE ... WHEN NOT MATCHED THEN INSERT — one statement for slowly changing feeds. Performance is file-pruning performance: co-locate join keys (ZORDER) so the merge scans fewer files.
3. Time travel + maintenance (5:00–9:00)
Read any version (versionAsOf, timestampAsOf); RESTORE moves the log pointer back atomically — never hand-delete files. OPTIMIZE compacts small files, ZORDER co-locates; VACUUM deletes unreferenced files, so retention must exceed your longest time-travel window (default 7 days; analysts needing 30-day travel need longer retention).
Key moments
- 1:40 — reading a
_delta_logcommit entry - 4:00 — MERGE anatomy for a subscription feed
- 7:00 — why VACUUM + short retention destroys recoverability
Check: A backfill corrupted v9; analysts need v8 now. Give the two commands (read + restore) and the VACUUM retention rule that keeps them safe.