Lab 1: Local PySpark Parquet ETL

# Lab 1: Local PySpark Parquet ETL (90 min) **Deliverable:** a Git repo with an idempotent `etl.py` + partitioned Parquet lake + `README.md` with row counts and `explain()` output. ## 1. Business context "ShopLite" dumps daily `orders.csv` (order_id, customer_id, order_date, status, amount). Build the bronze layer: clean, typed, partitioned Parquet that analysts can prune by date. ## 2. Requirements 1. `etl.py` runs with `spark-submit etl.py --date 2026-09-20` in local mode. 2. Drops null-`order_id` rows; casts `amount` to decimal; normalizes `status` to lowercase. 3. Writes `lake/bronze_orders/` partitioned by `order_date` (repartition to <= 8 files per date). 4. Re-runs are idempotent: same `--date` twice gives identical row counts (overwrite the date partition). 5. `README.md` includes: row counts in/out, rejected-row count, `df.explain()` excerpt, and file listing of one date partition. ## 3. Grading rubric (100 pts) - Correctness & idempotency (30): re-run safe; counts reconcile. - Partitioning & file sizing (25): date partitions, no tiny-file explosion. - Code quality (20): lazy-plan discipline, no stray `collect()`, typed schema. - README evidence (15): counts + plan excerpt + listing. - Error handling (10): bad rows quarantined to `lake/_quarantine/`, not silently dropped. ## 4. Stretch Add a `--full-refresh` flag that rewrites all partitions, and log per-partition counts to `lake/_audit/etl_log.json`.