AI Data Platform Series: Silver Layer Spark Structure Streaming
This is the fifth post in the AI Data Platform series, following Just Streams: Real-Time Data Pipelines on OCI (the series intro), Setting Up the AI Data Platform Environment (stage 0), OCI Streaming and the Stream Producer (stage 1), and Bronze Layer: Spark Structured Streaming (stage 2). That last post landed every arrival prediction as-is into an append-only Delta table, tfl.bronze.arrivals_bronze. This post covers stage 3: cleaning that data up, keeping only the latest prediction per vehicle/stop/line/direction, and upserting it into a proper silver table.
