rgoussu.dev
rgoussu@goussu: ~/library/data-storage
❯ ~/library/data-storage cat data-engineering.md

Data engineering

# Moving and shaping data at scale — batch vs. streaming, ELT and the modern warehouse, dimensional modeling, orchestration, and data quality.

Conceptsaved 2026-08-08 #data-engineering#etl#streaming#warehouse#analytics

Overview

Data engineering is the discipline of getting data from the systems that produce it to the people and models that need it — reliably, on schedule, in a shape fit for analysis. The field has converged on a recognizable stack: ingest (CDC, events) into cheap storage, transform inside the warehouse (ELT), model dimensionally for consumption, orchestrate it all as versioned, tested code. A backend engineer meets it at the edges constantly; knowing the shape of it turns "throw it over to the data team" into a designed interface.

Key points

  • Batch vs. streaming is latency vs. complexity: scheduled batch (hourly/daily — Spark, warehouse SQL) covers most analytics honestly; streaming (Flink, Kafka Streams) buys minutes-to-seconds freshness at the price of state, watermarks, and late-event handling ([event time vs. processing time] — the hard part). The old lambda architecture (both, reconciled) is giving way to kappa-ish "stream first, replay the log" where the log is durable (brokers & streaming).
  • ETL became ELT: load raw into the warehouse/lake first, transform there with SQL — compute is elastic, raw history is replayable, and transformations become versioned code (dbt as the tool that made SQL pipelines testable, documented, and reviewed like software).
  • The storage landscape: columnar warehouses (Snowflake, BigQuery — see the OLTP/OLAP split in databases), data lakes (object storage + Parquet), and the lakehouse convergence — open table formats (Iceberg, Delta) adding transactions, schema evolution, and time travel to files on S3, ending the lake-vs-warehouse either/or.
  • Dimensional modeling still earns its keep: facts (events, measures) and dimensions (who/what/where, denormalized) in a star schema — optimized for the questions analysts actually ask; slowly changing dimensions (type 2: validity ranges) for "what did we know then". Wide tables and semantic layers layer on top, not instead.
  • Ingestion: CDC from operational databases (Debezium — the outbox/CDC machinery reused), event streams, and third-party ELT connectors; idempotent, replayable loads over "run it again and hope".
  • Orchestration & quality: DAG schedulers (Airflow, Dagster) with retries, backfills, and idempotent tasks as the design constraint; data quality as tests on data (freshness, volume, nulls, referential checks — dbt tests, Great Expectations), lineage for "where did this number come from", and data contracts pushing schema ownership back to producers (API design for datasets).
  • To explore: streaming joins & exactly-once sinks, reverse ETL, feature stores (the ML consumer), cost engineering in pay-per-query warehouses.

Practice

  • dbt jaffle shop (source) — dbt's canonical toy project: raw seeds → staging → marts with tests and docs; teaches the ELT workflow and SQL-as-versioned-code in an afternoon.
  • Star-schema drill (source) — dimensionally model a domain you know cold (your expenses, a side project's events): pick the grain, split facts from dimensions, add a type-2 slowly changing dimension; teaches Kimball modeling as decisions, not diagrams.
  • Event-time windows with Kafka Streams (source) — build a small windowed aggregation, then feed it out-of-order and late events and watch results change with the watermark strategy; teaches the event-time vs. processing-time gap that makes streaming hard.
  • End-to-end ELT pipeline (exercise) — the full stack on a laptop: ingestion, warehouse transforms, dimensional marts, orchestration with backfills, data tests, and CI; the build project that connects every key point above.

Related