GCP Data Platform
The entire company's analytics, from 20+ source systems to live dashboards on a 10-minute loop, designed and operated by one person.
The full data platform for a multi-channel retailer, owned end to end: ingestion, warehouse, orchestration, and serving.
Context & Goal
A multi-channel retailer: dozens of corporate and franchise stores, three e-commerce storefronts, and Amazon channels. The goal was company-wide analytics and operational data products, not just BI, on fast, cost-controlled models that multiple teams can trust.
The entire platform runs in a single GCP project and is designed, built, and operated by one engineer.
The pipeline, end to end
From a webhook at a register to a number on a dashboard, on a 10-minute intraday loop.
Dozens of corporate and franchise stores, three e-commerce storefronts, Amazon channels, and the ERP.
16 Cloud Functions plus Cloud Run services land POS, e-commerce, and body-scanner webhooks, with Pub/Sub fan-out to downstream consumers.
Typed staging layers, partitioned and clustered marts, idempotent loads.
400+ models over 225 declared sources, orchestrated on 22 cron schedules: a nightly build chain plus 10-minute intraday pipelines.
Syncs to a Cloud SQL Postgres replica powering the ops platform, plus Klaviyo events, a Merchant Center feed, closeout emails, and Slack alerting.
Results
- Cut BigQuery infrastructure costs roughly 40% through partitioning, clustering, and query-pattern optimization.
- Live sales for every store on a 10-minute loop, from POS webhook to dashboard. Intraday revenue used to wait for the next day's ERP sync.
- Unified data access for 6+ departments on consistent, trusted metric definitions.
- Consolidated a patchwork of managed ELT onto Dagster-owned ingestion. Airbyte retired, Fivetran down to a single connector, every pipeline under one orchestrator.
What I built
Ingestion, warehouse, orchestration, and every rail that serves data back out.
BigQuery warehouse
400+ dbt models over 225 declared sources: typed staging layers, partitioned and clustered marts, idempotent loads.
Dagster orchestration
Self-hosted with Docker on a GCE VM: 22 cron schedules, ephemeral run containers, a daily overnight build chain, and 10-minute intraday sales pipelines.
Event-driven ingestion
16 Cloud Functions plus Cloud Run services landing POS, e-commerce, and body-scanner webhooks in BigQuery, with Pub/Sub fan-out to downstream consumers.
Serving syncs
From BigQuery to a Cloud SQL Postgres replica that powers the Internal Ops Platform, the company's BI layer.
Reverse ETL rails
Klaviyo purchase and profile events, a Google Merchant Center local-inventory feed, nightly revenue-accurate store closeout emails, and Slack alerting for discount fraud and payment anomalies.
An AI access layer
MCP servers exposing the warehouse and ERP to LLM agents, plus an LLM-maintained ops wiki that re-ingests documentation from every repo's CI.
Who It Serves
Every department below reads from the same models and the same metric definitions.
Architecture in detail
Hover a node to trace its connections; click a node for details. The diagram scrolls sideways on smaller screens.