GCP Data Platform
The entire company's analytics, from 30 source systems to live dashboards on a 10-minute loop, owned end to end.
The full data platform for a multi-channel retailer, owned end to end: ingestion, warehouse, orchestration, and serving.
Context & Goal
A multi-channel retailer: dozens of corporate and franchise stores, three e-commerce storefronts, and Amazon channels. The goal was company-wide analytics and operational data products, not just BI, on fast, cost-controlled models that multiple teams can trust.
The entire platform runs in a single GCP project and is designed, built, and operated by one engineer.
The pipeline, end to end
From a webhook at a register to a number on a dashboard, on a 10-minute intraday loop.
Dozens of corporate and franchise stores, three e-commerce storefronts, Amazon channels, and the ERP.
16 Cloud Functions plus Cloud Run services land POS, e-commerce, and body-scanner webhooks, with Pub/Sub fan-out to downstream consumers.
Typed staging layers, partitioned and clustered marts, idempotent loads.
500+ models over 225 declared sources, orchestrated on 22 cron schedules: a nightly build chain plus 10-minute intraday pipelines.
Syncs to a Cloud SQL Postgres replica powering the ops platform, plus Klaviyo events, a Merchant Center feed, closeout emails, and Slack alerting.
Results
- Cut BigQuery infrastructure costs roughly 40% through partitioning, clustering, and query-pattern optimization.
- Live sales for every store on a 10-minute loop, from POS webhook to dashboard. Intraday revenue used to wait for the next day's ERP sync.
- Unified data access for 6+ departments on consistent, trusted metric definitions.
- Consolidated a patchwork of managed ELT onto Dagster-owned ingestion. Airbyte retired, Fivetran down to a single connector, every pipeline under one orchestrator.
What I built
Ingestion, warehouse, orchestration, and every rail that serves data back out.
BigQuery warehouse
500+ dbt models over 225 declared sources: typed staging layers, partitioned and clustered marts, idempotent loads.
Dagster orchestration
Self-hosted with Docker on a GCE VM: 22 cron schedules, ephemeral run containers, a daily overnight build chain, and 10-minute intraday sales pipelines.
Event-driven ingestion
16 Cloud Functions plus Cloud Run services landing POS, e-commerce, and body-scanner webhooks in BigQuery, with Pub/Sub fan-out to downstream consumers.
Serving syncs
From BigQuery to a Cloud SQL Postgres replica that powers the Internal Ops Platform, the company's BI layer.
Reverse ETL rails
Klaviyo purchase and profile events, a Google Merchant Center local-inventory feed, nightly revenue-accurate store closeout emails, and Slack alerting for discount fraud and payment anomalies.
Knowledge-base agents
Agents on the Claude Agent SDK and Vertex AI read BigQuery and maintain the ops wiki, an Astro Starlight site that re-ingests documentation from every repo's CI. Every change the agents make lands as a pull request for human review. Alongside them, MCP servers expose the warehouse and ERP to LLM agents.
Who It Serves
Every department below reads from the same models and the same metric definitions.
Architecture in detail
Hover a node to trace its connections; click a node for details. The diagram scrolls sideways on smaller screens.