Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Data layer: ECC, Xano, Shopify and Google Cloud compared with Databricks, Snowflake and Fabric

The data layer in this book is ECC as the source, Xano as the system of record, Shopify as the storefront and Google Cloud (BigQuery, Merchant, Ads) for analytics and activation. Cloudflare Workers carry it at the edge. The usual alternative is a lakehouse or warehouse (Databricks, Snowflake or Microsoft Fabric) fed by ELT tools.

ECC + Xano + Shopify + Google CloudDatabricksSnowflakeMicrosoft Fabric
Shopify inShopify webhooks to Google Cloud Pub/Sub, then a BigQuery subscription into a table. BigQuery Data Transfer Service also has a Shopify connector, in Preview since 2026-01-28.Third-party ELT (Fivetran, Airbyte, Rivery, dlt), or Shopify → EventBridge → S3 → Auto LoaderThird-party ELT, or Shopify → EventBridge → S3 → SnowpipeNo native Shopify route: HTTPS webhooks into an Azure Function or Eventstreams, or ELT
Real timeShopify webhooks reach the Worker and Xano in seconds; Pub/Sub reaches BigQuery in near real timeNear real time only through an AWS event route you buildSame as DatabricksEventstreams, with an intake you build
Consent enforced in the pipeYes. The Worker verifies the webhook, checks consent and writes the ledger before anything lands. BigQuery gets only changed consent rows (crm_us.consent_log).Not by default. ELT copies what the API returns; policies apply after the data lands.Not by default. Masking and row-access policies apply after the data lands.Not by default. Purview policies apply after the data lands.
One consent decisionThe release gate in Xano, in front of every Google payload about a personBuilt in notebooks or Unity Catalog policies, downstream of the copyBuilt with masking and row-access policies, downstreamBuilt with Purview policies, downstream
ECC data inThe Rust converter in this book, from DD03L-shaped exports: typed and tested, no S/4HANA neededSAP connectors exist; the strongest assume S/4HANA and SAP Business Data CloudSAP connectors through partners; the same S/4HANA biasSAP mirroring and connectors; the same S/4HANA bias
Google activation (Merchant, Ads, Customer Match)Native: Merchant API, Google Ads and BigQuery are all Google, behind the release gateReverse-ETL tools (Hightouch, Census)Reverse-ETL toolsReverse-ETL tools
System of recordXano for customers, consent and products; the warehouse holds analytics onlyThe lakehouse tends to become the recordThe warehouse tends to become the recordOneLake tends to become the record
ResidencyPer-market BigQuery datasets (crm_us first); Xano instances per regionPer workspace regionPer account regionPer capacity region
Cost shapeWorkers, a Xano plan, BigQuery storage and queries: small to startCompute clusters plus ELT seatsCredits plus ELT seatsCapacity units plus ELT seats
Where it winsShopify and Google merchants, ECC shops without S/4HANA, consent-first commerceHeavy ML, data science teams, large-scale SparkEnterprise BI, data sharing, an existing Snowflake contractMicrosoft-standard enterprises: Power BI, Entra, Purview

One typed record, any destination

The converter turns ECC rows into Rust types once, with Serde derives on every struct. After that, the destination is a serializer, not a new integration: the same Product or customer record is written as whatever the target reads.

DestinationFormatToday
Google Merchant API v1JSON (productInputs)Built: src/merchant_api.rs, tests/merchant_api.rs
Merchant feed fileThe specification’s names and textBuilt: src/merchant.rs
XanoJSON records and CSV uploadsBuilt: xano-buildout
GraphQLschema.graphql from the same mapBuilt
BigQueryJSON rows (insertAll) or Pub/Sub messagesThrough the CRM Sync Worker
Databricks, Snowflake, FabricParquet or IcebergProposed: needs the Arrow / Parquet crates
Kafka, Pub/Sub, EventBridgeJSON or Avro messagesProposed

What stays fixed whatever the destination: names come from lib/naming.js, types from the DDIC (NUMC keeps its leading zeros, DATS 00000000 is None, amounts keep their scale), and the tests fail if a field goes missing. What doesn’t change either: any record about a person passes the release gate first. A new destination is a new output, never a way around consent.

Feeding the other three, governed

None of the three gets a consent-checked Shopify feed out of the box: ELT tools copy whatever the Admin API returns, and EventBridge or Pub/Sub deliver every event whatever the person’s consent. The same Worker intake can feed them, so a Databricks, Snowflake or Fabric shop keeps its warehouse and gets a governed Shopify feed:

Shopify webhook ─▶ Cloudflare Worker (verify HMAC, enforce consent, write ledger) ─▶ Xano (system of record)
                                                                          └─▶ Parquet / Iceberg in R2 or the client's S3
                                                                                └─▶ Databricks · Snowflake · Fabric · BigQuery

Today the Worker streams to Xano and BigQuery only. The Parquet / Iceberg export is proposed, not built. Cloudflare R2 can serve Iceberg tables through its catalog, which Snowflake and Databricks both read, so the export needs no new storage.

Cloudflare Basin compared with BigQuery

Cloudflare’s lake (now Basin: Basin Pipelines, Basin Catalog and Basin SQL, formerly Pipelines, R2 Data Catalog and R2 SQL) is open storage and a catalog at the edge, with no egress fees and a read-only query engine. BigQuery and the three platforms above are compute platforms: they write, transform, run ML and apply fine-grained access rules.

Cloudflare Basin (R2 + Iceberg)BigQuery
StorageParquet in R2, with an Iceberg REST catalog any Iceberg engine can readNative storage; BigLake Iceberg tables and external tables as options
IngestBasin Pipelines: streaming events with SQL transforms, fed straight from WorkersData Transfer Service connectors (Shopify, YouTube, Merchant Center, Ads), streaming, loads
QueryBasin SQL: distributed, read-only (Apache DataFusion); no Workers binding, queried by fetch()Full SQL with DML, BigQuery ML, Gemini
WritesPipelines, or an external engine (PyIceberg, Spark)INSERT, MERGE, UPDATE
EgressNone on R2Charged when data leaves Google Cloud
Fine-grained accessToken scope per bucket and catalog; no row-level security or column maskingRow-level security, policy tags, dynamic masking
ResidencyBucket jurisdiction (R2 has an EU jurisdiction); check the catalog and query engine follow itRegional and multi-region datasets
MaintenanceYou enable compaction and snapshot expiry; file count drives query timeManaged

What each is for in this layer

JobBetter fit
Edge events, consent and gate receipts, cheap long-term logsBasin
Shopify, YouTube, Merchant and Ads joins, segments, BigQuery MLBigQuery
One open copy of ECC extracts, read by several engines without egressR2 and Iceberg
Heavy transforms and an enterprise catalogDatabricks or Snowflake

Two consequences:

  • Basin has no column or row policies, so the release gate does that job. Keep personal data out of Basin, or pseudonymise it before Pipelines writes it: cohorts, receipts and hashed keys, not identities. See Function gating across clouds.
  • BigQuery doesn’t read R2 directly, as far as the current docs show (BigQuery Omni covers AWS S3 and Azure). Data for BigQuery is copied to Google Cloud Storage or loaded; Snowflake and Databricks read R2 through its S3-compatible API.

Basin was renamed recently and its docs are mid-migration: check pricing, limits and jurisdiction support on developers.cloudflare.com before quoting them.

Reading the table fairly

  • “Not by default” is not “can’t”. Each of the three can enforce policies once data has landed. The difference is where: here the check runs before anything is copied, and the ledger records why.
  • The ECC row is where this layer differs most. The other three platforms’ SAP paths mostly assume S/4HANA; this one works from ECC as it runs today. See An alternative to a full S/4HANA upgrade.
  • BigQuery’s own Shopify connector is in Preview; Fivetran and others list generally available connectors on the Google Cloud Marketplace. Sources: Load Shopify data into BigQuery, BigQuery release notes.