Data layer: ECC, Xano, Shopify and Google Cloud compared with Databricks, Snowflake and Fabric
The data layer in this book is ECC as the source, Xano as the system of record, Shopify as the storefront and Google Cloud (BigQuery, Merchant, Ads) for analytics and activation. Cloudflare Workers carry it at the edge. The usual alternative is a lakehouse or warehouse (Databricks, Snowflake or Microsoft Fabric) fed by ELT tools.
| ECC + Xano + Shopify + Google Cloud | Databricks | Snowflake | Microsoft Fabric | |
|---|---|---|---|---|
| Shopify in | Shopify webhooks to Google Cloud Pub/Sub, then a BigQuery subscription into a table. BigQuery Data Transfer Service also has a Shopify connector, in Preview since 2026-01-28. | Third-party ELT (Fivetran, Airbyte, Rivery, dlt), or Shopify → EventBridge → S3 → Auto Loader | Third-party ELT, or Shopify → EventBridge → S3 → Snowpipe | No native Shopify route: HTTPS webhooks into an Azure Function or Eventstreams, or ELT |
| Real time | Shopify webhooks reach the Worker and Xano in seconds; Pub/Sub reaches BigQuery in near real time | Near real time only through an AWS event route you build | Same as Databricks | Eventstreams, with an intake you build |
| Consent enforced in the pipe | Yes. The Worker verifies the webhook, checks consent and writes the ledger before anything lands. BigQuery gets only changed consent rows (crm_us.consent_log). | Not by default. ELT copies what the API returns; policies apply after the data lands. | Not by default. Masking and row-access policies apply after the data lands. | Not by default. Purview policies apply after the data lands. |
| One consent decision | The release gate in Xano, in front of every Google payload about a person | Built in notebooks or Unity Catalog policies, downstream of the copy | Built with masking and row-access policies, downstream | Built with Purview policies, downstream |
| ECC data in | The Rust converter in this book, from DD03L-shaped exports: typed and tested, no S/4HANA needed | SAP connectors exist; the strongest assume S/4HANA and SAP Business Data Cloud | SAP connectors through partners; the same S/4HANA bias | SAP mirroring and connectors; the same S/4HANA bias |
| Google activation (Merchant, Ads, Customer Match) | Native: Merchant API, Google Ads and BigQuery are all Google, behind the release gate | Reverse-ETL tools (Hightouch, Census) | Reverse-ETL tools | Reverse-ETL tools |
| System of record | Xano for customers, consent and products; the warehouse holds analytics only | The lakehouse tends to become the record | The warehouse tends to become the record | OneLake tends to become the record |
| Residency | Per-market BigQuery datasets (crm_us first); Xano instances per region | Per workspace region | Per account region | Per capacity region |
| Cost shape | Workers, a Xano plan, BigQuery storage and queries: small to start | Compute clusters plus ELT seats | Credits plus ELT seats | Capacity units plus ELT seats |
| Where it wins | Shopify and Google merchants, ECC shops without S/4HANA, consent-first commerce | Heavy ML, data science teams, large-scale Spark | Enterprise BI, data sharing, an existing Snowflake contract | Microsoft-standard enterprises: Power BI, Entra, Purview |
One typed record, any destination
The converter turns ECC rows into Rust types once, with Serde derives on every struct. After that, the destination is a
serializer, not a new integration: the same Product or customer record is written as whatever the target reads.
| Destination | Format | Today |
|---|---|---|
| Google Merchant API v1 | JSON (productInputs) | Built: src/merchant_api.rs, tests/merchant_api.rs |
| Merchant feed file | The specification’s names and text | Built: src/merchant.rs |
| Xano | JSON records and CSV uploads | Built: xano-buildout |
| GraphQL | schema.graphql from the same map | Built |
| BigQuery | JSON rows (insertAll) or Pub/Sub messages | Through the CRM Sync Worker |
| Databricks, Snowflake, Fabric | Parquet or Iceberg | Proposed: needs the Arrow / Parquet crates |
| Kafka, Pub/Sub, EventBridge | JSON or Avro messages | Proposed |
What stays fixed whatever the destination: names come from lib/naming.js, types from the DDIC (NUMC keeps its leading
zeros, DATS 00000000 is None, amounts keep their scale), and the tests fail if a field goes missing. What doesn’t
change either: any record about a person passes the release gate first. A new destination is a new output, never a way
around consent.
Feeding the other three, governed
None of the three gets a consent-checked Shopify feed out of the box: ELT tools copy whatever the Admin API returns, and EventBridge or Pub/Sub deliver every event whatever the person’s consent. The same Worker intake can feed them, so a Databricks, Snowflake or Fabric shop keeps its warehouse and gets a governed Shopify feed:
Shopify webhook ─▶ Cloudflare Worker (verify HMAC, enforce consent, write ledger) ─▶ Xano (system of record)
└─▶ Parquet / Iceberg in R2 or the client's S3
└─▶ Databricks · Snowflake · Fabric · BigQuery
Today the Worker streams to Xano and BigQuery only. The Parquet / Iceberg export is proposed, not built. Cloudflare R2 can serve Iceberg tables through its catalog, which Snowflake and Databricks both read, so the export needs no new storage.
Cloudflare Basin compared with BigQuery
Cloudflare’s lake (now Basin: Basin Pipelines, Basin Catalog and Basin SQL, formerly Pipelines, R2 Data Catalog and R2 SQL) is open storage and a catalog at the edge, with no egress fees and a read-only query engine. BigQuery and the three platforms above are compute platforms: they write, transform, run ML and apply fine-grained access rules.
| Cloudflare Basin (R2 + Iceberg) | BigQuery | |
|---|---|---|
| Storage | Parquet in R2, with an Iceberg REST catalog any Iceberg engine can read | Native storage; BigLake Iceberg tables and external tables as options |
| Ingest | Basin Pipelines: streaming events with SQL transforms, fed straight from Workers | Data Transfer Service connectors (Shopify, YouTube, Merchant Center, Ads), streaming, loads |
| Query | Basin SQL: distributed, read-only (Apache DataFusion); no Workers binding, queried by fetch() | Full SQL with DML, BigQuery ML, Gemini |
| Writes | Pipelines, or an external engine (PyIceberg, Spark) | INSERT, MERGE, UPDATE |
| Egress | None on R2 | Charged when data leaves Google Cloud |
| Fine-grained access | Token scope per bucket and catalog; no row-level security or column masking | Row-level security, policy tags, dynamic masking |
| Residency | Bucket jurisdiction (R2 has an EU jurisdiction); check the catalog and query engine follow it | Regional and multi-region datasets |
| Maintenance | You enable compaction and snapshot expiry; file count drives query time | Managed |
What each is for in this layer
| Job | Better fit |
|---|---|
| Edge events, consent and gate receipts, cheap long-term logs | Basin |
| Shopify, YouTube, Merchant and Ads joins, segments, BigQuery ML | BigQuery |
| One open copy of ECC extracts, read by several engines without egress | R2 and Iceberg |
| Heavy transforms and an enterprise catalog | Databricks or Snowflake |
Two consequences:
- Basin has no column or row policies, so the release gate does that job. Keep personal data out of Basin, or pseudonymise it before Pipelines writes it: cohorts, receipts and hashed keys, not identities. See Function gating across clouds.
- BigQuery doesn’t read R2 directly, as far as the current docs show (BigQuery Omni covers AWS S3 and Azure). Data for BigQuery is copied to Google Cloud Storage or loaded; Snowflake and Databricks read R2 through its S3-compatible API.
Basin was renamed recently and its docs are mid-migration: check pricing, limits and jurisdiction support on developers.cloudflare.com before quoting them.
Reading the table fairly
- “Not by default” is not “can’t”. Each of the three can enforce policies once data has landed. The difference is where: here the check runs before anything is copied, and the ledger records why.
- The ECC row is where this layer differs most. The other three platforms’ SAP paths mostly assume S/4HANA; this one works from ECC as it runs today. See An alternative to a full S/4HANA upgrade.
- BigQuery’s own Shopify connector is in Preview; Fivetran and others list generally available connectors on the Google Cloud Marketplace. Sources: Load Shopify data into BigQuery, BigQuery release notes.