No ETL still needs a contract

“Query the lakehouse as a graph” sounds like a connector problem. Point at Parquet, infer a schema, draw some relationships, and enjoy the absence of an ingestion job. The connector is the easy part. The harder question is what one row records and which distinctions the graph must preserve.

A purchase row may represent an order, an order line, a payment, a revision, or a denormalized combination of all four. Two rows with the same customer and product may describe repeated purchases or repeated versions of one event. A graph can model either interpretation with great confidence. Confidence is not a substitute for the source contract.

ShrikeDB separates source registration from mapping publication. Registration describes files and versions. A mapping states how columns become identities, properties, and relationships. Supported Cypher and SPARQL source paths then read the registered Parquet directly. This removes the mandatory ETL copy while keeping the semantic decision explicit and reviewable.

Every row has a meaning

Before publishing a mapping, finish one sentence: “One row records one ______.” Then answer four more:

• Which columns identify that thing?
• Are those identifiers unique within a file, a source system, or the whole organization?
• Does a repeated identifier mean another event, another version, or duplicated delivery?
• Which measures belong to the row, and which belong to a parent entity?

If the source contains order lines, an order-line identity preserves repeated products and quantities. A customer-to-product edge may help a recommendation query, but it cannot replace the events if finance later asks for purchases, refunds, or sequence. Flattening early often creates an attractive graph with no reliable denominator.

Automated profiling can suggest keys and relationship candidates from types, null rates, and samples. It cannot decide whether invoice 42 in Europe is the same invoice as 42 in North America. It cannot know whether a status row supersedes an earlier row or records a second event. Those are business facts, which is unfortunate only if one hoped column names would develop judgment.

The perfectly valid $200 mistake

Order 81 has two lines worth $40 and $60. The order table stores a $100 total. Join the order to both lines, carry the order total into each row, and sum it. The result is $200.

The join is correct. The aggregate uses the wrong grain. Adding DISTINCT to the amount is no repair because two different orders can both total $100. The identity that must remain distinct is the order, not the numeric value.

Write the calculation in nouns before writing it in a query: sum line amounts over line identities, or select qualifying order identities once and sum their totals. If the query also joins payments and shipments, decide which entity each measure belongs to before all three branches multiply one another.

This example matters because large graph errors often look exactly like correct business results. The total is plausible. Every relationship exists. Only the unit of aggregation is wrong. Source mappings should expose grain clearly enough that a query author can see that boundary.

Identity must survive the shortcut

Two systems both contain customer 42. If their IDs occupy different namespaces, merging them creates one fictional customer connected to unrelated orders, addresses, and accounts. Graph traversal spreads that mistake efficiently. It is among the few performance improvements nobody requested.

The reverse error fragments one real customer into several nodes and hides repeated behavior. Keep source-qualified identity first. Represent resolution as a separate, versioned assertion with supporting evidence. The analyst can then ask how a result changes if two records do not identify the same entity.

Direct source queries also need a source version. A file path is not a snapshot contract if an upstream job can replace its contents. Record the object identity, modification state, or manifest version used by the query. When a result changes, the operator should be able to separate a new source version from a new query or mapping.

This becomes critical when a source-backed branch joins stored graph state. A coherent result needs explicit publication boundaries for both sides. ShrikeDB reports the selected mapping and source-read diagnostics so an application can preserve that context with the answer.

A practical source workflow

Start with a query small enough to check by eye. If the reviewed mapping exposes paid_amount and billing_provider_npi, a bounded Cypher projection can inspect them:

CALL lakehouse.scan('YOUR_PUBLISHED_MAPPING_ID')
YIELD paid_amount, billing_provider_npi
RETURN billing_provider_npi AS provider,
       paid_amount AS amount
LIMIT 100

Then move through a deliberate sequence:

1. Register. Identify the local files and record the source version.
2. Profile. Inspect types, nulls, sampled distinctness, and candidate joins.
3. Define. State row grain, namespaces, relationship direction, and measure ownership.
4. Challenge. Test duplicate keys, missing values, late versions, and one-to-many branches.
5. Publish. Make the reviewed mapping available to the bounded query paths.
6. Observe. Inspect projected columns, predicate work, source bytes, and result cardinality.

The equivalent SPARQL path can expose mapped concepts as triples and bindings. The language differs; the contract does not. Both query paths should identify the mapping they used and remain bounded to the operations the source adapter can execute correctly.

When to query and when to store

Read in place when the source remains authoritative, the question projects a useful subset, and freshness matters more than repeated submillisecond access. It works especially well for exploration, bounded filtering, schema review, and graph questions that start from a selective set of source rows.

Store native state when the workload repeatedly traverses the same relationships, needs transactional updates with other modalities, depends on low-latency indexes, or must survive source replacement under a database-controlled history. The choice can differ by dataset. A large event archive may remain in Parquet while active entities, reviewed relationships, embeddings, and recent observations live in ShrikeDB.

The valuable architecture is not “everything stays in the lake” or “everything enters the database.” It is the ability to postpone a copy until the workload justifies one, while preserving the semantics needed to move between the two paths.

No ETL is a useful operational result. No data contract is merely a faster route to an argument about why the total changed.