Agentic AI on Google Cloud, Part 6: Graphs in Production at Scale

Parts 1 to 5 gave us an assistant that answers from documents, live APIs and verified evidence. This post adds a graph, but only the kind that earns its place in production: a large, continuously updated graph that answers relationship questions quickly, safely and at a cost you can predict.

The series. Part 1: GCP map and first agent. Part 2: knowledge preparation and ingestion. Part 3: verification and citations. Part 4: testing, security and monitoring. Part 5: CI/CD with GitHub Actions and Terraform. Part 6 (this post): a production-scale graph on GCP.

1. First, do you need a graph at all?

Graphs are fashionable, and fashionable tools are expensive to run. Before any design, decide honestly. Suppose a customer asks, "Which policy version applied to my claim, and was the provider part of a network group?" To answer, you must hop from a customer to a policy, to its versions, to a claim, to a provider, to a group. Each hop is a join, and the number of hops is the point: that is a multi-hop relationship question, and it is what graph databases are built for.

Question type	Best tool
"What does the policy say about X?" (find a passage)	Vector retrieval (Part 2)
"Total claimed this year" (aggregate, one or two joins)	SQL in BigQuery or the transactional database
"What is the claim's status now?"	The authoritative API, never a copy
"Which version applied on the service date, and who is linked to whom across four or more hops?"	Graph
"Which providers in a group have unusual claim volume?" (staff analytics)	Graph or BigQuery, depending on latency needs

A good test: write down your thirty most important questions. If fewer than a handful need three or more hops, or if SQL with indexed joins answers them within your latency budget, do not build a graph yet. A graph adds a second copy of your data, a synchronisation pipeline, a new query language and a new on-call burden. The rest of this post assumes you have passed that test.

2. What "large scale" means, and what breaks

For this series, imagine a national insurer: tens of millions of customers, hundreds of millions of claims, a few billion relationships, thousands of changes per second at peak, and an assistant that must answer in about a second. At that size, a graph prototype that worked on a laptop fails in specific, predictable ways:

Problem	What happens	Defence in this post
Supernodes	One hospital group connected to millions of claims; a query that touches it explodes	Start queries from small nodes, cap traversals, aggregate instead of listing
Deep traversals	Each extra hop multiplies the intermediate rows	Fixed, reviewed query templates with depth and row limits
Hot keys	Sequential ids send all writes to one place	Random UUID keys
Stale projection	The graph disagrees with the source system	Versioned idempotent sync, freshness metric, daily reconciliation
Data leaks	A traversal crosses from one customer to another	Customer-anchored queries; cross-customer queries are staff-only and aggregate
Runaway cost	Free-form queries scan huge parts of the graph	The model never writes queries; timeouts and row caps

3. Choosing the engine on GCP

Google Cloud gives you several ways to hold a graph. The decision is not "which is best" but "which job is each good at". The two native options use the same query language, GQL, the ISO standard Graph Query Language, which describes patterns such as "a customer who owns a policy that has a claim".

Option	Good for	Watch out for
Spanner Graph	The online graph: low-latency traversals behind the assistant, strong consistency, horizontal scaling, GQL combined with SQL, plus vector and full-text search over graph data	It is a database you size and operate; check edition and availability for your region
BigQuery Graph	The analytical graph: batch pattern analysis over full history, defined directly on existing tables	Preview at the time of writing (introduced April 2026); not for per-request serving. Confirm its status before depending on it
Third-party graph database (for example Neo4j on GCP)	Teams that need a mature graph ecosystem and algorithm libraries	Another vendor and another system to secure, size and keep in sync
Relationship tables in SQL (Part 2's approach)	Up to two or three hops, moderate scale	Deeper traversals become unreadable and slow

This post uses Spanner Graph for the online path and mentions BigQuery Graph for analytics. The reasoning is operational: the assistant needs predictable latency under concurrent load, and Spanner scales storage and throughput by adding compute while keeping consistency. A graph defined over SQL tables also means the same data can be read relationally, which helps reconciliation.

Figure 1. Production graph architecture. Events flow from the source systems through Pub/Sub into a sync service that writes Spanner Graph. The assistant reads it only through guarded, named queries. BigQuery holds history for analytics and for reconciliation.

The rule that keeps the architecture honest. The source systems stay authoritative. The graph is a projection: a fast, query-friendly copy of relationships. It can tell the assistant that a claim belongs to a policy version, but for anything involving money or a current status the assistant re-reads the source system, exactly as in Part 3.

4. The schema, and how the assistant infers

Before the build steps, here is the whole model on one page. Three ideas appear in it: nodes (the things), edges (the relationships between them) and properties (the few facts attached to each, such as a date or a status). Each node type has a key, and each edge points from a source node to a destination node, so "Customer Owns Policy" is a one-way arrow from a customer to a policy.

Figure 2. The graph schema. Green nodes belong to one customer; blue nodes are shared reference data. Solid edges come from source systems; the dashed edge is derived by a rule.

Read the figure as a map of the questions the assistant can answer. From a customer you can walk to their policies and claims. From a claim you can walk to its provider and the provider's group. From a policy you reach its versions, each with the dates it was valid and the document_id of the contract text. Nothing else is stored: no names, no addresses, no document bodies. Those stay in the source systems and are fetched through their own APIs when needed.

What "inference" means here

People use "inference" loosely with graphs, so be exact about what the system does. Here it means reaching a conclusion by following stored facts through a fixed rule, in a way that is repeatable and explainable. It does not mean a language model guessing a relationship. Figure 3 follows one real question through the graph.

Figure 3. An inference in practice. The question "which policy text applies to claim CLM-1042?" is answered by walking Customer, Policy, Claim, then applying a date rule to the policy's versions. The result selects a document in the knowledge base, so Part 2's retrieval quotes the right clause.

Notice what happened. Nothing in the data says "version 4 applies to this claim". The graph holds the claim's service date and each version's validity interval, and a rule combines them: the version whose interval contains the date wins. That is the inference. Its answer then feeds the vector search of Part 2, which filters by that version's document_id. Graph for structure, vectors for text, the claims API for status.

Five kinds of inference, and one that is not allowed

Kind	Example in this system	Where it runs
Path (traversal)	Claim, to its policy, to the policy's owner	A named query at request time
Temporal	The policy version valid on the service date	A named query, or a derived edge
Derived edge	AppliesVersion from a claim to its governing version	A batch rule that writes an edge with rule_id and derived_at
Aggregate	Claims per provider in a group over 90 days	A staff-only query returning counts
Semantic bridge	Graph picks document_id; vector search retrieves the passage	Agent tools chained in order
Model guess (forbidden as fact)	"These two providers seem related"	Never stored, never cited as evidence

The last row matters most. Trust comes in three levels, and the system keeps them apart:

Level	Origin	How it is used
Stored fact	A source system, copied by the sync	Authoritative for relationships; money and status still re-read at the source
Derived fact	A deterministic rule over stored facts, with a rule id	Reproducible and explainable; recomputed when inputs change
Model guess	A language model's opinion	May suggest where to look; never written to the graph or cited as evidence

Compute on read, or store the result?

For the version rule you have a choice. You can compute it every time a customer asks (a query with a date filter), or materialise it once as the dashed AppliesVersion edge. Neither is always right:

Approach	Strength	Cost
Compute on read	Always current; no extra storage; no invalidation to manage	A multi-hop query on every request
Materialise as a derived edge	Single-hop lookup; fast for analytics and bulk use	Can go stale; needs a recompute job and cleanup of old edges

A sound default: compute on read for customer requests, where a single claim keeps the query small, and materialise for bulk or analytical use, where you would otherwise repeat the same traversal millions of times. If you materialise, always record which rule produced the edge and when, and delete a rule's old edges for a claim before writing new ones so a corrected date can never leave a stale edge behind:

-- Rule version_by_service_date_v1: a claim is governed by the policy version whose validity
-- interval contains the claim's service date. Run for claims changed since @since (batch it by claim id range).
-- Step 1 removes edges this rule produced earlier for those claims, so a corrected date cannot leave a stale edge.
DELETE FROM ClaimAppliesVersion
WHERE rule_id = 'version_by_service_date_v1'
  AND claim_id IN (SELECT claim_id FROM Claim WHERE source_updated_at >= @since);

INSERT INTO ClaimAppliesVersion (claim_id, policy_version_id, rule_id, derived_at)
SELECT cl.claim_id, v.policy_version_id, 'version_by_service_date_v1', CURRENT_TIMESTAMP()
FROM Claim cl
JOIN PolicyHasClaim phc ON phc.claim_id = cl.claim_id
JOIN PolicyHasVersion phv ON phv.policy_id = phc.policy_id
JOIN PolicyVersion v ON v.policy_version_id = phv.policy_version_id
WHERE cl.source_updated_at >= @since
  AND v.effective_from <= cl.service_date
  AND (v.effective_to IS NULL OR v.effective_to >= cl.service_date);

The rule_id is also what makes the system auditable. When the assistant says "your claim falls under version 4", the evidence ledger can carry the rule and its timestamp, so a reviewer can see not just what was concluded but how.

5. Hands-on: build and run the graph

Nine steps, from design to operations. The code is in the project zip under graph/, src/graph/ and src/graph_sync/, and the graph is off by default so existing deployments are unaffected.

Step 1. Design the model for scale

The next job: decide what lives in the graph, because every extra property multiplies storage and write cost. Three rules guide the design. Ids and relationships only: names, addresses and documents stay in the source systems; the graph holds keys and the few attributes traversals filter on (status, dates). That keeps it small and limits privacy exposure. Random keys: use UUID strings, never sequential numbers, so writes spread across the cluster instead of piling onto one machine. Time on the edges that need it: policy versions carry effective_from and effective_to, so "which version applied on the service date" is a query, not a guess.

For the insurer, the nodes are Customer, Policy, PolicyVersion, Claim, Provider and ProviderGroup, and the relationships are Owns, HasVersion, HasClaim, ByProvider and MemberOf, plus one derived relationship, AppliesVersion, explained in Section 4. Notice the link to Part 2: each PolicyVersion carries the document_id used in kb.chunks, so the graph decides which version and vector retrieval then quotes that version's clause.

Step 2. Provision Spanner with Terraform

The next job: create the database, sized deliberately. Spanner capacity is measured in processing units (1000 units is one node). Start modestly, load-test in Step 8, then raise it; do not guess. The Terraform in the project adds the instance, the database and two narrow roles behind enable_graph, so enabling it is a reviewed pull request like any other change.

# infra/terraform/envs/dev/terraform.tfvars
enable_graph = true      # creates a Spanner instance; costs money from the moment it exists
# modules/platform: graph_processing_units = 1000 (tune after load testing)

The assistant's identity gets roles/spanner.databaseReader; the sync service gets roles/spanner.databaseUser. Again, the service that talks to customers cannot write to the graph. Check the Spanner documentation for the edition that includes Spanner Graph, and pick a regional or multi-region configuration to match your availability and residency needs.

Step 3. Create the schema, with traversal in mind

The next job: lay out the tables so that the most common traversals read data stored together. In Spanner, an interleaved table is stored physically next to its parent row. Placing each edge table inside its source node's table makes forward traversal (customer to policies) cheap, and an interleaved index does the same for the reverse direction (policy to its owner). Then a CREATE PROPERTY GRAPH statement declares which tables are nodes and which are edges.

-- Edge tables are interleaved in their SOURCE node's table so forward traversal reads co-located rows.
CREATE TABLE CustomerOwnsPolicy (
  customer_id STRING(36) NOT NULL,
  policy_id STRING(36) NOT NULL,
  source_updated_at TIMESTAMP NOT NULL,
) PRIMARY KEY (customer_id, policy_id),
  INTERLEAVE IN PARENT Customer ON DELETE CASCADE;

CREATE TABLE PolicyHasVersion (
  policy_id STRING(36) NOT NULL,
  policy_version_id STRING(64) NOT NULL,
  source_updated_at TIMESTAMP NOT NULL,
) PRIMARY KEY (policy_id, policy_version_id),
  INTERLEAVE IN PARENT Policy ON DELETE CASCADE;

CREATE TABLE PolicyHasClaim (
  policy_id STRING(36) NOT NULL,
  claim_id STRING(36) NOT NULL,
  source_updated_at TIMESTAMP NOT NULL,
) PRIMARY KEY (policy_id, claim_id),
  INTERLEAVE IN PARENT Policy ON DELETE CASCADE;

CREATE TABLE ClaimByProvider (
  claim_id STRING(36) NOT NULL,
  provider_id STRING(36) NOT NULL,
  source_updated_at TIMESTAMP NOT NULL,
) PRIMARY KEY (claim_id, provider_id),
  INTERLEAVE IN PARENT Claim ON DELETE CASCADE;

CREATE TABLE ProviderMemberOf (
  provider_id STRING(36) NOT NULL,
  group_id STRING(36) NOT NULL,
  source_updated_at TIMESTAMP NOT NULL,
) PRIMARY KEY (provider_id, group_id),
  INTERLEAVE IN PARENT Provider ON DELETE CASCADE;

-- DERIVED edge: written by a rule (graph/derive_applies_version.sql), never by source events.
-- rule_id and derived_at make every inference explainable and recomputable.
CREATE TABLE ClaimAppliesVersion (
  claim_id STRING(36) NOT NULL,
  policy_version_id STRING(64) NOT NULL,
  rule_id STRING(64) NOT NULL,
  derived_at TIMESTAMP NOT NULL,
) PRIMARY KEY (claim_id, policy_version_id),
  INTERLEAVE IN PARENT Claim ON DELETE CASCADE;

-- Reverse traversal (child -> parent) needs interleaved indexes on the destination side.
CREATE INDEX PolicyOwnerByPolicy ON CustomerOwnsPolicy (policy_id, customer_id), INTERLEAVE IN Policy;
CREATE INDEX ClaimPolicyByClaim ON PolicyHasClaim (claim_id, policy_id), INTERLEAVE IN Claim;

CREATE PROPERTY GRAPH InsuranceGraph
  NODE TABLES (Customer, Policy, PolicyVersion, Claim, Provider, ProviderGroup)
  EDGE TABLES (
    CustomerOwnsPolicy SOURCE KEY (customer_id) REFERENCES Customer (customer_id)
      DESTINATION KEY (policy_id) REFERENCES Policy (policy_id) LABEL Owns,
    PolicyHasVersion SOURCE KEY (policy_id) REFERENCES Policy (policy_id)
      DESTINATION KEY (policy_version_id) REFERENCES PolicyVersion (policy_version_id) LABEL HasVersion,
    PolicyHasClaim SOURCE KEY (policy_id) REFERENCES Policy (policy_id)
      DESTINATION KEY (claim_id) REFERENCES Claim (claim_id) LABEL HasClaim,
    ClaimByProvider SOURCE KEY (claim_id) REFERENCES Claim (claim_id)
      DESTINATION KEY (provider_id) REFERENCES Provider (provider_id) LABEL ByProvider,
    ProviderMemberOf SOURCE KEY (provider_id) REFERENCES Provider (provider_id)
      DESTINATION KEY (group_id) REFERENCES ProviderGroup (group_id) LABEL MemberOf,
    ClaimAppliesVersion SOURCE KEY (claim_id) REFERENCES Claim (claim_id)
      DESTINATION KEY (policy_version_id) REFERENCES PolicyVersion (policy_version_id) LABEL AppliesVersion
  );

Apply it as a migration, not through Terraform, because schema changes need ordering and review:

gcloud spanner databases ddl update insurance-graph --instance=agent-graph --ddl-file=graph/schema.sql

Spanner Graph's DDL and GQL syntax evolve, so validate this file against the current documentation, in a test database, before using it.

Step 4. Load history, then keep it fresh

The next job: fill the graph and keep it correct. There are two flows. The backfill loads existing history in bulk, normally from BigQuery tables through a batch pipeline such as Dataflow, with each table loaded in parallel and edges after their nodes. The change flow applies each business event as it happens: the source systems publish a change event to Pub/Sub (directly, or through change data capture, which turns database changes into events), and a small Cloud Run service applies it.

The hard part of the change flow is that events can be duplicated and can arrive out of order. The upsert below handles both: it creates the row only if it does not exist, then updates it only if the incoming event is newer than what is stored.

def apply_event(txn, ev: dict) -> str:
    spec = validate_event(ev)
    table, pk, ts = ev["table"], ev["pk"], ev["source_updated_at"]
    cols = spec["cols"]
    where = " AND ".join(f"{c} = @{c}" for c in spec["pk"])
    if ev["op"] == "delete":
        txn.execute_update(f"DELETE FROM {table} WHERE {where} AND source_updated_at <= @source_updated_at",
                           params={**pk, "source_updated_at": ts},
                           param_types={**{c: T[cols[c]] for c in pk}, "source_updated_at": T["TIMESTAMP"]})
        return "deleted"
    data = {**pk, **ev.get("values", {}), "source_updated_at": ts}
    names = list(data)
    ptypes = {c: T[cols[c]] for c in names}
    # 1) create the row if it does not exist, 2) update it only if this event is newer than what is stored
    txn.execute_update(f"INSERT OR IGNORE INTO {table} ({', '.join(names)}) VALUES ({', '.join('@' + c for c in names)})",
                       params=data, param_types=ptypes)
    sets = ", ".join(f"{c} = @{c}" for c in names if c not in spec["pk"])
    n = txn.execute_update(f"UPDATE {table} SET {sets} WHERE {where} AND source_updated_at < @source_updated_at",
                           params=data, param_types=ptypes)
    return "updated" if n else "ignored_or_inserted"

Two safeguards matter at this scale. Table and column names come from an allow-list (schema.py), so a malformed event cannot become a SQL injection. And a rejected event is acknowledged rather than retried forever, while a transient failure returns an error so Pub/Sub retries it and then moves it to a dead-letter queue, as in Part 2. For very high event rates, move the same logic into a streaming Dataflow job; the idempotent upsert stays the same.

Deletes use the same timestamp guard. For privacy erasure requests, publish a delete event for each affected node and verify afterwards, with the reconciliation in Step 7, that nothing remains.

Step 5. Let the application ask only reviewed questions

The next job: decide who may write queries. At scale, the answer is firm: the language model never writes graph queries. A model-generated query can scan the whole graph, cross from one customer to another, or simply be wrong. Instead the application keeps a small registry of named, parameterised queries that engineers have reviewed and load-tested. Each declares its parameters, its row cap and its scope:

QUERIES = {
    "list_customer_claims": NamedQuery(
        gql="""GRAPH InsuranceGraph
MATCH (c:Customer {customer_id: @customer_id})-[:Owns]->(p:Policy)-[:HasClaim]->(cl:Claim)
RETURN cl.claim_id AS claim_id, cl.status AS status, p.policy_id AS policy_id
ORDER BY cl.submitted_at DESC
LIMIT 50""",
        params={"customer_id": "STRING"}, columns=("claim_id", "status", "policy_id"), scope="customer"),

    # The graph answers "which policy VERSION applied on the service date"; Part 2 retrieval then
    # filters kb.chunks by that document_id, so the right clause is quoted.
    "policy_version_for_claim": NamedQuery(
        gql="""GRAPH InsuranceGraph
MATCH (c:Customer {customer_id: @customer_id})-[:Owns]->(p:Policy)-[:HasClaim]->(cl:Claim {claim_id: @claim_id})
MATCH (p)-[:HasVersion]->(v:PolicyVersion)
FILTER v.effective_from <= cl.service_date AND (v.effective_to IS NULL OR v.effective_to >= cl.service_date)
RETURN v.document_id AS document_id, v.version AS version, cl.service_date AS service_date
LIMIT 5""",
        params={"customer_id": "STRING", "claim_id": "STRING"},
        columns=("document_id", "version", "service_date"), scope="customer", max_rows=5),

    # Cross-customer pattern: AGGREGATES only, never other customers' claims. Staff tools only.
    "provider_group_claim_counts": NamedQuery(
        gql="""GRAPH InsuranceGraph
MATCH (g:ProviderGroup {group_id: @group_id})<-[:MemberOf]-(pr:Provider)<-[:ByProvider]-(cl:Claim)
FILTER cl.submitted_at >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 90 DAY)
RETURN pr.provider_id AS provider_id, COUNT(cl) AS claims_90d
GROUP BY pr.provider_id
ORDER BY claims_90d DESC
LIMIT 50""",
        params={"group_id": "STRING"}, columns=("provider_id", "claims_90d"), scope="internal"),
}

    ...

Look at the scopes. A customer query must start from the authenticated customer, so it can only walk outward from that person's own data. An internal query may cross customers, but returns aggregates only (counts per provider) and is rejected for any customer-facing caller. A guard checks all of this before anything reaches Spanner:

"""Pure-Python checks that run before any graph query leaves the application."""
from src.graph.queries import QUERIES, NamedQuery


class GraphGuardError(Exception):
    pass


def validate(name: str, params: dict, caller_scope: str, customer_id: str | None) -> NamedQuery:
    q = QUERIES.get(name)
    if q is None:
        raise GraphGuardError("unknown query")
    if q.scope == "internal" and caller_scope != "internal":
        raise GraphGuardError("query not allowed for this caller")
    if set(params) != set(q.params):
        raise GraphGuardError("unexpected or missing parameters")
    if q.scope == "customer":
        if not customer_id or params.get("customer_id") != customer_id:
            raise GraphGuardError("customer mismatch")
    return q

The client then runs the query with a timeout, a row cap and a bounded-staleness read, which is faster and cheaper than a fully current read and fine for relationships, since decisions about money still re-check the source system:

def run_named_query(name: str, params: dict, *, caller_scope: str, customer_id: str | None = None,
                    timeout_s: float = 5.0) -> dict:
    q = validate(name, params, caller_scope, customer_id)
    # Bounded-staleness read: cheaper and faster than a strong read. Relationships may lag by seconds;
    # decisions about money or status re-read the authoritative system instead.
    with _database().snapshot(exact_staleness=timedelta(seconds=10)) as snap:
        result = snap.execute_sql(q.gql, params=params,
                                  param_types={k: TYPES[v] for k, v in q.params.items()}, timeout=timeout_s)
        rows = []
        for r in result:
            rows.append([str(v) if v is not None and not isinstance(v, (int, float, str)) else v for v in r])
            if len(rows) >= q.max_rows:
                break
    return {"columns": list(q.columns), "rows": rows}

Guidance from Google's tuning documentation shapes how such queries are written: start a traversal from the lowest-cardinality node (a specific claim, not "all providers"), and for graphs with supernodes limit how many edges a query follows from a node. Build those limits into the templates, and test each one against a realistic hub node.

Step 6. Connect the graph to the agent, and to verification

The next job: give the assistant two small tools and keep them inside the evidence model of Part 3. The tools take no identity from the model; they read the customer from session state, like the claims tool.

from google.adk.tools import ToolContext
from src.graph.client import run_named_query
from src.graph.guard import GraphGuardError


def _run(name, params, tool_context):
    cust = tool_context.state["authenticated_customer_id"]
    try:
        out = run_named_query(name, {"customer_id": cust, **params}, caller_scope="customer", customer_id=cust)
    except GraphGuardError:
        return {"success": False, "error_code": "NOT_ALLOWED", "retryable": False}
    except Exception:
        return {"success": False, "error_code": "GRAPH_UNAVAILABLE", "retryable": True}
    return {"success": True, "source": "graph", **out}


def list_my_claims(tool_context: ToolContext) -> dict:
    """List the signed-in customer's claims with their status. Use when the customer
    asks about their claims in general, or does not give a claim id."""
    return _run("list_customer_claims", {}, tool_context)


def get_policy_version_for_claim(claim_id: str, tool_context: ToolContext) -> dict:
    """Find which policy version (document_id and version) applied on the service date of
    one of the signed-in customer's claims. Use before searching policy text for that claim."""
    return _run("policy_version_for_claim", {"claim_id": claim_id}, tool_context)

The pattern for answering "what does my policy say about this claim" becomes a chain: the graph tool returns the applicable document_id and version, the policy search filters on that document, and the claims tool supplies the current status. Each result enters the evidence ledger (graph results appear as graph:... sources), so the verifier can check the answer and the customer sees a citation. One more caution that belongs in the instructions and the verifier's thinking: a graph edge is evidence of a relationship, not proof of a business conclusion. "Claim is linked to this policy version" is a fact; "therefore it is covered" is a judgement that needs the policy text.

To turn it on: set AGENT_ENABLE_GRAPH=true (Terraform sets this together with the instance and database names when enable_graph is true).

Step 7. Prove the graph is fresh and correct

The next job: measure what you promised. Two numbers matter. Freshness is how far behind the graph is: the sync service logs the source_updated_at of each event, so the gap between now and the newest applied timestamp is a metric you can chart and alert on. Correctness is whether the graph matches the source. A daily reconciliation compares counts and checksums between the history in BigQuery and Spanner; BigQuery can query Spanner directly through a federated connection, so the check is a single SQL statement per table:

-- compare row counts between the BigQuery history and the live graph
WITH src AS (SELECT COUNT(*) AS n FROM `history.claims` WHERE NOT is_deleted),
     graph AS (SELECT n FROM EXTERNAL_QUERY(
        "projects/PROJECT/locations/REGION/connections/spanner_graph",
        "SELECT COUNT(*) AS n FROM Claim"))
SELECT src.n AS source_rows, graph.n AS graph_rows, src.n - graph.n AS gap
FROM src, graph

A small, stable gap during busy periods is normal because the two sides are read at slightly different moments. A growing gap is a defect. Alert on both freshness and gap, and keep the dead-letter queue visible, since a stuck event is often the cause.

Step 8. Load-test before launch, and watch the right signals

The next job: find the limits on purpose, instead of finding them in production. Build a test with a realistic query mix (mostly the customer queries, a few staff aggregates) and realistic data skew, including the biggest provider group you have, because supernodes only misbehave when they exist in the data. Ramp traffic with a load tool and watch four things:

Signal	What to look for
Query latency, p95 and p99	Per named query, not averaged; the slowest template sets your tail
Spanner CPU utilisation	Keep headroom; Google publishes recommended maximum utilisation targets for regional and multi-region instances, so scale up before you reach them
Hot spots	Spanner's key and query diagnostics tools show uneven load; sequential keys are the usual cause
Rows scanned per query	A sudden jump means a template or a data change turned a narrow traversal into a wide one

Repeat the test after any schema change and after any new named query. Make it a gate in the pipeline from Part 5: a new query template cannot merge until it has a latency and row-count budget and a test against a hub node.

Step 9. Protect the graph and operate it

The next job: make it safe to run for years. Security first. The customer-facing identity can only read, queries are anchored on the authenticated customer, cross-customer patterns exist only as aggregate staff tools, and the graph holds ids rather than personal details. Enable audit logging for data access on the database, and propagate erasure requests as delete events.

Then the incident you should rehearse. Tuesday morning: the p99 latency alert fires for policy_version_for_claim. Traces show that a hospital merger last night attached two million claims to one provider group, and a staff query that lists providers in the group is scanning it. The fix path: the customer-facing queries are unaffected because they start from a single claim; the staff aggregate gets a tighter row cap and a time window; you add the group to the load test as a permanent supernode case; and you raise capacity only if the signals still call for it. Total customer impact: none, because the design kept the expensive question away from customers.

Alert	First action
Graph freshness behind its objective	Check the sync service and the dead-letter queue; replay stuck events
Reconciliation gap growing	Find the table and the time range; reprocess from history; look for a source change that skipped events
Query p99 or rows scanned rising	Identify the template; look for a new hub node; tighten the cap or the window
Spanner CPU near target	Add processing units; investigate hot spots before adding more
Any guard rejection from customer callers	Treat as a possible bug or abuse attempt; investigate the traces

6. Cost, limits and gotchas

Topic	What to know
Cost model	You pay for provisioned compute and storage whether or not queries run. Start small, scale by measurement, and set a budget alert before enabling the graph
Two copies of the truth	Every relationship exists in the source system and the graph. Design the sync and reconciliation first, not last
Schema changes	Adding a column is easy; changing keys is a migration project. Choose keys carefully in Step 1
Preview features	BigQuery Graph was in preview at the time of writing; do not put a preview service on a customer-facing path
Consistency	Bounded-staleness reads trade freshness for speed. Never use them for decisions about money or current status
Cross-region	Multi-region improves availability and costs more and slows writes; choose from your availability target, not by default
Free-form queries	Fine for engineers in a console against a replica or test database; never for the model or for customers

7. The updated project, and what is next

The project zip now includes the graph schema (graph/schema.sql), the guarded query layer (src/graph), the sync service (src/graph_sync), the optional agent tools, Terraform behind enable_graph, a derived-edge rule (graph/derive_applies_version.sql) and 14 passing unit tests that cover the guard and the sync validation without needing any cloud access.

What has and has not been checked. The Python compiles and the offline unit tests pass. Nothing has been run against a real Spanner database: the DDL, the GQL queries (including FILTER, TIMESTAMP_SUB and the multi-MATCH form), the Terraform resources and the edition setting all need validation in a sandbox against current documentation. The sync service has no Dockerfile or CI wiring yet; deploy it the way Part 2's ingestion service is deployed.

Natural next steps: a streaming Dataflow version of the sync for very high event rates, Spanner change streams as an event source, vector search over graph nodes for hybrid retrieval, and staff-facing analytical tools on BigQuery Graph once it is generally available.

References: Spanner Graph documentation, Tuning Spanner Graph queries, Designing a Spanner Graph schema, BigQuery Graph.