Delta Trigger: Connecting Data Changes to Agents

Organizations do not suffer from a lack of systems. They suffer from a lack of timely, shared reaction when something important changes inside those systems.

A contract clause is revised. A support severity jumps. A feature flag flips in production. A campaign budget crosses a threshold. A vendor risk score worsens. A policy exception is approved. Each of those facts lands in a table somewhere: CRM, ticketing, product analytics, procurement, legal workflow, knowledge stores. People and agents across the business need to notice it, interpret it, and act, usually through the lens of what the company has already learned about situations like it.

Most agent systems still wake up on a schedule. Every five minutes, every hour, every night, they re-scan a database, re-read a lake, or poll an API, and then decide whether anything interesting happened. That model is simple to ship and expensive to operate. It reprocesses the quiet majority of the data in order to find the noisy minority. It welds "how often we look" to "how fast we react." And when something does change, the agent usually has only a current snapshot, not a crisp before and after.

Delta Trigger is the other architecture: a detection, propagation, and delivery engine that watches Delta Lake tables through Change Data Feed, evaluates a declarative set of triggers, and publishes durable, multi-tenant events onto a message bus that agent runtimes already know how to consume.

Agents stop polling the company's memory. The memory emits change, and the bus wakes whoever should care.

Each consumer gets the same contract: this entity changed in this way, here is the before and the after, here is a dedup key, here is which organization it belongs to. This piece covers why that pattern exists, how it is built, the architecture decisions that were genuinely hard, and what it unlocks once business memory and agents are first-class consumers of change.

Modern platforms write operational and institutional state into tables: cases, contracts, accounts, work items, signals, policies, vendor records, feature metrics, derived knowledge. Domain agents care about transitions in those tables, not about the tables themselves.

A transition looks like this:

• A matter's status moved to under review, or to approved.
• A new high-priority customer case was inserted.
• A risk or health score crossed a boundary.
• A budget, discount, or liability field changed by more than a threshold.
• A critical field was cleared, or an exception flag was set.
• A document or knowledge record was linked to a new entity.

Those are state changes, not bulk reads. They are also how organizational memory stays alive. Something happened, the company should notice it, relate it to prior outcomes, and decide what to do next.

If all you have is batch jobs and scheduled agents, you pay three taxes:

• A latency tax. Reaction speed is bounded by the poll interval, and the poll interval is bounded by what you are willing to spend.
• A compute tax. Every idle poll still scans or samples something. You pay for the silence.
• A semantics tax. Without pre and post images, "what changed?" becomes guesswork or an expensive point-in-time reconstruction.

Change data capture solves all three for data pipelines, and has for years. Debezium and the outbox pattern are mature, well-understood infrastructure. What those systems rarely solve cleanly is the agent connection: multi-tenant scoping, GitOps-managed trigger definitions, durable delivery with deduplication, and a clean ownership boundary so that the agent runtime keeps execution (ledgers, budgets, fencing, human approval) while a thinner library owns the single claim that this row transition matched this rule.

Delta Trigger is that thinner layer. It is the change plane underneath a company-wide memory and agent fabric, not a gadget bolted onto one department's workflow.

Institutional memory is layered: short-lived context, episode records of what happened, durable knowledge about people and entities, and patterns of what tends to work. Those layers are only useful if they update when reality does, and if the right specialists wake up when an update matters for their particular job.

One change plane can feed all of them. The same event, say a strategic account flag flipping on, means something different to success, to legal, to product, and to finance. Each agent reads the transition through its own skill and its own memory. None of those teams has to invent a separate poller to find out that the flag flipped.

Delta Trigger is not that memory. It is the nervous system: reliable detection that something in the shared stores changed in a way worth attention.

Delta Trigger is the change-to-event plane between Delta Lake and agent infrastructure. It owns six things.

It deliberately does not own:

• Agent scheduling, concurrency fencing, or run ledgers.
• Prompt text, output schemas, or domain business logic.
• The memory graph itself, or which function's skill interprets an event.
• A second, weaker runtime that competes with the real agent framework.

That boundary is the product. Detection stays generic and reusable across every function. Execution and interpretation stay where budgets, retries, governance, and human approval of memory writes already live.

To make agents event-native on shared business data. Teams already materialize operational truth and derived knowledge into Delta-style tables. Agents that maintain or act on organizational memory need to run when that truth changes, not when a cron happens to fire. Ad-hoc full-table diffs do not scale across many tenants, many tables, and many functional triggers.

One change plane, many functions. Legal, product, marketing, procurement, and customer teams should not each maintain a private CDC stack. A shared library and bus, with function-specific trigger definitions and function-specific agents, keeps the infrastructure common and the interpretation specialized. That is the split that actually holds up over time.

To separate "something happened" from "run this agent." Detection has different service levels, different failure modes, and a different scaling axis than agent execution. A publish-side monitor shards by tenant. Consume-side runners scale with work. Mixing them into one process creates coupling you will regret under load, especially when one event fans out to several domain agents.

To make trigger policy reviewable. Triggers are policy: which table, which primary key, which condition, which event name. Encoding them as YAML under a global directory and per-organization directories lets operators and domain owners review, validate in CI, and hot-reload without redeploying agent code for every new condition. Adding "legal exception granted" should not require a platform release when the vocabulary already exists.

Because the failure modes are silent and expensive. Watermarks advanced before durable publish. Business predicates pushed into the wrong layer of the change feed. Duplicate agent runs without idempotency keys. Silent full-history replay after a lost watermark. Those failures corrupt trust in memory and automation much faster than they corrupt a dashboard, and a dashboard at least looks wrong when it is wrong.

To get multi-tenant delivery without wiring every consumer to every producer. One shared bus, subjects shaped as tenant, trigger, and version, and consumers that subscribe either per-tenant or across tenants with a wildcard. Application code always receives an explicit tenant id on the event, never one inferred from a path convention.

domain tables, CDF enabled
        |
        v
monitor (publish side)
registry, watermarks,
pairing, predicates
        |
        v
JetStream stream
triggers.{org}.{name}.v1
        |
        v
trigger router
one event, many agents
        |
        v
domain runners
process, propose memory

Two deployables fall out of that shape:

• The monitor sweeps tenants and triggers, and publishes events.
• The router and runners consume events and execute agents with full runtime semantics, including human-approved writes back into memory where that is required.

The library provides the monitor loop and the queue primitives. The agent runtime implements the router and the runners. Domain teams own their skills, their prompts, and which memory writes they propose. Three owners, three clean seams.

The change feed layer

Delta Change Data Feed adds three metadata columns to every change row: _change_type, _commit_version, and _commit_timestamp. The change type is one of insert, update_preimage, update_postimage, or delete. That fourth value is the detail everything else hangs on: for a state transition, a single logical update arrives as two physical rows.

So the change feed layer:

• Reads a version range starting from the watermark.
• Optionally pushes down a structural filter, on change type only.
• Pairs pre and post images by commit version and primary key.
• Matches insert-only triggers for the new-record case.
• Computes a globally unique dedup key for every fire.

That dedup key is the spine of idempotency for everything downstream, including the question of whether this episode has already been written into memory.

The predicate engine

Predicates are an expression tree, evaluated in Python after pairing:

• Comparisons against fields in the before image or the after image.
• Boolean composition with AND, OR, and NOT.
• Set membership.
• Cross-image comparisons: absolute and percentage deltas, and threshold crossings.

YAML compiles into the same tree the runtime evaluates, so there is one semantics, not two. A missing column raises loudly, because schema drift is a configuration error and not a silent false negative. The difference between those two behaviors is the difference between an alert and an outage nobody noticed.

What the engine cannot do is deliberate. No aggregates over windows. No temporal absence, as in "no activity for thirty days." No cross-table joins. Those need a different execution model, either a scheduler or a second read, and pretending otherwise produces a query engine with a trigger-shaped API. So "no reply in thirty days" stays a scheduled check, and "severity just became critical" is a natural change-feed trigger. The engine stays row-local and honest about its ceiling.

The registry

name: case_severity_critical
scope: global
table: cases
primary_key_cols: [case_id]
fires_on: state_transition
predicate:
  eq: {field: severity,
       value: critical}

name: policy_exception_created
scope: global
table: policy_exceptions
primary_key_cols: [exception_id]
fires_on: new_record
predicate:
  eq: {field: status, value: active}

Per-tenant overrides and disables live under a per-organization directory. A hot-reloading registry fingerprints directory content, so operators can ship YAML without bouncing processes, with a torn-read guard for concurrent deploys. Tree validation collects every structural violation in a single CI pass rather than failing on the first one.

Global triggers encode company-wide policy: always notice critical severity. Tenant overrides encode local reality: this business unit disables a noisy trigger, or uses a different status vocabulary. Both are reviewable in a pull request, which is the entire point.

The monitor loop

Each poll tick, for one trigger:

• Load the watermark and choose an ending version, chunked.
• Read the change feed for that range.
• Pair and match rows.
• Evaluate predicates.
• Invoke a synchronous handler per event, in order.
• Only if every handler succeeded, advance the watermark.

That last step is load-bearing. Partial success must never skip commits, because skipped commits become holes in institutional awareness, and a hole in awareness does not announce itself.

The orchestrator

The orchestrator sweeps active tenants against effective triggers, wires the publish call as the handler, accumulates statistics for events published, publish errors, catch-up state, and poll timings, and exposes hooks for metrics tagging. Callers inject which tenants to sweep and how to shard them, how logical table names map to physical URIs, the publisher, the watermark store, and the hooks.

The library hardcodes no multi-tenancy layout beyond those protocols. Logical tables can be cases, contracts, campaigns, vendors, feature metrics, or knowledge nodes, whatever the company's systems of record actually materialize.

The queue

JetStream rather than plain publish and subscribe, for five reasons:

• Durable streams with retention windows.
• Pull consumers with explicit acknowledgement, which is in fact the only acknowledgement mode pull consumers support.
• Competing consumers for horizontal scale.
• Message-level deduplication through the Nats-Msg-Id header, set to the dedup key.
• A dead letter path once delivery attempts exceed the configured maximum.

One caveat worth stating plainly, because it is the kind of thing that gets assumed away: JetStream's duplicate window defaults to two minutes and is configurable per stream. It protects you from a publisher retry storm. It does not protect you from a redelivery an hour later, or from a deliberate replay next Tuesday. Broker-level deduplication is a convenience. The dedup key carried in the payload is the actual guarantee, and downstream writes have to honor it.

Subjects are tenant-scoped:

triggers.{org}.{trigger}.v{n}

Consumers commonly filter on one trigger across all tenants, or pin a single tenant for hard isolation. Different agent fleets subscribe to different trigger names without sharing process memory.

There is no library-owned dispatcher. Consumption looks like this:

bridge.pull_and_process(
  durable_name="router-case-crit",
  filter_subject=event_subject(
    "*", "case_severity_critical"),
  handler=route,
)

The agent runtime's router reconstructs the typed event and enqueues work on its own transport, rather than running a second half-built runtime inside the trigger library. One event can become several run requests: the customer agent and the compliance agent, from the same transition.

The optional analysis base

For the common shape, an idempotent model analysis written back to a table, a base class provides a dedup-aware write path through an atomic merge rather than a read-then-write, tolerant JSON parsing of model output, and schema-filtered fields. Concrete agents, whether contract risk narratives, case root-cause drafts, campaign postmortems, or vendor risk briefs, live outside the library. Concurrent writers for the same dedup key collapse to one row. Concurrent distinct events do not lose data. That is how derived memory stays consistent when the runners scale out.

One: the detection library is not the agent runtime

Ship a focused package for change feed to events to bus. Leave scheduling, fencing, budgets, ledgers, and domain interpretation to the agent framework and the function-specific skills.

An earlier version of this design put a dispatcher and an agent handler inside the library. Cross-referencing it against a production agent runtime design surfaced the collision immediately: two competing stories for "pull work and run code," and the library's version was materially weaker. Deleting the dispatcher kept the dependency one-way and prevented the kind of architecture drift that is invisible until the day you need to change one of the two.

Two: synchronous handlers, with a sync bridge over async NATS

Handlers are plain synchronous callables that raise on failure, because watermark advancement depends on that contract. The NATS Python clients are asyncio-only, so a sync bridge runs a dedicated event loop on a background thread and exposes a synchronous publish and pull facade.

"Advance the watermark only after durable publish" has to mean synchronously durable from the monitor's point of view. Hiding async inside the monitor would either break the raise-on-failure guarantee or invite fire-and-forget bugs, and a fire-and-forget bug in this position leaves a memory system quietly believing nothing happened. The bridge is therefore permanent architecture, not a temporary hack. Any future asyncio-only queue would need the same shape.

Three: never push business predicates into the scan

The change feed read may receive structural filters on change type only, such as pre and post images, or inserts. Business predicates, severity equals critical, status equals approved, run after pairing.

The reason is specific and nasty. Push a postimage-shaped predicate into the scan and it silently drops the preimage rows. Pairing needs both images. The trigger never fires, and nothing errors. There is no exception, no log line, no lag metric. In a memory system that is worse than latency, because entire classes of institutional event simply disappear.

A trigger that never fires and never errors is the worst failure mode in the system, because it looks exactly like a quiet week.

The right response is a test that deliberately pushes the unsafe predicate and asserts that pairing breaks, so the trap stays documented in executable form. Slightly more rows get materialized per chunk. Correctness beats scan-time cleverness.

Four: hardened watermarks and chunked catch-up

Four rules, each one paying for a specific incident:

• Save the watermark only after every handler in the chunk succeeds.
• Cap the commits processed per poll, with a sane default.
• Write watermark files atomically, to a temp file then a rename.
• Fail hard on a corrupt watermark file rather than silently resetting to the beginning of time.

While catching up, the monitor polls faster and exposes a catch-up signal. Crashes mid-range must not skip events. A multi-day outage must not exhaust memory by materializing the whole gap at once. And a silent full-history replay is an operational incident rather than a convenience, because a replay re-wakes every subscribed function at once. The trade-off accepted here is at-least-once delivery, which is exactly why the dedup key exists.

Five: tenant identity is explicit, end to end

The tenant id is a required field on the event and a token in the subject. It is never re-derived from table path conventions at consume time.

A shared bus across many organizations or business units is the default deployment, not the exotic one. Consumers should not need knowledge of storage layout to know who an event belongs to, and subject-level access control can enforce isolation at the infrastructure layer in a way that application-side filters cannot guarantee on their own. That matters a great deal when legal data and customer data ride the same platform. The redundancy between subject and payload is on purpose: the subject is for routing and access control, the field is for application code.

Six: competing consumers, no custom partition ring

Horizontal scale on the consume side is N identical processes sharing a durable name, and JetStream splits the work. Partition managers and leader election are unnecessary for this workload when handlers are idempotent.

The honest caveat is that ordered-per-key delivery is not guaranteed across a shared pull group. Correctness comes from keys and merges, not from global order. A memory writer that keys on the dedup key is safe under scale-out. A memory writer that assumes ordering is a bug waiting for its first busy afternoon.

Seven: publish-side sharding belongs to the caller

The set of tenants to sweep is a callable. Sharding by hash of tenant id is an operational pattern, not library magic, because different deployments have different sources of truth for which tenants are active. The one hard requirement is that watermark stores must be tenant-scoped, so two shards never clobber each other on the same trigger name.

Eight: idempotent writes use merge, not check-then-act

Analysis-style agents write with an atomic Delta merge on the dedup key, with exponential backoff and full jitter under commit contention.

Read-then-write, as in "have I seen this event before," races under concurrent runners. Delta uses optimistic concurrency control and will raise a concurrent-append conflict when two writers touch the same files, which is the expected outcome of two runners merging into the same table. Plain exponential backoff without jitter then re-collides, because both retries wake at the same moment. Full jitter is the fix, and it is worth knowing that this is a measured result rather than folklore. Duplicate rows in a memory table are not merely untidy. They break trust in what the company thinks it knows.

Nine: production needs a durable watermark store

A JSON file store is fine for a single VM with local disk. Ephemeral compute, whether serverless or diskless pods, must inject a production watermark store backed by Postgres, Delta, or a key-value store. Losing watermarks does not look like a crash. It looks like everything re-firing since the beginning of history, which is a much more expensive way to find out.

A production-shaped flow, end to end:

• The monitor publishes a trigger event after the predicate matches and the publish is durable.
• The router pulls with a durable consumer filtered by trigger name, across all tenants or pinned to one.
• The router reconstructs the typed event and enqueues one or more run requests on the agent runtime's work transport.
• A runner claims the work with fencing, builds a run context including any coalesced events, and calls the agent.
• The agent reads institutional memory, drafts analysis or actions, and proposes writes, often under human approval when the write updates shared memory.
• Failures retry under the runtime's policy. Poison messages land in the dead letter queue once delivery attempts are exhausted.

There is an important separation between the test model and the production model. In tests, the router may call the agent directly, to prove routing works. In production, the router stops at enqueue, because runners own concurrency and ledger semantics. Blurring those two is how a clean design quietly acquires a second scheduler.

Agents that only need analysis written to a table subclass the analysis base. Agents that need multi-tool workflows, retrieval over long-term memory, or cross-system actions implement the full runtime contract, and still receive the same typed event objects.

These are illustrative rather than exhaustive. The point is breadth: one change plane, many functions, one shared memory fabric.

Legal and compliance. A contract risk score crosses a threshold, which wakes a risk narrative agent that compares clauses against prior positions and assembles an escalation pack for counsel. A policy exception record is inserted, which wakes a compliance checklist agent and enriches the audit trail. A jurisdiction or governing-law field changes, which triggers a conflict check against the regulatory playbooks already in memory.

Customer-facing teams. Case severity becomes critical, which produces a war-room brief with commitments recalled from past episodes and suggested next actions. Account health drops through a boundary, which wakes a playbook agent grounded in similar recoveries. A commitment field is updated on a case, which writes the commitment into memory and schedules the follow-through reminder.

Product. Feature adoption drops by more than a set percentage, which wakes a research agent that correlates support themes with release notes. Launch status moves to generally available or to rolled back, which produces a cross-functional brief for marketing, support, and docs. A critical bug label is applied, which links prior similar defects out of institutional memory.

Marketing. Campaign budget utilization crosses a limit, which produces a spend-risk brief with reallocation options. A segment membership rule changes for a high-value cohort, which triggers a messaging consistency check against approved language. A content asset is rejected or expires, which cascades into dependent campaigns and open tasks.

Procurement. A vendor risk tier worsens, which wakes a risk agent that summarizes open purchase-order exposure and recalls alternative vendors from memory. Payment terms or renewal dates change, which produces a coordination brief for finance and procurement. A new preferred-vendor record arrives, which starts an onboarding checklist and a policy alignment pass.

Finance and operations. A material amount delta lands on a forecast or accrual line, which wakes an explainability agent with linked source episodes. A control exception flag is set, which assembles an evidence pack alongside prior exception outcomes. A cost center is reassigned, which scans dependent systems of record for access and workflow impact.

People and internal operations. Role sensitivity or access tier changes, which wakes an access-review agent. A policy acknowledgement goes overdue, modeled as a status transition, which nudges with the correct policy version pulled from the knowledge store. An org hierarchy change for a critical owner cascades ownership across open cases, contracts, and projects.

The pattern underneath all of them

The highest-leverage use is not any single agent. It is shared episodes. A change fires a trigger. Domain agent A interprets it for its function and proposes a memory write: the signal, the context, the recommended action. Later, domain agent B in a different function retrieves that episode when a related entity changes. Over time, patterns form, and the organization starts to know that when X transitions this way, Y interventions worked.

Delta Trigger does not invent those patterns. It guarantees the first step, noticing X, is reliable, multi-tenant, and identical for every function that depends on the same underlying tables. Everything compounding above it depends on that first step not being probabilistic.

The runbooks are unglamorous: redeploy monitors, rotate credentials, repair a corrupt watermark deliberately, and replay from a known version only with eyes fully open, because a replay re-notifies every subscribed function at once. Watermark age against current table version is the single most valuable signal on that list, because it is the only one that catches the failure where everything looks healthy and nothing is moving.

What the system is not, stated as clearly as what it is:

• Not a general stream processor. No windowed aggregates, no complex event processing language.
• Not an agent framework. No schedules, no ledgers, no prompt registry.
• Not the memory graph. It delivers change; skills and runtimes interpret and, with approval, write.
• Not unconstrained fan-out. Only declared triggers fire, and the rest of the noise stays in the lake.
• Not a place to push business filters into the scanner. Correctness forbids it for transitions.
• Not a single-function product. The same plane serves legal, customer teams, product, marketing, procurement, and everyone after them.

Those non-goals are what keep the thing understandable and testable. A system that says yes to all six of them is a data platform, and a data platform is a different project with a different budget.

• Events over snapshots for agent activation and memory freshness.
• At-least-once plus dedup keys over fragile exactly-once myths.
• Pair before you filter on business columns.
• Advance progress only after durable success.
• Tenant identity is explicit from subject to payload to handler.
• Policy as YAML, interpretation as skills. Triggers are GitOps. Agents are domain programs.
• One-way dependency. The trigger library never imports the agent framework.
• One change plane, many functions. Shared infrastructure, specialized lenses.
• Prove the dangerous claims with tests: unsafe pushdown, concurrent merges, competing consumers, TLS hang modes.

The extensions that preserve the architecture rather than fight it:

• Additional queue backends behind the same publisher and pull protocols.
• Pluggable watermark stores as first-class production defaults, not as an injection you remember on the way to production.
• Trigger classes for scheduled absence checks, still outside pure change-feed detection.
• Stronger subject-level access control per tenant credential.
• Richer coalescing and multi-agent fan-out policy in the router.
• Closer coupling to pattern memory: not only that this changed, but that this change matches a known outcome class.

The bet underneath all of them stays the same. Shared business tables become a living memory surface. Change feeds become the nervous system. Agents across every function become reflexes with context and governance, instead of scheduled full-table guesswork owned by whichever department got there first.

Agents are only as timely as their inputs. Polling a lake is a compromise that works right up until tenant count, table size, functional breadth, and reaction commitments stop cooperating, and those four rarely stop cooperating one at a time.

Delta Trigger treats the change feed as the source of truth for what changed, a predicate registry as the language for what matters, watermarks as the contract for how far we have safely gone, and a durable bus as the link to who should act. Agents plug in as consumers of typed transition events, each carrying a before, an after, a tenant identity, and a dedup key, while the agent runtime keeps ownership of execution and of how those events compound into durable organizational memory.

Build the detection plane carefully. Keep it boring and correct. Connect every function deliberately. Then every meaningful row transition becomes shared awareness, rather than the next full-table scan, and rather than a signal only one team ever sees.