The Mess Is the Moat

At DataRobot I had a front-row seat to a strange spectacle: some of the most accurate predictive models ever put in front of enterprises, walking into organizations and accomplishing far less than the math said they should.

The models were rarely the problem. The problem was everything around them. A churn model would flag an account, and by the time anyone acted, the context that made the prediction actionable had evaporated. A model would be trained on historical data soaked in the company's own past decisions, and would faithfully learn those decisions back. And when something finally worked, nobody could prove which intervention deserved the credit, so nobody doubled down. I left that chapter of my career with a conviction I have carried into everything since: in the enterprise, the mess is not an obstacle on the way to the value. The mess is the terrain the value lives in.

Parts 1 and 2 of this series argued for a decision-and-outcome observability layer built from decision episodes. This part is the honest accounting: the analog world is messy in five distinct ways, each needs different treatment, and none of them makes the loop impossible. In fact, whoever handles the mess well ends up with the moat, because the mess is exactly what generic AI cannot see.

The central finding from everything I have researched and lived: messy human work is not unlearnable. But "messy" is five different problems wearing one word, and treating them as one problem is how enterprise AI initiatives die politely in steering committees.

Partial observability. Some of the most important state is never entered into any system. People know the customer sounded worried. They know a particular executive will reject the proposal. There was a hallway conversation. A manager deliberately waited because of a political constraint. Research on knowledge work keeps finding the same fragmentation: information scattered across platforms before you even count what lives only in heads.¹

LLMs change the economics here, because email, chat, documents, meetings, and notes can now be transformed into semantic signals at scale, which is exactly what Microsoft is productizing as a workplace context layer for agents.² But there is a discipline that must survive the excitement: inferred state must remain inferred state. A good architecture distinguishes four kinds of knowing, and never lets them blur:

Observed: CRM stage moved 10:13
Stated: "Legal is the blocker"
Inferred: intent = unblock
  contract review, conf 0.77
External: signed 16 days later

Collapse those four into equally trusted "data" and you have manufactured false certainty at industrial scale.

Latent intent. Behavior does not tell you why. Two salespeople offer the same discount; one is defusing churn, the other is accelerating a year-end purchase. Same action, different meaning. There are exactly three ways to recover intent, and a serious system uses all three with provenance attached. Explicit intent is best and expensive: ask at the important moments, "what are you trying to accomplish?" Contextual inference is cheap and uncertain: a model reads the communications, the role, the state. Retrospective rationale is useful and biased, because humans reconstruct their reasons after seeing results. The design principle: do not make humans keep a diary. Ask for a few seconds of explicit signal exactly where uncertainty is high and the decision is consequential. That turns human input into active learning instead of continuous reporting.

A salesperson makes twenty meaningful choices before a deal closes. A product change moves churn six months later. This is the credit-assignment problem, and sequential decision-making research treats delayed, noisy feedback as a fundamental obstacle, not an inconvenience.³ The practical answer is the micro-result, which shortens the feedback horizon by asking better questions in sequence:

Q1: Did the specialist
    clear the objection?
Q2: Does clearing it predict
    move to procurement?
Q3: Does that pathway track
    higher win probability?
Q4: Would the specialist have
    helped vs not? (costly)

Those are four different levels of knowledge, and the genius of the ordering is that Q1 is answerable this week while Q4 may take a year of experiments. Most organizations try to answer Q4 first, fail, and conclude the whole thing is impossible.

Which brings me to the most liberating sentence in this series: you do not need causal identification for most closed-loop applications.

Suppose history shows that incidents with a particular symptom cluster usually end in rollback. The system can say: 38 similar incidents were observed; 31 eventually rolled back; median time to rollback was 42 minutes; the ones that rolled back within ten minutes recovered faster. That is precedent retrieval, not causal proof, and it is already valuable. Patterns are sufficient for retrieving similar cases, spotting recurring motifs, detecting deviations, predicting likely next states, forecasting delays, finding bottlenecks and rework loops, surfacing what experts usually do in comparable situations, and suggesting a low-risk next step. Process mining has run on exactly this logic for two decades; prescriptive process monitoring extends it toward runtime intervention while its own literature admits causality becomes essential as intervention stakes rise.³

Patterns tell you what usually happens. Causal methods tell you what will happen because you intervene. Pattern-first; causality where the intervention risk earns it.

But the moment the system starts recommending actions, history's selection effects come for you, and I have personally stepped on every one of these rakes.

Escalated support cases have worse outcomes than non-escalated ones, so a naive learner concludes: avoid escalation. Except agents escalated precisely the cases that were already hardest. Large discounts correlate with low win rates, because weak deals are the ones that need discounts; I watched that correlation nearly teach a sales organization exactly the wrong lesson. Experienced employees look slower on certain tasks, because they get the hardest cases. Customers who receive retention offers churn more, because only at-risk customers get offers. The moment recommendations change behavior, the ladder must be climbed deliberately: association, then adjusted comparison, then quasi-experiment, then randomized experiment where ethical and practical.

The fifth mess is the one software never has: the system under measurement reads its own dashboard. Make response time the KPI and people optimize response time, not resolution. Make closure the KPI and cases close prematurely. Publish the "workflow of high performers" and people mimic the visible workflow without the judgment underneath. Goodhart's law, but with feelings and org charts.

And here the stakes stop being technical. Research on process-mining adoption finds that what managers read as insight, workers read as monitoring. The OECD surveyed more than 6,000 mid-level managers across six countries: 90% of US managers reported their firms using at least one algorithmic-management tool, and nearly two-thirds of the managers using such tools reported at least one concern, most commonly unclear accountability and inability to follow the algorithm's logic.⁴ Regulators have noticed too: the UK ICO treats productivity software as worker monitoring with data-protection obligations, and the EU AI Act places employment systems that monitor performance or allocate work based on behavior into high-risk categories with worker-notification duties.⁵⁶

So the enterprise loop has two possible futures, and the terrifying part is that the technical architecture is almost identical for both: organizational learning infrastructure, or industrialized employee surveillance. Governance decides which one gets built. The design principle I will not compromise on: instrument the work before instrumenting the worker. Prefer "this procurement exception was resolved through these steps with this outcome" over "employee 174 typed 46% faster than employee 281." The first creates organizational intelligence. The second creates fear, and fear teaches the system lies.

Here is the twist I did not expect when I started pulling on this thread: while we debate whether this loop should exist, a dozen companies with completely different logos are already building its components, each convinced they are in a different market. Line their architecture diagrams up side by side, and something remarkable appears.