Blog / agentic security

The Substrate Problem: What AI Agents Actually Need to Investigate

Agents got better at using tools when interfaces started declaring what they meant. The same principle applies one layer down, to the data an agent reasons over, and security data is where it applies hardest.

Published

8 min

The observation everyone building agents has already made

Point an agent at an arbitrary REST API and it has to do archaeology. What endpoints exist? What does status=3 mean? The agent reads whatever documentation you pasted into context, guesses, calls, gets a 400, adjusts. It reverse-engineers an interface someone already understood and never wrote down in machine-readable form.

Point the same agent at an MCP server and the archaeology stops. Discovery becomes a protocol operation: the client sends tools/list and gets back what exists. Each tool carries a name, a description, an inputSchema in JSON Schema, and optionally an outputSchema — and where one is declared, servers MUST return conforming structured results and clients SHOULD validate them. The spec names the purpose plainly: strict validation, type information, guiding models to parse returned data properly.

None of that makes the model smarter. It removes a category of work. The agent stopped having to infer the interface.

The honest version of that observation

The naive reading is wrong, and the people who designed the protocol published the evidence themselves. Anthropic's work on code execution with MCP shows agents connected to many servers burning hundreds of thousands of tokens on tool definitions before reading the request; routing calls through code cut one workload from roughly 150,000 tokens to 2,000. IBM Research found that paraphrasing a request while preserving its meaning dropped function-calling performance 8–19% across nine models, and that growing the toolkit from about 2.7 to 5.6 similar tools cost a further 1–19%.

So the claim is narrower than the enthusiasm suggests: declared structure moves work from inference to specification, and it pays only when that structure is well-chosen and bounded. Badly chosen, it makes agents worse.

One finding transfers directly to what follows: resolving opaque alphanumeric identifiers into semantically meaningful language "significantly improves Claude's precision in retrieval tasks." An opaque ID forces inference; a name that means something does not. The win was never the protocol. It was semantics.

The same problem, one layer down

Apply it to data rather than interfaces. An agent investigating a security incident is generally handed one of two substrates, and they fail in opposite directions.

Raw telemetry has all the detail and none of the semantics. Process, file, socket and authentication events arrive as a flat sequence in which the relationships that matter — this process descends from that one, this write landed on a path that survives container teardown — are recoverable only by inference. The agent reconstructs a structure the collector already knew and discarded on the way out.

That reconstruction is not free. Chroma's context-rot study, across 18 models from four families, found that models "do not use their context uniformly; instead, their performance grows increasingly unreliable as input length grows," even on trivial tasks. A single plausible-but-irrelevant distractor measurably hurt performance, and on LongMemEval, focused prompts of roughly 300 tokens beat full versions of about 113,000. Telemetry for one incident is, structurally, a distractor-dense haystack.

The processed alert fails the other way: semantics without provenance. Severity, a technique label, a generated narrative. The reasoning has collapsed into a conclusion the agent cannot inspect or dispute — which matters, because we have evidence about what a model's account of its own reasoning is worth. Anthropic found Claude 3.7 Sonnet mentioned hints it had actually used only 25% of the time on average, DeepSeek R1 39%; on sensitive framings, 41% and 19%. Models trained to exploit reward hacks admitted it under 2% of the time, fabricating justifications instead.

Both substrates lose the same thing from opposite ends. Raw telemetry keeps the evidence and drops the meaning. A generated summary keeps the meaning and drops the evidence.

One incident, three substrates

A build runner, Linux, mid-afternoon. A dependency pulled during a routine pipeline run executes a post-install script, which spawns a shell, appends a key to authorized_keys under the service account's home directory, and opens an outbound connection. The build finishes green.

Substrate one: raw events. Several thousand records, most of them the pipeline being a pipeline. Somewhere inside is an execve whose parent is the package manager, a write whose file descriptor the agent must resolve to a path itself, and a socket it must tie to a process by correlating identifiers across event types. Frontier models are reasonably capable at this. But the agent spends its attention budget on reassembly, in the conditions where attention degrades, and reassembly is where a wrong guess becomes a confident narrative.

Substrate two: the alert. "Suspicious credential file modification on build-runner-07. High confidence. A process modified SSH authorized_keys shortly after an anomalous outbound connection." The agent holds a story it cannot audit. Which process? Did "shortly after" mean two seconds or ninety? Is that ordering real or an artifact of the summary? What made the connection anomalous? Its options are to accept the conclusion or fall back to substrate one, in which case the alert bought a starting point and nothing else.

Substrate three: a behavioral representation. The same activity, semantics declared rather than implied:

The difference is not subtle. "What persisted?" is a field rather than a derivation. But the last line matters most: the agent can assert that a key persisted, state separately that it cannot characterize the destination, and know which of those it is entitled to say.

What the evidence actually supports

The closest published test is AuditBench (Anand et al., June 2026): 51 attack-investigation scenarios across Linux and Windows, five frontier models, comparing raw logs from native OS collectors against a compact edge representation derived from provenance graphs. The result is real and it is modest. The structured representation achieved equal or better performance for several models at lower computational cost. Not transformative. Equal or better, cheaper.

The same paper is candid about what structure did not fix: no model dominated across tasks, and models produced overly suspicious verdicts, with false-positive rates reaching 80% on some lateral-movement tasks. Adjacent work agrees: knowledge-graph grounding reported at least 26.5% improvement over chain-of-thought baselines while producing interpretable traces, and a 2026 threat-intelligence evaluation found hybrid graph-text retrieval improved multi-hop answer quality up to 35% over vector RAG.

Read together, this is not proof that structure makes agents smarter. It is evidence that structure changes cost and traceability, and sometimes accuracy. That is the whole claim.

The part that matters most: representing what we did not see

Start with why models guess. Kalai et al. argue that language models hallucinate because training and evaluation reward guessing over acknowledging uncertainty: models are optimized to be good test-takers, and on most benchmarks a confident wrong answer scores better than an abstention.

Now consider what that means for security data. In most telemetry, the absence of a field and the absence of the behavior look identical. "No persistence was established" and "persistence was not observed" arrive in the same shape: nothing. A model pushed toward confident completion resolves that ambiguity in the assertive direction, and gives no sign it made a choice at all.

Abstention does work when uncertainty is available. Meta and TUM researchers found that in-dialogue uncertainty caught roughly half of unanswerable questions while rejecting only 10% of valid ones, and that rejecting the 5% most uncertain TriviaQA samples raised accuracy from 84.4% to 86.0%. Availability is the condition, and structure alone does not supply it: the threat-intelligence evaluation above found graph-only pipelines abstained worse on unanswerable questions. Explicit relations improved reasoning over what was present and did nothing for what was absent.

Which is the argument. Absence has to be represented, not inferred. A behavior that states "no endpoint telemetry between 10:42:31 and 10:44:02" hands the agent a boundary it can respect. An agent cannot reason about a gap it cannot see. Our hypothesis is that this measurably reduces confabulation about unobserved activity. [hallucination-rate comparison — to be measured]

Where this argument is weak

Normalization is expensive and brittle. Berkeley researchers arguing for parser-free querying of security logs note that one major platform maintains roughly 1,000 active parsers and ships more than 200 updates monthly, which they estimate at over 4,000 engineer-hours a year. Their on-demand extraction reports a 3.6× lower error rate than manual analyst scripting on complex queries, and 0.981 F₁ against structured parsing's 0.999 on simple ones. Their sharper point: schemas impose an expressivity ceiling, since you can only ask what the schema anticipated.

We concede the economics. But two things get called "structuring." Mapping vendor log formats onto a common event schema is parsing, and that critique lands. Deriving actor, capability, action, effect, persistence, relationship and observability is a semantic layer above parsing, answering a different question: extraction finds records matching a description, but it does not produce a reviewable account of what happened and what remained. Complementary, not rival.

Frontier models keep getting better at raw text. True, and the accuracy delta is narrowing; AuditBench's structured representation was equal for several models, not dominant. One study across 9,649 experiments and 11 models found serialization format did not significantly affect aggregate accuracy (χ²=2.45, p=0.484) while model capability produced a 21-point gap. (Single-author preprint; weight accordingly.) That measured syntax — YAML versus JSON versus Markdown — which is not our claim; ours is whether a field's meaning is declared or inferred. Even so, the honest response is to stop arguing about accuracy. The durable benefits do not depend on model capability at all: cost, traceability, and the ability to reconstruct why a conclusion was reached. A model that gets the right answer from raw logs still cannot hand you a reasoning path anyone can verify.

Every schema is a commitment, and novel techniques are where it is most likely wrong. This one we concede without a counterattack. A typed field asserts shape, not truth. Whatever a representation does not model is invisible downstream, and one that infers an actor or effect wrongly does not merely lose signal, it manufactures a confident distractor — and in security, part of the input is written by the adversary. Two mitigations, neither a refutation: behavioral assertions retain links to originating telemetry, so the abstraction is a lens rather than a replacement; and observability is the schema admitting its own limits, distinguishing "the schema says no" from "the schema could not see."

Long context removes the need for compression. Partly fair; much 2023-era advice is obsolete. But the binding constraint is signal density rather than window size, degradation shows up well before advertised limits, and enterprise telemetry exceeds any foreseeable window anyway. Cost scales with tokens whether or not accuracy holds.

The experiment we intend to run

No published work tests this representation specifically as agent context. The chain above is analogical, and we would rather it were not. The experiment is cheap: same model, same incidents, three input conditions — raw telemetry, alert summary, structured behavior — scored on root-cause accuracy, confabulation about unobserved activity, and context consumed. We intend to run it. We have not, and we will publish the numbers whichever way they fall. [experiment results — to be added when benchmarked]

Why we put the behavior layer where we did

As agents take on more investigative work, the question stops being "can the model figure it out" and becomes "can we check what it concluded, and why." Only the second gets harder as agent responsibility grows.

A behavior layer between telemetry and detection is a bet on the second question. It keeps what an investigator actually asks — who acted, what it could do, what it did, what changed, what remained, and how much of that we could see — inside the representation the decision comes from, rather than generating it afterward from a model asked to narrate itself. Telemetry, behavior, detection, investigation, response. Security signals become behavior, behavior becomes understanding, understanding becomes a decision someone can defend.

What we are not claiming

Understand the behavior behind the threat.