Harness engineering is the layer between an AI model and a real side effect. It checks the action, runs it, and returns the result without asking the model to police itself.
What is a harness in AI agent systems?
A harness is what turns a model's proposed action into an actual action. The model may decide to send an email, query a database, or run code. The harness checks the request, limits what it can reach, executes it, and returns the result. That separation matters. A model can suggest an unsafe action even when the system prompt says not to. The execution layer still needs its own rules.
Why the harness matters
Once an agent can change something outside its own context, the interesting question is no longer whether the model can choose a tool. It is who gets the final say before the side effect happens.
There are four practical concerns: isolation, narrow permissions, risk classification, and failure handling. A container or microVM can limit the blast radius. Scoped credentials limit what a successful call can touch. A fixed risk rule makes a decision reproducible. Timeouts and rate limits need a defined outcome instead of turning into a half-finished agent run.
Matimo is opinionated about one of these in particular: risk should come from the operation being requested, not from a second model making a guess about it. The other controls still matter, but without a predictable decision path, the word “governed” gets fuzzy very quickly.
How Matimo engineers this
This post is the third in a series on the engineering surfaces that decide whether a production agent behaves: what you tell the model, what you show it, what it can touch, how it reasons, and how its steps compose. Introducing Matimo.ai calls this surface “Harness: Matimo OSS.”
Deterministic risk classification, not a guess
Ask an agent platform how it decides whether a tool call is risky, and often the answer is another model. The calling model can self-assess, or a separate judge can sit in front of the executor. Matimo takes the less fashionable route: risk is fixed by the operation itself.
| Risk level | What triggers it |
|---|---|
| Low | HTTP GET, HEAD, OPTIONS |
| Medium | HTTP POST, PUT, PATCH |
| High | DELETE, shell/command execution, or requires_approval: true |
| Critical | function execution (arbitrary code) |
A GET is low risk because it is a GET. The classification does not depend on which model proposed it or what wording appeared in the request. That gives an auditor a rule to inspect instead of a model opinion to reconstruct.
The six-step pipeline
MatimoInstance.execute() is the harness. It is a fixed pipeline rather than a collection of per-tool hooks. Every call from a supported integration, including LangChain, CrewAI, the Matimo SDK, and its Model Context Protocol (MCP) server, passes through the same six steps:
- Policy check.
canExecute()evaluates the tool's risk and the caller's context. The result isallowed,denied, orpending_approval. - HITL / quarantine resolution. For
pending_approval, Matimo checks the approval manifest, then calls the registered human-in-the-loop callback. With no callback, it rejects the call. A timeout rejects it too. - Approval scan. Matimo checks
requires_approval: trueand scans command or SQL parameters for content that should trigger approval even when the tool was not statically flagged. - Auth injection. Credentials resolve in a fixed order: per-call overrides,
MATIMO_-prefixed environment variables, then the plain variable name. An unresolved placeholder stops execution with anAUTH_FAILEDerror naming the missing variable. - Executor dispatch. The declared
execution.typeselects exactly one ofHttpExecutor,CommandExecutor, orFunctionExecutor. - Result. The result goes back to the caller, or the pipeline throws a typed
MatimoError.
The order is not negotiable per tool. A call cannot skip policy because someone marked it trusted at the last second, and it cannot skip credential injection because it came from a different integration. LangChain, Matimo Workbench, and npx matimo mcp all use the same shape of pipeline.
Nine rules that gate untrusted tools
The policy check above evaluates every call, but a separate, stricter set of nine fixed rules specifically gates tools loaded from untrusted paths, meaning anything an agent created at runtime via matimo_create_tool, not the first-party provider packages Matimo ships. These are deterministic checks against the tool's own definition, not judgment calls about intent:
- SSRF (server-side request forgery) protection, checked twice: once against the raw URL at creation time, again against the fully resolved URL immediately before the HTTP request fires, so a placeholder like
{host}can't slip an internal IP past the first check and get substituted in later. - Reserved-namespace protection, blocking an agent from naming a tool something like
matimo_*and shadowing a built-in. - A credential allowlist, blocking a tool from referencing a credential it wasn't explicitly permitted to use.
- HTTP method and domain allowlists, restricting which methods and which hosts an agent-created tool can reach at all.
- Forced-approval and forced-draft-status rules, which strip an agent's ability to self-approve or self-publish a tool it just wrote.
Any critical- or high-severity violation rejects the tool outright before it's ever registered. Scope matters here: these nine rules govern agent-created and untrusted tools specifically. They're not a blanket restriction on what a first-party provider package can do; a trusted, developer-authored tool can legitimately use function-type execution when the job genuinely needs it. Trust origin is what decides the rule, not execution type on its own.
Where we're being precise on purpose
Two things about this harness are easy to overstate, and we'd rather undersell them than have a skeptical reader catch us rounding up.
The embedded-code path in FunctionExecutor is hardening, not sandboxing. Matimo recommends colocated .ts/.py executor files for function-type tools: real code, in a real file, that a developer wrote and reviewed. The SDK also supports an older path: embedding code directly as a string in a tool's YAML. That path is disabled by default and requires an explicit MATIMO_ALLOW_EMBEDDED_CODE=true opt-in. Even when enabled, the code is checked against a static regex blocklist (require(), import(), process, eval(), new Function(), global/globalThis, and a few others) before it's ever handed to new Function() for execution. That's real input hardening, and it does meaningfully narrow what a careless or malicious embedded snippet can reach. What it isn't is a sandbox: there's no VM boundary, no container, no process isolation around that execution. A regex blocklist can be worked around by a payload nobody thought to block; a VM boundary can't be argued with the same way. We call this "governed execution with hardening" on purpose, not "sandboxed," because that word would claim an isolation guarantee this path doesn't have. If your threat model requires true isolation, put this tool type behind your own container or VM boundary. Don't rely on the blocklist as that boundary.
Retry and backoff aren't wired into the HTTP executor yet. The tool YAML schema has room for an error_handling block (retry, backoff_type, initial_delay_ms) as a documented convention for tool authors, but as of this writing, HttpExecutor doesn't read or act on it. A failed HTTP call returns a normalized MatimoError once; it doesn't retry with backoff on your behalf. If a tool you're calling needs resilience against transient failures or rate limits, that logic has to live in your own calling code today, not inside the harness. We'd rather say that plainly now than have someone discover it the hard way against a flaky third-party API.
Why this is the shape we picked
The alternative to a fixed pipeline is a flexible one: let each tool, or each team, decide its own order of operations, adding or skipping steps as needed. That's more convenient in a demo and much harder to reason about in production, because "governed" stops meaning one thing. A tool call is auditable in Matimo specifically because the same six steps ran in the same order, regardless of which tool, which framework, or which tenant triggered it. That's the actual bar for a harness layer: not "can this stop a bad action" (execution-time policy governance broadly does that, and we've written about what that layer needs to include separately), but "can you prove, after the fact, that the same rules applied every time." A judgment call from a model can't clear that bar no matter how good the model is. A fixed pipeline can.
This harness sits underneath the rest of the series too. Earlier posts covered what you tell the model, in What Is Prompt Engineering?, and what you show it, in What Is Context Engineering?. Neither one matters if the layer actually touching your systems can't be trusted to behave the same way twice. The next post in the series covers the fourth surface: how Matimo's reasoning engines decide what to try next when a call fails or a plan needs to change.
Frequently asked questions
What is a harness in AI agent systems?
A harness is the software layer between an AI model and the outside world. It takes a model's decision to call a tool, such as sending an email or querying a database, and actually carries it out: checking whether the action is allowed, running it, and returning the result. The model does the reasoning; the harness does the doing, and it's responsible for making sure the doing is safe.
Is Matimo's execution sandboxed?
Mostly hardened, not fully sandboxed. Matimo recommends real code files reviewed by a developer for tool logic, and every call runs through a fixed policy and approval pipeline. The one exception is an older, opt-in path that lets a tool embed code directly in its YAML definition: that code is checked against a static regex blocklist before it runs, which is real hardening, but there's no virtual machine or container boundary around it. That's why we call it "governed execution with hardening," not "sandboxed."
How does Matimo decide which tool calls need approval?
Risk is set by the shape of the call, not by a model's guess: HTTP GET/HEAD/OPTIONS are low risk, POST/PUT/PATCH are medium, DELETE and shell commands are high, and arbitrary code execution is critical. High-risk and flagged calls are also checked against an approval manifest and, if needed, sent to a human-in-the-loop callback. If no callback is registered, the call is rejected by default rather than allowed to proceed.
Does Matimo retry failed tool calls automatically?
Not yet. The tool YAML schema has fields for retry and backoff settings, but Matimo's HTTP executor doesn't currently read or act on them. A failed call returns a single error instead of retrying, so retry logic against a flaky API has to live in the calling code today.
What are the nine rules that gate untrusted tools?
They are a fixed set of checks Matimo runs specifically on tools an agent creates or loads from an untrusted path, not on first-party tool packages. They include double SSRF (server-side request forgery, where a request is tricked into hitting an internal address) checks, a reserved-namespace rule that blocks naming collisions with built-in tools, a credential allowlist, HTTP method and domain allowlists, and rules that stop an agent from approving or publishing its own newly created tool.
Why doesn't Matimo let a model judge tool-call risk directly?
Because that judgment can't be reproduced or fully explained after the fact: run the same call twice through a model-based judge and the two answers can differ. Matimo fixes risk classification to the call's execution shape instead, so the same action is always classified the same way, and an auditor can point to a rule rather than a model's opinion.
Get in touch
If you want to see the pipeline directly rather than take our description of it, Matimo OSS is MIT-licensed and free: npm install matimo or pip install matimo. The full platform, including the reasoning engines and governance layer built on top of this harness, is Matimo Workbench, generally available with a free tier. Questions about how the harness handles a specific case in your stack? Contact us directly.