The question
AI tools connect model decisions to files, APIs, and sensitive operations. Metadata can describe a tool as read-only while its handler still changes state. A suspicious description is a reason to investigate. It is not evidence of an exploit.
So the question for this test was narrow: when this tool is called, does a file actually appear on disk?
What I built
Gerfach combines passive inspection with controlled active tests. Protocol adapters normalize what a server exposes, modules run probes through the adapter, and reports attach the evidence another engineer needs to check a finding.
- MCP adapterDiscover the local tool and normalize its schema.
- Module runnerRun state-oracle and invoke through the adapter.
- Filesystem oracleSnapshot, diff, and match the probe canary.
- Report pipelineAttach evidence and emit terminal and JSON output.
Other adapter families: A2A, REST, GraphQL, gRPC, OpenAI function calling, and LangChain tools (schema inspection only). JSON and SARIF reporting are implemented; this investigation publishes JSON. For this case study a small local fixture and replay were added around the scanner. The scanner itself was unchanged.
The test, step by step
The fixture is intentionally inconsistent: it claims read-only and allows writes. Press play, or pick a step.
A promise in the metadata
The local MCP fixture exposes write_file with readOnlyHint: true. Its initial handler still permits writes inside a temporary directory. That inconsistency is intentional.
tool: write_file
annotations: { readOnlyHint: true }
DEMO_ALLOW_WRITE=1A guided replay of captured artifacts from 20 September 2026. This is not a live scan or a screen recording. Playback starts only when you choose.
The canary appeared in both the filename and its content. The scanner emitted CHAMBER-FS-001, verified by its filesystem state oracle.
Inspect the filesystem deltaThe same module ran against a fresh directory with the write gate disabled. One check passed and the report contained no findings.
Inspect the rerun JSONWhat this shows, and what it doesn’t
Shows
- A write happened in the initial fixture, confirmed on disk.
- This runtime gate prevented the tested write.
- The directory allowlist stayed intact.
Does not show
- Path traversal. The raw report’s CRITICAL / CWE-22 label is broader than what was demonstrated.
- An unauthorized exploit, or general safety of any target.
- Network egress, which was not tested. No Docker services or external targets were used.
- Detection accuracy or adoption. The module and test counts describe scope only.
Three engineering decisions
One contract, different transports
Adapters normalize discovery into ServerCapabilities. Connection and invocation stay protocol-specific: the MCP adapter calls the SDK; the REST adapter maps OpenAPI operations into tools. The audit layer works against the same model without pretending every protocol has the same capabilities.
connect(options: AdapterConnectOptions): Promise<ServerCapabilities>;
invoke(name: string, args: Record<string, unknown>, timeoutMs?: number): Promise<unknown>;
disconnect(): Promise<void>;Observe the effect, keep the proof
A tool declaration suggests capability. A call demonstrates reachability. A success response is still only a claim. The state oracle compares filesystem snapshots and correlates the delta with a unique canary to establish an independently observed side effect.
{
"id": "CHAMBER-FS-001",
"toolName": "write_file",
"verifiedBy": "state-oracle",
"summary": {
"added": 1,
"modified": 0,
"deleted": 0
},
"matchedIn": [
"path",
"content"
]
}Make the failure repeatable
The repository includes Docker fixtures and local mocks. This demonstration uses stdio and temporary directories to isolate one behavior. Existing regression tests check both the emitted finding and the artifact on disk; the portfolio runner also asserts the before and after result.
expect(finding.id).toBe("CHAMBER-FS-001");
expect(finding.severity).toBe(Severity.CRITICAL);
expect(evidence.verifiedBy).toBe("state-oracle");
expect(existsSync(fsDiffFile)).toBe(true);
const artifact = JSON.parse(readFileSync(fsDiffFile, "utf-8"));
expect(artifact.version).toBe(1);
expect(artifact.delta.changes.length).toBeGreaterThan(0);
expect(JSON.stringify(artifact)).toContain("chamber-state");Raw evidence
The scanner source is private. These are the files the page is built from, unedited apart from redacted local paths.
Show the captured CLI output
--server "bun scripts/gerfach/fixture.mjs" --active
--modules state-oracle --chamber-run-dir <run>
--no-plugins --context sandboxed --probe-delay 0
--format terminal --output <report.json> --accept
MCP Audit ──────────────────────────────────────────────────────────────────────── Target: gerfach-portfolio-local-fixture v1.0.0 Protocol: 2025-11-05 (mcp / stdio) Surface: 1 tools, 0 resources, 0 prompts Duration: 181ms ──────────────────────────────────────────────────────────────────────── Risk Overview MEDIUM CRS: 0.484 UMRI: 47.76 Legacy score: 70/100 SSVC: Track* CRIT 1 HIGH 0 MED 0 LOW 0 INFO 0 Check Summary pass 0 warn 0 fail 1 error 0 skip 0 total 1
Verbatim excerpt, typeset for readability. Read the complete output ↗
Repository snapshot checked 20 September 2026. 93 test files exist; only the three relevant files were run for this case study.