Lucio Durán
lucioduran.com

Projects / MCP Contract Harness

MCP Contract Harness

Read-only MCP server and independent clients for checking stdio negotiation, pagination, errors and cancellation, with actual wire messages and reproducible reports.

Author: · Python 3.11.9 / Node 24.8.0 / MCP 2025-06-18 · License: MIT

Page published: · Page updated:

Repository · Client report

Problem and relevance to agents

A successful initialize response does not establish that an integration handles a second page, cancellation or errors. This project checks those contracts against a specific MCP version and preserves exchanged messages.

Agents need tool discovery, complete resource retrieval and a distinction between protocol and execution errors. This implementation exposes those boundaries through client checks and transport traces.

Architecture

  1. Bounded server. A Python standard-library server implements a subset of MCP 2025-06-18 over stdio JSON-RPC. It exposes three fictional notes and two read-only tools without filesystem or network access.
  2. Three clients. A custom Python client, official Python SDK 2.3.0 and official TypeScript SDK 1.32.1 connect to the same server. SDK clients negotiate from 2025-11-25 to the supported version.
  3. Wire recording. A proxy records actual lines from SDK clients before forwarding them. The custom client records its own transport. Traces preserve direction and JSON-RPC messages.
  4. Contract validation. An additional validator checks correlation, cursor chains, unanswered cancelled requests and a subsequent answered ping. Tool execution failures use isError; protocol failures use JSON-RPC errors.

What this demonstrates

  • Implementation of a versioned MCP contract without a server-side SDK dependency.
  • Measured interoperability with three clients, explicit per-client coverage and inspectable messages.
  • Validation of negative cases, pagination and cancellation beyond successful initialization.

Published results

The published report records three clients exercising one local server. The custom client completed 13 scenarios and each SDK client completed eight. All three negotiated MCP 2025-06-18.

The recorded run uses Python 3.11.9 and Node 24.8.0 on an Intel Xeon W-3223. Every client traverses resource and tool pages, reads a note, checks results and errors, submits an invalid cursor and cancels an operation before sending another ping.

Recorded client runs

Client report ·
ClientVersionChecksMessagesProtocol
Custom reference0.1.013342025-06-18
Official Python SDK2.3.08232025-06-18
Official TypeScript SDK1.32.18232025-06-18

These are fixed correctness scenarios for a stdio subset. The 13, 8 and 8 checks do not represent identical coverage or a client ranking. The result certifies neither full MCP conformance nor compatibility with graphical hosts.

The matrix identifies checks performed only by the custom client: pre-initialization errors, unknown methods and resources, invalid JSON and unknown cancellations. Interoperability matrix and scope.

Documented negative tests alter a page, the negotiated version, a duplicated response and a response to a cancelled request. The client report is not a statistical defect-detection measurement. Interoperability matrix and scope.

Reproduce

The server and raw client use the Python standard library. SDK reproduction requires Python 3.11.9, Node 24.8.0, npm 11.6.0 and the repository lock files. No models, accounts or paid APIs are required.

The first command runs the custom client against a real subprocess. Full reproduction installs pinned dependencies, verifies saved traces and writes a new run into an unused directory. Do not use Python -O: clients rely on contract assertions.

Run the custom client

git clone https://github.com/lucioduran/mcp-contract-harness.git
cd mcp-contract-harness
git checkout 44e77c071525127cd7e9ef5008b1b9f41e02ad07
python -m clients.raw

Reproduce all three clients

python -m venv .venv
. .venv/bin/activate
python -m pip install --require-hashes -r requirements-dev.lock
npm ci --ignore-scripts
python -m clients.reproduce verify --output results/reference
python -m clients.reproduce run --output build/reproduced
python -m clients.reproduce verify --output build/reproduced

Scope and limitations

  • HTTP, OAuth, subscriptions, prompts, sampling, elicitation, progress and resource templates are unsupported. This is not MCP certification.
  • ChatGPT, Claude Desktop, IDEs and other graphical hosts were not tested. SDK compatibility does not establish compatibility with those hosts.
  • Cancellation uses a deliberate 400 ms delay and a silence window of at least 600 ms. It does not prove all scheduling races or absence of leaks under load.
  • Limits are 64 KiB per message, 32 active requests and 10,000 identifiers per session. Performance, backpressure, exhaustive fuzzing and adversarial resource exhaustion were not evaluated.
  • Tool annotations are descriptive. The actual restriction comes from reading a compiled catalog; there is no security audit or sandbox for arbitrary tools.
  • Traces and hashes support consistency inspection. They do not authenticate who performed the measurement or constitute independent review.

Sources and revision

Values on this page come from the linked report. They are not live measurements. The report records the executed revision; the reviewed revision includes the subsequent publication of that report.