Adding MCP problem type basic fuzzing infrastructure#1628
Open
guidorc wants to merge 33 commits into
Open
Conversation
Introduces a new problem domain for blackbox testing of MCP (Model Context Protocol) servers, following the same architecture as the existing GraphQL and RPC problem domains. New components under core/src/main/kotlin/.../problem/mcp/: - McpAction (abstract base), McpToolCallAction (tools/call), McpResourceReadAction (resources/read), McpInputParam, McpUriParam - McpIndividual: chromosome wrapping sequences of MCP actions - McpCallResult: per-action execution record storing isError flag - client/McpClient (interface), client/HttpMcpClient (JSON-RPC 2.0 over HTTP with pagination), client/McpDTOs - service/McpSampler: @PostConstruct discovery via tools/list, resources/list, resources/templates/list; gene tree building from JSON Schema; random and smart (ad-hoc) sampling - service/McpFitness (abstract base), service/McpBlackBoxFitness: per-action scoring using isError flag and successful reads as signals - service/McpBlackBoxModule: Guice bindings mirroring GraphQLBlackBoxModule Unit tests cover action construction, individual copy/mutation, and HttpMcpClient JSON-RPC parsing via WireMock stubs.
- Add ProblemType.MCP (experimental) to EMConfig enum - Add bbTargetUrl validation for MCP black-box mode - Route ProblemType.MCP to McpBlackBoxModule in Main.init() - Add getAlgorithmKeyMcp() supporting RANDOM, MIO, MOSA, WTS, SMARTS - Dispatch MCP algorithm key in Main.run() - Bind NoTestCaseWriter in McpBlackBoxModule EvoMaster can now be invoked with: --blackBox true --problemType MCP --bbTargetUrl <url>
Connection failures during @PostConstruct were surfacing as an uncaught InvocationTargetException labelled as an EvoMaster bug. Wrap the listTools() call so a failed connection to the MCP server produces a clean, user-readable SutProblemException instead.
Servers may return 400/404/405 for MCP methods they don't support (e.g., resources/list on a tools-only server). Previously this caused an IOException from HttpURLConnection.inputStream on non-2xx responses, crashing the sampler during initialization. Extracted openConnection() helper and made post() return null on 400/404/405, which callers treat as an empty/error result.
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* Setting up MCP Problem Type Skeleton * remove db related size from MCP individual * Fix for resource read action id Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * feedback * adjust resource read action parameters --------- Co-authored-by: guido-rodriguez_sfemu <guido.rodriguez@salesforce.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Setting up MCP Problem Type Skeleton
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adding the basic search infrastructure for blackbox fuzzing of Model Context Protocol (MCP) servers. The approach is: connect to the target server, discover its capabilities via the standard MCP handshake, automatically construct a mutable input representation (gene tree) for each tool and resource from their JSON Schema definitions, then use a search algorithm to evolve a sequences of calls that maximise coverage; reaching each capability and obtaining a non-error response
Components
MCP client (HttpMcpClient, McpClient, McpDTOs)
An HTTP-based client that implements the MCP JSON-RPC protocol: initialization handshake, tool invocation (tools/call), static resource reads (resources/read), and capability discovery (tools/list, resources/list, resources/templates/list).
Action model (McpAction, McpToolCallAction, McpResourceReadAction)
Two concrete action types mirror the two MCP interaction patterns: calling a named tool with structured arguments, and reading a resource by URI (including template URIs with mutable parameters).
Individual (McpIndividual)
A test case represented as an ordered sequence of McpActions, integrating with EvoMaster's existing EnterpriseIndividual and mutation infrastructure.
Sampler (McpSampler)
On startup, connects to the target MCP server, performs the handshake, and discovers all tools and resources. Each tool's JSON Schema input definition is converted into an ObjectGene tree used by the search engine to mutate arguments. Pre-built single-action individuals ensure every capability is exercised at least once before random search begins.
Fitness function (McpBlackBoxFitness)
Evaluates individuals by executing their actions sequentially and scoring coverage targets based on observable protocol signals: 1.0 for a successful tool call or resource read, 0.5 for a server-reported error (partial credit keeps the search gradient useful). Exceptions break the action sequence early.
McpBlackBoxModule
Wires all the above into EvoMaster's dependency injection container and registers the problem type in the main entry point.
Follow Up
client calls is needed.
(explicit JSON Schema format → type switch → name-based heuristics) to generate valid values for these fields, matching the approach used by the REST builder.