Skip to content

fix(repos)!: stop mirroring file bodies in get_file_contents structured output - #3500

Open
jbayardo wants to merge 1 commit into
mainfrom
jbayardo-fix-get-file-contents-duplication
Open

jbayardo wants to merge 1 commit into
mainfrom
jbayardo-fix-get-file-contents-duplication

Conversation

@jbayardo

Copy link
Copy Markdown

Summary

On protocol 2026-07-28, get_file_contents returned every file twice: once as the embedded resource in content and again as text/blob in structuredContent. Structured output now carries only the resource uri and mimeType; the body stays in content. Modern JSON text is also encoded without HTML escaping, so it equals JSON.stringify(structuredContent).

Why

Clients that forward both content and structuredContent to the model pay for each file twice. In Copilot CLI the mirror is ~41.5% of get_file_contents tokens: ~1.07 GB/day of structured content across 81K of 241K daily calls. Without it, ~9.4K/day of the 19.7K results that spill past 20 KiB would stay inline.

Separately, json.Marshal escapes <, > and & as \u003c, \u003e and \u0026. Copilot CLI only drops the structured JSON when the text is byte-equal to JSON.stringify(structuredContent), so all-text results containing those characters (e.g. pull_request_read, search_code, web_search, search_repositories) were also sent twice.

MCP spec context: modelcontextprotocol/modelcontextprotocol#1624, modelcontextprotocol/modelcontextprotocol#2200.

Fixes #

What changed

  • RepositoryResourceOutput keeps only uri and mimeType. repositoryContentOutput no longer copies the file body, and the text/blob oneOf is removed from the output schema.
  • applyModernStructuredText encodes with SetEscapeHTML(false) and then unescapes \u003c/\u003e/\u0026 inside JSON strings. The second step is needed because the SDK hands over an already-escaped json.RawMessage. Escaped backslashes are handled, so a literal \u003c in the data stays as-is.
  • Tests:
    • The get_file_contents wire test now expects structured output equal to the legacy blocks minus the resource bodies, with no text or blob.
    • Schema cases updated.
    • New tests cover unescaped modern text and the unescaping scanner's edge cases.
  • get_file_contents.snap regenerated; docs/typed-tool-schemas.md documents both rules.

Options considered

Option Outcome
Metadata-only structured output (chosen) Keeps the typed outputSchema for discovery. The file body is no longer duplicated, and every client still receives it in content.
Drop outputSchema for get_file_contents Removes typed output for the tool entirely, which is a bigger contract change.
PreserveHandlerContent Only skips the text rewrite; structuredContent is still emitted, so it does not fix the duplication.
Omit structuredContent only for file reads Not viable: with an outputSchema, the spec requires structuredContent, and the TS SDK client throws when it is missing.

Measurements (real API, stdio binary, github/github-mcp-server@main, protocol 2026-07-28)

Bytes of content + structuredContent:

Request Before After
docs/typed-tool-schemas.md 4,251 + 4,263 4,251 + 313
README.md 123,084 + 123,096 123,084 + 296
pkg/octicons/icons/repo-light.png 852 + 864 852 + 306
pkg/github (directory) 73,351 + 69,198 unchanged
pull_request_read #3437: text equals JSON.stringify(structuredContent) ❌ (1 HTML escape) ✅

Legacy protocol (2025-11-25) results are byte-identical before and after.

MCP impact

  • No tool or API changes
  • Tool schema or behavior changed — the get_file_contents output schema and structuredContent no longer include text/blob. This is a contract change for consumers that read file bodies from structuredContent (typed outputs shipped in v2.0.0); they should read the embedded resource in content. Go library consumers lose RepositoryResourceOutput.Text and .Blob. Modern JSON text no longer HTML-escapes <, > and &; it parses to the same value.
  • New tool added

Prompts tested (tool changes only)

Exercised over stdio with a JSON-RPC harness (protocol 2026-07-28 and 2025-11-25), not through an LLM client:

  • "Show me docs/typed-tool-schemas.md in github/github-mcp-server" (text file)
  • "Get README.md from github/github-mcp-server" (large text file)
  • "Fetch pkg/octicons/icons/repo-light.png" (binary file)
  • "List pkg/github in github/github-mcp-server" (directory)
  • "Get PR Fix UTF-8 encoding in repository resource URIs #3437 in github/github-mcp-server" (pull_request_read; HTML escaping)

In every modern-protocol case, structuredContent validated against the advertised outputSchema with Ajv (the TS SDK's validator). I also confirmed in the Copilot CLI source that it still sends the embedded resource body to the model.

Security / limits

  • No security or limits impact
  • Auth / permissions considered
  • Data exposure, filtering, or token/size limits considered — this roughly halves get_file_contents result size on the modern protocol and lets clients dedupe all-text results. No data is newly exposed or filtered.

Tool renaming

  • I am renaming tools as part of this PR (e.g. a part of a consolidation effort)
    • I have added the new tool aliases in deprecated_tool_aliases.go
  • I am not renaming tools as part of this PR

Note: if you're renaming tools, you must add the tool aliases. For more information on how to do so, please refer to the official docs.

Lint & tests

  • Linted locally with ./script/lint: gofmt -s -l clean; golangci-lint run (v2.14.0, matching CI) on ./pkg/inventory/... ./pkg/github/... reported 0 issues.
  • Tested locally with ./script/test: go test ./... -count=1 passed. I ran it without -race because cgo isn't available on this Windows machine; CI runs -race. go run ./cmd/github-mcp-server generate-docs produced no changes.

Docs

  • Not needed
  • Updated (README / docs / examples) — docs/typed-tool-schemas.md now covers non-HTML-escaped modern text and metadata-only structured output for resource results. README needed no changes.

get_file_contents returned each file twice on protocol 2026-07-28: once in
the embedded resource and again as text/blob in structuredContent. Clients
that forward both to the model paid for every file twice. Structured output
now carries only the resource uri and mimeType; the body stays in content.

Modern JSON text for all-text results is also encoded without HTML escaping,
so it matches JSON.stringify(structuredContent) and clients can dedupe it.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@jbayardo jbayardo changed the title Stop mirroring file bodies in get_file_contents structured output fix(repos)!: stop mirroring file bodies in get_file_contents structured output Oct 14, 2026
@jbayardo
jbayardo marked this pull request as ready for review October 14, 2026 23:07
@jbayardo
jbayardo requested a review from a team as a code owner October 14, 2026 23:07
Copilot AI max review requested due to automatic review settings October 14, 2026 23:07

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Breaking structured-output and exported Go type changes require maintainer confirmation of downstream compatibility.

2 open findings
What changed in this PR

Reduces duplicate payloads in modern MCP tool results while preserving legacy responses.

Changes:

  • Keeps file bodies in content, with only resource metadata in structured output.
  • Removes HTML escaping from modern structured JSON text.
  • Updates tests, schema snapshots, and documentation.
File Description
pkg/​inventory/​typed_output.go Normalizes HTML escapes in modern JSON text.
pkg/​inventory/​typed_output_test.go Tests HTML encoding and escape handling.
pkg/​github/​typed_repository_outputs_test.go Verifies metadata-only structured resources.
pkg/​github/​repository_tool_types.go Removes resource bodies from exported types and schemas.
pkg/​github/​__toolsnaps__/​get_file_contents.snap Records the metadata-only output schema.
docs/​typed-tool-schemas.md Documents serialization and resource-body behavior.

🧠 Review effort: Max


💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment on lines +400 to +401
if !bytes.Contains(data, []byte(`\u00`)) {
return data
Comment on lines +79 to +80
`get_file_contents` keeps file bodies in the embedded resource and reports only
its `uri` and `mimeType` in `structuredContent`, so clients that forward both
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants