release: add coverage and Xcode MCP guidance #129
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,21 @@ | ||
| # Code Coverage Contract | ||
|
|
||
| Code coverage is execution evidence: it shows which instrumented code ran during a test run. It does not prove behavior is correct, tests assert the right outcomes, or UI and device paths are adequately verified. | ||
|
|
||
| ## Default Policy | ||
|
|
||
| - Report coverage by target and file, and include function-level detail when the tool provides it. | ||
| - Keep test failures, coverage-collection failures, and missing-report failures as separate results. | ||
| - Default to reporting only. Do not invent a global coverage percentage, fail a build, exclude a source path, or claim changed-line coverage without an explicit repository policy. | ||
| - Treat generated, vendored, test-target, and platform-glue sources as visible in the first report. Exclude them only with a checked-in, human-readable reason. | ||
| - Do not commit generated coverage reports by default. Keep local reports ephemeral and upload CI reports only when that repository's CI policy asks for it. | ||
|
|
||
| ## Required Evidence Packet | ||
|
|
||
| Every coverage result should state the toolchain and coverage command; configuration, scheme or package, and destination when applicable; report artifact path and format; target and file summary; exclusions and their reason; whether the result is comparable with a baseline; and test, collection, and report-extraction failures independently. | ||
|
|
||
| ## Comparison And Gates | ||
|
|
||
| Compare coverage only when the source revision, toolchain, build configuration, selected targets, destination, and exclusion policy are compatible. A raw project-wide percentage is not a substitute for changed-file or changed-line evidence. | ||
|
|
||
| Add a CI gate only after the repository chooses one concrete metric: an overall minimum, target minimum, changed-file minimum, or changed-line minimum. Record the policy and allowed exceptions in the repository; keep the Apple Dev Skills default report-only. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,31 @@ | ||
| # SwiftPM Code Coverage | ||
|
|
||
| Use SwiftPM coverage for a package-first test surface. Follow the shared [`code-coverage-contract`](../../../shared/references/code-coverage-contract.md) for evidence, artifacts, comparisons, and gates. | ||
|
|
||
| ## Collection And Report Discovery | ||
|
|
||
| Run coverage as part of the test command: | ||
|
|
||
| ```bash | ||
| swift test --enable-code-coverage | ||
| ``` | ||
|
|
||
| After a successful collection, ask the active toolchain for the exported JSON location instead of hard-coding a `.build` path: | ||
|
|
||
| ```bash | ||
| swift test --show-codecov-path | ||
| ``` | ||
|
|
||
| Treat `--show-codecov-path` as a separate report-location query. Do not assume every SwiftPM release accepts it combined with `--enable-code-coverage`, and do not split coverage into a separate build plus `swift test --skip-build` path unless that exact toolchain has been verified. | ||
|
|
||
| ## Reporting | ||
|
|
||
| Read the exported JSON from the reported path and summarize the targets and files it contains. Preserve the original artifact path in the evidence packet. If the file is missing, unreadable, or does not contain the expected coverage payload, report extraction failed; do not substitute a guessed `.build` location or a percentage inferred from test output. | ||
|
|
||
| SwiftPM coverage is appropriate for package targets and their ordinary tests. It does not replace app-host, XCUITest, simulator, physical-device, or Xcode test-plan coverage. Hand off to `xcode-testing-workflow` when those surfaces matter. | ||
|
|
||
| ## Failure Boundaries | ||
|
|
||
| - A passing `swift test` without an exported coverage JSON is a coverage-collection failure, not a zero-coverage result. | ||
| - Keep package coverage reports out of version control unless the package has an explicit artifact policy. | ||
| - Do not create a coverage gate without a repository-owned metric and exclusion policy. | ||
|
Comment on lines
+27
to
+31
There was a problem hiding this comment. 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win 🧩 Analysis chain🏁 Script executed: #!/bin/bash
set -euo pipefail
# Inspect the target file with line numbers.
sed -n '1,120p' plugins/apple-dev-skills/skills/swift-package-testing-workflow/references/code-coverage.md | cat -n
# Find related coverage wording in the skill.
rg -n "coverage|swift test|show-codecov-path|enable-code-coverage" plugins/apple-dev-skills/skills/swift-package-testing-workflow -SRepository: gaelic-ghost/socket Length of output: 11478 🏁 Script executed: #!/bin/bash
set -euo pipefail
# Pull the relevant SwiftPM docs text from the web for the exact flag semantics.
python3 - <<'PY'
import urllib.request
url = "https://docs.swift.org/swiftpm/documentation/packagemanagerdocs/swifttest/"
with urllib.request.urlopen(url, timeout=20) as r:
html = r.read().decode("utf-8", "replace")
for needle in ["--enable-code-coverage", "--show-codecov-path", "code coverage"]:
idx = html.find(needle)
print(f"\n=== {needle} @ {idx} ===")
if idx != -1:
start = max(0, idx - 500)
end = min(len(html), idx + 1200)
print(html[start:end])
PYRepository: gaelic-ghost/socket Length of output: 2751 🏁 Script executed: #!/bin/bash
set -euo pipefail
# Check whether the specific file exists in this repo snapshot and show its exact contents if accessible.
test -f plugins/apple-dev-skills/skills/swift-package-testing-workflow/references/code-coverage.md && \
sed -n '1,120p' plugins/apple-dev-skills/skills/swift-package-testing-workflow/references/code-coverage.md | cat -nRepository: gaelic-ghost/socket Length of output: 2139 Scope the missing-report rule to coverage-enabled runs. The failure-boundary bullet reads as if any passing 🤖 Prompt for AI Agents |
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
| @@ -1,13 +1,13 @@ | ||||||
| --- | ||||||
| name: xcode-testing-workflow | ||||||
| description: Guide Swift Testing, XCTest, XCUITest, XCUIAutomation-oriented mechanics, code coverage, xctestplan matrices, accessibility-verification follow-through, test filtering, retries, diagnostics, and test-specific fallback work in existing Xcode-managed projects and workspaces. Use when Xcode-aware execution is needed and the task is primarily about testing rather than build or run work. | ||||||
| --- | ||||||
|
|
||||||
| # Xcode Testing Workflow | ||||||
|
|
||||||
| ## Purpose | ||||||
|
|
||||||
| Use this skill as the primary execution workflow for test-focused work in or around Xcode-managed projects and workspaces. Keep it focused on Swift Testing, XCTest, XCUITest, XCUIAutomation-oriented mechanics, code coverage, `.xctestplan`, destinations, launch arguments, interruption handling, attachments, accessibility-verification follow-through, Instruments profiling, `xctrace` trace capture, filters, retries, diagnostics, and test-specific Debug/Release validation instead of broad build/run or toolchain work. `scripts/run_workflow.py` is the runtime entrypoint for MCP-first test execution, official CLI fallback planning, and the remaining `.pbxproj` warning boundary when mutation enters project-file territory. | ||||||
|
There was a problem hiding this comment. 🎯 Functional Correctness | 🟠 Major | ⚡ Quick win Describe The document states elsewhere that the script applies configuration, safeguards, and fallback planning while MCP execution remains agent-side. Calling it the entrypoint for “MCP-first test execution” is contradictory and may send users to a script that cannot perform the promised MCP execution. Proposed wording-`scripts/run_workflow.py` is the runtime entrypoint for MCP-first test execution, official CLI fallback planning, and the remaining `.pbxproj` warning boundary when mutation enters project-file territory.
+`scripts/run_workflow.py` is the runtime entrypoint for runtime configuration, official CLI fallback planning, and the remaining `.pbxproj` warning boundary when mutation enters project-file territory; MCP execution remains agent-side.📝 Committable suggestion
Suggested change
🤖 Prompt for AI AgentsThere was a problem hiding this comment.
For Xcode coverage requests such as Useful? React with 👍 / 👎. |
||||||
|
|
||||||
| ## When To Use | ||||||
|
|
||||||
|
|
@@ -41,6 +41,7 @@ Use this skill as the primary execution workflow for test-focused work in or aro | |||||
| - preserve its project-appropriate logging, telemetry, and SwiftUI architecture guidance | ||||||
| 4. Run `scripts/run_workflow.py` to apply runtime configuration, `.pbxproj` warning safeguards, and CLI fallback planning. | ||||||
| 5. Use the focused references for the right testing surface: | ||||||
| - `references/code-coverage.md` for Xcode coverage collection, `.xcresult` artifacts, `xccov` reporting, comparison, and the Xcode 27 MCP boundary | ||||||
| - `references/xctestplan-configurations-and-matrix.md` for `.xctestplan`, launch-argument matrices, named configurations, and Debug/Release test coverage | ||||||
| - `references/xcuitest-and-xcuiautomation.md` for UI automation mechanics, waits, interruption handling, activities, and attachments | ||||||
| - `references/ui-accessibility-verification.md` for accessibility-specific runtime verification expectations and coordination with `apple-ui-accessibility-workflow` | ||||||
|
|
@@ -119,6 +120,7 @@ Use this skill as the primary execution workflow for test-focused work in or aro | |||||
| - `references/workflow-policy.md` | ||||||
| - `references/mcp-tool-matrix.md` | ||||||
| - `references/cli-fallback-matrix.md` | ||||||
| - `references/code-coverage.md` | ||||||
| - `references/xctestplan-configurations-and-matrix.md` | ||||||
| - `references/xcuitest-and-xcuiautomation.md` | ||||||
| - `references/ui-accessibility-verification.md` | ||||||
|
|
||||||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -7,6 +7,8 @@ | |
| - `xcodebuild -workspace <ws.xcworkspace> -scheme <scheme> build` | ||
| - Xcode tests: | ||
| - `xcodebuild test -project <proj.xcodeproj> -scheme <scheme> -destination '<dest>'` | ||
| - Coverage: `xcodebuild test -project <proj.xcodeproj> -scheme <scheme> -destination '<dest>' -enableCodeCoverage YES -resultBundlePath <artifacts>/<run>.xcresult` | ||
| - Coverage report: `xcrun xccov view --report --json <artifacts>/<run>.xcresult` | ||
|
|
||
| ## Runtime and tooling | ||
|
|
||
|
|
@@ -30,6 +32,8 @@ | |
| - `swift build` | ||
| - Test: | ||
| - `swift test` | ||
| - Coverage: `swift test --enable-code-coverage` | ||
| - Coverage report location: `swift test --show-codecov-path` | ||
| - Run executable target: | ||
| - `swift run <target>` | ||
| - Package commands: | ||
|
|
||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For coverage-only requests such as
collect coverage, the skill now advertises code coverage here, butscripts/run_workflow.pystill only infers test work from words liketest/xctestandbuild_commands()still emits plainswift testrather thanswift test --enable-code-coverageplus the report-location query. That makes the documented coverage path disagree with the command-planning entrypoint: agents either block when the user says “coverage” without “test”, or run tests without collecting coverage when the request is inferred as testing.Useful? React with 👍 / 👎.