Release Watchdog¶

How Pulp detects — and now repairs — release failures.

The important change: release health is owned by one reconciler that fixes things, not by a fleet of watchdogs that only report. Four reporting watchdogs (release-guard, release-health, release-cli-watchdog, release-draft-stuck-check) were retired after filing 413 issues in two weeks while fixing nothing — recovery was always a human running gh workflow run by hand.

They failed in two distinct ways, both worth remembering before adding another reporter:

Their grace windows were shorter than the pipeline. They alarmed after 15/30/45/60 minutes on a pipeline that really takes 70–165+ minutes, so they fired on healthy releases that were still building. Roughly half of all issues filed self-resolved.
Dedupe silently broke, so one condition minted issues forever. release-health.yml created issues with a --label release-health that does not exist in this repo; the labelled create failed, an unlabelled fallback fired, and its dedupe/auto-close lookups (which filter by that label) could never match their own issues. It opened a fresh "🚨 Release pipeline DOWN" issue every two hours, forever — 69 of them, none closable. Its title also embedded an escalating count, so they were not even title-identical. (It had a second latent bug: gh api --paginate without --slurp emits one JSON document per page, which is not valid JSON — once the repo passed 100 releases, parsing failed and every tag looked unreleased.)

If you add a watchdog, it must (a) know whether the thing it is judging is still running, and (b) be able to find its own previous issue. Otherwise it becomes a firehose.

The layers¶

Layer	Trigger	Failure mode caught	Median detection
0. release-path PR gate	PR touching release-path files	prebuilt-Skia / link-order / CMake breakage at PR time (#1962)	5-15 min (pre-merge)
0. Windows CLI compile + pre-tag check	PR or main push that can reach the MSVC CLI build	an MSVC-only error under `#ifdef _WIN32` that every Clang/GCC gate accepts	pre-merge (advisory); pre-tag (blocking)
0. Agent-capability pre-tag check	SDK tag about to be cut	a release whose agent-capability counter is still pending (`agent_capability_manifest.py --require-assigned`)	pre-tag (blocking)
1. Workflow lint	PR review	bad YAML / bad `uses:` / bad shell	seconds (pre-merge)
2. Auto-release watchdog	`workflow_run` completion	auto-release.yml runtime failure (any cause)	1-2 minutes
3. Cadence check	`schedule` every 30 min	version bumped on main but no tag created	≤45 min
4. Release reconciler	`schedule` every 30 min	tag exists but never published — and REPAIRS it; reports a compile error at once and any tag unpublished after 2 h, naming the failing leg	≤30 min
5. Release content gate	each native release leg + release finalizer	correctly named archives containing missing, stale, wrong-version, non-executable, or invalidly signed products	pre-publish
6. Deletion tripwire	`release: [deleted]`	a human/agent deleted a release or draft — burned tag name, destroyed assets	~1 minute

Layers 0-3 are prevention and detection. Layer 4 is the only one that changes release state, and it can only ever drive a release forward.

Layer 6 — Release-deletion tripwire¶

File: .github/workflows/release-deleted-tripwire.yml

Deleting a release is close to irreversible in two non-obvious ways: deleting a release that was ever published permanently burns its tag name (GitHub reserves an immutable release's tag_name forever, so every later publish attempt for that tag fails with HTTP 422: tag_name was used by an immutable release — the only recovery is a new patch tag), and deleting a draft destroys its attached assets (the binaries still survive as the building run's workflow artifacts for ~90 days — gh run download <run-id> recovers them, so a failed publish never destroys the build output by itself).

No Pulp automation deletes releases — the reconciler is re-dispatch-only and its NeverDestructive test asserts there is no delete path — so every deletion is a human or agent-driven gh call. The tripwire files (or updates, deduped by exact title) a tracking issue within a minute of the deletion, naming the tag, the actor, and the correct recovery, while the actor is still at the keyboard. It only reports; it never recreates or retags. The publish finalizer in release-cli.yml classifies the matching downstream symptom: an immutable-tag 422 at the --draft=false flip is reported as permanently unrecoverable for that tag, with the new-patch-tag recovery named, instead of surfacing as a bare HTTP error after a full multi-platform build.

Layer 5 — Published-product content gate¶

Files: tools/scripts/release_artifact_contents.py and the versioned tools/scripts/release_product_matrix.json, wired into .github/workflows/release-cli.yml.

An outer asset name and checksum do not prove the archive contains the product users expect. Each release platform therefore validates its CLI and SDK archives against one checked-in product matrix before upload. The final release job runs the same matrix against the assets it downloads back from the GitHub release draft immediately before publication.

The contract is deliberately exact where stale build output is dangerous:

CLI archives contain only pulp, pulp-cpp, pulp-import-design, pulp-mcp, the import-design browser-capture runtime, and the platform WebGPU runtime (with .exe/DLL names on Windows).
Every SDK carries the complete public pulp-* library target set plus the VST3, CLAP, and LV2 development surfaces; Darwin also requires Audio Unit.
A vanished target is missing, while a target left over from an old staging directory is rejected as stale/unexpected.
version.txt, sdk_build_type.txt, safe archive paths, and Unix executable modes are verified from the archive bytes.
Darwin re-signs installed Mach-O files after CMake's install-time RPATH rewrite and runs codesign --verify --strict on every shipped executable and dylib.

The negative controls in test_release_artifact_contents.py remove the format library, inject a retired target, add an unexpected CLI payload, substitute the wrong version, and force code-signature verification to fail. The watchdog is only considered wired while all of those broken fixtures are observed failing. The content contract begins above the current v0.759.0 release (v0.759.1 is the SemVer floor); older backfills predate this matrix and remain governed by their historical outer-asset contract. A backfill uses the tag's own matrix when present, so future product-matrix evolution is evaluated against the contract versioned with the source being rebuilt. A manual dispatch with source_ref resolves the matrix from that source ref rather than from the version label. Matrices before the declarative CLI fields retain their historical contract: v0.764.0 introduced the import-design payload, while earlier matrix-governed releases use the original three-binary CLI contract.

Required gate liveness (post-merge coverage)¶

Files: .github/workflows/required-gate-liveness.yml and tools/scripts/required_gate_liveness.py.

Ruleset drift checks prove which contexts GitHub is configured to require. They do not prove a workflow or job still matches any commit. On every push to main (with a twice-hourly scheduled backstop), the liveness audit loads required contexts from .github/rulesets/main-protection.json and queries GitHub's check-runs for the exact merged SHA. Every required context must have a completed successful check-run on that SHA. A path filter that silently stops matching, a renamed job, a pending check, or a failed check makes the workflow fail and opens or updates one stable incident; the incident closes on recovery. A skipped duplicate from a separate cache-warming push does not shadow a real executed gate on the same SHA, but an only-skipped context still fails closed.

test_required_gate_liveness.py supplies the negative controls: an absent path-filtered gate, a check attached only to another SHA, pending and failed checks, an only-skipped context, and an empty required-check contract must all fail. A regression control also proves that a later skipped duplicate cannot erase an executed success.

Layer 4 — Release reconciler (detection AND repair)¶

.github/workflows/release-reconcile.yml + tools/scripts/release_reconcile.py.

Every 30 minutes it compares desired state (every recent SDK tag has a published release) against actual state, and drives the difference to zero by re-dispatching release-cli.yml for any tag that is stuck.

The decision rules — decide() in release_reconcile.py, unit-tested in test_release_reconcile.py:

Tag state	Action
Published, with every required asset	nothing
Published but MISSING assets (at/above the asset-contract floor)	incident — a published release is immutable, so it needs a new patch tag
A `release-cli` run is queued/in-progress	nothing, at ANY age
Younger than the grace window, no run yet	nothing
Unpublished, but a NEWER version already shipped	nothing — see "superseded" below
No release, no live run, not superseded	re-dispatch `release-cli.yml` (up to 3 attempts)
A draft was left behind	re-dispatch — the re-run re-drives the draft to published
Retry budget exhausted	ONE incident issue per stall episode, updated in place, closes on recovery
A failed leg whose log carries a compiler error against a source file	incident at once, no re-dispatch — see "Compile errors are tag-immutable"
`Package CLI` failed with the same first error on two runs of one tag	incident at once, no further automatic re-dispatch — see "A repeated package failure opens the circuit"
Unpublished `DROUGHT_HOURS` (2) after tagging, with a failed leg or no run at all	listed in the incident, naming the failing legs; still re-dispatched if on the retry path

Every entry in the incident names the failing legs of the tag's newest completed run (<job> › <first failed step>, e.g. CLI windows-x64 › Build), so the issue says which platform broke without anyone opening the run.

Compile errors are tag-immutable. The sources are frozen at the tag, so a compiler diagnostic against one fails identically on every re-run. TAG_IMMUTABLE_COMPILE_ERRORS matches MSVC file(line,col): error C2xxx–C9xxx: and Clang/GCC file:line:col: error: and returns the first diagnostic as the signature. It deliberately does not match C1xxx fatal errors (out of heap space, cannot open include file), fatal error:, internal compiler errors, or linker errors, all of which a resource limit or an interrupted fetch can cause; those stay on the retry path. Before this rule, three tags that failed their Windows legs on one MSVC C2668 were each re-dispatched on every sweep — a full release matrix per attempt — while no incident opened.

A repeated package failure opens the circuit. A Package CLI step that fails with the same first error on two runs of one tag keeps failing until something changes. The build inputs are frozen in the tag, and a dispatched run's packager is main's copy (the backfill compatibility overlay), which a sweep cannot tell has been fixed. So the circuit opens instead of re-dispatching hourly. The incident names the error, someone dispatches once after the fix lands on main, and a passing run clears it. One failure stays on the retry path, since a runner can die mid-package. v0.963.0 to v0.965.0 each failed Package CLI (Unix) with "wgpu native library not found" and were re-dispatched hourly, seven ~2-hour matrices, before this rule.

Job logs are fetched with gh api --allow-escape-sequences. Logs carry ANSI colour codes, and the runner's gh refuses to print such a response without the flag. Without it every log read as empty: no failure could be classified, no tag-immutable circuit could open, and every incident entry read "no error line". A fetch failure is printed in the reconciler's run output, never swallowed. The first error is taken from the failing step (##[group]Run … up to ##[error]Process completed) before the whole log, because earlier steps can print error-shaped lines and still succeed.

One issue per stall episode. A stall opens a new issue titled Release stalled: <oldest unresolved tag> unpublished (<date>). Later sweeps edit it in place and comment once per tag that joins. Recovery closes it, and a closed episode is never reopened, so the next stall reads as news. The long-lived incident this replaced (#6880) had been reopened so often since July that its reopen for v0.963.0 went unnoticed.

The drought rule does not alarm on a slow build. An age threshold alone is exactly what made the retired watchdogs fire on healthy releases. A tag counts only when its newest completed run has a failed leg, or when it has no live run at all; a first build that is merely taking 150 minutes is left alone. A tag that is being re-dispatched after a failure keeps counting, so the incident does not close and reopen around every retry.

"Superseded" is not the old reaper. The reaper cancelled in-flight runs and deleted drafts for any tag older than the latest published release — destructive, and racing releases that were merely slow. This rule only declines to start a speculative rebuild of a tag whose users are already served by a newer release. It never touches release state, and a live run always outranks it. Without it, the reconciler's first sweep would have re-dispatched twelve superseded tags at once, saturating the very runners the current release depends on.

The asset-contract floor (ASSET_CONTRACT_FLOOR) exists for the same reason. Releases from before the Intel darwin-x64 pair and SHA256SUMS legitimately lack them; holding them to today's contract would flag a pile of perfectly good historical releases — a brand-new false-alarm firehose. Raise the floor when the required asset set changes; never lower it.

Two properties are load-bearing:

A live run outranks everything, at any age. A release that has been building for four hours is slow, not stuck. The predecessor of this workflow — a "supersede reaper" in auto-release.yml — kept concluding otherwise and cancelling in-flight releases, which is how 11 of 18 tags in July 2026 were destroyed with all their binaries built green.

It cannot destroy anything. There is no cancel path and no delete path, and a test (NeverDestructive) asserts the source contains neither. Recovery by destroying release state is the bug this workflow exists to undo.

Re-dispatch runs release-cli.yml with --ref <tag> and version=<tag>. The ref must be the tag: release-cli's "Dispatch must run on the tag it publishes" step refuses any dispatch whose commit is not the tag commit, because the build provenance would otherwise name a commit consumers do not pin. A dispatch from main fails that step in seconds without building anything. If a tag's own workflow is broken, the fix is a new tag cut from the fixed main. Re-dispatch is idempotent: the finalizer no-ops on an already-published tag.

Every incident entry says what to change¶

An incident line that only names a failing leg sends every reader into the run log to rediscover the cause. tools/scripts/release_failure_diagnosis.py reads the failing job's log once and the reconciler appends, under each tag:

the failing workflow run (linked), job and step;
the first error line — the first specific error, never the generic Process completed with exit code 1;
a cause class: pinned-asset digest mismatch, pinned-revision mismatch, compile error, re-dispatch from the wrong ref, or unclassified;
the fix and how to verify it.

For a pinned-asset digest mismatch (ERROR: sha256 mismatch from fetch_skia_for_release.py) the fix names the pin file and line (tools/deps/manifest.json, the release_assets key), the pinned and observed digests, and the provenance check to run before repinning: the upstream release asset's live digest from the GitHub API, the asset-replacement entry in that release's notes, and for Windows archives the msvc-toolchain.json inside the zip, which must carry real toolset/SDK/compiler values (no unknown, no error). Verification is the next SDK tag's Release CLI run passing the same step and publishing.

The reconciler also checks the upstream digest itself. When the upstream asset now carries exactly the digest the failed run observed, the mismatch is a fact about the tag's frozen pin, so the circuit opens at once instead of spending re-dispatches; when it does not (a truncated or corrupt download is possible), the tag stays on the retry path and the entry says the mismatch is unconfirmed.

The root cause is taken from the newest run that failed for a reason other than the dispatch-ref guard. During the v0.952.0–v0.955.1 outage every re-dispatch failed only that guard, the incident listed the guard as the failing leg five times, and the actual sha256 mismatch was never named.

Sign and Release failures are reported too. That workflow runs on every tag push independently of release-cli.yml, so its failure never blocks a publish and is invisible to every published/unpublished check. The reconciler reads each tag's newest Sign and Release run and lists a failed one with the same diagnosis.

Accepted archives with broken provenance. The "do not repin to an archive whose msvc-toolchain.json has unknown/error values" step holds for every future repin, except archives listed in PROVENANCE_EXCEPTIONS in release_failure_diagnosis.py, keyed by repo, release tag, asset and digest. The one entry is the chrome/m153 win-arm64 archive pinned by Pulp PR 10170 (sha256 18e57481…a9c7): its provenance file is known broken, its contents were verified by archive comparison, and it is Skia 37c5ad6a rather than the release lock's 8b8c3872. The same is true of the old pin: both arm64 uploads built 37c5ad6a, per built_skia in the resolve-skia logs of skia-builder runs 37596862943 and 38178800952 (provenance finding: https://github.com/danielraffel/skia-builder/pull/32#issuecomment-6135800920). For that archive the incident names the exception instead of the rule, so it never tells an agent to undo the repin.

The incident body is bounded. GitHub rejects an issue body over 65,536 characters, and a rejected edit would fail every sweep. incident_body() keeps the newest five tags' diagnoses in full (each capped at 8,000 characters), collapses older tags to one line, and folds the oldest one-liners into a single line that still names every tag if needed, staying under 60,000 characters.

A tag joining the incident posts a comment. The incident is one issue, edited in place, and an edit notifies nobody — which is how five broken releases sat in an already-open incident. Each sweep that adds a tag to an open incident comments with the newly unresolved tags.

Replaying a past incident. release_reconcile.py --tag vX.Y.Z [--tag …] evaluates only those tags, never re-dispatches or edits the issue, and prints the incident body it would write. The workflow exposes this as the tags dispatch input. test_release_failure_diagnosis.py replays the recorded v0.955.1 outage (Release CLI run 38214661512) as the positive case and the published v0.951.0 as the negative control.

Layer 1 — Workflow lint (pre-merge)¶

File: .github/workflows/workflow-lint.yml

Runs on any PR that touches workflows/actions or a release-policy input covered by its regression suite. In particular, changes to any CMakeLists.txt or tools/cmake/** run the SDK archive/matrix parity test, so a new installed library—or a target changing between interface-only and archive-bearing—cannot survive until the multi-platform release build before being rejected as unexpected. Executes three checks:

yamllint against relaxed profile. Catches syntactic errors and flags most structural issues.
Structural yaml.safe_load on every workflow file. Dumb-but-decisive: catches the #501 class specifically (block-scalar terminated by a less-indented content line).
actionlint via the raven-actions/actionlint reusable action. Catches GitHub Actions-specific issues: unknown uses: refs, deprecated action versions, shell escaping bugs, etc. Custom self-hosted runner labels are declared in .github/actionlint.yaml; changing that file also triggers this workflow so the declaration cannot drift silently.

Failure means the PR cannot merge until fixed. Running locally:

# yamllint
pip install yamllint==1.35.1
yamllint -d relaxed .github/workflows/

# structural parse (catches #501-class bugs)
python3 -c "import yaml, pathlib; [yaml.safe_load(p.read_text()) for p in pathlib.Path('.github/workflows').rglob('*.y*ml')]"

# actionlint (brew / go / prebuilt)
brew install actionlint
actionlint

Layer 2 — Auto-release watchdog (runtime)¶

File: .github/workflows/auto-release-watchdog.yml

Triggers on workflow_run completion for auto-release.yml. Fetches the run's job count via gh api and classifies the outcome:

success — if a tracking issue is open from a prior failure, close it with a recovery note
job_failure — one or more jobs failed; open or update tracker
workflow_file_error (0 jobs + failure) — GitHub rejected the file; open or update tracker with a dedicated message explaining that gh run view will not have logs

Tracking issue title: Auto-release workflow failed — RELEASES BLOCKED. One issue, edited in place, auto-closed on recovery — mirrors the #475 close-path pattern used by the orphan-branch and deps-drift sweeps.

auto-release.yml only tags. CHANGELOG.md is not a release artifact these layers watch: it is rendered in the version-bump commit, so a failed render cannot block a tag, and the next bump retries it (see versioning.md § Release pipeline).

Layer 3 — Release cadence check (invariant)¶

File: .github/workflows/release-cadence-check.yml

Runs every 30 minutes (plus workflow_dispatch). Scans main commits in the last 24h that changed CMakeLists.txt. For each commit that actually modified the VERSION line, checks:

Has the commit been on main for longer than the grace window (default 15 min, to let auto-release finish)?
Does a tag vX.Y.Z matching the bumped version exist, pointing at this commit or a descendant?

If the grace window has expired and no tag exists → add to findings and open/update a tracking issue titled Release cadence: version bumped but no tag.

This layer is cause-agnostic. Whether auto-release failed because of a YAML bug (Layer 1 would catch), a runtime job failure (Layer 2 would catch), a missing secret (neither of the above might catch), a forgotten manual step, or a GitHub outage — the invariant fires because the symptom (missing release) appears.

It checks only that the tag exists. A tag whose release never published, or published without its assets, passes this layer: during the v0.952.0–v0.955.1 outage every tag existed and the cadence check stayed green while no release shipped. Whether each tag's release published with the required assets — and why not, when it did not — is Layer 4's job (the release reconciler, above).

release-path PR gate (pre-tag prevention, issue #1962)¶

File: .github/workflows/release-path-pr-gate.yml

Sibling to fix/feat-needs-bump. fix/feat-needs-bump catches "user- facing change merged without a bump"; the release-path PR gate catches the bigger structural gap that #1962 surfaced: the release-build path is never tested at PR time.

PR build.yml builds Pulp from source via FetchContent — it never runs tools/scripts/fetch_skia_for_release.py, never builds the SDK tarball, never links the prebuilt Skia archives. So every breakage to the prebuilt-Skia path (chrome/m144 fontconfig undefineds, SkUnicode core/icu link-order, future Skia bumps that change asset layout) sails through PR green and only detonates post-tag, when release-cli.yml is the only workflow exercising that code path.

The PR gate runs the exact release-cli.yml build steps — fetch_skia_for_release.py, the PULP_REQUIRE_GPU_FOR_SDK=ON configure, cmake --build … --target pulp-cli, and a pulp-cpp --version smoke — for the two platforms that surface release-path regressions first:

linux-x64 — GNU ld is the strictest static-link environment. fontconfig undefineds, SkUnicode core/icu order bugs, anything involving missing --start-group/--end-group shows up here before macOS or Windows even notice.
darwin-arm64 — sanity check that we don't ship a Linux-only gate that misses macOS-only regressions (Metal framework drift, AppKit symbol changes, etc.).

Triggered only when a PR touches files in the release-path scope: tools/scripts/fetch_skia_for_release.py, tools/deps/manifest.json, tools/cmake/Find*.cmake, tools/cmake/Pulp*.cmake, tools/cli/CMakeLists.txt, core/{canvas,render,view}/CMakeLists.txt, CMakeLists.txt, release-cli.yml. Most PRs (view / docs / examples / plugin) skip this gate entirely so iteration speed is unaffected.

The macOS leg has its own PULP_RELEASE_PR_GATE_MACOS_RUNS_ON_JSON selector, falling back to the legacy shared PULP_RELEASE_MACOS_RUNS_ON_JSON only while a fleet migrates. Production uses mutually exclusive Tart CI class labels: tagged Release CLI / Sign and Release jobs request pulp-release-tagged, while this PR-time gate requests pulp-release-pr-gate. One release supervisor serves both classes but mints a JIT runner for only the highest class with demand, so an older PR gate cannot claim capacity while a tagged release is waiting. GitHub retains FIFO within each class.

If release-cli.yml's job structure ever drifts from this gate, the gate is lying. Mirror any structural change to release-cli.yml here (or refactor both into a shared composite action). The "Mirror release-cli.yml's Linux deps step verbatim" comment in the workflow calls this out.

Windows CLI compile (MSVC) and the pre-tag check¶

Files: .github/workflows/windows-cli-compile.yml, tools/scripts/windows_cli_compile_scope.py, tools/scripts/windows_cli_release_precheck.py (called from auto-release.yml).

Every required build gate compiles with Clang, and the PR-time Linux lanes use Clang or GCC, so code under #ifdef _WIN32 is compiled by nothing until release-cli.yml builds a tag on windows-latest. An MSVC-only error there passes every PR check and then fails the CLI windows-x64 / CLI windows-arm64 legs of every tag until it is fixed; each such release builds for ~50 minutes, skips its release job, and publishes nothing. That is how v0.897.3, v0.898.0 and v0.899.0 went unpublished on 2026-10-03 (a helper defined twice under _WIN32 in tools/cli/kit_profile_verification.cpp, MSVC C2668).

PR time (advisory). Windows CLI compile (MSVC) configures exactly like the release's Windows legs (test_windows_cli_compile_workflow.py fails on drift) and builds pulp-cli and pulp-mcp with Ninja, cl.exe and a weekly ccache. A scope job on ubuntu first runs the classifier and the Windows job runs only when the change can reach that compile: tools/cli/, tools/mcp/, tools/cmake/, or C/C++ outside test/, examples/ and external/ that names a Windows platform path or carries _WIN32 / _WIN64 / _MSC_VER / WIN32_LEAN_AND_MEAN / NOMINMAX. That was 39 of 556 merges in the two weeks before it landed, one hosted Windows job each — not the four-per-PR load that made build.yml drop Windows from PR heads. It is not a required check, but it is shaped to be one: the scope job always reports and an irrelevant change skips the compile job, so promotion needs a merge_group trigger and the ruleset entry, not a redesign. Its boundaries: it does not compile Windows code outside the CLI's link closure, and MSVC strictness in a file with no Windows marker is only caught here when the file is under tools/cli/ or tools/mcp/ (the nightly cross-platform-check.yml builds everything).

Release time (blocking). The same workflow runs on every push to main that touches the CLI's source trees, without cancelling earlier pushes, and posts a Windows CLI compile verdict check on the pushed commit. That job runs only when the Build CLI targets (MSVC) step ran and fails only when that step failed, so runner or bootstrap trouble leaves no verdict rather than a red one. Before auto-release.yml creates an SDK tag, windows_cli_release_precheck.py walks main's first-parent history back from the tagged commit (commits that touched those trees, and the up-to-four a merge-queue batch can push above one) and reads the newest verdict. A failure withholds the vX.Y.Z tag (the plugin tag is never held), with an SDK tag withheld annotation and step summary naming the failing run. It reads check runs (checks: read), so auto-release.yml still holds no actions scope and cannot cancel anything. The version stays untagged, which Layer 3 reports, and the first push to main after a clean compile tags the then-current version, because tagging compares HEAD's version with the latest tag. Missing or unreadable evidence never withholds a tag. Set the repository variable PULP_RELEASE_WINDOWS_PRECHECK=off to disable the check.

The pre-tag check cannot stop the first tag after a bad merge if that tag is cut before the merge's compile finishes; Layer 4 then reports that tag as soon as its Windows legs fail.

Agent-capability pre-tag check¶

Branches leave MANIFEST_REVISION and SURFACE_INVENTORY_VERSION at their base value and the version-at-land bump assigns them (agent_capability_manifest.py --assign-pending, best-effort inside the bump commit). If that assignment failed, main still carries a pending counter: a revision number that names no published contract. auto-release.yml runs --require-assigned before tagging the SDK and withholds the tag when anything is pending, with an SDK tag withheld annotation and summary. The plugin tag is never held.

Recovery when the assignment keeps failing. The bump run's log names the failure (could not assign pending capability counters, with the tool's last lines, or timed out after 300 s). Fix the cause, then assign by hand: on a branch from main, run python3 tools/scripts/agent_capability_manifest.py --assign-pending and land the result as an ordinary PR. Its one-step advance over the pending base is exactly what the check accepts, and --status on the branch must report every counter assigned.

What happens to the withheld version. The tag decision tags HEAD whenever its version is newer than the latest tag (auto_release_decision.py). So the withheld vN is tagged by the first push to main after the counters are assigned, if main is still at vN at that point, on that commit rather than on the bump commit. If another SDK bump lands first, main moves to vN+1, that bump assigns the counters and is tagged, and vN is skipped for good: nothing tags an older version later.

fix/feat-needs-bump (PR-time prevention, issue #1009)¶

The watchdog layers above all react to a stranded release — a user-facing fix that merged without a bump. The structural fix is to catch it at PR time, before the merge ever happens. That lives in .github/workflows/version-skill-check.yml via the --require-bump-for-fix-feat flag on tools/scripts/version_bump_check.py.

What it does: On PR triggers, parses both ${{ github.event.pull_request.title }} and the live commit-derived signals contributed by the PR range. Explicit reverts cancel their target signals; reverting a revert restores its target signal. If either carries the Conventional Commits prefix ^(fix|feat)($[^)]*$)?!?:\s, it asserts that EITHER:

A commit in the PR's diff range has subject chore: bump versions (the canonical subject pulp pr writes when a bump was applied), OR
A commit in the range carries a top-level Version-Bump: skip reason="..." trailer (with non-empty reason).

Otherwise hard-fails with a message that suggests both fix paths.

What it does NOT do: the per-surface verdict pipeline is unchanged. Internal-only fixes whose heuristic verdict is "patch (advisory)" still get a bump injected by pulp pr — but if the merge bypasses pulp pr and the bump never lands, this check catches it.

Motivating incident: 2026-04-30, PR #1008 (fix(view): on(id,'click',fn) auto-wires View::on_click) merged at 02:36:45Z via gh pr merge after a force-push had short-circuited shipyard pr's version-bump step. The existing watchdogs all reported green: auto-release.yml decided SHOULD_TAG=0 and exited successfully (correct outcome for a no-bump merge). The release-cadence-check.yml looks for bumps without tags, not the inverse. The fix landed on main but consumers couldn't reach it until the catch-up bump PR (#1011) merged.

Required branch protection¶

The version-skill-check GitHub workflow runs this check on every PR. The canonical repository makes it load-bearing through branch protection on main:

Required check: Versioning & Skill-Sync / Enforce version & skill sync

The checked-in intent lives in .github/rulesets/main-protection.json; keep the live ruleset aligned with it. Normal merges then refuse any PR whose title or commit subjects signal fix: / feat: without either the bump commit or the skip trailer, regardless of squash / rebase / merge-commit mode. Administrators can still bypass branch protection when the live ruleset permits it, so the post-merge detector remains load-bearing.

The same required check writes an Expected release tags run summary for the PR queue. It predicts SDK and plugin tags from the PR head, fetched tag state, the repository's one-commit-subject/multi-commit-PR-title squash policy, and sticky Release: skip metadata. Tag creation remains a post-merge action in auto-release.yml; the queue report is prediction, not a pre-created tag.

Layer 3 backstop in `auto-release.yml`¶

Intent-trailer model is LIVE (2026-07-17): the Stranded fix/feat detector step now exports PULP_ACCEPT_INTENT_TRAILERS=1, so a fix/feat that merged intent-only (its number assigned moments later by the version-at-land --push bot) reads as COVERED, not stranded — otherwise it would false-warn on every intent-only merge. It stays --no-merges-scoped: a stray intent on a re-sync merge commit, or a fix/feat with NO intent, still correctly strands. The "intent present but the bot never applied it" case is covered by the separate pending-intent liveness alarm (pending-intent-liveness.yml). See docs/guides/version-at-land-cutover.md.

If the PR-time gate is bypassed somehow (force-push race, admin merge, unknown-unknown), auto-release.yml has a final backstop step (Stranded fix/feat detector) that runs after the tag-or-not decision. It applies the same live-signal classifier to the whole pushed range and maps each signal back to its release surface. That catches plain merge tips, multi-commit rebases, re-reverts that restore an older fix: / feat: signal, and mixed pushes where (for example) a plugin tag does not cover an unbumped SDK fix. Explicit Release: skip and range-wide top-level Version-Bump: skip opt-outs remain silent; a sticky per-surface Release: skip counts as intentional coverage only for that surface when its bump belongs to the pushed range. The bump commit is the coverage boundary: signals at or before it are covered, while a later fix remains uncovered. Recovery also retains independent fix/feat levels per surface and respects an explicit numeric Version-Bump: <surface>=<level> verdict. The tracker records those exact levels and passes them back through --recover-levels, so covered feature work cannot inflate a later fix recovery. For multi-commit squashes, the auto-release guard also recognizes embedded source skip-trailer lines before GitHub's co-author footer; that footer prevents ordinary interpret-trailers parsing from seeing the nested source trailers.

What counts as a declared bypass¶

Scanning a whole squash body for a trailer means a trailer that is only being quoted — a friction report, a guide to this very grammar, a PR body pasted into a commit — is sitting in the same text as a real declaration. Telling them apart is not optional here: a bypass read from a quoted example withholds a release tag, and a release that does not happen reports nothing anywhere.

Indented and >-quoted prose moves the trailer off column zero, so a line-start anchor excludes it. A fenced code block does not, so the fence is masked before the scan. Both rules live in tools/scripts/gate_common.py, and .github/workflows/auto-release.yml reaches them through tools/scripts/release_trailer_guard.py rather than matching trailers itself:

# `skip` or `no-skip` on stdout; exit 2 when the commit cannot be classified
python3 tools/scripts/release_trailer_guard.py --query release-skip --ref HEAD
python3 tools/scripts/release_trailer_guard.py --query version-bump-skip --ref HEAD

An exit 2 fails the guard step. Neither guess is safe: "no bypass" publishes an unwanted tag, and "bypass" withholds a wanted one silently.

The same helpers back the PR-time gate (_range_has_version_bump_skip_trailer() in version_bump_check.py) and the pre-merge tag prediction (pr_release_tag_report.py), so all three layers accept exactly the same values — including the requirement that a Version-Bump: skip carry a non-empty reason="...", and the rule that a per-surface Version-Bump: <surface>=skip is not a whole-release opt-out. tools/scripts/test_release_trailer_guard.py runs the workflow's own guard step, extracted from the YAML, against real commits. For an unbumped live signal, it:

Emits a ::warning:: annotation visible in the workflow run UI.
Opens a tracking issue titled release: stuck — fix/feat merged without bump (<sha>) with the release-stuck label and step-by-step recovery instructions.

The tracker is keyed on the tip SHA so multiple stranded merges produce distinct issues — each needs its own catch-up bump PR. Unlike version-keyed release watchdogs, a SHA-keyed tracker cannot be auto-closed from its title alone because the affected surface is not encoded there — but it is encoded in the body. watchdog-reaper.yml (daily at 06:00 UTC, plus immediately after every successful Release CLI run — the event that actually resolves most open trackers; the sweep chains via workflow_run because the publish uses GITHUB_TOKEN, whose events cannot trigger other workflows) parses the body for the full tip SHA and the uncovered surfaces, then auto-closes the tracker once a later release tag for every uncovered surface contains the stranded commit (git tag --contains <tip>, filtered per surface: SDK tags vX.Y.Z, plugin tags plugin-vX.Y.Z). That means the missing bump has since shipped and consumers can reach the change. The close decision lives in the pure, unit-tested tools/scripts/reap_stranded_tracker.py (tools/scripts/test_reap_stranded_tracker.py); it fails safe — an unparseable field or any still-unshipped surface leaves the tracker open, so a genuinely active strand is never closed. This keeps the SHA-keyed trackers from orphaning forever (102 had piled up by 2026-07, burying real signal). Its generated recovery command starts from fetched origin/main, preserves the historical analysis range, and passes the explicitly uncovered surface list. Current main is the version-arithmetic base, while --recover-stranded-release ignores a historical marker-only chore: bump versions; later version movement, a stale marker, or an already released sibling surface cannot produce a successful no-op or duplicate tag.

Manual override¶

All three watchdog layers honor the standard Release: skip reason="..." trailer already documented in CLAUDE.md — the skip flag makes the auto-release step decline to tag, Layer 2 treats the workflow run as a normal success, and Layer 3 sees no VERSION change so never fires.

Version-Bump: skip reason="..." is also honored as a release-skip by the auto-release.yml guard (pulp #1308 follow-up). Authors use this trailer for fix/feat changes that legitimately don't bump SDK or plugin versions — typical cases:

JS-only changes to packages/pulp-react/ (versions independently via packages/pulp-react/package.json + npm publish)
Docs / refactors accidentally typed as fix: / feat:
Test-infra changes that mention a fix in their subject

Without honoring this trailer here, the post-merge stranded-fix detector would fire on every such merge and demand a follow-up bump PR — even though the author already declared no SDK/plugin bump is needed. The PR-time gate (version-skill-check.yml) already accepts this trailer for the same reason; the post-merge layer now matches.

The two trailers are still semantically distinct:

Release: skip reason="..." — opt out of this release tag (e.g. the change is part of a multi-PR series; tag the last one).
Version-Bump: skip reason="..." — declare no SDK/plugin bump needed (the change is genuinely not user-facing for those surfaces).

In both cases, the auto-release guard now treats them as legitimate opt-outs and won't synthesize a stranded-fix tracker.

Follow-up hooks (optional, not required)¶

A future enhancement could call the same tools from .githooks/pre-push so linting fires before a branch even reaches GitHub. Not strictly needed — Layer 1 in CI is sufficient — but reduces iteration time for contributors who touch CI workflows frequently.

2026-05-03 (issue #1375) — release-cli.yml runs for v0.74.0 and v0.74.1 both died at the same point on windows-arm64: mid-Priming shared Yoga source cache... with exit 127 (no further log output). v0.74.0's GitHub release ended up with only the plugin .pkg files (those come from sign-and-release.yml); v0.74.1 had no GitHub release at all (pulp sdk install --version 0.74.1 404'd). No watchdog alerted — both Layer 2 (auto-release) and Layer 3 (cadence) reported green because auto-release.yml itself ran fine and a tag existed for both. Fix: retry-on-failure around every shared-source priming call in setup.sh (so a transient 127 doesn't strand a release), plus a parallel Layer 2b watchdog (release-cli-watchdog.yml) keyed on per-tag SDK-asset presence.
2026-04-30 (PR #1008 → issue #1009) — fix(view): ... merged via gh pr merge after shipyard pr short-circuited its bump step (force-push race). auto-release.yml saw no version movement and exited successfully (SHOULD_TAG=0). All three watchdog layers reported green because none of them watch for the inverse case (success-without-tag after a user-facing merge). Fixed by the fix/feat-needs-bump PR-time gate plus the auto-release.yml backstop step documented above.
2026-04-20 (PR #501 → #510) — YAML indent bug rejected auto-release at workflow-file level; all 8 runs in the following day failed silently. Layer 1 would have caught this at PR review; Layer 2 would have alerted minutes after #501 merged.
v0.15–v0.18 MSVC include bug — release builds silently produced unusable artifacts for 24h because no gate verified the produced binary actually ran. The cadence check would not have caught this (tag was created), but the feedback_silent_release_failure memory in ~/.claude documents the shape; a future Layer 4 could smoke-test the produced binary.