CI Validation

Pulp validates branches on macOS (local), Ubuntu (SSH), and Windows (SSH) validation lanes before merging.

On a pull-request head, the affected-test step runs tests selected from the diff and from binaries rebuilt by that head's governed build. A marker written just before the build bounds the warm Ninja log; older branch outputs do not add tests. Docs-only diffs add no rebuilt-binary tests. The rebuilt set counts only when it is narrow: at most eight test executables, and no regeneration of build.ninja or CMakeCache.txt during the build. A cold, reconfigured or stale warm directory rebuilds far more than the diff reaches, and the step then adds no rebuilt-binary tests and logs why.

The macOS retarget lane keeps the untrusted-runner boundary for contributor code and invokes tools/ci/governed-build.sh inside that namespace for its build. This preserves the lane's runner isolation while applying the same bounded worker and build-directory locking policy as the other build emitters.

The advisory Windows release-path, MIDI Services, and BLE compile gates use PowerShell for their job steps but invoke bash tools/ci/governed-build.sh for each CMake build. The governor supplies the worker bound; its build-directory lock applies on POSIX runners (Git Bash/MSYS skips that lock because native and POSIX paths cannot be reconciled safely). The main Windows matrix follows the same rule, so custom runner selectors cannot bypass the worker governor.

Setting up a dedicated machine as a persistent CI runner? See self-hosted-runner.md for the walkthrough + first-run gotchas (git-lfs hook conflict, Xcode license, Apple Clang version skew).

Fork routing in YAML is defense in depth, not access control

Browser-source fidelity on the local ARM gate

The required macOS ARM job installs Pulp's checksum-pinned Chrome for Testing archive into its disposable job directory and exports PULP_DESIGN_BROWSER. This is required for generic HTML source-to-native tests: the browser capture is the visual reference for Skia lowering. The local runner image does not need a mutable global browser, but the workflow must fail if the pinned archive cannot be downloaded, verified, extracted, or executed; a skipped browser comparison is not a passing fidelity gate.

The browser-capture Node tests are intentionally split by execution contract: dependency-free units, the serial real-Chromium integration file (with its own 600-second CTest timeout), and the esbuild-backed materialized-runtime canonicalization case. The required Linux leg runs the locked npm ci --prefix tools/import-design/jsx-runtime step before CMake configure, which makes the dependency-backed case visible to CTest. The required macOS lane performs the same locked install before running its affected-test sweep. Because the automatic Linux selector may use a clean Proxmox VM without a system Node installation, build.yml first provisions the exact Node 22.15.0 toolchain with actions/setup-node@v7 and a lockfile-keyed npm cache; that version supplies the zstdDecompressSync API used by the .fig lane.

The arm64 macOS legs of build.yml (required gate) and build-macos.yml (retarget) also configure PULP_GPU_AUDIO_EXACT_PROVIDER_PROOF=ON: they compile the GPU-audio hardware-evidence binaries but never run them, because their tests carry the gpu-proof label that every gate ctest selection excludes, as does Shipyard's macOS test stage (.shipyard/config.toml [validation.default.overrides.macos], --label-exclude) and the changed-surface runner. A guard step fails the gate unless the configured gpu-proof tests equal tools/ci/gpu_proof_tests.txt and none of them is in the gate's own selection. Hardware receipts remain owner-run on a physical GPU. Source-only or offline configurations without that node_modules/esbuild installation still register the dependency-free suites, but they do not claim the canonicalization proof.

The macOS runner is chosen by the resolver in build.yml, which normally honors PULP_LOCAL_MACOS_RUNS_ON_JSON and routes to the M1/M3/M5 event-class JIT VM pool. All three hosts serve the required gate on equal terms — M1 waits 10 minutes before taking Pulp work, which is latency policy, not an inability to serve it. For a pull request whose head branch lives in another repository, both self-hosted selectors are ignored and the leg falls through to the GitHub-hosted macos-15 label.

This checked-in routing is useful defense in depth, but it is not a security boundary: pull-request workflow YAML is contributor-controlled and can remove its own guard. The local Macs hold the Developer ID signing keychain and the notary key (~/.config/pulp/secrets/), and PULP_LOCAL_MACOS_RUNS_ON_JSON is a repo variable — variables, unlike secrets, do resolve for fork runs. Without the guard, one "Approve and run" click on a fork pull request could otherwise execute contributor code on the credentialed machines. The real boundary must be an organization runner group restricted to selected trusted workflow refs (or an equivalent trusted dispatcher). Until then, do not add private pools to automatic PR routing.

The leg is rerouted rather than skipped, so a fork contributor still gets a real macOS result on a clean throwaway runner. Note that the required macos check is posted by the local lane, so a fork PR still cannot merge on its own — the maintainer adopts the commits onto an in-repo contrib/* branch and ships that.

Same-repo pull requests, pushes, and workflow_dispatch runs are unaffected. Covered as defense in depth by tools/scripts/test_fork_pr_runner_routing.py (ctest: fork-pr-runner-routing), which runs the resolver the workflow actually embeds; that test does not prove the runners are inaccessible.

A pull request whose fork was deleted is a fork too. Its github.event.pull_request.head.repo.full_name is null, so a check that only treats a non-empty, foreign head as a fork lets it read the runner variables; !github.event.pull_request.head.repo.fork passes it the same way, because !null is true. Every place that routes pull requests therefore counts a missing head repository as a fork:

• build.yml's resolver and coverage.yml;
• tools/scripts/resolve_runs_on.py, which sends any fork pull request to its GitHub-hosted fallback and needs PR_HEAD_REPO from its caller;
• every job that reads a self-hosted-capable variable straight into runs-on, through the expression (github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository) && vars.X || '"<hosted label>"'.

tools/scripts/fork_runner_guard_check.py (workflow-lint, gates.sh) fails a workflow that drops either.

The physical Intel lane is advisory and isolated

The Intel Mac mini serves nightly-intel.yml through an ephemeral JIT supervisor, not the persistent required-gate pool. Its selector is exact:

self-hosted,macOS,X64,pulp-intel-native,pulp-host-macmini

PULP_NATIVE_INTEL_RUNS_ON_JSON is intentionally unset until the host passes tools/ci/native-intel-runner.sh --check and a manual dispatch proves a cold workspace can claim and finish the job. While unset, the native job uses macos-15-intel. To pilot without changing the variable, dispatch nightly-intel.yml with use_physical_intel enabled; that boolean maps internally to the exact selector above and cannot target another pool. To roll back after enabling it, unset the variable and redispatch any job already queued for the local labels; GitHub does not reroute an assigned job.

Before starting the supervisor, create a dedicated organization runner group for the Mac mini, restrict it to Generous-Corp/pulp and the protected default-branch .github/workflows/nightly-intel.yml, and set its numeric ID as PULP_NATIVE_INTEL_RUNNER_GROUP_ID in the LaunchAgent. The supervisor refuses the default group and an unset/non-numeric ID, then reads the organization runner-group API and requires the group to contain only this repository and only nightly-intel.yml@refs/heads/main. The GitHub credential therefore needs runner-group read access as well as repository runner administration. Prove that a workflow revision from a PR branch cannot target the group before enabling the repository selector; labels alone are not an access boundary.

The GitHub credential and controller must never share a uid with workflow jobs. The login account runs only the controller and holds gh/ghapp auth. Jobs run as the fixed hidden service identity pulp-ci (uid 499, primary group staff), which owns only its current disposable job root. Its directory-service home is the root-owned /var/empty, its login shell and authentication are disabled, and the worker supplies a new private HOME and TMPDIR for each job. It must not be an administrator and must not be able to write the controller checkout or worker shim. Do not copy the controller account's GitHub credential into it.

Creating that OS boundary is a one-time administrator operation:

1. Confirm uid 499 is unused, then create the fixed non-login identity. These are deliberate directory-service writes, so inspect the first command before continuing; do not choose another uid and weaken the fixed-identity check.

dscl . -list /Users UniqueID | awk '$2 == 499 { print; found=1 } END { exit found ? 1 : 0 }'
sudo dscl . -create /Users/pulp-ci
sudo dscl . -create /Users/pulp-ci RealName 'Pulp native Intel CI worker'
sudo dscl . -create /Users/pulp-ci UniqueID 499
sudo dscl . -create /Users/pulp-ci PrimaryGroupID 20
sudo dscl . -create /Users/pulp-ci NFSHomeDirectory /var/empty
sudo dscl . -create /Users/pulp-ci UserShell /usr/bin/false
sudo dscl . -create /Users/pulp-ci IsHidden 1
sudo dscl . -create /Users/pulp-ci AuthenticationAuthority ';DisabledUser;'
sudo dscl . -create /Users/pulp-ci Password '*'

Do not create /Users/pulp-ci, enable automatic login, or enable remote login for this identity. 2. Put the shared, Apple-signed Xcode at /Applications/Xcode.app and accept its license once. Put a verified, unconfigured GitHub Actions runner archive at /usr/local/share/pulp-native-intel/actions-runner-mini, verified CMake, Ninja, ccache, and Git LFS tools under /usr/local/share/pulp-native-intel, and a prewarmed cache at /usr/local/share/pulp-native-intel/ccache. The commands exposed in bin/ may be relative symlinks into the same trusted root (for example, into a complete CMake bundle); no link may escape it. Recursively set Xcode and this entire trusted root to root:wheel, then remove group/world write bits:

sudo chmod -RN /Applications/Xcode.app \
  /usr/local/share/pulp-native-intel
sudo chown -R root:wheel /Applications/Xcode.app \
  /usr/local/share/pulp-native-intel
sudo chmod -R go-w /Applications/Xcode.app \
  /usr/local/share/pulp-native-intel

The golden runner must never be configured or run in place. Jobs consume the warm ccache read-only with ccache depend mode explicitly disabled through CCACHE_NODEPEND=1 (decision 20); they use an ephemeral writable temp directory and cannot poison the cache. To refresh it, stop the LaunchAgent, build a new cache in a staging directory, install that directory as root:wheel without group/world write, run --check, and only then restart the controller. 3. Install the checked-in lifecycle shim immutably:

sudo install -d -o root -g wheel -m 0755 /usr/local/libexec
sudo install -o root -g wheel -m 0755 \
  tools/ci/native-intel-runner-worker.sh \
  /usr/local/libexec/pulp-native-intel-worker

1. Use sudo visudo -f /etc/sudoers.d/pulp-native-intel to install this narrow rule, replacing daniel only if the controller login is different:

daniel ALL=(root) NOPASSWD: /usr/local/libexec/pulp-native-intel-worker --check, /usr/local/libexec/pulp-native-intel-worker --clean, /usr/local/libexec/pulp-native-intel-worker --run

The root-owned shim accepts only those three fixed operations. Before each job it removes any job-installed crontab, kills leftover uid-499 processes, removes that uid's state from the host's mutable data roots (including macOS temp roots, /Users/Shared, and /Library/Caches), removes the fixed job root, and copies the immutable golden runner into it. The private per-job HOME, TMPDIR, ccache temp directory, runner, and workspace are all uid-owned only for the lifetime of that job. After the runner exits the shim kills leftovers and removes all of them again. Runner executables never survive into the next job; only the root-owned read-only warm ccache does. The worker also requires macOS's com.apple.atrun service to remain disabled and removes any uid-499 at/batch jobs before serving another workflow. JIT registration, runner-group verification, stale-registration removal, and all authenticated GitHub API calls remain in the controller. The shim receives only the ephemeral one-job JIT payload on standard input, drops to pulp-ci, runs run.sh with a clean environment, and contains no GitHub client or persistent credential. Both sides fail closed if the fixed account identity is absent or altered, the worker is mutable or has a write-granting ACL, passwordless delegation is absent, the shared Xcode/tools/cache have unsafe ownership, permissions, or escaping symlinks, or group verification fails. Xcode signature, Gatekeeper, license, xcodebuild, and clang probes run only after dropping to uid 499, so a root-only success cannot mask an unusable worker toolchain. A later identity, ownership, signature, or toolchain validation failure returns a terminal configuration status and leaves the controller alive but its lane offline; launchd therefore does not turn an integrity failure into a restart/retry storm. Restart the controller only after correcting the failed preflight.

The LaunchAgent template is tools/launchd/pulp-native-intel-runner.plist.template. RunAtLoad and KeepAlive restore the controller after the login account logs in. FileVault prevents the startup volume from mounting unattended after a cold power cycle, so the host is not available until a person unlocks it. Keep this lane advisory and do not weaken FileVault or give it any of the required ARM64 gate labels. The controller prefers ghapp when it is installed and otherwise uses its own authenticated rootless gh. The job account receives neither client nor token.

The Linux x64 lanes run on macpro (Proxmox)

Ephemeral x86_64 Proxmox VMs run on macpro — a Late-2013 Mac Pro (Xeon E5-1650 v2, 6c/12t, 31 GB) repurposed as a Linux CI host. The repository-scoped pool retains its existing operator-dispatch behavior and five generic labels. The provider additions below do not change workflow routing: protected PR and merge-group Linux remain GitHub-hosted until a separate routing change is reviewed and enabled.

There are three distinct service roles:

• pulp-ephemeral-pool@.service is the existing repository-scoped pool. It keeps the optional per-slot /etc/pulp/linux-runner-group-%i.env contract and the legacy bridged network. Do not replace it with a shared protected-role environment; retain generic capacity for existing operator dispatches.
• pulp-trusted-ephemeral-pool@.service loads /etc/pulp/linux-trusted-runner-group.env, selects policy trusted, prefix pulp-ci-ephemeral, and label pulp-auto-linux-x64. Its group must be named pulp-trusted-build, contain only Generous-Corp/pulp, and select exactly protected-main build.yml, pr-safe-linux.yml, vellum-freeze-check.yml, and version-skill-check.yml.
• pulp-pr-safe-ephemeral-pool@.service loads /etc/pulp/linux-pr-safe-runner-group.env, selects policy pr-safe, prefix pulp-pr-safe-ephemeral, and label pulp-pr-safe-linux-x64. Its group must be named pulp-pr-safe-build, contain only Generous-Corp/pulp, and select exactly protected-main pr-safe-linux.yml.

Both protected roles register at organization scope only after verify_linux_runner_group.py proves the exact group name, repository, and workflow allowlist. Use a distinct root-owned mode-0600 organization token with runner read/write permission. The only helper alternative is an exact root-owned, non-group/world-writable /usr/local/bin/ghapp. Capability labels are rejected without a verified organization group.

Install the supervisor, role wrappers, verifier, units, and network helper before enabling either protected role:

The scheduled reaper deliberately does not read the long-lived runner token. Before enabling its timer, install Shipyard's GitHub App helper at the exact root-owned, non-symlink, non-group/world-writable path /usr/local/bin/ghapp. The token-file mode below remains supported by the pool supervisor, but it is not a reaper authentication fallback; if the helper is missing or insecure, recovery exits before inspecting or changing any VM.

apt-get update
apt-get install -y gh
install -d -o root -g root -m 0755 /usr/local/lib/pulp
# From a checkout of main on the host (or copied there):
tools/ci/proxmox-host-health.sh --install "$PWD"
/usr/local/sbin/pulp-proxmox-host-health.sh          # must print HEALTHY

tools/ci/proxmox-host-health.sh holds the one table of which repository file is installed at which host path (--manifest prints it). --install copies by that table, saves the previous copies under /root/pulp-ci-host-backup-<UTC>, and renames each file into place so a running supervisor keeps its open inode. Never copy a single file by hand: the supervisor, role wrappers, verifier and reaper are a matched set, and a partial copy is how the host ran a hand-forked supervisor that no clone could be reaped from.

The same script, run with no arguments, is the drift guard. pulp-ephemeral-reap.service runs it after every reaper pass (ExecStartPost). It compares each installed file's git blob id with the same path on main through the GitHub App helper, and also reports any pool slot stuck failed and a thin pool within 10 points of the governor's disk limit. Any finding fails the reaper unit, so drift shows in systemctl --failed within 15 minutes. A run that cannot reach GitHub exits 2 (UNVERIFIED), never 0. A merge to main that changes one of these files makes the host report drift until it is reinstalled, which is the intent: the host should run what main says.

Drift is never installed automatically, because a merge to tools/ci should not become root code on the host until someone chooses to run it. The guard instead makes the reinstall a single verified command:

• /var/lib/pulp-ci-host/drift.state records when drift was first seen and the main commit it was compared against. A later merge updates the commit but keeps the first-seen time, and the next drift-free check deletes the file.
• On drift, the check stages the table's files at that exact commit under /root/pulp-deploy-staged/<sha>, verifies each file's blob id against the commit, and deletes any other stage. A fetch that does not verify stages nothing.
• pulp-proxmox-host-health.sh --install-staged installs that stage after verifying it again against main as it is at that moment. It refuses when main has moved past the stage or a staged file changed, so it installs exactly what the guard compared against. The UNHEALTHY line prints this command whenever a stage is ready.
• pulp-proxmox-host-health.sh --json runs the same check and prints one JSON object (schema: 1). It includes state (healthy, unhealthy or unverified), ref_sha, drift.first_seen and drift.age_seconds, each drifted file with its installed and expected blobs, failed slots, disk fill, the stage, and reinstall_command. It uses the same exit codes as the text check.
• The reaper unit runs the check as --json-out /run/pulp-ci-host/health.json. That prints the text check to the journal as before, and also writes the JSON to that file (mode 0644, replaced by rename) on every outcome, including unverified. Fleet monitoring reads the file over the existing root SSH alias and never runs the check itself. /run is cleared at boot, so the file is absent until the first reaper pass after a reboot. If the path cannot be written, the previous object stays with its old checked_at. A reader therefore treats an absent or stale file as unknown, never as healthy. It reports drift older than an hour, any non-healthy state, and a checked_at older than 30 minutes, since the reaper runs every 15 minutes.
• tools/ci/proxmox_host_watch.py is that reader. It runs every 15 minutes from one non-gate fleet Mac (tools/ci/com.danielraffel.pulp.proxmox-host-watch.plist.template). It reaches the host over ssh by its fleet name and only reads. Each problem gets one issue in the private infrastructure-issues repository. The issue is updated in place and closed when the problem clears. It covers four signals:
• a pulp-* unit that has been failed for over 15 minutes (a slot at its restart limit), shown with its last error line. For no free clone id in A..B, it also names the orphan clone that holds the slot's VMID.
• health.json that is absent, unreadable or stale. This counts as unknown, never as healthy.
• a non-healthy state, or drift older than an hour, with the reinstall_command.
• a VM the reaper has skipped on 4 or more passes over at least 45 minutes while no online runner registration names it. A live job's lease skip stays quiet because its runner is online.

An observation that could not be read never closes an issue. Every issue carries what, evidence, likely cause, fix, safe-to-auto-apply and verify.

Create separate root-owned role environments; never share one:

install -d -o root -g root -m 0755 /etc/pulp
install -d -o root -g root -m 0700 /root/.config/pulp/secrets
printf 'PULP_LINUX_RUNNER_GROUP_ID=%s\n' "$TRUSTED_GROUP_ID" \
  | install -o root -g root -m 0600 /dev/stdin \
      /etc/pulp/linux-trusted-runner-group.env
printf 'PULP_LINUX_RUNNER_GROUP_ID=%s\n' "$PR_SAFE_GROUP_ID" \
  | install -o root -g root -m 0600 /dev/stdin \
      /etc/pulp/linux-pr-safe-runner-group.env
install -o root -g root -m 0600 /path/to/org-runner-token \
  /root/.config/pulp/secrets/gh-org-runner-pat

The protected roles require the Proxmox firewall and three no-uplink /30 bridges. The helper leaves vmbr0 byte-identical, creates vmbr-ci200..202 with controller addresses 10.240.<VMID>.1/30, routes guests at 10.240.<VMID>.2/30, and installs one source-scoped NAT rule per bridge. The supervisor proves exact controller SSH ingress, default-deny ingress, private/reserved and IPv6 egress denial, and L2/IP source isolation before registration.

/usr/local/sbin/configure-proxmox-ci-network --dry-run
/usr/local/sbin/configure-proxmox-ci-network --apply
/usr/local/sbin/configure-proxmox-ci-network --verify
systemctl daemon-reload
systemctl enable --now pulp-ephemeral-reap.timer
systemctl enable --now pulp-trusted-ephemeral-pool@1.service
systemctl enable --now pulp-pr-safe-ephemeral-pool@1.service

Before enabling the timer on a host that already has clones in the managed VMID range, run the reaper in report-only mode and classify each reported legacy generation. The reaper will never mutate a pre-upgrade clone whose Proxmox description lacks the updated supervisor's exact pulp-runner-scope=... provenance, because the old supervisor did not persist whether --keep was requested. Preserve an intentionally retained clone by creating its root-owned mode-0600 generation marker under /var/lib/pulp/ephemeral-runner-keep; only after proving a legacy clone is disposable may an operator add its exact repository or organization recovery scope to the Proxmox description. Newly allocated clones carry that scope by construction, so scheduled recovery is automatic after the migration boundary.

--apply and --verify both assert a default-deny egress policy per isolated bridge, not merely that the bridge exists. MASQUERADE is address translation, not filtering: with NAT alone and Proxmox's stock -P FORWARD ACCEPT, a guest on 10.240.20x.2/30 still reaches 192.168.86.0/24, it just arrives looking like the host. The managed policy denies RFC1918 first, allows the uplink, permits established return traffic, and terminates in its own catch-all DROP so the chain policy is never what decides. The post-up/pre-down hooks restore and remove it alongside the NAT rule, so an ifdown/ifup cycle cannot leave a bridge up without its policy.

This matters most for the PR-safe pool, which exists to run unreviewed contributor code. Do not enable pulp-pr-safe-ephemeral-pool@N on a host where --verify does not prove the egress policy.

Start one instance of each protected role first and retain a generic pool instance. Verify the live group and exact role labels before adding capacity. For rollback, stop and disable only the protected role units, wait for their disposable guests to be absent, then run configure-proxmox-ci-network --rollback. Rollback refuses to remove a bridge with an attached guest and restores the prior IPv4-forwarding state.

Add an isolated lane for another repository

The same supervisor can serve another repository without sharing Pulp's group, labels, runner name, golden, or VM range. Install proxmox-ephemeral-pool@.service, then create one root-owned mode-0600 profile per slot under /etc/pulp/proxmox-runner/. A profile must set every identity explicitly:

TARTCI_RUNNER_REPO=Generous-Corp/vellum
TARTCI_RUNNER_GROUP_ID=123
TARTCI_RUNNER_GROUP_NAME=vellum-pr-safe-build
TARTCI_RUNNER_WORKFLOW=.github/workflows/build.yml
TARTCI_RUNNER_LABELS=self-hosted,Linux,X64,vellum-build-linux-x64,vellum-host-macpro
TARTCI_RUNNER_NAME_PREFIX=vellum-ci
TARTCI_PROXMOX_VM_NAME_PREFIX=vellum-ci
TARTCI_PROXMOX_GOLDEN=9006
TARTCI_PROXMOX_CLONE_BASE=203
TARTCI_PROXMOX_CLONE_MAX=203
TARTCI_RUNNER_GITHUB_AUTH_MODE=token-file
TARTCI_ORG_RUNNER_PAT_FILE=/root/.config/pulp/secrets/vellum-org-runner-pat

The verifier requires the named non-default group to contain only that repository and allow exactly the named workflow at refs/heads/main. The labels must include self-hosted,Linux,X64 and must not reuse a pulp-* capability label. VMIDs are restricted to 1..254 because the isolated address is derived as 10.240.<VMID>.2/30; every repository receives a disjoint range and matching vmbr-ci<VMID> bridge.

Both TARTCI_RUNNER_GITHUB_AUTH_MODE and TARTCI_ORG_RUNNER_PAT_FILE are mandatory for a non-Pulp profile. Generic profiles never inherit Pulp's authentication mode or organization PAT path; omitting either value fails before any runner or VM is created.

The network helper owns one exact contiguous range. To expand it, first stop the protected role units, wait until every managed guest is absent, roll back the currently installed range, then apply the expanded range. Use that same range for later verification or rollback. For the example above, replace Pulp's three-bridge contract with the four-bridge contract that also contains 203:

systemctl stop 'pulp-trusted-ephemeral-pool@*' \
  'pulp-pr-safe-ephemeral-pool@*'
/usr/local/sbin/configure-proxmox-ci-network --rollback
TARTCI_PROXMOX_CLONE_BASE=200 TARTCI_PROXMOX_CLONE_MAX=203 \
  /usr/local/sbin/configure-proxmox-ci-network --dry-run
TARTCI_PROXMOX_CLONE_BASE=200 TARTCI_PROXMOX_CLONE_MAX=203 \
  /usr/local/sbin/configure-proxmox-ci-network --apply
systemctl daemon-reload
systemctl enable --now proxmox-ephemeral-pool@vellum-1.service

Runner registration uses GitHub's JIT endpoint. The management credential stays in a root-owned mode-0600 host file (or the verified root-owned GitHub App helper), and the one-use encoded configuration reaches the guest through a mode-0600 stdin transfer. Each boot appends a generation UUID to the stable slot prefix, avoiding stale-name registration conflicts while generation-fenced cleanup still binds deletion to the exact clone. Registration visibility and broker heartbeat waits are bounded, and diagnostic tails are credential- sanitized.

Keep the repository's workflow selector hosted until a live proof records the exact eligible job claim, expected labels, one-job completion, deregistration, VM destruction, and firewall-policy removal. A healthy local runner is not a fallback after labels have been assigned: GitHub cannot retarget a queued job. Never route pull_request_target or secret-bearing jobs to this pool.

ssh macpro                       # 192.168.86.43, Proxmox VE 9.2
qm list                          # 9xxx = pulp-linux-golden* (templates)
systemctl status 'pulp-ephemeral-pool@*'
journalctl -u 'pulp-ephemeral-pool@1' -f

The supervisor and its systemd unit are versioned here as tools/ci/proxmox-ephemeral-runner-linux.sh and tools/ci/pulp-ephemeral-pool@.service; the host copies live at /usr/local/sbin/ and /etc/systemd/system/. The script's GOLDEN= names the template in use — read it rather than trusting a number written down here, since re-baking a warmer golden mints a new id.

Organization runner-group configuration for the generic pool is per slot at /etc/pulp/linux-runner-group-<slot>.env, and only per slot. The unit loads no shared group file: one would eventually move every Restart=always instance into a restricted group and delete the repository-scoped capacity that release and operator dispatches depend on. Each numbered file must either set the reviewed group ID or set PULP_LINUX_RUNNER_GROUP_ID= explicitly for repository-scoped dispatch mode; a slot with no file stays repository-scoped. Migrating a host that still carries the pre-per-slot /etc/pulp/linux-runner-group.env: write a numbered file for every enabled slot first, reload and restart the services, verify each slot's registration scope, and only then delete the legacy shared file. Doing it in that order avoids both a no-capacity migration window and silently converting every slot to a group that does not admit an operator workflow.

Golden + disposable clone. The golden carries the dependency set, prebuilt Skia (external/skia-build/.../libskia.a), a warm ccache, the uncredentialed gh executable used by preamble/alias jobs, and the shared FetchContent source cache that setup.sh consults via PULP_SHARED_FETCHCONTENT_SOURCE_DIR. That last one is not optional: with it empty, every job re-clones three.js (~2.2 GB of history) before it can compile. Each job gets a linked clone (copy-on-write, ~28 s to boot), starts an ephemeral JIT runner, takes exactly one job, and the clone is destroyed. Nothing accumulates, so nothing needs cleaning — and the cache a job inherits cannot be poisoned by the job before it. This closes the reused-build-dir class outright, which matters because build.yml sets clean: false on self-hosted runners.

The supervisor publishes a root-owned per-generation lease while it owns a clone. pulp-ephemeral-reap.timer is the crash-recovery backstop: after one hour it considers only an ownerless Pulp slot, then requires the exact GitHub registration to be idle, one Runner.Listener --jitconfig, no worker or configuration process, and an empty _work. Execution rechecks the unchanged VM config under the VMID allocation lock, then fences dispatch by deregistering the runner, and logs each of the four steps as FENCE <vmid> n/4:

1. Read the runner and require busy=false.
2. DELETE the registration. GitHub refuses to delete a runner that is running a job, so a refusal leaves the clone for the next pass, and a success proves no job held it.
3. Require the runner to return 404, and require that no in-progress job in the repository names it. The second check guards against a 404 that only means the wrong id was read.
4. Re-probe the guest for no worker and an untouched _work, then destroy it.

A finished JIT clone (listener exited, registration gone) is reaped once the guest's identity matches the host's recorded generation. JIT runners delete .runner when their one job ends, so for such a clone the guest probe reads the name from the listener logs under _diag, and only when every log agrees on one name. Before this, every finished JIT clone failed with post-job guest identity does not match the host generation and stayed until someone removed it by hand.

A label fence cannot work here: a JIT runner's labels are read-only, so replacing them never removes a routing label. The supervisor's own shutdown uses the same deregistration fence, and hands a runner GitHub reports busy to its deferred cleanup. Missing, duplicate, unreachable, busy, or otherwise ambiguous evidence always preserves the VM. Run pulp-ephemeral-reap.sh without arguments for a non-mutating report. After a controller reboot leaves an onboot=0 clone stopped, recovery accepts only its one exact generation-bound GitHub registration in offline and busy=false state, deregisters that exact ID under the same VMID lock, and then destroys the clone. Online, busy, duplicate, or unreadable stopped-clone registrations remain preserved. An operator's explicit --keep disposition is generation-bound under /var/lib/pulp/ephemeral-runner-keep, so it survives a host reboot; a newly allocated generation clears only the old marker for its own VMID while holding the allocation lock.

One generic slot runs by default, pulp-ephemeral-pool@1.service; systemd restarting a slot is what provisions the next clone. One slot holds 8 GB while it waits idle, and the Windows VM (300, 10 GB) is sized to fit beside it: the governor admits both with about 9.5 GB to spare, and refuses a second Linux slot while Windows runs. Add @2 only when work is routed here, and check macpro-governor.sh status first.

Repository-scoped clones retain their deterministic network identities: 200..202 map to 192.168.86.251..253 and stable locally administered MAC addresses. Protected-role clones use the isolated 10.240.<VMID>.2/30 identities. Do not return the generic pool to random clone MACs. Each short-lived MAC retains a DHCP lease after its VM is destroyed, and normal CI volume can exhaust the LAN lease pool. The GitHub runner registration remains unique per invocation; stable network identity must not become a static Actions runner name.

Resource governance, mirroring the tiers in CLAUDE.md:

• Tier 0 — per-VM cores=4 cpulimit=4 cpuunits=50 balloon=0, hypervisor-enforced. cpuunits=50 is below the default so build VMs yield to the host; balloon=0 pins memory so a build is never squeezed mid-link.
• Tier 1 — /usr/local/sbin/macpro-governor.sh (status / can-start / can-start-new), versioned as tools/ci/macpro-governor.sh. The hypervisor reserve is derived from the host: one thread in six (at least 2) and one eighth of memory (at least 4 GB), which is 2 threads + 4 GB on macpro. MACPRO_GOVERNOR_* variables override a derived value. Disk is the third axis: it refuses while the pve/data thin pool is at 85% data or 75% metadata, or when the fill cannot be read. Every REFUSE line is followed by one because cpu|memory|disk: ... line per refusing axis, and the supervisor copies them into its journal. A refused supervisor waits ~270 s before exiting 75, so a full host retries about every five minutes instead of every 30 s. Each pool unit also carries a restart ceiling (StartLimitBurst=40 per hour); a slot that trips it stays failed until systemctl reset-failed. Memory is a hard limit; CPU allows 1.5x overcommit. That asymmetry is deliberate: an OOM mid-link yields a truncated object file that reads like a compiler bug, while CPU contention only costs time. Every clone is admitted through it, so nothing can oversubscribe the host.

Routing rollback: this provider-only change does not enable protected-event routing, so hosted Linux remains in effect. If a later routing change is active, remove its protected selector before stopping these role services. Existing operator dispatches keep their generic-pool rollback: unset PULP_LOCAL_LINUX_RUNS_ON_JSON and redispatch. Once a local job has been assigned, runs-on has no live fallback.

Registration uses a fine-grained PAT at /root/.config/pulp/secrets/gh-runner-pat (mode 600, root) with only Administration: read/write, minting a single-use JIT configuration per job. That host credential never enters a guest. Jobs that call gh authenticate with the short-lived GITHUB_TOKEN injected by Actions; the golden must not contain a persistent gh login in any supported config or credential store.

Routing the Linux advisory lanes to macpro

Three advisory Linux lanes can run on the self-hosted x86_64 host instead of GitHub's pool. Measured cost on hosted runners, per PR:

Lane	Variable	Hosted wait	Hosted run
GCC compile (core, Linux)	PULP_LOCAL_GCC_RUNS_ON_JSON	63.2m	11.3m
IWYU (Linux, Clang)	PULP_LOCAL_IWYU_RUNS_ON_JSON	4.2m	5.2m
Public headers standalone	PULP_LOCAL_HEADERS_RUNS_ON_JSON	9.7m	2.9m

About 96 job-minutes of hosted load per PR. None is a required check, so a red result here never blocks a merge — which is why they are the right lanes to move first.

Each falls back to its GitHub-hosted label when the variable is unset, so the workflow change is inert until a variable is set. Flip them one at a time and watch a full cycle: runs-on has no automatic fallback once a variable is set, so a lane pointed at a stopped pool queues indefinitely rather than erroring. Rollback is unsetting the variable.

gh variable set PULP_LOCAL_IWYU_RUNS_ON_JSON \
  --repo Generous-Corp/pulp \
  --body '["self-hosted","Linux","X64","pulp-build-linux-x64","pulp-host-macpro"]'

A label match is not enough: check the runner group first. The Mac Pro pool registers into a restricted organization runner group, and a group that does not list a workflow will never assign a job to it. The label set above is a subset of what the pool advertises, so the job looks routable and then queues forever. The two installed pool roles and what their groups admit:

Pool role	Extra label	Group	Group admits
proxmox-trusted-ephemeral-runner-linux.sh	pulp-auto-linux-x64	pulp-trusted-build	build.yml, vellum-freeze-check.yml, version-skill-check.yml, each at refs/heads/main
proxmox-pr-safe-ephemeral-runner-linux.sh	pulp-pr-safe-linux-x64	pulp-pr-safe-build	pr-safe-linux.yml@refs/heads/main

Neither group lists iwyu.yml, header-self-contained.yml, or gcc-compile-gate.yml, so setting one of the three variables above routes that lane into a group that cannot admit it. Before flipping one, confirm the consuming workflow is in the group's selected workflows:

ghapp api orgs/Generous-Corp/actions/runner-groups/3 --jq '.selected_workflows'
ghapp api orgs/Generous-Corp/actions/runner-groups/3/runners --jq '.total_count'

Start with IWYU: it is the cheapest of the three, so a mistake costs the least. Check capacity first with ssh macpro /usr/local/sbin/macpro-governor.sh status. Slot count is live host state, not a constant: individual @N slots get masked and unmasked as the fleet is worked on, and every routed lane queues behind Linux (x64) for whatever slots exist before it queues against GitHub. Read the count at the moment you flip rather than trusting any number written here:

ssh macpro 'systemctl list-units "*ephemeral-pool@*" --all'

What a merge group classifies against

classify decides whether a merge group needs the native matrix by diffing the group against main's tip, not against merge_group.base_sha. With entries queued ahead, base_sha is the previous speculative group, and a passing group lands every entry ahead of it, so the decision has to cover the whole batch. tools/scripts/resolve_classify_base.py walks the first-parent chain from base_sha down to merge_group.base_ref; a chain it cannot walk fails closed to the native build. Receipt reuse (protected-receipt-reuse) is refused for a group whose first parent is not that main tip, because a receipt certifies one pull request's tree on main and the verifier must load from protected code.

The merge-group HOLD check (tools/ci/merge_group_hold_check.py) and the queue circuit breaker (tools/ci/queue_breaker.py) scan the same batch. The classify job's "Resolve the merge group's main base" step runs the protected base branch's copy of the resolver once, before any queue entry's code runs, and the HOLD check, the breaker and the classification all take its answer, so a HOLD, a fix or a queue-unblock on an entry further ahead in the batch counts for the group that lands it. When the base does not resolve, classify builds natively, the HOLD check fails the group, and a live breaker fails the group too, rather than narrowing either scan back to base_sha.

The preamble slot

build.yml's preamble jobs (resolve-provider, classify, protected-receipt-reuse) finish in seconds but gate every merge-group run. Pointed at the build labels, they queue behind hours-long Linux builds. macpro therefore runs one dedicated tiny slot for them:

Unit	pulp-ephemeral-pool@preamble.service
Profile	tools/ci/linux-runner-group-preamble.env → /etc/pulp/linux-runner-group-preamble.env (mode 0600)
Clone	VMID 203, 1 core, 2048 MB, golden 9005, 192.168.86.254, MAC 02:50:55:4c:50:03
Labels	["self-hosted","Linux","X64","pulp-preamble-macpro"]

The profile reuses the generic supervisor. That gives it the same lease, generation description, reaper coverage, and deregistration on shutdown as the build slots. Three supervisor settings make it small and disjoint:

• TARTCI_PROXMOX_CORES / TARTCI_PROXMOX_MEMORY_MB set the clone size (default 4 cores, 8192 MB).
• TARTCI_PROXMOX_GUEST_INDEX_BASE sets the VMID the LAN identity is indexed from (default the slot's own CLONE_BASE). The preamble profile sets it to 200, so VMID 203 takes .254 instead of colliding with slot 200's .251 and MAC.

The slot carries neither pulp-build-linux-x64 nor pulp-host-macpro, so no build can take it. Point a workflow at it only through PULP_PREAMBLE_RUNS_ON_JSON. Changing that variable is a repository settings change; check that the slot is registered first:

ghapp api repos/Generous-Corp/pulp/actions/runners \
  --jq '[.runners[] | select(any(.labels[]; .name == "pulp-preamble-macpro"))] | length'

Its 2 GB fits beside one build slot and the 10 GB Windows VM, but a second build slot does not, so check macpro-governor.sh status before enabling one.

The Windows x64 slot

macpro also runs one Windows x64 runner. It owns a single VM and resets it to a golden snapshot before and after every job, instead of cloning: a Windows guest with the MSVC toolchain is a 130 GB disk set, so a clone per job would exhaust the thin pool. Nothing a job writes reaches the next job, and a job cannot change the golden.

Unit	pulp-windows-ephemeral-pool@1.service
Supervisor	tools/ci/proxmox-ephemeral-runner-windows.sh → /usr/local/sbin/pulp-windows-ephemeral-runner.sh
Profile	/etc/pulp/windows-runner-1.env: PULP_WINDOWS_VMID=300, PULP_WINDOWS_GOLDEN_SNAPSHOT=<name>
VM	VMID 300, 4 cores, 9216 MB, Windows Server 2022 evaluation, LAN DHCP
Labels	["self-hosted","Windows","X64","pulp-build-windows-x64","pulp-host-macpro"]
Routing	tools/scripts/windows_runner_selector.py --route; PULP_LOCAL_WINDOWS_X64_RUNS_ON_JSON overrides the default selector. Distinct from the ARM64 QEMU PULP_LOCAL_WINDOWS_* lanes

What runs on it

Windows x64 jobs default to this lane, and GitHub-hosted Windows runs only when it is opted into. No pull request reaches the lane. The selector checks for a fork first, before any selector or variable, and a pull request whose head repository is missing counts as a fork. The two pull-request-triggered jobs below (control-shipping-native and coverage) never send a pull request to the lane either: without the opt-in they leave Windows x64 out. Each routed workflow resolves its runner with windows_runner_selector.py --route, which emits runs_on, route (self-hosted, hosted or skip) and route_reason:

Workflow	Job	Lane by default on	Hosted when
cross-platform-check.yml	Windows build + test	the nightly, a dispatch	opted in
software-gpu-proof.yml	Windows WARP proof	the nightly, a dispatch	opted in
control-shipping-native.yml	windows-x64 leg	the nightly, a dispatch	opted in (the only way a pull request runs it; otherwise deferred)
coverage.yml	Coverage report (Windows, Clang)	a schedule while the lane is idle, a dispatch	opted in (the only way a pull request runs it)
android.yml	windows-x64 row	the nightly, a dispatch	opted in (which also adds it to a push)
freshness-check.yml	Build validation, windows-x64	a tier-2 dispatch	opted in
build.yml	Windows (x64)	a dispatch	opted in (windows-2022)

The default selector is ["self-hosted","Windows","X64","pulp-build-windows-x64"]. It leaves out pulp-host-macpro, so a second Windows x64 host carrying the same capability label would serve the same jobs.

The lane is one VM that runs one job at a time, and nothing on it is a required check (tools/scripts/test_windows_x64_routing.py pins both). So a scheduled coverage leg is bounded with --max-backlog 1: while another Windows x64 job is queued or running, it is left out instead of queued. A pull request's control-shipping-native windows-x64 leg, left out unless hosted is opted into, is named in the consolidated evidence (deferred_legs, complete: false). A dispatch always queues. The backlog probe counts jobs carrying pulp-build-windows-x64 in other queued and in-progress runs; when it cannot read them it counts zero and the job queues.

Three switches choose hosted:

• -f allow_hosted_windows=true on a build.yml dispatch;
• PULP_ALLOW_HOSTED_WINDOWS=1, which also lets a push run build.yml's and android.yml's Windows legs;
• PULP_LOCAL_WINDOWS_X64_RUNS_ON_JSON set to a hosted label such as '"windows-latest"', which turns the lane off: each routed job goes back to its own hosted image and nothing else changes. Use this one to take the lane out of service.

Some jobs stay hosted on purpose. windows-cli-compile.yml mirrors the release build on windows-latest, and its push verdict feeds auto-release.yml's pre-tag check. The three build.yml compile gates (MSVC release-path, MIDI Services, BLE) track windows-latest's moving toolchain, and they are already opt-in. release-cli.yml builds the release itself. The windows-arm64 legs have no self-hosted x64 equivalent.

The golden must carry every shell a routed job uses. shell: pwsh steps need PowerShell 7 (pwsh), not only Windows PowerShell; without it those jobs fail at their first pwsh step.

Each invocation:

1. Waits while any registration named pulp-win-ephemeral-<vmid>-* is busy (exit 75), and deregisters idle leftovers through the same fence as the Linux supervisor.
2. Stops the VM and rolls it back to the golden snapshot.
3. Asks macpro-governor.sh can-start <vmid> for admission. A refusal waits about five minutes, then exits 75.
4. Boots the VM, reads its address from the QEMU guest agent, and connects over SSH as an administrator. The guest's host key is pinned under the alias pulp-windows-vm-<vmid> in /var/lib/pulp/windows-runner/known_hosts.
5. Mints a repository-scoped JIT configuration with the existing runner credential, writes it into the guest over stdin, and runs one job.
6. Fences the registration and rolls back again.

A runner that stays online and idle for 45 minutes (PULP_WINDOWS_RUNNER_IDLE_RECYCLE_SECONDS, default 2700; 0 turns it off) is deregistered through the same fence and the invocation exits 0, so systemd registers a fresh one. A JIT session that idles for hours can stay online while GitHub stops assigning it jobs: on 2026-10-25 a matching job queued for hours beside an idle registration. The fence deletes the registration before the session is stopped, so a runner that took a job in that window refuses the delete and keeps it.

The VM belongs to the slot. Anything else that stops it (qm shutdown 300 from another tool or session) kills the running job. The supervisor watches the VM while a job runs. When it stops without the supervisor stopping it, the supervisor ends the job wait at once, then fences and resets as usual. It reports what stopped the VM in three places:

• the journal line VM 300 stopped externally during the job: ...;
• syslog, tag pulp-windows-runner;
• one line per event in /var/lib/pulp/windows-runner/external-stops.log.

Each report names the Proxmox stop task with its user and time, and every address that opened a root SSH session in the two minutes before it:

ssh macpro tail /var/lib/pulp/windows-runner/external-stops.log

It registers at repository scope, like the generic Linux pool, so it serves only workflows that name its labels. Labels reserved for automatic PR routing (pulp-auto-*, pulp-pr-safe-*) are refused.

Memory is sized so the slot fits beside both Linux build slots and the preamble slot: 8192 + 8192 + 2048 + 9216 MB is within the governor's 27968 MB, so a cycling Linux slot is always re-admitted. Do not raise the VM's memory without rechecking macpro-governor.sh status.

What the golden snapshot carries

• the toolchain (VS Build Tools, CMake, Ninja, Git with bash, Python with a python3 alias, LLVM clang++, which setup.sh probes for a C++20 compiler);
• the x64 Actions runner at B:\actions-runner, on the second disk, which Defender excludes;
• root@macpro's key in C:\ProgramData\ssh\administrators_authorized_keys;
• every root-folder scheduled task disabled, and Windows Update set to NoAutoUpdate, so the image cannot change itself during a job;
• long paths enabled, for Windows and for Git.

qm rollback reverts every disk of the VM, the 30 GB B: disk included: a marker file written to both C: and B: before a rollback was gone from both after it. Recheck that if a disk is ever added outside local-lvm, because a disk on storage without snapshots would carry state from one job to the next.

What the golden removed from the earlier interactive VM

VM 300 was used interactively before it became a CI slot. Its state at that point is the snapshot pre-ci-dedicate-20261022. To recover anything below without disturbing the CI slot, make a full clone from that snapshot (qm clone 300 <new-vmid> --snapname pre-ci-dedicate-20261022 --full, after checking thin-pool room with macpro-governor.sh status). The golden changed only this:

• Scheduled tasks disabled (none deleted): CaptureCurrent, ci-setup, CodexIntelAudio20261018, DialogOnce, DismissCleanX64UI20261015, DismissReaperModalX6420261015, DismissReaperModalX6420261015b, Enum4104, Enum5512, EnumOnce, GuiProbe, GuiProbe2, GuiTest, IntelEnumNow, IntelTest, LuaProbe, MicrosoftEdgeUpdateTaskMachineCore, MicrosoftEdgeUpdateTaskMachineUA, OpenActionsX6420261015, OpenCurrentProject, pulp-build, PulpCloseDialogs, PulpReaperInteractive, PulpSourceRefreshWeekly, PulpSpectrX64Smoke, PulpTsconTemp, PulpVS18UpdateWeekly, PulpVSUpdateWeekly, ScreenshotOnce, SpectrCurrentNormalOnce, SpectrCurrentProjectOnce, SpectrCurrentUIOnce, SpectrCurrentX64Diagnose20261017, SpectrCurrentX64Exact20261017, SpectrCurrentX64Exact20261017b, SpectrCurrentX64PackageSmoke, SpectrEnum, SpectrIntelLive, SpectrIntelLiveD0, SpectrIntelShot, SpectrNormalScanEditorX6420261015, SpectrReaperCurrentProject, SpectrReaperInteractive, SpectrReaperWetProject, SpectrRunLua2X6420261015, SpectrRunLua3X6420261015, SpectrRunLuaX6420261015, SpectrTaskProbe, SpectrX64CreateDry20261015, SpectrX64CreateWet20261015, SpectrX64DummyRender20261015, SpectrX64EmptyRender20261015, SpectrX64OfflineRender20261015, SpectrX64RenderDismiss20261015, SpectrX64RenderPid20261015, SpectrX64SmokeInteractive.
• Deleted from C:: C:\pulp-probe (an earlier build trial, 19 GB).
• Deleted from B: (PulpBuildCache), every top-level entry: the Pulp FetchContent caches (pulp-fetchcontent, pulp-fc-cache-20261015), the Pulp and Spectr build, package, install-backup, REAPER and diagnostic trees dated 2026-10-13 to 2026-10-17, vst3sdk, skia-m153-x64-20261015, receipts, temp, and the loose .cmd/.ps1/.log/.zip/.patch/.bundle/.dmp files beside them.
• Kept: every user account, including ci and pulp-rdp, and their profiles; C:\ci-setup; the Spectr trees on C:; the installed toolchains.

Refreshing the golden

Because nothing updates itself, the golden goes stale on purpose. To refresh it:

ssh macpro systemctl stop pulp-windows-ephemeral-pool@1    # waits for a running job
ssh macpro 'qm rollback 300 <current-golden> && /usr/local/sbin/macpro-governor.sh can-start 300 && qm start 300'
# Update in the guest over SSH: VS installer update, runner version, tools.
# Verify: cl, clang++, cmake, ninja, python3, bash and git all answer, and
# B:\actions-runner\bin\Runner.Listener.exe --version prints the new version.
ssh macpro 'qm shutdown 300 && qm snapshot 300 ci-runner-golden-<YYYYMMDD> --description "<what changed>"'
# Point the slot at it, then restart.
ssh macpro "sed -i 's/^PULP_WINDOWS_GOLDEN_SNAPSHOT=.*/PULP_WINDOWS_GOLDEN_SNAPSHOT=ci-runner-golden-<YYYYMMDD>/' /etc/pulp/windows-runner-1.env"
ssh macpro systemctl start pulp-windows-ephemeral-pool@1

Use a new dated name every time and keep the previous snapshot until a job has run green on the new one; rolling back to it is the recovery. The evaluation license expires 180 days after installation (slmgr /dlv shows the time left), so install a fresh evaluation into a new golden before then.

Windows build caches: what the workflows read, and the operator runbook

Both Windows lanes keep nothing between jobs (the macpro slot rolls every disk back; the m5s ARM64 lane discards its overlay), so a cache only helps when it is baked into the golden or served by the host. The workflows already read the three hooks below; each is inert until a runner sets it:

Variable (runner job environment)	Read by	Unset
PULP_SKIA_CACHE_ROOT	build.yml configure (PulpSkiaCache.cmake) and the cross-platform-check.yml Windows Skia step (fetch_skia_for_release.py --cache-root)	build.yml uses ~/.cache/pulp/skia; cross-platform-check fetches into external/skia-build as before
PULP_SHARED_FETCHCONTENT_SOURCE_DIR	setup.sh (fetchcontent_cache_root)	%LOCALAPPDATA%\Pulp\fetchcontent-src
PULP_CCACHE_REMOTE_STORAGE	tools/ci/ccache_remote_storage.py, which sets CCACHE_REMOTE_STORAGE	nothing is written; ccache, when present, uses only its job-local cache

Both Windows jobs build with Ninja and cl from the developer environment (tools/ci/windows_build_env.ps1), because the Visual Studio generator ignores the compiler launcher: under it ccache counted cacheable 0, against 2394 for the same build under Ninja. The script uses a ccache already on PATH, installs one with choco when choco exists, and otherwise builds without a compiler cache and says so (the macpro golden has no choco).

ccache_remote_storage.py gives write access only to code already on main: push and schedule on refs/heads/main, workflow_dispatch on refs/heads/main, and merge_group. Every other job gets the same URL with ccache's |read-only attribute. That attribute is honored by ccache, not enforced by the server, so it only separates honest jobs. Today the local Windows lanes run only operator dispatches. If pull-request code is ever routed to them, give PR jobs a separate pool and a storage endpoint they cannot write, as the Linux pulp-trusted-* and pulp-pr-safe-* pools are separated.

Operator runbook. Apply this when the fleet freeze lifts. None of it is applied yet, and every step is a host or golden change.

Golden prerequisites, as the x64 golden stands today: it has Windows PowerShell 5.1 but no PowerShell 7 (a shell: pwsh step fails with pwsh: command not found), and no choco. The workflows are written for that: tools/ci/windows_build_env.ps1 runs under shell: powershell and needs neither. Any new Windows step for these lanes must use shell: powershell or bash unless the golden gains PowerShell 7. Tools the jobs need (ccache below) must be baked in, not installed per job.

1. Bake the Skia/Dawn generation and the source prime into the x64 golden. Follow "Refreshing the golden" above. While the VM is booted from the current golden, in the guest as ci:

git clone --depth 1 https://github.com/Generous-Corp/pulp.git C:\pulp-bake
cd C:\pulp-bake
$env:PULP_SHARED_FETCHCONTENT_SOURCE_DIR = 'C:\pulp-cache\fetchcontent-src'
bash ./setup.sh --ci --deps-only
python tools/scripts/fetch_skia_for_release.py windows-x64 --cache-root C:/pulp-cache/skia
python tools/scripts/fetch_skia_for_release.py windows-x64 --cache-root C:/pulp-cache/skia --validate-only

Then add these lines to B:\actions-runner\.env. The runner exports its .env to every job.

PULP_SKIA_CACHE_ROOT=C:/pulp-cache/skia
PULP_SHARED_FETCHCONTENT_SOURCE_DIR=C:/pulp-cache/fetchcontent-src

Remove C:\pulp-bake afterwards. Install ccache in the guest too (the portable release zip from ccache's GitHub releases, unpacked into a directory on the machine PATH). Check thin-pool room with macpro-governor.sh status first: the source prime is about 2.2 GB and the Skia generation about 0.4 GB. Snapshot under a new dated name and switch PULP_WINDOWS_GOLDEN_SNAPSHOT, as in "Refreshing the golden".

Because the caches live on a snapshot disk, a job can read them but anything it writes there is rolled back. A pin bump therefore costs a fetch on every job until the next bake, never a stale build: the Skia generation is keyed by the asset sha256 and setup.sh re-fetches a source whose ref changed.

1. Serve a shared ccache from the lane host. On macpro, run an HTTP server that supports ccache's http remote storage (GET, PUT and HEAD; nginx with dav_methods PUT; create_full_put_path on; is enough). Root it on host storage outside local-lvm snapshots, listen only on the address the slot's VM reaches (pin the VM's DHCP lease), and prune it on a timer, since the server does not evict. Then add PULP_CCACHE_REMOTE_STORAGE=http://<host address>:<port>/pulp-windows-x64 to B:\actions-runner\.env in the golden. Use one path per architecture and toolset.

2. The m5s ARM64 lane needs the same golden steps on .shipyard/vm-image.windows.toml's image (vcvarsarm64.bat is selected automatically), plus a reviewed change to that manifest's [cache] mode = "guest-local" before any shared storage is attached. It also needs its qemu-runner-windows LaunchAgent reinstalled: it is not running today.

3. Prove it. Use at most three cross-platform-check.yml dispatches with windows_only=true and the macpro selector (no macOS gate VM is taken):

4. Skia: the log shows OK: immutable Skia cache generation ready at C:/pulp-cache/skia/... and no Downloading ->.
5. Sources: there are no Priming shared ... source cache lines.
6. ccache: run the same commit twice, cold then warm. The second shows ccache remote storage: read-write and a hit rate near the cacheable count. The Build step needs a compiling main.

Windows runs nightly, not per merge

Windows is billed at 2x on GitHub-hosted runners and gates nothing — no Windows context appears in main's required checks, so the merge queue never waits for it. Measured across 12 runs it was roughly 90% of billable Actions spend, and with max_entries_to_build=2 each merge cycle ran it twice.

It now runs on schedule and workflow_dispatch only. Coverage did not move to nobody: cross-platform-check.yml already builds and tests Windows nightly, and its tracking-issues job find-or-creates a per-platform issue on failure, reopens a closed one, and auto-closes it on recovery. So a Windows regression is caught, filed as a work item, and picked up deliberately — instead of consuming queue capacity that the required checks are waiting behind.

Need Windows on a specific change before the nightly? Dispatch it:

ghapp workflow run build.yml --ref <branch>

A build.yml dispatch always includes the macOS leg, so it also takes a macOS gate VM; its Windows leg runs on the self-hosted Windows x64 lane. To run only the full Windows suite, dispatch the nightly instead. Its Windows job takes the lane by default and an optional runner selector, which pins a specific runner without touching the gate hosts:

ghapp workflow run cross-platform-check.yml --ref <branch> -f windows_only=true \
  -f windows_runner_selector_json='["self-hosted","Windows","X64","pulp-build-windows-x64","pulp-host-macpro"]'

The schedule passes no inputs, so the nightly runs on the self-hosted Windows x64 lane, or on windows-latest when hosted is opted into (see "What runs on it"). A run with either input set never opens or closes the per-platform tracking issues, because it did not measure the nightly's configuration.

A small Resolve Windows runner job checks the selector before any runner is chosen. It runs tools/scripts/windows_runner_selector.py, which refuses anything that is not a Windows selector, such as macOS or Linux labels, the gate pool, or the preamble slot, so a pasted selector cannot put a Windows build on a gate VM. Such a dispatch also gets its own concurrency group per run, so a long proof on main never holds back a pending nightly.

This is a deliberate trade: up to ~24 h of latency on a Windows regression, in exchange for merge-queue capacity and spend. Revisit if Windows parity becomes an active workstream rather than a background one.

The FetchContent cache had to point at a real path

build.yml restored and saved three FetchContent paths and none of them ever populated — a different reason on each platform. On Linux the cached ~/.cache/Pulp/... did not match CMake's lowercase ~/.cache/pulp/...; on Windows the cached path carried an extra Cache/ segment versus $LOCALAPPDATA/Pulp/fc; and off Windows the sources never left <build>/_deps anyway, because pulp_configure_fetchcontent_base_dir returns early unless WIN32 (it is a MAX_PATH workaround for MSBuild, not a cache).

The fix caches <build>/_deps — where FetchContent already writes — rather than relocating it. Do not "improve" this by setting FETCHCONTENT_BASE_DIR to a path outside the build tree. PulpWclap.cmake and PulpWebUi.cmake resolve CHOC from <root>/build*/_deps/choc-src; moving it produces ERROR: configured CHOC source not found under build-macos/_deps and fails the required macOS gate. That was tried and reverted.

Worth it because three.js is a 2.2 GB git clone, fetched whenever PULP_BUILD_TESTS and PULP_ENABLE_GPU are both ON — the default on pull_request and merge_group. Measured on an ephemeral Linux runner with an otherwise identical tree: 414 s cold configure against 119 s warm.

If a dependency pin changes and a stale cache is suspected, the key includes hashFiles('setup.sh'); bump that or clear the Actions cache to force a refetch.

Primary: Shipyard

Shipyard is Pulp's primary CI tool. It delivers exact SHAs via git bundles, runs your build/test commands on each platform, and gates merges on per-SHA evidence.

The agent-capability manifest gate compares a branch with protected capability history through origin/main and intentionally fails closed if that ref is missing. A depth-1 pull_request checkout contains only GitHub's synthetic merge commit, so build.yml fetches the event-pinned pull_request.base.sha into that local ref before CTest. This must be the event SHA, not a moving fetch of current main. Shipyard PR validation instead arrives through workflow_dispatch, whose payload has no protected base SHA, so that path retains the explicit protected main fetch.

The Shipyard mac target (.shipyard/config.toml, [targets.mac] + [validation.default]) is a local Debug build with examples ON plus the full ctest run, on the Mac that runs shipyard. It is opt-in (default = false): shipyard pr and shipyard ship push the branch, open or find the PR and arm auto-merge, then leave the verdict to the required GitHub checks ([governance] required_status_checks) and print validation: delegated. Nothing is queued and no ship-state is written, so a missing ship-state for a Pulp PR is expected, not an orphan. The lane is never silently absent: pulp status, shipyard status, shipyard ship-state list/show, shipyard landing and shipyard doctor all print mac: opt-in, not run (GitHub required checks decide), and Shipyard never probes it, counts it in landability, or reports it as a validation gap unless it is requested. Request the lane explicitly when you want it:

shipyard pr --target mac                  # push/open/arm, then also run the local lane
shipyard ship --pr <n> --target mac       # run it on an existing PR
shipyard run --targets mac                # validate the current branch only

It was made opt-in on 2026-09-30 (decisions contract row #9): after 2026-09-02 it passed 1 of 147 runs, failed 64 of 64 in the week before the change at about 42 host-hours on one Mac, and PRs merged on the required checks regardless. Opt-in needs Shipyard 0.242.0 or later, which fleet hosts receive through Shipyard's own auto-update; an older binary ignores the key and runs the lane as before.

When requested, its test step is ctest ... --repeat until-pass:2 --label-exclude "validation|slow|performance|bench|quality-lab" — it excludes the long slow tests, the example plugins' validation format-validators (reported by the path-filtered, currently advisory example-validation lane), and the relative-timing / CPU-budget / benchmark tests (performance|bench|quality-lab), and retries a single flake once so timing-flakes don't redden the gate. The perf/ratio tests are excluded (2026-07-21, mirrors build.yml) because they tolerate steady load but flake under the load variance of the Studio's 2 concurrent build VMs (cap=2) — a perf gate can't live on a cap=2 runner; it belongs in a dedicated cap=1 nightly/perf lane. The full lane model — what runs where, the label taxonomy, and how to route a new test — is docs/guides/test-lanes.md.

The same profile resolves its CMake interpreter through tools/ci/find_python311.py and passes the result as Python3_EXECUTABLE. Apple's command-line tools still expose Python 3.9, which can configure the project but cannot run the tomllib-based decisions-contract tests; the selector uses an installed 3.11+ interpreter or an existing uv 3.12 runtime and fails before the hour-long Debug build if neither exists.

./tools/install-shipyard.sh              # install pinned version
./tools/install-shipyard.sh --status     # compare installed vs pinned
shipyard run --targets mac                # validate current branch on the opt-in local lane
shipyard pr                               # create/track the PR and arm auto-merge; required checks decide
shipyard cloud run build <branch>         # dispatch to Namespace
shipyard rescue <PR>                      # recover a wedged PR's queued runs (dispatches a branch-tip replacement; re-run a required pull_request run instead)
shipyard runner watch --kill-hung-workers # prevent self-hosted runner wedges
shipyard update --check --json            # report installed vs latest

A local main-refresh push honours refresh_branch

[merge] refresh_branch = "only-if-conflicting" in .shipyard/config.toml governs Shipyard's own ghapp update-branch calls. A local git merge origin/main && git push moves the PR head the same way, cancelling the in-flight required macos run (build.yml cancels superseded PR runs), while the merge queue re-tests the merge result anyway. The pre-push hook runs tools/scripts/refresh_push_check.py before every other gate, including under PULP_SKIP_PREPUSH=1, so those pushes are visible:

Setting	Effect
PULP_REFRESH_PUSH_POLICY=warn (default)	warn on a pure refresh of an open PR with no reason to refresh
PULP_REFRESH_PUSH_POLICY=refuse	fail the push instead; PULP_ALLOW_REFRESH_PUSH=1 allows one
PULP_REFRESH_PUSH_POLICY=off	log only
PULP_REFRESH_PUSH_LOG	log path (default ~/.local/state/pulp/refresh-pushes.jsonl)

A refresh is justified, and never warned about, when the PR is CONFLICTING, a required check on its head is failing, or the queue removed it at its current head. A push counts as a pure refresh only when every new branch commit is a merge with a parent on origin/main whose tree equals Git's clean automatic merge; a conflict-resolution merge is logged but never warned about. Pushes with no merge from main cost two git rev-list calls and write nothing. The GitHub lookup (one query, plus a required-ness query only when a check has failed) shares a 5 s budget and fails open. Each log line is one JSON object: ts, repo, branch, pr, head_before, head_after, main_merges, other_commits, pure_refresh, mergeable, failing_required, ejected_at_head, gate_in_flight, lookup, policy, mode, decision, via.

Gate cost per merged PR

.shipyard/config.toml [metrics.gate_cost] pins the repository, workflow (build.yml) and required gate job (macos) for shipyard metrics gate-cost, so the proxy needs no flags and two people get the same answer:

shipyard metrics gate-cost --since 7d

It reports gate runs and gate-minutes per merged PR, merge-queue batch fullness, and receipt reuse for the macos target, each with its sample size and named gaps. A flag overrides the matching key for one run.

Runner timing metrics

Pulp does not store CI timing history in the Pulp CLI or MCP server. When a checkout uses Shipyard, and optionally tartci for disposable local VMs, Shipyard owns the timing database and query surface:

• Shipyard can import GitHub Actions job timings and local command evidence.
• tartci can optionally emit per-VM runtime records for macOS, Linux, and Windows VM lanes: boot/setup/run/cleanup durations, labels, host, provider, golden/cache hints, outcome, and failure class.
• Shipyard imports those tartci records into its local metrics store and exposes agent-readable summaries, slowest lanes, trend/drift checks, comparisons, and placement advice.

This is mainly for agents watching Pulp CI over time. It gives them enough history to answer "is this runner behaving normally?", "did boot/build time regress?", "which lane should I monitor next?", and "is this worth investigating or just within the usual range?" Humans can use the same commands for high-level platform comparisons, but no observability service is required.

The shipyard metrics commands require a Shipyard build that includes the metrics subcommand, which every Shipyard since v0.81.4 provides; the pin in tools/shipyard.toml is newer, so no separate binary is needed.

# Enable VM runtime records on tartci hosts or LaunchAgents.
export TARTCI_RUNTIME_MEASURE=1
export TARTCI_RUNTIME_GH_ENRICH=1

# Inspect tartci's local VM timing records.
tartci runtime recent --repo Generous-Corp/pulp --limit 20 --json
tartci runtime summary --repo Generous-Corp/pulp --json

# Import both GitHub Actions and tartci VM timing into Shipyard's metrics store.
shipyard metrics import github --repo Generous-Corp/pulp --limit 50 --json
tartci runtime export --repo Generous-Corp/pulp --since-days 14 \
  | shipyard metrics import tartci --json

# Agent-friendly queries.
shipyard metrics summary --project pulp --json
shipyard metrics slowest --project pulp --limit 20 --json
shipyard metrics watch --project pulp --since 14d --json
shipyard metrics advise --project pulp --json

Use the Shipyard and tartci docs for setup details; this guide only records how Pulp expects agents and contributors to consume the optional integration. Without tartci, shipyard metrics import github and manual/command metrics still work for GitHub-hosted or SSH-backed CI lanes.

Build-speed scorecard

shipyard metrics import github keys each self-hosted job by its ephemeral runner name, so a per-host p50 from it is a p50 over one-job "hosts", and it records no step timings. tools/scripts/build_speed_scorecard.py ingest fills that gap for the required macos gate: it records each gate job against its PHYSICAL host (m1/m3/m5, from the runner name) plus per-step and queue rows, the merge queue's enqueue→merged and PR opened→merged latency, and merge_group run outcomes. Step, queue and latency rows go to their own projects so they never inflate pulp's worker-minute totals:

Project	Targets
pulp	macos-gate/<event>, macos-gate/merge_group/receipt-reused (hosted placeholder), macos-gate/merge_group/placeholder/{receipt-reused,skip-safe,unknown} (what the placeholder's annotation says it stood for), macos-gate/<event>/no-runner-cancel (cancelled before any runner took it; duration = the wait), release/<workflow> (release-lane jobs that ran on a gate runner)
pulp-gate-steps	macos-gate/<event>/{queue,Configure,Build,Test,Fast tier,SDK contract}, release/<workflow>/queue
pulp-merge-queue	pr/enqueue-to-merged, pr/last-enqueue-to-merged, pr/open-to-merged, pr/gate-minutes (self-hosted gate minutes one merged PR cost, PR heads + merge groups, every conclusion), pr/ejected/<reason> (one removal from the queue before merging; duration = queue time it threw away), merge-group-run

Ingest is idempotent (rows already in the store are skipped by external id), so it can run on a schedule. "Is this build slower than usual on this host?" is then shipyard metrics watch --project pulp-gate-steps --json.

# Backfill (first run) or top up; run from a Pulp checkout so ghapp resolves.
python3 tools/scripts/build_speed_scorecard.py ingest --since 14d

# The scorecard: local build, PR pipeline, fleet. Add --json for machines.
python3 tools/scripts/build_speed_scorecard.py report --since 7d

# Before/after against the recorded baseline (rendered by bench_diff.py).
python3 tools/scripts/build_speed_scorecard.py report --since 7d \
  --baseline planning/bench/build-speed/2026-09-23/baseline.json

# The local section needs an existing Ninja build dir; it never builds one.
python3 tools/scripts/build_speed_scorecard.py report --build-dir build --runs last

# Gate job and step p50/p90 either side of one instant (a merge, an incident's
# end), within the --since window, plus time to merge and gate cost.
python3 tools/scripts/build_speed_scorecard.py report --since 2026-09-17 \
  --split 2026-09-24T13:12Z --no-local --no-fleet

# A clean after-window that skips a rollout and stops at a fixed end.
python3 tools/scripts/build_speed_scorecard.py report --since 2026-09-17T13:12Z \
  --split 2026-09-24T13:12Z --after-from 2026-09-27T12:00Z --until 2026-09-29T12:00Z \
  --no-local --no-fleet

Wall-clock waits mostly track load, so they cannot say whether a change worked. proxies is the before/after verdict instead: for each mechanism a change targets it counts that mechanism directly from job labels and runner names, run cancellations and head-SHA changes, PR timelines, failed-job logs, and each host's tartci supervisor events (~/.tartci/state/macos-fleet/<lane>/events.jsonl, read over ssh with ls/cat only). Every row names its source and n, carries a control on the same instrument that must be non-zero (a zero control prints INSTRUMENT BLIND and exits 3), and says "insufficient sample" below 20 units a side instead of a percentage. Wall time is printed beside the rows as context. --exclude START/END drops an infrastructure incident from both sides; a link error inside it is classified infra_host_cache. Raw inputs are cached (~/.cache/pulp/build-speed/proxies.json), so --no-collect re-renders with other windows without another sweep.

python3 tools/scripts/build_speed_scorecard.py proxies --since 2026-09-17T13:12Z \
  --split 2026-09-22T00:00Z --after-from 2026-09-27T12:00Z --until 2026-09-29T12:00Z \
  --exclude 2026-09-26T08:23Z/2026-09-27T01:12Z

--split also renders a "Time to merge and gate cost" table: PR open→merged and first/last enqueue→merged p50/p90, merged PRs per day and hour, PRs per merge-queue push, gate-minutes per merged PR, cancelled gate-minutes (per day and per merged PR), receipt reuse rate, merge_group run failure rate, ejections per 100 merged PRs (with reasons), gate jobs cancelled before any runner took them, release-lane queue and gate-runner minutes, and gate queue p50 per host. Everything is normalised per day or per merged PR so windows of different length compare. Receipt reuse counts only placeholders whose annotation says a receipt was reused: a skip-safe merge group never needed the native gate and is reported separately. shipyard metrics compare is not a substitute here: it ignores --before and compares against all history before the split (Shipyard#618).

The drift section comes from shipyard metrics watch, which halves a fixed window, so a window that spans a change mixes both regimes. --split compares either side of an explicit instant instead. It reads the metrics list rows the pipeline section already fetched, because shipyard metrics compare splits only on whole days ago and cannot filter by target.

On a pull-request head that ran the full suite green, the receipt issue step (protected_merge_receipt.py issue) now announces its result: a protected receipt issued notice and job-summary line on success, or a protected receipt NOT issued warning carrying the issuer's reason (and a summary line) on failure. The step stays continue-on-error; the merge group then validates in full. A missing receipt with no such line is itself a defect to report.

The gate's Build step keys the iOS compile gate on its input digest: tools/ci/ios_gate_digest.py digests every tracked file the gate's three iOS configures can read (product sources, examples, CMake, dependency pins, the gate's own scripts, the Three.js bundler and its npm lockfile, the build wrapper, and the directories the root CMakeLists.txt adds on iOS) plus the Apple toolchain and node, and looks the digest up among passing runs (repository-wide ios-gate-ok-<digest> artifacts, 14-day retention). The gate is skipped only when a receipt for the same digest was written by a trusted gate run: this repository's build.yml, on a merge_group or same-repository pull_request run, with a marker naming the same digest and run (decisions contract row 23). One run in ten (by run id) and every schedule/push run is a control that runs the gate anyway, and every lookup failure runs it. The job summary says which happened (iOS compile gate: SKIPPED — input digest … passed in trusted run N). A lookup that could not be answered is reported as such, never as a miss: the summary reads receipt lookup failed: HTTP 401 …, the job carries an ios-gate-receipt-lookup warning, and the notice gains a lookup_error field, while no matching receipt for this digest and no trusted PASS receipt for this digest mean the lookup worked. Artifact archives redirect to blob storage, which refuses a forwarded token with 401; every receipt reader downloads through protected_merge_receipt.py's download_archive(), which drops the token on that cross-host hop. The pulp-ios-gate-shadow/v1 notice records skipped, control_run, would_skip or run, then ran_ok or ran_failed. The safety number is control_run/would_skip followed by ran_failed: it was 0 over the shadow window and must stay 0. If it is ever not, set the repository variable PULP_IOS_GATE_DIGEST_MODE=shadow (annotate only, no workflow edit) and add the missing input before switching back; off disables the lookup.

Script-driven ctests declare their inputs in test/ctest_script_inputs.json, generated by tools/scripts/script_test_inputs.py --build-dir <dir> --write (entry script, transitive local imports or sources, repository paths the command line and the scripts name) and kept fresh by the pr-fast script-test-inputs-drift check, which is diff-scoped: only drift in scripts the change under test touches fails it; drift from main's own movement is an advisory note (--check --full lists everything). The list is independent of the local configure: tests registered from examples/** are excluded (the gate builds with examples OFF, the scope recorded in docs/status/consumption-profiles.json), and an entry generated into the build tree is recorded as ${CMAKE_BINARY_DIR}/..., so any configured build of the tree writes the same bytes. The affected-test shadow selects a declared test only when one of its inputs changed; a test without an entry (nested cmake builds, tests with no command) keeps the fail-closed "any script surface changed" rule.

Compiled tests declare the checkout files they open at run time with pulp_test_data(<suite> PATHS ...) next to their registration (tools/cmake/PulpTestData.cmake); configure writes <build>/test/test-data/<exe>.inputs.json plus an index of every test executable's sources, and the generator folds them into the same list under executables (kind: compiled). An executable with a source that reads the checkout (names PULP_SOURCE_DIR, test/fixtures, or a definition pointing into the checkout) but no declaration is data: undeclared: the shadow selects it on every change, and script-test-inputs-drift fails a pull request that adds a new undeclared source, so the backlog only shrinks. A test that runs or loads another built target declares it with pulp_test_spawns(<test> <target>...), which adds the add_dependencies edge after every tool directory has been read. An inline if(TARGET <tool>) in test/cmake is evaluated too early and creates no edge. An executable whose sources call a process API with no such edge and no reviewed pulp_test_spawns(<test> NONE REASON "<why>") is spawns: undeclared, and the shadow selects it on every change. So is one whose code names a built program (a string such as "pulp-cpp") that no edge reaches and no reviewed pulp_test_spawns(<test> NOT_RUN <target>) covers, and one whose declared tool this configuration does not build.

The declarations come from a static scan, so a nightly read audit (.github/workflows/read-audit-nightly.yml, tools/ci/read_audit.py) measures them. On a GitHub-hosted Linux runner it builds the tree, runs every compiled test executable's ctest registrations under strace -f, and diffs the tracked checkout files they open, list or probe against the executable's declared inputs. It uses the selector's own prefix-or-glob match, and CMake files count as covered because a CMake change already reruns everything. Listing a directory above a declared input (Python's import scan of its own script directory) is covered, but opening a file there never is. A submodule gitlink is a directory, not a file. Each undeclared access is reported with its executable, its test and the program that made it. Before the tests, a control program goes through the same ctest and strace path. It reads one undeclared and one declared tracked file, and the run fails with no verdict unless only the undeclared one is flagged. The report names what it cannot see: macOS-only executables, targets that did not build, script-driven tests, and relative paths whose directory was unknown. Each run ends with one read-audit stage0: line and a stage0 block in the report. The verdict is clean only when the control was flagged, every declared executable this platform registers was audited, and no audited executable has a finding. incomplete means the run cannot vouch for itself. The nightly passes --fail-on-findings, so its exit follows that verdict: 0 for clean, 1 for findings, 2 for incomplete. A red nightly therefore means the streak of clean runs is broken. The streak counts clean cron days on main, and only the schedule speaks for a day: the counted run is that day's run with event schedule, and the day is the UTC date that run was created (GitHub fires the cron hours late, so the cron hour says nothing). A dispatch counts in its place only when that day's scheduled run was cancelled and the dispatch ran on the same head sha after it that day, which is the per-ref cancel-in-progress replacement. Every other dispatch is a canary precondition or a control and never counts. A day with no counted run is a gap and resets the streak like a finding. .github/workflows/read-audit-cadence-check.yml (tools/ci/read_audit_cadence.py) rebuilds that ledger every 30 minutes from each run's own stage0 verdict, writes each day's counted run, every run it did not count and why, and the streak to its summary and artifact, and opens a tracking issue naming any recent gap. A clean run's stage0 block also publishes covered, the executables it observed with no finding. The key manifest (tools/ci/executable_keys.py --audit-report) marks any executable outside that set always_run audit_uncovered, and every executable when no clean report is handed in. So a lane never skips a test whose reads no audit has observed, such as the macOS-only executables Linux does not register. The rederive command in .shipyard/config.toml passes the same report as --audit-report {audit_report}; the flag and the key code that accepts it change together, because a host re-derives with the base's command against the base's key code.

After the full ctest run, a merge-group macos job also annotates the affected-test set in shadow mode (pulp-affected-tests-shadow/v1, from tools/ci/affected_tests_shadow.py): the ctest entries the build graph and recorded header dependencies say the group's change could reach, with script-driven tests counted as affected whenever any script surface changed and every test selected when a CMake file changed. A compiled test is also affected when a program it runs or loads at run time is: a tool it spawns through $<TARGET_FILE:x> or a path it computes, reached through the add_dependencies(<test> x) edge that the CMake codemodel records and Ninja keeps only as order-only (tools/ci/spawn_closure.py; without a readable codemodel every test is selected). It selects nothing; the number to watch is failed_outside_selection, which must stay at zero over a long window before selection could gate anything.

After a successful merge-group build, the macos job also measures cross-VM test-binary identity (pulp-binary-identity-shadow/v1, from tools/ci/binary_identity_shadow.py measure): it hashes this build's test executables the way a receipt does and compares them path by path with the PR head's receipt when one exists, annotating identical/differing counts and whether the group's tree equals the head's. Per-binary receipt reuse is only possible if identical approaches 100% on tree-identical groups; read the aggregate with binary_identity_shadow.py report --repository <o/r>, which says "waiting on receipts" until 20 such groups have been compared.

The same merge-group step also computes per-test content-hash receipts in shadow mode (tools/ci/test_receipts_shadow.py, notice pulp-test-receipts-shadow/v1). Each ctest entry gets a key: its normalised command and properties, the bytes of its executable and of every file its command names, the toolchain, the CI policy files, and then for a script test the blob ids of its declared inputs (test/ctest_script_inputs.json), or for a compiled test the runtime surface (every tracked non-compiled, non-doc file) and the build's non-test executables (compiled tests read both through PULP_SOURCE_DIR/PULP_BUILD_DIR). The job writes the keys of the tests that passed, and the names that failed, as the test-receipts-macos artifact, only from merge-group runs. It then reads the last 20 trusted receipts and reports how many selected tests a receipt-keyed skip WOULD have skipped and their test-seconds (job summary: "Per-test receipts (shadow): would skip N of M tests"). A failed receipt request is counted apart from a refused receipt (lookup_errors and lookup_error in the notice, a test-receipts-shadow-lookup warning, and receipt lookup failed: <status> in the summary), so receipt_runs: 0 from a broken lookup cannot pass for an empty history. Always-run, never counted: drift/lint/registry/ sync/guard/census/inventory tests and probes, pr-fast/GPU/host-labelled tests, script tests that declare a top-level directory, unkeyable tests (undeclared scripts, nested builds), and anything that failed in a receipt run read; a CMake change or a control run id would force a full run. Nothing is skipped. The safety number is would_skip_failed (would-skip tests that failed in the same run): it must stay 0 over a long window, and binary identity must approach 100%, before enforcement is proposed as a contract amendment.

Every macos job, pull-request head and merge group alike, also uploads a reuse replay record (tools/ci/reuse_record.py, artifact reuse-record-macos, or reuse-record-macos-attempt-N on a re-run, kept 90 days): tests.jsonl with one record per test (run and pull request, head, base and merge tree, suite, outcome, attempts, duration, executable, runner image fingerprint, and the per-test receipt key where the merge-group shadow computed one), identity.json with each executable's sha256 and the digest of its runtime closure (the non-system dylibs and frameworks it loads, a script's interpreter), and job.json with the context and sizes. job.json names the host that wrote it (platform, darwin-arm64 or linux-x86_64). An alias macos job runs on Linux, so a reader that wants the macOS build refuses any record whose platform is not darwin-*. The runner image fields take the SDK version and build from xcrun, and from the SDK's own SDKSettings.json and SystemVersion.plist when xcrun cannot answer. The digest does not depend on which source answered, and runner_image.probe records why a probe failed. toolchain (toolchain_identity()) is what a compile depends on besides its sources. Its fields are the reuse key and are digested: - the compiler CMake's check recorded (id, version, its --version line); - the SDK version and build; - effective, what configure made of the compiler and flags, read from CMakeCache.txt (CMAKE_<LANG>_COMPILER, _FLAGS, linker flags, sysroot, deployment target); - build_env, environment read while compiling (CCACHE_* settings, SOURCE_DATE_EPOCH, ZERO_AR_DATE, PULP_OFFLINE_BUILD).

effective holds absolute paths, such as the compiler and sysroot, so a gate VM record and an m3 record have different digests even when their clang is identical. Each host's records match only that host's records, by design.

Two parts are recorded but never keyed. configure_env holds the live values of the variables CMake reads at configure time (CC, CXXFLAGS, SDKROOT, ...), which can differ from those the directory was configured under. diagnostic.target, the compiler's -print-target-triple, names the host OS rather than the deployment target. Only the listed variables are ever read, because the record is uploaded and the environment holds secrets. Each is recorded as its value, as unset, or as "" when set but empty.

A value no probe found is left out rather than written as "unknown", both here and in runner_image.fields, and readers take an absent key as unknown. missing names the required keys that were absent (the SDK pair is required on macOS), and an incomplete identity warns. A run outside GitHub Actions (the local mac lane) passes --run-kind lane --run-id <id>. The id is required, and the commit, tree and parent come from --source-root's HEAD. dirty records whether git status --porcelain was non-empty, since a dirty lane record did not test its commit. The gate and the lane compute all of this with the same code.

For a reader that selects records by JSON pointer: - /platform is the host. - /toolchain/digest and /toolchain/complete are the toolchain key. - /dirty is lane runs only. - /suites/full is present when a full suite was recorded. - /link_members/usable and /object_deps/usable are booleans. An absent or null summary means the file was not written. - /problems is empty exactly when nothing in the record is incomplete.

A test executable's hash depends only on its inputs because every test executable and module is linked with -Wl,-objc_stubs_small (tools/cmake/PulpTestSuite.cmake, behind check_linker_flag). Without it, Apple's linker laid out the ObjC selector stubs (__TEXT,__objc_stubs) differently from one link of identical objects to the next, so binaries pulling in SDL3 changed bytes with no input change. The test-link-determinism ctest relinks pulp-test-standalone-rt twelve times and fails on more than one hash. It skips only when the configure found the linker without the flag, and then names that linker. Shipped products are linked unchanged.

The record hashes every registered test executable, not only those the job ran, so a fast-tier pull-request head's hashes are there for the merge group's binary-identity shadow, which reads the head's reuse record when the head issued no receipt (annotation field source: reuse-record). With --link-members it also hashes every module the build linked (link members' kind: module), because a test runs the modules it loads. Those modules go into the same identity.json /executables map, keyed by <build>/-relative real path and marked "kind": "module". An entry with sha256 null and an error has no hash. After a failed build only the executables that ran are hashed, and no modules. job.json /identity/usable is true when the map hashed anything. The macOS gate also configures with -DPULP_RECORD_LINK_MAPS=ON (tools/cmake/PulpLinkMaps.cmake): every link runs through tools/ci/link-members-launcher.sh, which adds -Wl,-map, returns the linker's status, and for an executable, a loadable module (a -bundle link: plug-in bundles, LV2 binaries, reload probes) or a shared library keeps only the map's object list and the link arguments under <build>/link-members/ (the map, megabytes of symbol table, is deleted). The linked bytes are identical with and without it. link-members-<sha>.json in the record then lists, per link (each entry's kind: executable, module or shared; schema pulp-link-members/v4), the archive members it pulled, with whole set on archives the link line force-loads (-force_load, -all_load, -ObjC), and in shared the shared libraries the build produced that the link names. Partial links (-r) are only counted, as unrecorded. Every reader calls link_members.unusable() first and treats an unknown schema or kind, an unrecorded link, or shared libraries in a map from before v4 as no record at all, and the record job warns.

A build-produced shared library's content reaches the binaries that load it without changing their inputs or bytes, so it is not keyed as their input. link_members.shared_scope() names those loaders, executables and modules alike. The key manifest marks them always_run shared_link, and the replay leaves them out of its link map. Loaders of system libraries only are not in scope, because the runner image names those.

A change to a pin file (tools/deps/manifest.json, tools/cmake/PulpDependencies.cmake, tools/cmake/PulpFetchContent.cmake, a FetchContent block of the root CMakeLists.txt, or a *_SDK_REF assignment in setup.sh, which pins the SDKs it clones outside the tree) marks always_run dependency_pin only the executables that build against a dependency it moved on this platform. tools/ci/dependency_pins.py names the moved dependencies: manifest entries by name (documentation fields and another platform's release asset do not count), and CMake blocks by the anchors in tools/ci/dependency_pin_map.json, compared after comments and the branches this platform does not take are dropped. An executable reaches a name through its codemodel dependency closure or an archive its recorded link pulled. Anything it cannot name moves every executable: a file that does not parse, a change outside any mapped block, any change to PulpFetchContent.cmake or to a root FetchContent block, a name the map lacks (the map's unmapped list names those on purpose, each with its reason, and a test fails on a linking dependency in neither), or a mapped name this build shows no target or archive for. The key manifest's producer records the verdict as dependency_pins. job.json counts the loaders as /link_members/shared_loaders. object-deps-<sha>.json (tools/ci/object_deps.py, --object-deps) holds, per object file, the in-tree headers Ninja recorded it including (ninja -t deps, as <src>/ and <build>/ paths, headers under <build>/_deps left out), and which objects each static archive member comes from. A changed header then reaches exactly the objects that included it in the build that ran. A build with no Ninja log writes no file and warns. An object Ninja marks STALE is listed in stale with no headers, and a reader treats it, or any object the file does not list, as changed. codemodel-<sha>.json (tools/ci/codemodel_digest.py) holds, per CMake target, digests of its source list, compile groups, link line and the ctest registrations that run its artifact, read from the file-API codemodel reply the configure step requests, with build and source roots written as <build>/ and <src>/ so the same configuration digests identically on every VM. A generated part digests the CONTENT of every build-tree file the target compiles or included when it compiled (Ninja's dependency log, ninja -t deps), so a VERSION bump re-keys only the targets that include the configured version header, and a target whose generated marker source or compile definitions embed a build identity re-keys on every configure, by declaration rather than by learning it from history. Without a Ninja log the record warns that generated headers are not keyed.

When a merge-group macos ctest fails, the job also annotates a flake exoneration verdict in shadow mode (pulp-flake-exoneration-shadow/v1, from tools/ci/flake_exoneration_shadow.py): for each failing test, whether it failed on at least two other heads in the last 24 hours (other runs' ctest-logs-macos artifacts) and passes on main's tip, judged by the tip's required gate job, Chromium's OCCURS_ON_OTHER_CLS. It exonerates nothing; the job still fails. Main evidence that was not actually read (a tip whose gate did not execute the suite, a failing-test list that could not be downloaded, a red gate that named no test) is "main unknown" and never exonerates, and a deterministic test (label pr-fast, or named in tools/ci/drift_fast.json) is never eligible. Read exonerated_only failures ÷ merge-group failures, and main_evidence_read ÷ exonerations, which must be 1, with unique_cause > 0 and the base-red streak count unchanged as controls.

Receipt reuse in a merge group downloads the PR head's exact-tree receipt and verifies the run it came from. GitHub leaves pull_requests empty on runs of App-opened pull requests (most of this repository's), so the verifier binds such a run to its pull request through the commit (commits/<head>/pulls) and accepts only when exactly one open pull request from this repository carries that head, its base is the protected base, the run is a successful pull_request run of build.yml, and the pull request number equals the merge-group entry parsed from the queue ref. Any other shape refuses with a commit-pull lookup: reason that the shipyard-receipt-decision annotation carries, so refusals caused by the binding are countable. The verdict is per target: the receipt's own job (macos) must have concluded success with its issue and publish steps green, and every required context in .github/rulesets/main-protection.json must be green on the head; an advisory job's red (hosted Linux) does not refuse, a red required context does and is named.

The fleet section reads each host's ~/.local/state/pulp/host_vitals.json (one ssh <host> cat per host). The host-vitals sensor publishes a build snapshot there — ccache hit rate and fill for the host cache and the gate cache (cumulative for the cache directory; the per-job figure for one gate run is the ccache per-job: line that build.yml's "Ccache stats" step prints from tools/ci/ccache_job_delta.py, the difference between the --print-stats snapshots taken before and after the Build step — the cumulative "Hits" line a job prints is the host's history and cannot show a cold or poisoned cache on that job), gate-VM count and memory, tartci's installed commit (the sealed launcher bundle's source_commit, else the fleet install record, as tartci fleet-macos self-update reports it), executing generation and checkout, lease usage and host profile, wheelhouse contents. Each probe runs under a deadline (PULP_VITALS_PROBE_TIMEOUT, 15 s): under launchd a ccache -s against a cache on an external volume can block in open() indefinitely, and it then reads as null rather than freezing the published reading, which the sensor writes before the snapshot. A host whose installed sensor predates that snapshot is probed live with the in-repo host_vitals.sh --build-json and labelled so; re-run tools/scripts/install_host_vitals_sensor.sh on it to publish the snapshot. An unreachable host is reported as UNREACHABLE, never as zeros.

Every local build is recorded as local-build/{all,focused} in the pulp project (job governed-build) when shipyard is on PATH: the governed wrapper tools/ci/governed-build.sh (--provider governed-build) and the pulp build CLI (--provider pulp-cli) both call the one recorder, tools/ci/record_build_metric.sh, which owns the fields: wall time, granted -j (--profile j<N>), where the grant came from (--routing-decision), and for a focused build --workflow targets:<count>/<graph size>:<list>. The record is detached, silent and returns in tens of milliseconds; PULP_BUILD_METRICS=0 turns it off. pulp dev/pulp loop rebuilds that run through the C++ watch loop are not recorded yet.

For the local build itself, tools/scripts/build_time_report.py has three subcommands: log (edge-seconds by category from .ninja_log, deduplicated by edge so a multi-output edge counts once), trace (clang -ftime-trace aggregation by header, TU and area) and blast-radius (touch a file, dry-run a regeneration-free copy of build.ninja, restore the mtime). A plain ninja -n in a tree with CONFIGURE_DEPENDS globs stops at "Re-running CMake" and reports nothing, which reads as a zero blast radius; the tool strips those edges, checks that ninja printed every edge it planned, and exits 3 rather than reporting zeros when a touched compiled source (or its built-in control) produces no work.

Pulp intentionally pins Shipyard in tools/shipyard.toml even if your daily global shipyard is newer. Use shipyard pin bump --to vX.Y.Z for pin updates instead of hand-editing the file; newer Rust Shipyard releases changed the macOS asset shape to a signed/notarized .dmg, and the bump command keeps the version and asset metadata in sync.

The public Pulp installer does not install Shipyard or GitHub CLI (gh). That is intentional: ordinary Pulp users do not need either tool to create, build, run, or upgrade projects. They are source-checkout contributor tools. pulp pr defaults to Shipyard and fails with install/switch guidance if Shipyard is missing; contributors who prefer their own PR flow can set pulp config set pr.workflow github or manual. The github workflow uses gh directly and requires it to be installed and authenticated. Run pulp status to see the effective workflow and local tool health.

Optional local VM routing

Core Pulp development can also use local, disposable VMs through tartci. This is optional: a normal contributor can open a PR and let GitHub Actions run on hosted runners. The value of the local VM setup is faster feedback on trusted Apple Silicon hardware while keeping every job clean-per-run.

The current Pulp routing policy is intentionally kept in parseable TOML at .shipyard/ci-profiles/normal-local-fast.toml instead of copied into this guide. It names the PR, release, coverage, scheduled, and issue-on-failure policies and maps stable target IDs to concrete GitHub runs-on selectors. Shipyard owns orchestration and profile selection; tartci owns the local VM providers, goldens, host caches, and per-host status. Use the upstream docs for details:

• Shipyard profiles explain how profiles and fallback resolution work.
• tartci explains the Tart/QEMU VM lanes, tartci status --json, and tartci profile explain|plan.
• mac-ci-host-setup.md is the Pulp-specific host setup guide for joining the macOS VM pool.

When pulp build, pulp dev, or pulp loop run on a tartci-governed host, the CLI asks tartci for a host-core lease and caps CMake parallelism to the leased job count. Lease-backed builds also run through a POSIX process-group watchdog by default. The watchdog terminates a build that stays over its CPU budget long enough to threaten the shared host; set PULP_TARTCI_WATCHDOG=monitor to log over-budget samples without killing, or PULP_TARTCI_WATCHDOG=0 to disable the wrapper. Operators can tune PULP_TARTCI_WATCHDOG_INTERVAL_SECS, PULP_TARTCI_WATCHDOG_SAMPLES, PULP_TARTCI_WATCHDOG_TERM_GRACE_SECS, PULP_TARTCI_WATCHDOG_CPU_PER_JOB, and PULP_TARTCI_WATCHDOG_PYTHON per host.

Build classes: interactive vs background

tools/ci/governed-build.sh (which pulp build uses in a source checkout) runs every build in one of two classes, chosen by PULP_BUILD_CLASS:

Class	Who	Lease request	QoS	On denial
interactive (default)	pulp build, manual builds, release/packaging builds	the host's interactive share; any partial grant down to interactive_min_cores, waiting up to interactive_wait_secs for it	normal, always	leaseless -j2 at normal QoS (never the background floor)
background	every Shipyard-local stage in .shipyard/config.toml	the profile size, partial grants accepted	the host role's QoS (background on laptops)	the agent-floor lease, then leaseless -j2

Against a tartci that predates classes (no TARTCI_GOVERNOR_SCHEMA in tartci host-profile), both classes use the class-less lease contract, and an interactive build still never runs under taskpolicy -b or takes a floor lease. With no tartci at all the Tier-0 bound applies unchanged.

Where a host enables dynamic_lending, an interactive build may borrow the gate's idle reserved cores; a gate lease that needs them is admitted anyway and the borrower is moved to background QoS until the gate finishes (nothing is killed). Each host's knobs live in ~/.config/tartci/governor.toml; tartci governor show prints them with their sources, tartci governor set KEY=VALUE edits them, and tartci governor explain says what each class would be granted right now.

The macOS release VM lane and cross-lane priority

Release macOS legs (release-cli.yml, sign-and-release.yml, and release-path-pr-gate.yml's legacy fallback) route through PULP_RELEASE_MACOS_RUNS_ON_JSON, which is deliberately set to the base gate label set (self-hosted, macOS, ARM64, pulp-build, pulp-build-vm). TartCI's event-class-v2 gate slots mint runners only for Build and Test jobs that carry a class label, so no slot boots for a release job. GitHub, however, hands a queued job to any idle registered runner whose labels are a superset, so release jobs run on whichever gate runner comes free. Service is opportunistic: it has served every release since the variable was set, but it depends on concurrent gate demand and competes with PRs for the same slots. The contract records this as opportunistic_service, and static mode reports those rows OPPORTUNISTIC rather than REACHABLE (not blocking; the row goes UNSERVED and fails if no gate registration carries the labels):

PULP_RELEASE_MACOS_RUNS_ON_JSON  required expect    Release CLI  OPPORTUNISTIC  no registration mints for 'Release CLI'; idle runners minted for ['Build and Test'] carry the labels (...)
PULP_RELEASE_MACOS_RUNS_ON_JSON  required fallback  Release CLI  UNSERVED       no Generous-Corp/pulp registration advertises ['pulp-gate-fast']
PULP_RELEASE_MACOS_RUNS_ON_JSON  required rollback  -            HOSTED         GitHub-hosted allowlist

The dedicated pulp-build-vm-release lane (m5's legacy pulp-release slot, which yields to Build and Test) still exists but nothing routes to it, pending a redesign that folds the release classes into the v2 gate lane.

Unsetting the variable is not a rollback. The resolver chain (resolve_release_runners.py, and release-path-pr-gate.yml) falls through to PULP_LOCAL_MACOS_RUNS_ON_JSON, whose label set carries pulp-gate-fast. No registration advertises that label (build.yml rewrites it per event; the release workflows do not), so a release darwin-arm64 leg routed there queues forever.

The served break-glass rollback is to set the variable to the lane's break_glass_rollback, GitHub-hosted ["macos-15"] (ghapp variable set PULP_RELEASE_MACOS_RUNS_ON_JSON --body '["macos-15"]'). It schedules slower when the hosted pool is congested, but it always dispatches. Static mode fails when a required lane's unset fallback is UNSERVED and its contract names no served break_glass_rollback, so a rollback that cannot run cannot be documented as one. An opportunistic_service declaration never rescues a fallback or rollback row.

A release slot admits a job only when three independent gates all pass; labels are necessary but not sufficient:

1. Workflow allowlist — TARTCI_RUNNER_WORKFLOW_TIERS in the slot's LaunchAgent names the workflows it may serve (e.g. Release CLI, Sign and Release). A matching label with an unlisted workflow never boots.
2. VM-count cap — macOS allows 2 concurrent VMs per host (kernel quota), shared by every macOS lane on that host regardless of label.
3. Core-lease budget — the tartci per-host lease store admits or refuses by core count; tagged-release boots acquire at gate priority, so release and gate work contend first-come-first-served for the same budget.

Cross-lane priority is the tartci provider's opt-in yield hook: a slot whose LaunchAgent sets TARTCI_YIELD_TO_WORKFLOW_NAME (single-valued — a pipe-separated list is passed as one literal name and silently matches nothing) plus TARTCI_YIELD_TO_LABELS refuses to boot while the named workflow has queued or in-progress demand matching those labels. Enabling the hook on a subset of gate slots lets release work claim a VM slot promptly while the never-yielding remainder of the gate pool keeps serving PRs — the subset size is the gate-capacity floor, expressed purely in per-host config. For an advisory lane sharing a Pulp event-class-v2 host, include both pulp-build-merge-group and pulp-build-pr-head in the yield selector. The idle gate cannot preempt an already-running advisory VM, so ignoring PR-head demand could consume the last free slot just before strict merge-group demand arrives. Using only the base gate labels matches neither v2 class because each job requests its additional mutually exclusive class label.

One sharp edge to monitor for: the yield probe fails closed — any gh error while scanning the priority workflow's queue reads as "demand exists, keep yielding." A total outage is self-limiting (the main queue scan goes blind first and the supervisor self-restarts), but an asymmetric failure (main scan healthy, priority scan erroring) can hold a yielding slot down indefinitely. Before treating a long-yielding slot as real demand, corroborate with tartci leases status (is a release lease actually held anywhere?) and the per-runner event log, which distinguishes yielded_to_priority (detail carries the workflow and queue counts) from yielded_host_health, alongside per-slot heartbeats at ~/.tartci/state/macos/<runner>.state.json. Note the heartbeat write fails silently on a full disk, so heartbeat staleness is a freshness alarm of its own but the phase field cannot be a monitor's sole source.

Declared fleet state — which hosts carry which tartci slots — lives in the fleet manifest in the private planning repository, not in this guide. A slot present on a host but absent from the manifest is configuration drift, even when it works: it will not survive fleet reconciliation, and repairs to such a slot start by declaring it.

Persistent native Actions runners are covered by the manifest's actions_runner_policy key. The policy discovers configured runner directories from globs rather than names, pins one reviewed Actions runner version with automatic updates disabled, keeps system directories before Homebrew on the runner's captured .path, and locates RUSTUP_HOME/CARGO_HOME under the runner's own internal-APFS _toolcache. tools/fleet/verify.sh reports any deviation; tools/fleet/apply.sh repairs it only when no Runner.Worker is active, restarts an offline listener, and otherwise leaves a manual receipt for the bounded watchdog ladder. It never retries a workflow or transfers recovery to another host.

Shipping a PR: shipyard pr

shipyard pr is the single "ship this" orchestrator. Agents and humans should route every normal ship cycle through it rather than pairing gh pr create with shipyard ship manually. It:

1. Runs tools/scripts/skill_sync_check.py (hard-fails on missing SKILL.md updates).
2. Runs tools/scripts/version_bump_check.py --mode=apply to bump SDK / Claude plugin / marketplace versions consistently.
3. Commits the bump (if any) as chore: bump <surfaces>.
4. Pushes the branch, creates the PR, and records Shipyard tracking state.
5. Runs cross-platform validate + merge on green.
6. The auto-release workflow tags and publishes binaries on merge.

shipyard pr                              # primary ship path
shipyard pr --base develop/package-manager # ship to a develop branch
shipyard pr --title "..."                # override PR title
shipyard pr --dry-run                    # print the plan without executing

pulp pr is a compatibility wrapper that delegates to shipyard pr by default; it is valid, but guidance should name shipyard pr directly so humans and agents understand where PR tracking state lives. Its github and manual workflows are explicit local opt-outs and do not create Shipyard tracking state.

Direct gh pr create is an emergency/manual bypass only. If it is used, call out that the PR may not appear in Shipyard-managed state until it is reconciled or re-shipped through Shipyard.

Shipyard v0.3.0 workflow surface

Shipyard v0.3.0 adds stateful ship resume, SSH --resume-from staging, and incremental git bundles on top of the basic run / ship / cloud run surface above.

# Resume an interrupted ship
shipyard ship --resume                    # pick up where the last session left off
shipyard ship --no-resume                 # discard stale state and ship fresh

# Inspect in-flight ship state
shipyard ship-state list                  # self-describing inventory: PR, title, URL, tip SHA, dispatched run IDs
shipyard ship-state show <pr>             # full state for one PR
shipyard ship-state discard <pr>          # archive stale state

# Prune old ship state + evidence
shipyard cleanup --ship-state             # dry run — show what would be pruned
shipyard cleanup --ship-state --apply     # prune closed-PR state + aged records

# Fast test iteration on any target
shipyard run --resume-from build          # skip configure+setup, start at the build stage
shipyard run --resume-from test           # skip configure+build, run tests only

# The `windows` / `ubuntu` SSH targets are opt-in per machine and are NOT
# declared in .shipyard/config.toml — the commands below only work once you
# uncomment the matching block in .shipyard.local/config.toml (see
# .shipyard.local/config.toml.example). `shipyard targets list` shows what
# this machine actually has.
shipyard run --targets windows --smoke    # fast Windows-only preflight
shipyard run --targets windows --resume-from test   # ~2 min rerun vs ~15 min full

# Target and config inspection
shipyard targets                          # list configured targets with reachability
shipyard targets test windows             # probe a single target
shipyard config show                      # effective merged config
shipyard config profiles                  # list profiles plus the active one

Ship state lives at <state_dir>/ship/<pr>.json. Shipyard auto-resumes the next time you run shipyard ship on the same PR — it refuses to resume if the PR's head SHA or merge policy changed since the state was written, so a rebase or force-push deliberately forces a fresh ship.

--resume-from works on both local and SSH targets. On SSH targets, Shipyard probes the remote for a marker file proving the previous stage passed for the exact SHA, and skips earlier stages when it finds one.

Incremental bundles — SSH validation now sends only the git delta between the remote HEAD and the target SHA. Typical cycles drop from ~443 MB to a few KB. No configuration needed — Shipyard falls back to a full bundle automatically when the delta would be larger than the full pack.

Codex does not auto-review App-authored PRs

Codex's automatic code review fires on PR open only when the pull request's author is a GitHub User. A PR opened by a GitHub App is skipped. Because shipyard pr opens PRs as shipyard-local[bot], and that is the mandated path for agents, the default outcome is that agent-opened PRs merge with no review while human-opened ones are reviewed.

Codex reports the distinction itself. Its review-summary comment carries a "Review trigger" cell that reads PR opened on a User-authored PR and Manual request on an App-authored one — the App-authored PRs that were reviewed at all had been reviewed because somebody asked.

Only the automatic trigger is restricted; asking still works. A @codex review comment gets a real review on an App-authored PR, and it does so even when the comment itself comes from an App. (Codex replies to a bot commenter with "To use Codex here, create a Codex account and connect to github", which looks like a refusal and is not — the review runs anyway.) The skip is Codex-side and cannot be configured from this repository; there is no workflow trigger or github.actor guard here involved. What this repository can do is ask.

.github/workflows/codex-review-request.yml is that ask. On a PR opened by shipyard-local[bot] it posts the same @codex review comment a human would, using GITHUB_TOKEN and no privileged secret at all, then verifies a review actually completed when the connector is available. The check is advisory: missing credentials, API failures, unavailable connectors, and review timeouts emit warnings and pass so they never block an otherwise green PR.

The absence of a user PAT there is deliberate. A same-repository pull_request evaluates the workflow file from the PR's own revision, so any secret exposed to this job is readable by a PR that edits this file — and the PRs it runs on are exactly the unreviewed ones. An App identity is sufficient: a @codex review from one does produce a completed review.

It runs on synchronize as well as opened and ready_for_review. That is load-bearing rather than thorough: under this repo's up-to-date branch protection a PR is pushed to repeatedly, and on opened alone the commit that was reviewed and the commit that merges are different ones. Superseded runs are cancelled, because during a burst of pushes only the final head can merge.

The verification is the point. A mitigation that posts a comment and never checks whether anything came back can no-op in silence, which is the same failure it exists to correct. Three distinctions keep that check honest, and all three live in tools/scripts/codex_review_signal.sh (self-tested by test_codex_review_signal.sh, ctest codex-review-signal-selftest):

• Acknowledgement is not completion. Codex posts the summary comment and reacts with EYES the instant a review is requested, before it knows what it can do. Only **Completed** in the summary's status cell counts.
• Completion is per-commit. The summary names the commit it reviewed, and the check requires the PR's current head to be that commit. Without the binding, a review of an earlier push would answer for code nobody has seen — a real path, since the workflow can fire on opened and again on ready_for_review with commits in between.
• An unreachable API is not a finding. Any gh failure exits 2, distinct from the exit 1 that means "no review", and the workflow treats it as unknown rather than as a verdict.

THUMBS_UP is reported rather than required: it separates "reviewed, no findings" from "reviewed, left comments", which is worth printing, but it carries no commit and so cannot prove anything about a particular head.

The job checks out the base commit, never the PR. The checker decides whether a PR was reviewed, so running the PR's own copy would let an unreviewed change rule that it needs no review. The base copy is the reviewed one, and a PR that edits the checker is still judged by the version already on the branch it targets.

That has one consequence worth knowing: on the pull request that first adds the checker, the base commit has no copy of it, so the checker cannot run. The job reports that exit distinctly — "did not run" rather than "not reviewed" — and passes with a warning because review coverage is advisory.

This workflow requests reviews; it does not audit whether older PRs got one. .github/workflows/post-merge-review-sweep.yml remains the separate, scheduled sweep that collects bot review comments on already-merged PRs.

Keeping fleet Macs on current Shipyard

tools/shipyard.toml pins the Shipyard version tools/install-shipyard.sh and the workflows install; a public cloner runs install-shipyard.sh once and never thinks about this again. The local fleet Macs run Shipyard's latest release instead, and tartci keeps them there: tartci fleet-macos tool-freshness (refreshed by tartci's launchd watchdog every 30 minutes) compares the installed version with the latest release, applies shipyard update --to <tag> --refresh-daemon once a release is 30 minutes old, verifies the re-read version, and reports a tool behind for more than 12 hours as STALE in tartci pool status and tartci doctor fleet. Opt a host out with auto_apply = false under [tools.shipyard] in ~/.config/tartci/tool-freshness.toml. See tartci's runbook, "Shipyard and pulp CLI freshness".

v0.81.0 also gives the fleet watchdog an expected-host inventory independent of ephemeral runner names. Pulp declares the MacPro and Mac Mini active in .shipyard/config.toml; absence or insufficient online matches produces expected_host_unavailable. The planned MacBook Air is declared with active = false, so it remains visible without claiming capacity. Matching uses stable label subsets (pulp-host-macpro, pulp-host-macmini, and architecture), never a JIT runner identity. Inspect the combined view with:

shipyard runner fleet-status --repo Generous-Corp/pulp --json

The same report calls out Tart disk-floor and ccache-size admission failures and merge-group Linux jobs left on ubuntu-latest while online self-hosted Linux x64 capacity is idle.

Why the macOS gate hosts are not declared as expected hosts

The three Apple Silicon Macs that serve the required macos gate (m3, m5, m1) are deliberately absent from expected_host, and the reason is the matching rule above: they carry no host-identifying label. Every gate runner on all three registers the same set — self-hosted, macOS, ARM64, pulp-build, pulp-build-vm, pulp-gate-fast — and the labels that vary between them (pulp-build-studio, pulp-build-vm-secondary) describe a role, not a machine. m1's and m5's gate runners are label-identical. There is no pulp-host-m3 analogue to the pulp-host-macpro / pulp-host-macmini labels that make those two declarations work.

So an expected_host entry per machine would match the same pool three times: all three rows report online whenever any one of the machines is serving. That is worse than no declaration, because it turns a genuine partial degradation — a pool at a third of capacity with one host dead — into three green rows. Raising min_online pool-wide fails in the opposite direction: the gate pool is ephemeral JIT, so a healthy but idle host has zero runners registered and would alarm on every quiet period.

Per-host state for these three comes from the same report's hosts[] array instead, keyed by class (m1, m5, studio) and read from tartci host state over SSH rather than inferred from labels. It carries routable, free/cap, supervisor heartbeat age, and the disk-floor and ccache admission problems that keep a host from accepting work. tools/scripts/runner_topology.json owns the complementary question of which label set each lane is contracted to route to.

One limit worth stating plainly: fleet-status is a manual-inspection view. No workflow or script consumes it, so a declaration here pages nobody on its own. Detecting a partially degraded gate pool needs capacity measured against demand — a busy pool and a pool at a third of capacity look alike from host presence — and nothing implements that today.

Host resource governance

Pulp's local Macs are shared: CI validation builds run alongside agent and developer builds on the same host. Two of them melted in July 2026 — one CPU-bound, one memory-bound/OOM — from unbounded builds oversubscribing the machine. A per-host build-resource governor now bounds every build path. It is tiered:

• Tier 0 — always, zero config. The pulp CLI bounds build parallelism to min(cores, RAM_budget / 1.5 GiB) on every build it emits (pulp build/dev/loop, the local-SDK build). No lease store required; override the RAM axis with PULP_BUILD_MEM_BUDGET_MB. tools/scripts/build_parallelism_guard.py rejects a bare --parallel/-j (unbounded) anywhere in the repo, and — on the shared-host surfaces agents copy from (CLAUDE.md, .shipyard/config.toml, .agents/skills/**) — also rejects an explicit but whole-machine count (-j$(nproc) / -j$(sysctl -n hw.ncpu) / --parallel $(getconf _NPROCESSORS_ONLN)): it has a count, so it is not unbounded, but on a shared Mac it claims every core, so N concurrent builds request N × cores and starve each other. The rule is a property of the host, not the command — so the guard fires only where a static scan can prove the surface is shared. It does NOT scan .github/workflows/**, and not because a workflow leg never shares a box: a workflow's runs-on is resolved dynamically (often ${{ fromJSON(matrix.runs_on_json) }} or a repo var) and can point at the shared self-hosted Studios — Pulp's own macOS matrix leg resolves to PULP_LOCAL_MACOS_RUNS_ON_JSON, the Studios that host the required macos gate. A file scan cannot resolve that, so in a workflow the bound is the author's responsibility: route a self-hosted macOS leg through tools/ci/governed-build.sh (as build.yml's matrix Build step on its macOS and Linux legs and its intel-canary compile, examples-validation.yml, web-plugins.yml's gpu-audio-macos job, and format-baseline-diff.yml now do; build.yml's Windows leg keeps a literal, because a GitHub-hosted ephemeral runner shares with nobody and no lane runs the wrapper under MSYS bash today). The steer everywhere is pulp build / tools/ci/governed-build.sh, which take their -j from the governor.

A literal --parallel N in a workflow is the specific anti-pattern here, and not only because no one number fits every runner. It is also a silent ceiling: it keeps its value when a VM is resized, so the resize buys nothing and the no-op reads as "more cores did not help" rather than "the build was never asked to use them". build.yml's Build step carried --parallel 4 fleet-wide for this reason until it was replaced by the governor.

An explicit -j under the wrapper is a ceiling, not an override. tools/ci/governed-build.sh reads a -j N, -jN, --parallel N or --parallel=N on the build command (including a native-tool flag after --) as a request, takes the lower of it and PULP_BUILD_JOBS, sizes the lease to that, and rewrites the flag to the final share before running the command. So governed-build.sh cmake --build build -j4 runs at most four jobs and reserves four cores, -j64 under a 6-core lease runs six, and a bare --parallel gets the governed count instead of going unbounded. The log says so: build command's own parallelism flag set to the governed share -jN. Before this, the command's own flag silently won in both directions: a -j4 reserved and logged a profile-sized lease, and a higher -j ran past the lease. PULP_BUILD_JOBS remains the way to request a lower share without touching the command. A CMAKE_BUILD_PARALLEL_LEVEL you export is a ceiling in the same way: the lowest of it, the flag, and PULP_BUILD_JOBS sets the share, and the wrapper then exports that final share under the same name. (It used to overwrite the caller's value, so CMAKE_BUILD_PARALLEL_LEVEL=4 ran at the full lease.) A non-numeric or zero value is ignored.

A gate VM's -j is its lease. tartci sizes each macOS gate clone from the lane's VM lease: C vCPUs and (C-1) x 1536 x 4/3 MB of RAM, clamped to 8 GiB and 16 GiB. That memory figure is the exact inverse of the Tier-0 bound, min(cores, RAM x 0.75 / 1.5 GiB), so a guest lands on -j(C-1) unless the clamp moved it. Measured on 2026-09-23: m1 leases 3 vCPU/8 GiB and builds at -j3 (cores bind), m5 leases 6 vCPU/10 GiB and builds at -j5 (memory binds), and the Studio's 12 vCPU/16 GiB VMs build at -j8 (the memory ceiling binds). The [governed-build] line names the axis that bound it, and reads the lease from TARTCI_GUEST_CORES / TARTCI_GUEST_MEM_MB when the runner declares them; a declaration can only narrow what the guest sees. So a slow m1 or m5 gate is not a governor bug: speeding it up means a larger lease (the host profile's vm_pool_cores, and TARTCI_VM_LEASE_MAX_MEM_MB for the Studio), which trades against the agent builds and the second VM on that host. The non-Windows ctest step takes -j from a declared TARTCI_GUEST_CORES (narrowed to the visible cores, capped at 8) instead of the literal -j8 that ran eight tests on a 3-vCPU guest. A runner that declares no lease keeps -j8, so the change reaches each host only as that host's tartci starts declaring the lease — a per-host rollout, not a fleet flip. - Ninja link pool (every Ninja build). The governor bounds how many jobs a build runs; the pulp_link job pool (tools/cmake/PulpLinkPool.cmake) additionally bounds how many of them are links. Every Pulp executable and bundle links the same ~150 MiB of Skia/Dawn/SDL3 archives, so a wide link phase is memory-bandwidth-bound: links run slower each and peak RSS climbs with the number in flight, without the compiles needing to be throttled. The depth is one slot per 2 GiB of RAM, clamped to [2, 8] (a test link peaks near 0.45 GiB RSS; an 8 GiB gate VM gets 4 slots, the same as its governed -j, so the pool never throttles it). -DPULP_LINK_JOBS=N overrides the depth and -DPULP_LINK_JOBS=0 disables the pool. Makefile, Xcode, and Visual Studio generators have no job pools, so the setting is a no-op there. - Tier 1 — tartci per-host lease governor. On a host running a tartci lease store, builds and VM runners acquire a weighted core+memory lease before starting; admission is min(core-budget, memory-budget), so a build that would exhaust RAM is refused even when CPU is free. Each host derives a role budget from tartci host-profile: - dedicated-builder — a machine whose job is CI builds (largest core + memory budget). - dev-overflow — a shared dev machine that also takes overflow CI, running its VM lane at non-gate priority so it never starves the required macos gate. - light — a low-resource/travel host with a small budget. - Tier 2 — Orchard fleet VM placement (shadow phase). Fleet-level placement, wired but placing nothing yet. See the tartci runbook's Orchard section.

The mac local lane is the one that historically escaped the CLI: Shipyard's local backend runs .shipyard/config.toml commands directly on the host and does not pass through the pulp CLI. Every build stage in that config — default, parser, and smoke — and all POSIX CTest stages are therefore wrapped by tools/ci/governed-build.sh. It acquires a tartci build lease sized from the host profile, exports the granted CMake and CTest parallelism, runs the workload as a child process, and releases the lease on exit. CTest applies that bounded share while still honoring each test's RUN_SERIAL and RESOURCE_LOCK properties. Suites that open the real CoreAudio device use RUN_SERIAL; their PROCESSORS 8 value is a timing weight, not an assumption that the dynamically granted share is always eight. When tartci is absent (a build VM or a plain checkout), the wrapper uses the Tier-0 bound. A lease denial retries at reported free capacity, then uses the conservative floor if capacity disappears; it never fails the workload or piles onto a saturated host.

An uncatchable SIGKILL cannot run the wrapper's release trap. Recovery is still bounded without weakening admission: tartci's next leases acquire revalidates each owner's PID, process start time, and host boot identity under the store lock, removes dead/reused owners, and only then calculates available capacity. A stale heartbeat with a still-matching live owner is reported but retained; elapsed time alone never steals capacity from live work. (The smoke lane previously used a raw --parallel $(getconf _NPROCESSORS_ONLN) and so ran whole-machine on the shared Mac while the required gate validated alongside it; it now takes a governed share like the other lanes.) The version-controlled overrides.windows recipes keep a fixed --parallel 4 instead: they run under PowerShell with no wrapper-path or $(…) assumptions, and unbounded MSBuild link parallelism trips LNK1104 on ARM64.

pulp status reports the active tier with a Build governance: Tier N (…) line. Host-side setup and the deeper lease/role/memory-axis mechanics live in the tartci repo (scripts/leases.py, scripts/host_profile.py, tartci host-profile / tartci leases).

brew update in the macOS gate is advisory, not a gate

build.yml's macOS legs run brew update --quiet before installing ccache, because a runner with a stale Homebrew config makes the following brew install fail fast rather than update itself. That step is deliberately non-fatal: a tap fetch that misses the Homebrew CDN exits 1 with Error: Failed to download while leaving brew entirely usable, and the step sits ahead of every build in the required macos gate — so a CDN hiccup failed the required check on a branch whose code was fine. The Install ccache step that follows already retries behind its own brew update, which is what actually recovers the stale-config case, so ignoring the exit code here removes a flake without removing any coverage.

When triaging a red macos, Error: Failed to download in the brew step is therefore no longer a cause — read past it to the build and ctest output.

The update itself is skipped when ccache and ninja are both already on PATH, which is every tartci gate VM (the golden bakes both). Nothing later in the job runs brew install there, so the update only downloaded a portable Ruby and the Homebrew API data on every gate job. A runner missing either tool (Namespace, GitHub-hosted) still updates before installing.

Repository history and pinned archives from the tartci host cache

A gate VM is fresh for every job, so without help each job re-downloads the same bytes: a depth-2 checkout (~56 MiB) and the GPU-provenance --unshallow (~57 MiB), the pinned Chrome for Testing archive (187 MB), the prebuilt Skia archive (57 MB) and, when the iOS gate runs, the iOS simulator Skia slice (69 MB). tartci can mount a read-only host artifact cache and name it in TARTCI_ARTIFACT_CACHE (tartci docs/runbook.md, scripts/artifact-cache.sh). build.yml uses it only as an accelerator:

• Seed the checkout from the host git mirror runs before actions/checkout and pre-creates the workspace repository with $TARTCI_ARTIFACT_CACHE/git/<owner>/<repo>.git/objects as a Git alternate. checkout@v5 keeps a repository whose origin URL matches and deletes anything else, so with no cache, a warm workspace or any failure the job does the ordinary full fetch. The hydration step prints gate-git-transfer: local_object_kib=N alternates=yes|no: the repository bytes this job pulled.
• The Chrome step and fetch_skia_for_release.py look up the digest they already pin under $TARTCI_ARTIFACT_CACHE/sha256/, re-verify the bytes and download on a miss or mismatch. They log gate-artifact: chrome source=cache|network bytes=N and Copied from host artifact cache / Downloaded N bytes.

A pin bump therefore never breaks the gate; until the hosts' caches hold the new archive, jobs download it as before.

The visual-analysis Python dependencies are installed, then proved

build.yml installs tools/motion/visual/requirements.txt into the interpreter CMake configured, in a step that sits between Configure and Build on every platform that runs ctest. It reads Python3_EXECUTABLE out of $PULP_BUILD_DIR/CMakeCache.txt rather than trusting whatever python3 the shell resolves, because ctest launches the visual tests through that cache entry. Installing into a different interpreter would leave every one of them skipping while the install step reported success, so a missing cache entry fails the step outright instead of falling back.

It first asks pip install --dry-run --no-index whether the declared floor in requirements.txt is already satisfied, and stops there with no network contact when it is (see "The required gate must not need a third-party service at run time"). Only when something is missing does it install tools/motion/visual/requirements.lock, the hash-pinned resolution of requirements.txt, under --require-hashes, so every install lays down the same bytes and a new upstream release cannot change a gate verdict. Regenerate the lock (command in its header) whenever requirements.txt changes; tools/scripts/test_visual_python_deps_step.py fails if a declared distribution is not pinned and hashed there.

When an install is needed, the bytes come from, in order:

1. A tartci gate guest whose host keeps a pip wheelhouse gets TARTCI_PIP_WHEELHOUSE in its job environment, and the step installs from that read-only mount with --no-index — no relay, no PyPI. The host side is scripts/pip-wheelhouse.sh sync in tartci; an empty or absent wheelhouse changes nothing.
2. Otherwise, or if the wheelhouse cannot satisfy the lock, the package index, with three attempts spaced 20 s and 40 s apart (PULP_PIP_RETRY_DELAY_SECS).

PEP 668 hosts (Homebrew on the self-hosted Macs, Debian on the Linux legs) refuse a plain --user install and mark themselves with an EXTERNALLY-MANAGED file beside the stdlib; the step checks for it once and passes --break-system-packages only there.

A Tunnel connection failed: 403 Forbidden from this step is the gate guest's egress relay refusing PyPI, not a package or diff problem. That took down 27 m5 gate jobs on 2026-09-22/23 until tartci's relay allowlist gained pypi.org and files.pythonhosted.org; a populated wheelhouse removes the dependency.

Seven ctest registrations import numpy, Pillow or scikit-image and skip themselves when one is absent. A ctest SKIP is indistinguishable from a PASS in a green run, so before this step existed a lane that quietly lost a wheel reported success while the checks it was built for never executed. The visual-python-deps-present ctest closes that hole: it is the one registration in the set that deliberately carries no SKIP_RETURN_CODE, so an incomplete dependency set fails the suite and names the gap.

It reads the declared set from the requirements file rather than restating it, so adding a dependency there is enough to have it checked. Note that the declared set is wider than any single skip message admits: the two motion self-checks guard on Pillow and report only Pillow, but the analyzer they call also needs scikit-image, so installing numpy and Pillow alone moves them from one skip to another.

Running the suite locally without those wheels now produces one failing test with the install command in its output:

python3 -m pip install --user -r tools/motion/visual/requirements.txt

requirements-optional.txt pins opencv-python for full affine estimation. The analyzer falls back to a translation-only estimator without it, so it stays optional and is not part of the checked set.

Every build leg reports its toolchain against the manifest

Before configuring, each build.yml leg runs tools/toolchain/check_env.py --format github. It prints one ::warning:: annotation per tool whose version differs from tools/toolchain/manifest.toml (python3, cmake, ninja, node, rustc, clang-format, mise, and the npm lockfile installs), with the found version, its path and the fix. The step is report only: the manifest is in warn mode, so the script exits 0, and the step is continue-on-error. It reads versions and installs nothing. Because the gate VM, the hosted Linux and Windows runners and each host get their tools from different places, these annotations are where drift between them shows up first; for example, the hosted Linux leg's apt CMake is reported against the 4.3.3 pin, labelled as the manifest's declared Linux exception. See mise.

The step costs well under a second (0.45 s on the macOS gate VM, 0.10 s on hosted Linux), each version probe is capped at 5 s, and the step at 2 minutes, so a hung tool cannot stall a leg. It stays report only under bash -e whatever is missing: no Python on the runner, or a check that cannot read its expected versions (exit 3), becomes a warning annotation, never a failed step.

The build legs also set RUSTUP_AUTO_INSTALL=0: a rustup proxy would otherwise download, mid-build, whatever toolchain rust-toolchain.toml names when the runner lacks it. With it off, a missing toolchain fails the cargo step by name, and the image or the resolve step has to provide it. The "Provide the pinned Rust toolchain" step is that resolve step: tools/ci/provide_pinned_rust.sh --fallback-to-default runs before configure and logs which branch it took as rust: branch=<name>:

• present: the image has the exact Rust the generated experimental/pulp-rs/rust-toolchain.toml names, with its components and targets. Nothing to do; the steady state once the golden bakes the pin by name.
• stable-alias: the image's stable is that same release with every pinned part, so the job selects it by name (RUSTUP_TOOLCHAIN=stable) with no download. rustup refuses to toolchain link an official release name, so this is how an identical stable stands in for it.
• install: rustup downloads the pin, before the build, with three spaced retries.
• fallback: the download failed on a GitHub-hosted leg. The step warns, naming both versions, writes a step summary line, uploads its receipt as the rust-toolchain-fallback-<leg> artifact, and sets RUSTUP_TOOLCHAIN=stable (or the runner default) for the rest of the job.

Every branch writes a JSON receipt ($RUNNER_TEMP/pulp-rust-toolchain.json: pinned, branch, toolchain, rustc, reason, attempts) and the step outputs branch and receipt. The fallback is refused, and the step fails, on a self-hosted leg (the required gate's VMs) and on a pull request that changes experimental/pulp-rs. The gate VMs reach the network through a proxy that refuses static.rust-lang.org, so they can never download the pin: their images must carry it by name (present) or as an identical stable (stable-alias), and a gate log showing install means the golden drifted. When tartci bakes the pin by name into the live golden, the next change removes --fallback-to-default from build.yml and every leg fails closed (decisions contract row 29). A cargo without rustup (Homebrew Rust) ignores rust-toolchain.toml; the step warns and receipts it as no-rustup. The release lanes (release-cli.yml, release-path-pr-gate.yml) run the same script without the fallback.

Build-time Node dependencies are installed before the build, never inside it

The Three.js bundler (tools/scripts/bundle_threejs_for_jsc.mjs) runs as an iOS AUv3 POST_BUILD command inside the iOS compile gate and as the pulp_bundle_threejs_for_jsc_smoke ctest. It loads esbuild from tools/scripts/node_modules, and when that is absent its default is to run npm install on the spot. That is convenient in a local checkout and wrong in the required gate: an ephemeral VM starts without node_modules, so every gate run made a compile depend on registry.npmjs.org, and a DNS blip (getaddrinfo ENOTFOUND registry.npmjs.org) failed the macos check as an iOS compile error (exit 65) three times between 2026-09-27 and 2026-09-29, on three different hosts.

build.yml now splits the two:

1. The build job sets PULP_OFFLINE_BUILD=1. Under it the bundler refuses a missing esbuild with exit 3 and a message naming npm ci --prefix tools/scripts, instead of fetching. Any value other than empty, 0, false, no or off counts. Local builds leave it unset and keep the one-time install.
2. The Install build-time Node dependencies step, ahead of Build, runs npm ci --prefix tools/scripts --prefer-offline from the committed package-lock.json with three attempts spaced 20 s and 40 s apart (PULP_NPM_RETRY_DELAY_SECS). It first loads the installed esbuild, checks its version against the lockfile and runs one transform (which starts the platform binary), and stops there with no registry contact when that works. It succeeds only when the same probe passes after the install. Without node on PATH it does nothing, because CMake then registers neither the bundle step nor its tests.

A registry outage now fails that step, after its retries, with a message that says so, rather than surfacing twenty minutes later as a compile failure. No tartci host mount carries an npm cache today, so --prefer-offline only helps where the runner's own ~/.npm persists; an npm cache mount analogous to TARTCI_PIP_WHEELHOUSE would remove the registry from the gate entirely. tools/scripts/test_node_build_deps_step.py runs the step's real script against a stub registry, and pulp_bundle_threejs_for_jsc_offline proves the bundler never launches npm under PULP_OFFLINE_BUILD.

Lane timeouts — and why a timeout looks like a broken PR

Gate-VM job timeouts are a different clock. On the self-hosted macOS gate the host's tartci job_timeout (2 h) is what finally ends a hung job, and it ends as failure, not timed_out, because the VM is torn down under the runner. A hung step therefore holds a gate VM for two hours unless the step carries its own bound. build.yml bounds "Bootstrap repository dependencies" at 15 minutes (it takes 0.3 min p50 and at most ~1 min on gate VMs): one such hang on 2026-09-26 held an m3 gate slot for 2 h 11 min. When a gate job ends in failure near the 2-hour mark, read its steps list: the last in_progress step is the one that hung.

[targets.<name>] timeout_secs in .shipyard/config.toml bounds how long a validation lane may run. The mac lane is 14400s (4h) as of 2026-08-20, raised from 7200s after the earlier 3600s ceiling also proved too short.

The reason the value matters more than it looks: when a lane hits it, Shipyard reports

✗ Validation failed. PR #NNNN not merged.
    Target:  mac
    Error:   Validation timed out

with no per-target diagnostics. That is indistinguishable from a genuinely broken branch, and the natural response — re-push, or start debugging the diff — is wrong in both directions. Always read the lane log before believing the verdict:

tail -40 "~/Library/Application Support/shipyard/logs/<job-id>/mac.log"
grep -c "error:" "~/Library/Application Support/shipyard/logs/<job-id>/mac.log"

A log that ends mid-build at some percentage with zero error: lines was killed by the clock, not by your code.

Two properties worth knowing when reading a timeout:

• Queue wait is not charged against the budget. The clock starts when the lane starts, so a job that sat pending for 40 minutes still gets its full window. Check started_at vs completed_at in queue.json to tell queueing apart from a slow build.
• Warm build dirs are the difference between passing and timing out. On one loaded afternoon, a small change against a warm dir finished in 41 min and passed, while a broad core/view change was killed at 98% after 62 min and again at 67% after 113 min on a cold dir. Whether a branch lands should not depend on that, which is why the ceiling was raised rather than left to look like flakiness.

If you are running heavy work on the same machine — a VM, a parallel build — it competes with the lane directly. Shutting it down is a legitimate first move when a lane is timing out marginally.

Validation Profiles

Shipyard validates from a profile (shipyard run --pipeline <name>). Pulp's .shipyard/config.toml defines three:

Profile	When to use	What it runs
default	Most PRs. The lane every cross-platform target gates on.	Full setup → configure → build → test. Examples ON. Excludes the slow ctest label.
parser	PRs that only touch runtime-import parser code.	Same stages with PULP_BUILD_EXAMPLES=OFF; tests filter to --label-include parser-import. Skips plugin validators (auval / pluginval / clap-validator) and the broader format-adapter smoke surface.
smoke	Quick downstream-scaffold check after dependency or install-layout edits.	Configure + governed build only (both cmake --build steps go through tools/ci/governed-build.sh); runs the SDK-smoke export against a downstream scaffold.
gates	Version-bump / skill-sync gate scripts.	tools/scripts/skill_sync_check.py + tools/scripts/version_bump_check.py in report mode.

shipyard config profiles lists what is installed locally and which one is active.

Interrupted-build guard on the POSIX profiles

All three build profiles reuse a warm build/, and that directory is shared with whatever the agent or human is building in the same checkout. A validation killed mid-compile — which is how timeout_secs ends a run, with a SIGKILL that executes no trap or exit handler — leaves partial object files behind. The next incremental build links those against freshly compiled ones, mixing object layouts, and the result is heap corruption and SEGFAULTs in tests unrelated to the change. Each timeout seeded the next run's failure, so once the pattern started it sustained itself, and every symptom read like a bad diff.

The POSIX configure stages therefore run:

tools/ci/build-dir-sentinel.sh guard build 'cmake -S . -B build …'

and their build stages finish with tools/ci/build-dir-sentinel.sh clear build. guard arms the marker, runs configure, and recreates the directory on the next run if the marker survived; clear removes it once the build finishes. Windows profiles are excluded — they run under PowerShell, where the wrapper's path assumptions do not hold.

guard distinguishes a stage that failed from one that was killed, and the difference is worth real money here. A configure that exits non-zero on its own — a dependency-floor mismatch, a missing toolchain — produced no object files, so wiping its directory buys nothing and costs a cold rebuild, which is precisely what pushes the next run over the cap. Those clear the marker and keep the warm tree. A stage killed by a signal keeps it. The test for this is specific: the wrapper can outlive a kill that reaches only its child, so "we reached the line after the command" is not evidence of a clean exit — the shell's 128+signum convention is what the check reads.

Two things are deliberate and worth preserving:

• It is a file, not a handler. Nothing runs at kill time, so anything that must execute to record the failure is structurally blind to it. The marker is written beforehand and removed afterwards, so abnormal termination leaves it behind by default.
• It is armed before cmake, not after. build.yml's macOS lane arms its equivalent after configure; a run killed during configure escapes that, and a half-written CMakeCache.txt is its own kind of broken.

Do not lift this into a lane whose timeout the build cannot finish under. Arming a sentinel against a too-short cap produces an infinite wipe → cold rebuild → timeout → wipe loop, roughly an hour of a shared machine per cycle. [targets.mac] timeout_secs is 14400 for this reason; check it before changing either number. tools/scripts/test_build_dir_sentinel.py asserts both directions plus the arm/clear pairing — a profile that arms without clearing wipes its build dir on every run.

Auto-selecting the parser profile

tools/scripts/validation_profile_select.py classifies the current diff and prints parser or default:

# Default: diff HEAD against origin/main
shipyard run --pipeline "$(python3 tools/scripts/validation_profile_select.py)"

# Explicit diff base
shipyard run --pipeline "$(python3 tools/scripts/validation_profile_select.py --base origin/develop)"

# Operate on a literal file list (e.g. piped from gh pr diff)
gh pr diff <PR> --name-only \
  | python3 tools/scripts/validation_profile_select.py --paths-from -

# JSON envelope (profile + matched + unmatched)
python3 tools/scripts/validation_profile_select.py --json

The script returns parser only when every changed path falls inside the explicit parser-only scope (the standalone tools/import-design tool, tools/import-validation scripts, the packages/pulp-import-ir package, test/fixtures/imports/**, the parser test files in test/, the core/view/.../design_import* family, and the import-runtime JS). Any path outside that set forces default — the safety bias is toward broad validation.

To opt out for an individual run, pass --pipeline default explicitly.

Changed-surface risk selection (controlled canary)

The required macOS target also declares a schema-v3 changed_surface_selection policy. It classifies an exact diff into mandatory, affected, extended, or full validation and records both the selected test set and its reviewed CMake producer targets. A protected-base execution template may atomically replace the POSIX local build and test stages, but repository config cannot enable it. Shipyard reads the separate machine-global changed_surface_execution.mode switch, whose absent/default value is off. The controlled shadow_compare mode builds only the declared producer targets and runs selected tests first, then runs the unchanged full build and CTest corpus, returns the full path's status, and emits an immutable receipt with separate selected/full build and test wall times, registration counts, and failure-coverage classification. Thus the full path remains merge-authoritative throughout the canary.

The full build follows the selected-target build in the same locked warm tree, so its direct timer is explicitly recorded as the incremental remainder, not as an independent full-build baseline. The receipt's estimated total full-build duration is the selected-build duration plus that incremental remainder. Use that derived total for shadow speed comparisons; using the remainder alone would overstate the selected-build savings.

Each schema-v3 selection produces an execution-receipt schema that also binds the policy, selection, validation, workflow, literal-test, and literal-build- target digests plus a durable timestamp, so a later session can aggregate trials without reconstructing the originating agent. A receipt is graduation_eligible only when shadow comparison ran and both suites passed. missed_full_failure and selected_only_failure are explicit mismatched_non_graduation evidence. Two failures are failure_overlap_unproven, not graduation evidence, because terminal status alone does not prove both suites found the same failure. Receipt publication fsyncs both the file and containing directory.

Nothing pins the exact CTest inventory. A bounded run configures the protected base commit (the merge base the plan verified) in a scratch worktree under the build directory, with the same generator, Python and policy build_flags, and requires this tree's registrations to equal the base's. A configure lists every registration without building, so this costs one configure (about 20 to 30 seconds through the build governor), cached per base and flag set. The comparison uses the configuration-neutral projection in changed_surface_inventory.project_registrations: source, build and Python paths and the build type become tokens, discovered Catch2 cases are one row per executable (their case list is the executable's content, not its registration), and a build-tree program is folded to one token because an unbuilt base lists none; the run's receipt counts those name-only rows. A base that cannot be configured, a checkout whose merge base is not the plan's base, a registration left without a command after the build, or any difference from the base selects the full suite with the reason (inventory: base not recorded, inventory: base mismatch). The base configure runs with FETCHCONTENT_FULLY_DISCONNECTED=ON, so dependencies resolve from the machine's shared FetchContent source cache and a miss fails the configure instead of cloning mid-plan. That includes the prebuilt WebGPU runtime archive, which the webgpu dependency fetches through FetchContent: with its cache entry removed, the disconnected configure fails ("requires source directory for dependency wgpu-macos-aarch64-release to already be populated") and downloads nothing. Its "Fetching WebGPU implementation" status line prints either way. The one download FetchContent does not govern is the second WebGPU slice a universal (arm64;x86_64) build lipos in tools/cmake/PulpWgpuUniversal.cmake; a universal policy flag set or build cache therefore refuses before any configure, and a test keeps that the only raw file(DOWNLOAD) in the CMake tree. A cold lane build directory refuses by design ("no command after the build"): bounded runs need the lane's normal warm build. The cached base inventories live inside the lane's build directory, which pull-request code can write; that is acceptable while the lane runs PR code and is not a required gate, and must be revisited if it is ever promoted. A bounded plan never carries a registration change of its own: CMake and test/cmake/** are test-topology paths, which select the full suite. The required gate also records the same projection as registrations-<sha>.json in each job's reuse-record-macos artifact, recordable only when no registration lacks a command, for measurement; it is not on the validation path, because Shipyard's validation commands carry no GitHub credentials.

When the lane's full suite carries known reds, two failing legs are compared per test from each leg's ctest --output-junit report. The plan is matched_fail (never matched_pass) when the selected leg failed nothing the full suite passed and saw every full-suite failure inside its selection, and it graduates only if every full-suite failure outside the selection is an unexpired row of tools/ci/changed_surface_lane_reds.json, read from the protected base. That file is a policy path, so a PR editing it runs in full and cannot allowlist its own regression; owners delete their row when they fix the test.

The mandatory kernel always runs, including the selector's own changed-surface-policy-selftest (the policy tables, from source) and changed-surface-policy-inventory (the live-tree inventory check: it takes --build-dir only under PULP_CHANGED_SURFACE_INVENTORY_TARGET and runs bare elsewhere; the selftest takes no build-tree argument because it is also a source-lane test). Known build-system, CI, ABI, public-header, security, provenance, packaging, dependency, policy, and test-topology changes require the full suite; unknown paths fail safely to full as well. Reviewed bounded families cover Forge/DSP catalog projection commands, the isolated ChildProcess test source, and the Forge Rack module generator plus its safety contract; generated families cover Python tool scripts and skill docs (below). Each family names literal affected tests and any required extended tests; neighboring paths remain full. The declared inventory is tied to the required macOS Debug configuration, which enables PULP_CHANGED_SURFACE_INVENTORY_TARGET; optional Linux targets do not assert that platform-specific cardinality.

Python tool scripts and skill docs select through generated families. tools/scripts/changed_surface_script_families.py reads test/ctest_script_inputs.json (each script-driven ctest's entry and the inputs it imports or names) and writes one family per group of top-level tools/scripts/*.py files sharing the same readers into .shipyard/changed-surface-families.toml, which the selection names as its families_file (Shipyard reads it from the protected base and treats it as a policy path), plus agent-skill-docs for .agents/skills/*/SKILL.md and a script-surface-whole-tree family that adds the drift, lint, registry, sync, guard, census, inventory and probe tests to any of those changes. A script-surface-environment-bound family likewise adds every declared script test with a RESOURCE_LOCK or a gpu, browser-capture, audio-device or environment-bound label: their failures follow the host, not the diff. Label a test environment-bound when its outcome depends on timing, the toolchain or host state and no other label says so. A script stays unmapped, and so selects the full suite, when native code, a shell or JavaScript file, or CMake (other than as a test's entry) names it, transitively; when a reader needs a CTest fixture or runs a build product no non-example target produces; or when nothing in the authoritative corpus reads it. Each family also builds the targets whose products its readers run. A Python test script's body is not CTest registration, so tools/scripts/test_*.py is no longer a test-topology path; test/cmake/** and CMake files still are. The skill and guide prose that describe this policy are not policy inputs either. The generator, the script-inputs list and the inventory contract are.

changed-surface-script-families-drift (macOS) regenerates the file and compares it. Like script-test-inputs-drift it is diff-scoped: drift blocks a change touching a script, a skill doc, the script-inputs list, the generator or the config, reports otherwise, and never fails a merge group. Regenerate with python3 tools/scripts/changed_surface_script_families.py --build-dir <dir> --write after the script-inputs list. The build directory needs a CMake file-API codemodel reply; without one the check reports a skip.

The pre-push hook runs the build-free half, changed_surface_script_families.py --static --base origin/main: it predicts the mapping of each script the change adds, removes or re-reads from the committed script-inputs list at the merge base and at HEAD, and blocks a mapping flip the families file does not carry (about a second). Comparing against the base cancels what the prediction cannot see without a build (labels outside the authoritative corpus, fixtures, build products), and a reader-set change is reported rather than blocked. tools/scripts/gates.sh runs it before the configured check, which decides when the two disagree, and fails rather than reporting NOT CHECKED when the configured check cannot run on a change to a surface it blocks.

Documentation under docs/guides/**, docs/reference/**, docs/examples/**, and docs/validation/** selects only that mandatory kernel. Generated or authoritative state under docs/status/** remains fail-closed for this bounded selection rather than inheriting that rule (the separate mobile allowlist below treats it on its own terms).

The required Build-and-Test workflow also uses a separate, narrower mobile-safe allowlist to avoid an unrelated mobile compile tax. On pull requests and merge groups, only a diff whose every path matches ios_compile_skip_safe_paths may emit the exact ios_compile_required=false authorization. A bounded macOS test family does not inherit mobile-skip authority; each allowlist addition requires its own mobile-impact review. The allowlist covers docs/**, test/**, tools/scripts/test_*.py, .agents/skills/**, the Claude plugin manifests and CHANGELOG.md: the gate configures with PULP_BUILD_TESTS=OFF, so none of those reach it except the paths IOS_COMPILE_REQUIRED_PATTERNS in tools/scripts/classify_changes.py denies first (the gate's own scripts, the CoreMIDI harness sources, and each test//docs/ file a non-test CMake file names, re-derived from the tree by test_classify_changes.py). The macOS job then skips the two-SDK iOS compile step but still performs its ordinary desktop build and tests. Missing, malformed, empty, mixed, unknown, policy, CMake, CI, public header, non-test tools/scripts, or apple/** evidence runs the iOS gate.

Paths outside that allowlist no longer run it by default either: on pull requests and merge groups the gate runs only when a path reaches an iOS-only surface (ios_surface in classify_changes.py: apple/**, iOS examples/templates, iOS platform sources, any source that branches on an iOS target, non-test CMake, dependency pins, the simulator Skia fetcher, the gate's own wiring). Shared code is compiled for macOS by the same job, so a break there fails the macOS build; measured over 14 days the per-PR iOS gate ran 1,374 times and caught nothing the macOS build would have missed, at about 790 gate slot-minutes a day. The full gate runs nightly on main (.github/workflows/ios-compile-gate-nightly.yml, on the base self-hosted gate labels, opening one de-duplicated tracking issue on failure) and on every release tag (release-cli.yml ios-compile-gate, which the publish job requires). Pushes to main, manual runs, nightly/release workflows, and audits never accept this skip; their existing event policy remains unchanged. Keep the condition inside the required job: path-filtering the workflow or job would prevent the stable required context from reporting.

CTest display names are not identities: the authoritative target registers some names more than once. The inventory validator therefore fingerprints a canonical {name, executable, argv, working_directory, properties} composite and treats the suite as a multiset. Literal selection expands every composite with the requested name; missing commands, duplicate properties or duplicate composite identities require the full suite. Source-head, source-tree, and toolchain provenance remain in the emitted manifest for comparison. External executables use a portable basename in the registration fingerprint while the toolchain digest binds their raw and resolved paths plus a bounded content digest, so same-named tools remain distinguishable. Raw worktree paths and CTest registration order are intentionally excluded from portable identity. full_test_count is an upper bound for the declared literal tests and receipt telemetry, not a pin.

Do not promote selection from shadow to authoritative based on a few green runs. Graduation requires per-risk-class comparison evidence showing that the selected receipts agree with full validation, plus a separately reviewed policy change. Tests remain in the repository and continue to run in full for unknown/high-risk work, main, nightly, release, and audit surfaces.

Ordinary and changed-surface build-and-test stages serialize access to the same canonical build directory through tools/ci/build_dir_lock.py. Its persistent lock file lives in per-user host state, not beside build/, so acquiring the lock cannot dirty the exact source checkout that changed-surface execution verifies. The full canonical build path is hashed into the filename and independently bound inside the locked file; aliases converge, while equally named build directories in separate worktrees stay independent. Lock files intentionally remain after unlock so queued waiters keep one inode. Tests may set the trusted, absolute PULP_BUILD_DIR_LOCK_ROOT override; production uses durable per-user application state with owner-only permissions (or the user's inherited profile ACL on Windows), rather than an OS-purgeable cache or runtime directory.

Interactive builds take the same lock without waiting. tools/ci/governed-build.sh runs every cmake --build <dir> under build_dir_lock.py --no-wait, which refuses a second build into a directory another live build holds (exit 75) and prints that holder's pid, liveness, working directory and command. The lock is a kernel flock released when its holder exits, so a killed build leaves nothing stale; a holder exports PULP_BUILD_DIR_LOCK_HELD, so a locked stage may still call the wrapper for its own tree. Before any lease, the wrapper and the root CMake configure also refuse a source checkout under /tmp, /private/tmp or $TMPDIR (exit 3; tools/ci/checkout_location_guard.py), because such a tree misses the shared ccache on every compile. PULP_ALLOW_TMP_CHECKOUT=1 overrides that for a deliberate throwaway checkout, and GitHub Actions jobs are exempt.

Runs on main: cache warming and main's health

build.yml triggers on push: branches: [main] in addition to pull_request / merge_group / workflow_dispatch. That run gates nothing. Its job is to publish the GitHub-hosted Linux/Windows ccache and FetchContent caches that PR runs restore from. It has no macOS leg and runs entirely on GitHub-hosted runners.

It is needed because of how GitHub's cloud cache is scoped: a cache entry written by a PR run is visible only to that PR's own ref, so PR runs can never warm each other. Only a non-PR run on the default branch writes an entry every subsequent PR can read. Without the push trigger the Save … steps are unreachable and the matching Restore … steps are a permanent miss.

Each trigger runs a deliberately different slice of the matrix:

	PR run	merge_group run	push: main cache run
macOS matrix leg	yes	yes	no — main's health is the merge group's macos job
Linux matrix leg	yes	no — PR-head result is reused	yes (publishes the cache)
Windows matrix leg	no — see below	no — see below	yes (publishes the cache)
windows-{msvc-release,midi2,ble}-gate	no — see below	no — see below	no
required direct macos context	yes	yes	no
Writes to GitHub's cloud cache	no	no	Linux + Windows only

Why the push run has no macOS leg

main's macOS health is already measured, by the merge queue. The queue lands with the MERGE method, so the commit on main IS the merge-group head (checked over 12 consecutive first-parent commits of main: each had a merge_group Build and Test run whose head_sha equalled it). That run's required macos job built and tested main's exact commit. A push leg would test it a second time.

It also never did. Push runs carry no event class, so the leg kept the legacy base selector (…pulp-gate-fast), which the event-class-v2 pool does not serve. It queued for hours while holding the refs/heads/main concurrency group, and because push runs are exempt from cancel-in-progress, GitHub kept at most one more push run pending and cancelled it when the next merge arrived: 57 of the last 60 push runs were cancelled with zero jobs started, and the Linux/Windows cache-save steps ran at most once per stuck window. With the leg omitted, a push run finishes on hosted runners and the caches publish on every merge.

a2t-protected-event, a push-only job, runs on GitHub-hosted macos-15 for the same reason: its verifier needs only macOS, git and python3. a2t-headroom-issue, the other push-only job, runs on ubuntu-latest; it only reads an artifact and calls the GitHub API. tools/scripts/test_fork_pr_runner_routing.py (run by workflow-lint.yml, which installs PyYAML) walks every build.yml job reachable on push and fails if any routes to a runner that is not a GitHub-hosted label or a hosted lane variable in runner_topology.json, and runs the embedded resolver on a push event to prove its matrix is Linux + Windows only.

Caches are still saved only from GitHub-hosted Linux and Windows: the two Save … steps stay scoped runner.environment == 'github-hosted' && runner.os != 'macOS', which is narrower than the restore side on purpose, because the self-hosted Macs keep ccache and FetchContent on local disk between jobs.

tools/ci/test_build_matrix_contract.py (ctest build-matrix-contract) pins the guard: it asserts over the parsed syntax tree that the macOS matrix entry sits under exactly if EVENT_NAME != "push":, so gating it on any other event (which would drop the required gate itself) fails a test.

Base health detector (main-health-detector.yml)

A broken main is amplified rather than caught by a merge queue: every batch is main plus its entries, so every batch inherits the break, fails after a full gate, ejects its innocent entries, re-forms, and pays again. Nothing in a batch's own report says the base is the cause, so each batch reads as a new culprit in turn.

.github/workflows/main-health-detector.yml runs tools/ci/base_poison_detector.py right after every completed merge-group Build and Test run, green or red (workflow_run, completed, gated to event == 'merge_group'), on a schedule as a backstop, and on demand. A green group is usually a new tip, and without a read then a healthy tip never gets a verdict. The event trigger is the one that matters: GitHub throttles this repository's schedules to roughly one run every four hours whatever the cron says, longer than a batch's lifetime. A workflow_run job runs the default branch's copy of the workflow with base-repository permissions, so the detector checks out only the default branch (never the triggering run's head) and its token is read-only (actions, checks, contents, pull-requests, statuses). It runs on the preamble Linux runner, draws no macOS gate host, and reports only — pausing a re-forming batch or prioritising the fix is Shipyard's side and is not wired here. One detector runs at a time (group: main-health-detector, cancel-in-progress: false); a superseded tick loses nothing, because the next one reads main's current head, which is strictly more relevant than the head the cancelled tick would have reported on.

Where main's evidence comes from without a build. The merge group whose head sha is main's tip (main_evidence_source: head-sha). Under the MERGE method its runs tested exactly the commit on main. It is judged on every required status context — .shipyard/config.toml [governance] required_status_checks (six contexts: macos, Enforce version & skill sync, Build + prove + (owner-gated) deploy, Vellum freeze, Vellum trusted freeze, drift-fast) — gathered from the merge_group runs of every workflow in [landability] workflows: build.yml's jobs through the jobs API (their steps tell an executed macos suite from a reused receipt), the other workflows' jobs through one commits/<sha>/check-runs read kept to those runs' check suites (so a push run on the landed commit is never read as the gate), and the commit's statuses for a context published as a status. Never by a run's conclusion, which used to fold in advisory legs: while hosted Linux was failing every merge group, reading the run called every green tip red and turned the Linux failure into a fake batch streak. In a merge group the Linux matrix leg is now job-level continue-on-error, so it no longer turns the run's conclusion red, but a run conclusion still cannot say which required context failed. And never by macos alone: on 2026-09-29 the required drift-fast context was red on main for about fourteen hours while macos stayed green, the macos-only detector read healthy throughout, and every batch that failed drift-fast read as a pass, so the batch streak stayed 0. Replaying tip 31e644ed61b2 read-only now reports poisoned (drift-fast failed gpu-probe-historical-v1-acceptance on main and in 5 consecutive batches).

The tip is healthy only when every required context is present and green and macos executed the suite; a context that failed makes it red whatever the others say; a context still running or never reported leaves it unproven, and its reason names the context. The tip's runs are read at any status, because a group lands as soon as its required checks pass while advisory legs may still be running. When the tip is not evidence (a reused receipt, still running), an earlier merge group whose head tree equals main's tree is used instead (tree-identity): the tree determines what was built. When the tip has no merge_group run at all (an admin or direct push), the verdict is unproven and its reason names the tip. The signal says which source it used and carries main_head_sha. Batches are judged the same way, newest head first, so a batch counts toward the streak when any required context failed on it.

main's push run is evidence too. A push to main runs the Linux leg of Build and Test on main's exact commit (no macOS leg, and none of the other required contexts). When that leg's test step ran and it failed a test that the batch streak also shares, the verdict is poisoned with proof main-push-linux-failure+batch-streak, even when the merge group that landed the commit was green. That is the shape of a test that passes when its commit lands and fails on main afterwards: on 2026-10-23 main's tip 5489bbad landed through a green merge group, its push run then failed gpu-trace-overhead-acceptance-selftest on Linux, and eleven consecutive batches failed the same test on macOS, while the detector read healthy. The failing names come from the leg's ctest-logs-linux artifact. The streak keeps a Linux-only failure out (the batches run macOS, so a test only Linux fails is never in their shared set), a leg that ran no test step is not evidence, and a green push run is never read as health. The signal carries main_push_run_id, main_push_execution and main_push_failing_tests. tools/ci/fixtures/base_poison_2026-10-23.json replays that night.

Shipyard reads the signal. .shipyard/config.toml sets [base_health] auto_jump = "dry-run": shipyard base-health --act records "would jump PR #n" once per poisoned episode that names a candidate_fix_pr, in <state>/base-health/jump-decisions.jsonl on the host that runs it, and changes nothing. on (dequeue the fix and re-enqueue its head with jump: true) stays off until it checks a trusted approval on the head it enqueues and survives a failed re-enqueue.

Failing test names come from wherever the failing context records them: for macos, the ctest-logs-macos artifact (Testing/Temporary/LastTestsFailed.log); for a context that uploads no artifact (drift-fast), the ctest "The following tests FAILED" block of its job, read from the run log zip. Never the per-job log endpoint: ghapp api .../actions/jobs/<id>/logs refuses any response carrying terminal escape sequences and returns a short refusal instead, so a log scrape silently yields no failing tests — which reads as "the failure was not a test failure".

Replay an earlier state read-only with --tip-sha <sha> (judge that commit as main's tip) and --before <ISO time> (batch history as it stood then).

The signal. A ::notice title=base-poison-signal:: annotation carrying one line of compact JSON (schema base-poison-signal/v1), plus a job-summary table and a base-poison-signal artifact. Fields: status, proof, safe_to_pause_queue, tests, main_observed, main_run_id, main_evidence_source, main_head_sha, main_failing_tests, main_failing_contexts, main_contexts (one {context, state, conclusion, source, run_id, tests} per required context), batch_streak, batch_streak_tests, batch_streak_contexts, batch_streak_heads, batch_streak_runs, main_push_run_id, main_push_execution, main_push_failing_tests, candidate_fix_pr, likely_culprits, required_contexts_source, reason. Fields are only ever added under base-poison-signal/v1; Shipyard's reader keys on schema, status and safe_to_pause_queue.

likely_culprits (filled only with --name-fix-pr) lists the queued entries whose presence separates the merge groups that failed a test from the ones that ran it green over the last 24 hours (--history-since, capped at --history-limit runs), from queue_batch_attribute.py --history. It names a pull request to dequeue; candidate_fix_pr names one to prioritise, and the two are never merged. required_contexts_source is protection when every required context was judged, and macos-only-fallback when branch protection could not be read (the job summary then says the finding judged macos only).

status	Means	safe_to_pause_queue
healthy	every required context passed on main's tree and macos ran the suite	false
unproven	a required context is still running, missing, or macos ran nothing — main's health is unmeasured	false
suspected	a streak, or a main failure not shared by a streak (or naming no test)	false
poisoned	a required context, or the executed Linux leg of main's push run, failed a test on main that a streak of consecutive executed batches also failed	true

safe_to_pause_queue is the only field a consumer should act on. It is true for poisoned and for nothing else.

Two rules it deliberately refuses, because both are measured unsafe and both look convincing:

1. Cross-batch corroboration of an identical failure. Batches with disjoint single-entry memberships have failed with a byte-identical error at different bases while one of those entries was genuinely broken. That rule is structurally indistinguishable from the innocent case, so a streak alone is suspected with proof: batch-streak-only — a prioritisation hint, never proof, and it never pauses anything.
2. "No ctest block, therefore infrastructure." A compile or link error produces no ctest block either. A failure naming no test contributes an empty set and so cannot carry any verdict that names one.

It fails closed in every direction. A receipt-reuse green (three steps, no build, no test) is not evidence and reads as unproven rather than healthy; a cancelled run is not evidence, so cancelling doomed batches cannot turn a red base green; and absence of evidence is never health. A fix PR is named only on the attributor's decisive weight — an exact test-name to file-stem match with a single contender — because prioritising the wrong branch is worse than prioritising none.

tools/ci/test_base_poison_detector.py (ctest base-poison-detector-selftest) pins each of those refusals, and also asserts that a workflow actually invokes the detector: a correct rule nothing runs reads exactly like one that works, which is how a designated push lane once went 58 runs without producing a single observation.

Push runs are also exempt from cancel-in-progress: they share the refs/heads/main concurrency group, so cancelling a superseded one would kill its cache-save step exactly when main is busiest. PR runs still cancel.

Keep job-level if: gates on !cancelled(), never always(). A job gated on always() runs even when its run has been cancelled, so a superseded run keeps building, stays in_progress, and goes on holding its group. Every newer head of that PR then sits at pending with zero jobs, which is indistinguishable from runner starvation from the outside, and an ordinary POST .../cancel will not free it: only force-cancel bypasses always(). !cancelled() buys what these gates actually need, since it still evaluates when an upstream need failed or was skipped. Step-level always() is fine and is used deliberately for log upload. tools/scripts/test_build_workflow.py enforces the job-level rule.

A workflow_dispatch run of build.yml does not share a group with that branch's PR runs. The group keys on github.ref, which is refs/heads/<branch> for a dispatch but refs/pull/<n>/merge for a pull_request, so the two coexist and neither cancels the other. Anything that sweeps duplicate runs must key on the event as well as the branch, or it will cancel a deliberate manual dispatch as a phantom duplicate.

The classify job diffs an event-dependent base (tools/scripts/resolve_classify_base.py): a PR diffs github.event.pull_request.base.sha, a merge group diffs github.event.merge_group.base_sha, and a push diffs github.event.before. Those immutable event SHAs let the preamble use a depth-1 head checkout, fetch only a missing exact base object at depth 1, and compare the two trees directly. It must not fetch Pulp's full history merely to compute changed paths: on a reusable traveling runner that grew .git to 70 GiB and delayed a cheap classifier by more than 20 minutes. If the exact base is missing, malformed, or cannot be fetched, classification fails closed to a native build. The job has a 10-minute outer timeout so a disconnected roaming runner cannot occupy the preamble indefinitely. On a push, origin/main resolves to HEAD itself and the diff is always empty — so a docs-only merge is otherwise indistinguishable from a core merge, and the run never skips. A docs-only merge to main now correctly skips the whole matrix.

Wide non-native classification (opt-in, default off). With the repository variable PULP_CLASSIFY_WIDE_NON_NATIVE unset, classify behaves exactly as above. Set to 1, it passes --wide-non-native to classify_changes.py, which consults tools/scripts/wide_non_native.py for the files the base allowlist kept native. A file under tools/scripts/, tools/testing/ or tools/import-validation/ is admitted only when every tracked file naming its stem is inert (Markdown, docs/, planning/, .agents/, a workflow other than build.yml / build-macos.yml / .github/actions/**), a script run by tools/ci/source_selftests.json or tools/ci/wide_non_native_checks.json, the CMake registration of one of those tests, or another admitted file. Anything else keeps the native build: a gate-side ctest registration, tools/ci/**, tools/cmake/**, C++ or test sources, the classifier's own files, a failed or over-budget reference search. The gate's repository scanners that walk tools/ without naming files run from tools/ci/wide_non_native_checks.json on the required Enforce version & skill sync context whenever the variable is 1. .github/workflows/** and tools/ci/** are never widened. Over the 177 PRs merged 2026-09-21..27 the widening admits 9 more (19 skip-native instead of 10); tools/scripts/test_wide_non_native.py replays that window.

The classifier also establishes its interpreter explicitly. A macOS LaunchAgent normally sees only /usr/bin:/bin:/usr/sbin:/sbin; on M5 that made the preamble use Apple's Python 3.9 and fail importing tomllib, while the same merge group passed when M3 claimed it. build.yml prepends the Homebrew and user-bin locations, resolves one Python 3.11+ executable through tools/ci/find_python311.py, and uses that executable for every classifier and JSON parser in the job. Keep this fail-closed interpreter selection with the self-hosted preamble route; interactive-shell PATH is not fleet configuration.

One semantic fast path sits above that path classifier. A same-repository release/version-bump pull request may skip the native matrix and the required WebCLAP proof only when the protected base's tools/scripts/generated_version_bump_check.py verifies one open pull request, one signed release-bot commit whose parent is an immutable protected-main base, the fixed branch/title/message marker, and a candidate tree byte-identical to rerunning that protected base's version-at-land writer. The verifier resolves a merge-group candidate through exactly one associated pull and then fetches the detailed pull record. A bump queued behind exactly one earlier entry may still qualify in GitHub's observed #7771 topology: the event group must have exactly [prior cumulative group, generated candidate] as its parents, the candidate must have one original protected-main parent, and that SHA must be the prior group's first parent. The verifier is loaded from that immutable original parent, never from the speculative cumulative group. The writer and its derived generator's complete local executable dependency closure must be byte-identical across the range, every derived-file subprocess is rebound to the protected copy, and the complete event tree must equal the cumulative base plus only the regenerated version projection. Multiple candidate associations, nested/unknown topology, writer drift, missing protected code, GitHub API or signature errors, pagination ambiguity, or any additional byte retains ordinary validation. Version/skill enforcement and both Vellum gates still run, and pushes to main, releases, scheduled work, and manual dispatches retain their normal validation.

On a pull_request event the run checks out GitHub's synthesized refs/pull/N/merge commit, and a freshly opened bump PR's payload usually has a null or stale merge_commit_sha and no after, so by default the verifier cannot bind GITHUB_SHA to the event and every PR-event bump run takes the full native path (only merge groups are fast-pathed). Setting the repository variable PULP_BUMP_FASTPATH_PR_MERGE_REF=1 (unset by default) makes build.yml pass --accept-pr-merge-ref to the protected-base verifier, which then accepts that checkout only when it is the working HEAD, its parents are exactly [pull_request.base.sha, pull_request.head.sha] in that order, and its tree is byte-identical to the head's tree. Every other check above still applies, so fork PRs, non-bump branches, stale bases, force-push races, and merge commits with extra content all keep the full path. The variable is read by the workflow, never by PR content; a base verifier that predates the flag rejects it and the run keeps the full path.

When the exact projection refuses, the same verifier is run once more with --allowlist-diff: the signed single-commit bump is still required, but its content is bounded by the protected base's writer_allowed_globs (the writer's own allowlist) and the Version-Bump-Applied: marker instead of a byte comparison, and the allowlisted code/build files (CMakeLists.txt, agent_capability_manifest.py) may change only their version values. The outcome is still reported as taken. See version-at-land cutover.

Every run on the release bot's bump reports the outcome as a bump-fast-path notice on the classify job, and in its step summary: taken, or refused: followed by the reason. The reason is the verifier's own line (verifier: ...), shape: no trusted verifier base when the group or pull-request base resolves no trusted base (for example, main moved more than one queue entry past the bump's parent, a case the verifier is never asked about), or the protected base has no verifier. Only the bump is reported; other runs print nothing. tools/scripts/bump_fastpath_report.py counts the notices (--days 7 by default), and .github/workflows/bump-fastpath-weekly.yml runs it every Monday into its job summary. A run from before the notice existed counts as unreported, never as taken.

The four advisory workflows that pull_request: auto_merge_enabled starts (coverage.yml, control-shipping-native.yml, examples-validation.yml, intel-portability.yml) skip every job on release/version-bump: the bump's tree is main's plus version files, which they already ran on, and none of their contexts is required. Each job's condition is github.head_ref != 'release/version-bump' && (<its own condition>), so every other pull request, schedule and dispatch runs exactly as before. This also keeps bump pull requests off GitHub-hosted Windows runners.

The provenance boundary is the signing KEY, not the author. GitHub's valid SSH-signature record and the danielraffel account are necessary but not sufficient: every fleet agent's key is registered to that account and verifies too, and author/committer name and email are free text. The verifier extracts the public key embedded in the verified SSHSIG and requires its fingerprint to be listed in tools/scripts/release_bot_signing.json read from the protected base. That file records the public half of RELEASE_BOT_SSH_SIGNING_KEY (GitHub signing-key id 1289902, Actions-only, SHA256:1GDNVI...); rotating the secret means landing the new public key there first.

One admission writer feeds the merge queue

The steward on the configured mutation machine (m3, mutation_machine = studio) is the only writer that admits Pulp PRs. A launchd job runs shipyard runner steward --repo Generous-Corp/pulp --base main --arm-unqueued --apply every five minutes. It runs from a config-only sparse clone on the internal disk, because a launchd job that opens a path on /Volumes can block in the kernel on a privacy prompt nobody can see. gh must resolve to the ghapp wrapper so every arm passes the queue-arm guard and lands in the arm ledger. The job refuses to run otherwise. Shipyard re-reads the exact-head approval immediately before each enqueue and refuses on a HOLD, a missing approval, or unreadable state. The native queue then validates and merges.

.github/workflows/shipyard-merge-steward.yml is a read-only diagnostic. It has no schedule, no apply input and only read-scoped tokens, and it is not in .github/schedule-backstop.json. A second writer, scheduled or dispatched, would bypass the m3 guard and race the mutation-authority flag. Its census also failed under the Actions-minted token on every run, so it never admitted a PR.

[train] mode = "off". The native queue already batches up to five entries, and a second grouping layer added planning work without forming a train. shipyard train status and shipyard train metrics still report queue depth and approval-to-merge time.

A merge_group batch stops at the first failing test

The full suite is roughly 21,764 tests. A batch that has one failing test is already unlandable, so the remaining tests prove nothing the first failure did not, and a merge queue multiplies the waste: every batch inherits a broken base, every platform lane pays it, and each re-formed batch pays again. One observed cascade had its leader fail at test 101 and spend about eighteen more minutes proving it, across three lanes and more than ten consecutive batches.

So merge_group runs ctest with --stop-on-failure, and no other lane does. Every other lane runs to completion and reports every failing test; for the complete macOS list of a failing change, run it through a Shipyard workflow_dispatch.

--stop-on-failure composes with the existing --repeat until-pass:2 rather than defeating it. ctest stops only after the retries are exhausted, so a single flake still self-heals and only a test that fails every attempt ends the run. Under -j8 the stop is bounded by the parallel width: the tests already in flight finish, so a suite failing at test 101 runs about 8 more, not 21,663.

Both event-dependent ctest decisions (this flag, and the label set) live in tools/ci/ctest_gate_args.py so they are stated once and tested. tools/ci/test_ctest_gate_args.py (ctest ctest-gate-args-selftest) pins the rules AND asserts build.yml still calls the module and still passes both outputs to ctest, because a decision module that is correct but unreferenced reads exactly like one that works.

The label set follows the same split. performance, bench and quality-lab are relative-timing tests that survive steady load but not the load variance of a Studio running concurrent build VMs, so they are excluded wherever the suite runs on the shared self-hosted Macs, and kept on the steady GitHub-hosted Linux and Windows legs.

drift-fast: tree-reading drift checks on a hosted runner

Most merge groups that fail the required macos gate fail only Python registrations that read the checkout and the configured test graph, never a compiled binary: wide-non-native-selftest, the consumption-census-* checks, tools-registry-check, rack-generator-safety, prepush-refresh-push, script-test-inputs-drift, ctest-scheduling-contract and the source-selftest-lane-* pair. Over three days, 45 of 69 failed groups failed only those (or those plus known flakes). They surface only when a head is combined with the current tip, and on macos they run after a ~20-minute build that none of them needs.

.github/workflows/drift-fast.yml runs them on ubuntu-latest for every pull_request (the refs/pull/N/merge tree) and every merge_group, both of which are the tip combined with the head. The job bootstraps dependencies, configures exactly as the gate does (Release, examples off, GPU on) and does not build. It restores (never saves) main's Linux FetchContent cache, including the shared ~/.cache/pulp/fetchcontent-src that setup.sh clones into, and bounds a cold bootstrap at 20 minutes. Before configuring, it runs tools/scripts/hydrate_gpu_provenance_commits.py because the historical GPU probe acceptance registration walks commits outside the depth-2 event checkout. Without it that test raises ShallowCheckoutError on every run, a red check the macos gate never shares, so tools/ci/drift_fast.json lists the step under workflow_preconditions (each with a why) and tools/ci/drift_fast.py check fails when the workflow does not invoke it ahead of the driver. tools/ci/drift_fast.py run then runs the selection named in tools/ci/drift_fast.json: every member of ctest_labels (today the whole pr-fast tier, 99 tests) plus the listed registrations outside it, through ctest itself, so SKIP_RETURN_CODE, pass regexes, timeouts and resource locks behave as on the gate, with the gate's --repeat until-pass:2.

What it cannot check, and says so as NOT CHECKED lines rather than passing silently:

• ios-compile-gate-legs is registered only on Apple, so a Linux configure does not register it.
• consumption-census-drift and consumption-census-negative-contract compare against a recorded profile for this host. The census records only darwin-arm64-* profiles, so on Linux the comparison skips (exit 77); only a stale feature roster, which fails on every host, still fails here. The census content itself is still checked on macos.
• A selected test whose command names a build-tree file the configure did not create needs the build this job skips, so the driver drops it. Today that is project-package-mutation-control (pr-fast): a WILL_FAIL mutation control whose runner, without its pulp-test-project-package binary, would "fail as expected" and pass without testing anything.

It fails (exit 2) rather than passing vacuously when a listed label selects no registration or when ctest ran a different number of tests than were selected. drift-fast-selftest (tools/ci/test_drift_fast.py, on the source-selftest lane) pins the selection and verdict logic and holds the manifest and workflow to each other.

Adding a test: append it to tools/ci/drift_fast.json with a why. It must pass from a configure alone; a test that needs a built binary does not belong here. A red drift-fast should mean the same tree fails macos too. When a test is red only on this Ubuntu checkout (a shallow history, an Apple-only SDK clone the gate has and this runner lacks), fix its Linux precondition or name the step it needs under workflow_preconditions; do not let the lane carry a standing false alarm.

Local reproduction (configure only; no compile):

./setup.sh --deps-only --non-interactive
cmake -S . -B build-drift -G Ninja -DCMAKE_BUILD_TYPE=Release -DPULP_BUILD_EXAMPLES=OFF
PULP_SCRIPT_INPUTS_BASE=origin/main python3 tools/ci/drift_fast.py run --build-dir build-drift

tools/scripts/gates.sh does the configure half on its own for the one check most often broken by hand: when the diff can drift test/ctest_script_inputs.json and no current Ninja build exists, it configures build-gate (no compile) and runs the diff-scoped script-test-inputs-drift check, printing the --write command on drift. PULP_GATES_NO_CONFIGURE=1 opts out and reports the check NOT CHECKED.

On a warm M-series host this is about 60 s of configure plus about 40 s of tests. Set PULP_SCRIPT_INPUTS_BASE to scope script-test-inputs-drift the way CI does; without a base it compares every entry, and entries whose recorded path embeds a build-directory name report as stale.

It is required. drift-fast is a required status check on main, so a red run blocks the pull request and ejects the merge group, in minutes rather than after the ~20-minute macos build. It was promoted after 18 consecutive green merge_group runs with no false alarm, following two environment fixes that removed its only false alarms: the bounded GPU-provenance hydration above (gpu-probe-historical-v1-acceptance raised ShallowCheckoutError on the depth-2 checkout), and consumption-census-drift-description comparing only header roots the checkout actually provisions. The one caveat: in that window no merge group was genuinely drifted, so it has not yet shown a true catch at the merge-group stage; its pull-request runs have caught real drift (skip-not-pass-lint, script-test-inputs-drift) on heads that were then fixed before queueing.

Because it is required, it must report on every event it can see. A required check that never reports holds a pull request on "Expected — Waiting for status" forever. So the workflow triggers on every pull_request to main (drafts included, no paths: or paths-ignore: filter) and on every merge_group, and its job carries no job-level if:. tools/ci/drift_fast.py check fails if either invariant is broken, and drift-fast-selftest runs it. Step-level if: is fine: it skips a step, never the job. A pull request from a fork needs a maintainer's approval before any workflow runs, which holds every required check equally; see the contrib-intake skill.

Live enforcement is classic branch protection, not a ruleset; the checked-in .github/rulesets/main-protection.json and .shipyard/config.toml [governance] mirror it, and base-poison-detector-selftest fails when those two disagree (and the detector judges all six on main's tip). [governance] is the full six-context contract, because shipyard governance apply pushes exactly that list to branch protection. Removing it again is the reverse required_status_checks edit.

Every failed run names its cause: the advisory ci-cause job

When any job in a build.yml run fails, the ci-cause (advisory) job runs tools/ci/ci_cause_notices.sh. It reads each failed job's finished log through the Actions API and runs shipyard cause extract over it. The result is one ci.cause/v1 record per failed job, printed as a ::notice title=ci-cause:: annotation on the run: the failing test, compile error, runner failure or gate verdict, its class and a confidence.

• Why a separate job. A job's own log cannot be downloaded until the job has finished, so a step inside the failing job has nothing to read.
• Pinned and verified. The extractor is the Shipyard release Pulp pins in tools/shipyard.toml, the same one tools/install-shipyard.sh installs. Its SHA-256 must match the asset's line in tools/toolchain/shipyard.sha256, or nothing runs. A Shipyard pin bump therefore moves the extractor with it; a pin or checksum the script cannot read is reported and names nothing.
• Never a gate. The job is continue-on-error. A missing asset, an unreadable log or an extractor failure is reported and exits 0.
• Selftest. ci-cause-notices-selftest covers the script.

The records are the input to the root-cause dataset. Shipyard's scorecard --export-cases carries the same records, and shipyard cause archive keeps them append-only.

A green macos check does not always mean the suite ran

The required macos context is reported by more than one job. It can come from the macOS matrix leg, which builds and tests, or from a bootstrap job that claims the name without running anything, because the change touched no native build input or because an earlier run's protected receipt was reused. With PULP_MACOS_PR_GATING on, an unapproved pull-request head's macOS leg itself runs only its configure-only contract lane. Both look identical in the checks list.

A reused receipt produces a three-step job (Set up job, the bootstrap step, Complete job) reporting success. A pr_number dispatch adds the Tested refs/pull/<n>/merge at <sha> onto <base> step after the bootstrap step, so a skip-safe head's tier record can bind its approval to the merge commit; it runs no test. Reading one of those as evidence that the suite passed has already sent an investigation in the wrong direction for hours.

Two things address it. Each bootstrap job now writes a banner into its own job summary stating that no build and no test step ran, and why, so the void case is legible from the check itself rather than only from its step count. And tools/scripts/gate_suite_executed.py answers the question directly:

# executed / tier-only / built-but-untested / not-executed, per `macos` job
python3 tools/scripts/gate_suite_executed.py --run-id <run-id>

# exit nonzero unless the suite genuinely ran, for use in a script
python3 tools/scripts/gate_suite_executed.py --run-id <run-id> --require-executed

It classifies by the steps that actually ran, not by the conclusion, so it separates a real pass from a claimed one. Only a Test (<platform>) step is the suite. A pull-request head that ran only the fast deterministic tier or the affected-tests tier reads tier-only: the merge queue still runs the suite, and --require-executed does not accept it. It answers only whether the suite ran: an executed verdict on a failing job is still executed.

Attributing a failed batch to the PR that caused it

A merge_group batch is named for one pull request but contains every entry ahead of it, so the run title names a branch that is usually not the culprit.

# defaults to the most recent failed merge_group run
python3 tools/scripts/queue_batch_attribute.py [<run-id>]

It extracts the failing ctest names from the batch's macos job and maps each to the pull request whose files own it. Below its confidence threshold it reports LIKELY PRE-EXISTING ON MAIN and names nobody, which is the case that matters most: naming the wrong pull request sends someone to fix an innocent branch while the real break stays on main, failing every batch that forms. When it reports that, check main's own health first, which is what the push macOS leg above exists to tell you.

Letting an un-implicated head back into the queue

queue-arm-guard refuses a same-head re-enqueue after a failed_checks ejection, and branch-refresh-guard refuses the update-branch that would otherwise dodge it. Both are right on their own: re-queueing a head that broke a batch ejects its innocent batch-mates all over again, and a refresh buys nothing while cancelling and restarting a 40-minute required gate. Together they trap a pull request ejected by a defect that was never its own.

.shipyard/config.toml declares the way out:

[queue.attribution]
command = ["python3", "tools/scripts/queue_batch_attribute.py", "--certify"]

The guard runs that argv with --repo <owner/name> --pr <n> --run-id <ejecting run> appended, from the repository root, and allows the re-enqueue only when it exits 0 and prints a JSON object whose implicates_head is exactly false beside a verdict that positively names why — infrastructure or other_pull_request. A verdict recording what was not found certifies nothing, and neither does a missing or null implicates_head.

python3 tools/scripts/queue_batch_attribute.py --certify \
  --repo Generous-Corp/pulp --pr 8773 --run-id 35973715485

The first reading is the batch's chain ancestry. A queue ref is gh-readonly-queue/main/pr-<N>-<parent>, and <parent> is the commit the entry was stacked on — main's tip or the previous entry's group commit — so each batch is a one-entry experiment against its parent. Comparing only the required macos gate:

• parent passed (on main, or its group's macos went green) → the head is the culprit (implicates_head: true);
• parent failed and every test this batch failed, the parent failed too → the head is a neighbour: other_pull_request, implicated_pr = the ancestor whose own parent passed. A parent that failed before its suite ran (a Build failure) observed none of the tests, so it is looked through to the nearest ancestor whose suite ran; if that ancestor passed, nobody is named;
• parent failed but this batch failed a test the parent did not → culprit;
• every failing test is on the known-flake list (tools/scripts/queue_known_flakes.json) → known flake, certified as infrastructure;
• main's tip fails the same tests → pre-existing on main, infrastructure.

Known flakes are set aside before any comparison. A clearing chain verdict still refuses when another REQUIRED job failed (the chain compares macos only), when the head's diff cannot be read, or when the head's own diff owns a failing test — two entries can break the same test. The reading is recorded under chain in the JSON (classification, parent_state, the failing test sets); the guard reads only the top-level fields. On 69 failed macos groups over three days the file-ownership reading below named no culprit at all, while the chain reading classified 67.

When the chain cannot rule — an unreadable parent, a parent still running, a batch not named for this pull request — certification falls back to the per-step accounting. It is per failing step and exhaustive: every failing step of every failing job must be positively accounted for, and one unaccounted step refuses the whole verdict. Two accounts qualify.

• The step is a package-manager fetch, an artifact move, a cache warm, or runner-generated — and the head changes nothing that step reads. Both halves are required. A step name alone proves nothing, because a workflow step is repository content and a head can rewrite the step that failed; and the surface is scoped per step class, because a blanket one including CMakeLists.txt would refuse every version bump while a curl of a pinned wasi-sdk release plainly cannot fail because a version string moved.
• The failing step ran the suite and the ctest failures are owned by a different batch member — the attribution above, at its confidence threshold, naming exactly one culprit that is not this head.

Everything else refuses, including the case that looks most like a free pass: a batch that failed with no ctest block at all. "No test failure, therefore infrastructure" is the tempting rule and the wrong one — a compile or link error is the most common way a head breaks a batch and it produces no ctest block. pulp#8811 was ejected exactly that way, with its own required checks green, and its own later commit admits the head was broken.

Two candidate signals were evaluated against live runs and are deliberately not implemented, because each one certified a head that was at fault:

• The identical failure reproduces in a batch that does not contain this head. Disproven. Three batches with disjoint single-entry memberships (#8803, #8807, #8811) failed with a byte-identical CMake test-discovery error, which would have certified #8811 — the head that was broken. The structure is indistinguishable from the genuinely-innocent case, where four batches (#8891, #8892, #8896, and #8888's) shared one write_scenario_wav link error.
• The failure is present on main at the batch's base. Sound in principle — a main-branch run at the base contains no queue entry, so anything it fails is the base's — but not observable here, which is the more useful finding. build.yml's own push run at a queue base is cancelled by its concurrency group with zero jobs, so the one lane whose configuration matches the batch never reports on the base. The main-branch runs that do carry jobs at that sha are other workflows with other build configurations, and a failure matched across configurations does not transfer.

Measured on the case that motivated this: four batches with disjoint single-entry memberships, on two different bases and two different gate hosts, all failed to link pulp::test::audio::write_scenario_wav. The bases did not contain f4ee4ac3cc ("keep grouped wav bridge link self-contained"), a four-line test/cmake/app_audio_host_tests.cmake fix — so the base really was red, and none of the four heads was at fault. Main's Debug and sanitizer runs at that base linked both affected targets cleanly (585 and 1758 link lines, so the reading is not blindness), because they do not group that target the way the gate does.

So the obstacle to certifying this whole class is a CI-topology fact, not a missing rule: nothing publishes a same-configuration verdict on a queue base. A same-configuration verdict on each base would make the rule checkable and would have cleared all four.

One same-head retry after a network ejection

.shipyard/config.toml also opts Pulp into Shipyard's environment re-enqueue:

[queue.environment_requeue]
enabled = true

When the attributor above does not certify a same-head failed_checks ejection, Shipyard itself allows one re-enqueue of that head, with no new push, when every failing required check on the removal's merge-group commit is an Actions job whose every failing step printed a network signature (Could not resolve host, ENOTFOUND, ECONNRESET, Tunnel connection failed, Proxy CONNECT aborted, curl: (6) / (56), ...) within 60 output lines of that step's first ##[error]. A second ejection of the same head is refused as before. shipyard landing --pr <n> prints the verdict and each step's matching log line under ENVIRONMENT RE-ENQUEUE; read it before pushing a no-op commit.

This is the pip relay 403, pinned-Chrome curl: (56), cargo DNS and upload ECONNRESET class: over 2026-09-15..29 it caused 15 ejections, and 12 of them were followed by a push that changed nothing but paid a full macos gate. The chain rule above cannot clear them and actively implicates them: it compares outcomes, not causes, so a batch whose parent passed on a host that could reach PyPI and which then failed on an m5 VM that could not reads as culprit (--certify on #8678's first ejection, run 35815699794, and on #8911's cargo DNS ejection, run 36315085363, both return implicates_head: true). An attributor verdict that does not certify therefore does not veto the network retry; the retry rests on the failing step's own output. Two log details the reader handles: the Install visual-analysis Python dependencies step's own script comment quotes Tunnel connection failed: 403 Forbidden, and GitHub echoes that script into every job log, so script lines are never read as output; and the if: always() ctest steps after a failure print their own ##[error], so the failing step is located from the jobs API, not from the first error in the log. The rules and their rationale are in Shipyard's docs/ghapp-guards.md ("Environment re-enqueue"). The key takes effect once the fleet runs a Shipyard release that reads it; older releases ignore it.

Pull-request heads also run the tests their diff reaches

After the pr-fast tier, the Test what this pull request reaches step runs tools/ci/pr_head_affected_tests.py against the configured PR-head build: the diff is the merge ref's first parent to its head, the selection is pulp affected's projection (changed_surface_inventory.project_affected plus the Shipyard changed-surface families), and Configure writes the CMake codemodel query it reads. It is gating. It never takes the projection's all fallback, skips docs/skills/workflow-only diffs, adds the cmake-* fixtures for tools/cmake/** and wide-non-native-selftest for the wide tier's inputs or a test/ CMake registration, drops pr-fast members, and stops starting batches after ten minutes, listing the rest as skipped. Its annotation is pr-head-affected-tests (pulp-pr-head-affected-tests/v1): selected, passed, failed, skipped-for-budget and minutes, which is the per-PR extra cost to watch.

Exact PR receipts on an unchanged merge-group candidate

A pull-request head normally runs only the build and the pr-fast tier, which is what its required macos check means, and that issues no receipt. A head that is ready to land also runs the full suite: the Decide pull-request test suite step (tools/ci/ctest_gate_args.py --pr-suite) reads the live pull request just before testing and chooses the full suite only when it is armed for auto-merge, is still the pull request's head, and was merged onto the current tip of main. Anything else, including an unreadable pull request, keeps the fast tier. Each clause drops a run whose receipt could never be consumed, because the receipt names the exact head and base. A push after arming re-decides on its own run, so a new head gets its own full run and the old receipt simply never matches. Arming a pull request whose checks are already green enqueues it immediately and starts no new run, so it validates in full in the merge group as before. REST auto_merge reads null once the queue holds a pull request (decisions contract row 11); that only ever yields the fast tier.

Do not push a merge of main into a pull request to make it eligible. A head whose base has moved is still validated in full by the merge group, and a refresh push cancels and restarts the running macos gate, so it costs more than the receipt could save. Merge main only when the pull request is DIRTY or a required check fails because of a stale base.

That full run is evidence and, for an armed head, a queue-admission gate: its step is not continue-on-error when pr_suite is full, so a failure turns the required macos check red before the head can enter the queue. Ordinary pull-request runs still skip the full step, leaving macos as build + fast tier. The Surface ctest failures step still reports a failed armed run. That reporter always exits 0 (errexit off, every possibly-empty grep guarded): it runs whenever the Test step's outcome is failure, so a non-zero exit from it alone would turn the required check red for a failure the workflow deliberately does not gate on. tools/ci/test_build_matrix_contract.py executes the step's script under GitHub's bash -e -o pipefail. When it passes, the macOS or Linux matrix child publishes a two-day protected-validation-<target>-<head>-<base> receipt. The issuer requires the Test step's outcome to be success (its conclusion is always success under continue-on-error). The receipt binds the exact synthetic merge tree and parents, protected workflow/policy blobs, observed platform/toolchain identity, and SHA-256 identities for every CTest executable that was actually exercised. Receipt publication is an optimization after the normal build and tests; inability to publish does not weaken the PR gate.

On merge_group, the preamble loads the verifier from the candidate's exact protected-base parent. It accepts exactly one unexpired artifact from a successful pull_request run, verifies GitHub's archive digest, then derives a new decision bound to the merge-group SHA. The candidate must have exactly the same base, head, tree, and policy blobs as the validated PR checkout. Any API, history, artifact, schema, digest, base/head/tree, or policy mismatch leaves that target in the original native matrix. The required macos bootstrap may report success only from the new subject-bound decision; a PR receipt alone can never satisfy the merge-group check. Fork PRs use the same public Actions artifact contract and keep the existing hosted-runner trust boundary.

Each merge group's decision per target is published by the protected-receipt-reuse job as a job-summary table and as a check-run annotation titled shipyard-receipt-decision (one JSON object: verdict reuse/refuse, reason, source_run_id, and the receipt's selected/passed/skipped/inventory_count), so shipyard landing answers "did this merge actually run tests?" directly. A refusal carries the protected-base verifier's own reason (for example "receipt selection covers too little of the built test inventory"). The notes are rendered by the checked-out script after the protected-base verifier has decided; they never decide.

A receipt stands in for the group's whole suite, including the slow-affected proofs (such as agent-capability-installed-sdk) that the change classifier admits for some changes. The receipt's signed selection records whether they ran: the anchored ^slow$ label set means they did. The reuse step passes the group's own classification to the verifier as --affected-slow-required (an empty classifier output counts as true), but only when the protected-base verifier accepts that flag. A group that requires the proofs while the receipt's run skipped them is refused with "merge group requires the slow-affected proofs but the receipt's run did not execute them", and a verifier given no requirement treats the proofs as required.

A merge group whose commit is not two-parent is refused for both targets with the parent count it actually has (or "parents could not be read" when the history is unavailable). A merge group that changed no native build input is not evaluated at all: it publishes no decision annotation, only a plain receipt reuse not evaluated: merge group changed no native build input notice and a summary row. Those are the near-instant macos bootstrap jobs. Read the emitted ##[notice] lines, not a grep of the whole log: the job log also echoes the step's script, which contains every refusal message verbatim.

The test step of each gate job likewise emits a shipyard-test-tier annotation: fast on a pull-request head (only the pr-fast label tier runs there; the full suite runs in the merge queue), full where the full suite ran, receipt-reused / not-required from the no-suite bootstraps, and deferred from the contract lane of a pull-request head without macos-ci (see "macOS on a pull request waits for approval"), so a fast-tier or deferred green is never read as full validation.

Required checks trust the pull request's own workflow YAML

Every required context on main is posted by a GitHub Actions workflow, and a merge group runs the workflow YAML its pull request carries. A pull request that edits build.yml can therefore decide its own macos result, by forcing macos_reused=true or by rewriting the job outright. Only the receipt verifier is loaded from the protected base, and the pull request controls whether it is called. No check inside the repository can close this: anything a workflow runs, the pull request can remove. By Daniel's decision on 2026-09-30, required contexts are only as trustworthy as the pull request's own workflow YAML. There is no org-level required workflow and no required code-owner review.

What CI does instead is make it visible. required-check-machinery.yml runs on pull_request_target, so its definition and its script (tools/scripts/required_check_machinery.py) come from protected main. The pull request's head is fetched only to diff it. It posts the advisory, never required, check run Required-check machinery (advisory): conclusion neutral titled touches required-check machinery: <files> when anything is flagged, success titled touches no required-check machinery otherwise, so a flagged pull request is visibly different without reading as a failure to chase. The check's summary and the run summary list each flagged file with its reason, and a flagged run also emits a ::warning. If the report cannot be computed the job fails and posts no check run.

Reason	Paths
receipt reuse	build.yml, .agents/contract.toml, classify_changes.py, protected_merge_receipt.py, tools/ci/ctest_gate_args.py
required-check workflow	the workflow mapped to each required context in the ruleset, plus the local actions and reusable workflows it calls
merge rules	.github/rulesets/, .github/CODEOWNERS, and the report's own workflow and script

The required contexts come from protected main's committed .github/rulesets/main-protection.json, which a pull request cannot change for its own report and which needs no administration access to read. It is kept equal to the live required checks by the ruleset drift workflow (ruleset-drift-check.yml, with tools/scripts/test_ruleset_drift_config.py pinning the expected contexts) and to Shipyard's [governance] list by tools/ci/test_base_poison_detector.py. If the file is missing or malformed, or lists no required checks, every .github/workflows and .github/actions file is counted and the check's summary says so. A required context with no mapped producer widens the report the same way. Treat a flagged pull request as one whose required checks it can grade itself, and review those files before it is enqueued.

The A2T scope-history window announces itself before it fails

gpu-trace-overhead-acceptance-selftest runs on the required macos gate and fails every run once the A2T scope-touching revision count passes 75% of A2T_SCOPE_HISTORY_LIMIT. The count grows with ordinary traffic (shared docs are in scope), so the failure has a date. The guard test writes its measurement to PULP_A2T_HEADROOM_OUT, which the Test step sets, and the Announce A2T scope-history headroom step turns it into one annotation: ::warning title=a2t-scope-history-headroom from 60% of the limit, naming the re-pin, a ::notice with the numbers below that, and a warning when the measurement is missing after the suite ran. The fix is always the re-pin in decisions contract row 22: move only A2T_SCOPE_HISTORY_BASE on protected main, never the limit.

An annotation is easy to miss, and main once reached the 75% failure with the warning printed and unread. So every push to main also keeps one tracking issue. The Linux leg uploads the measurement as the a2t-scope-headroom artifact, and the advisory a2t-headroom-issue job runs tools/ci/a2t_headroom_annotation.py --reconcile-issue. From the same 60% mark (headroom below 20% of the guard's budget), it opens the issue, or updates it in place. The title carries the check name, the count and limit, and the re-pin, and the body the full instruction. The job closes the issue on the first push after a re-pin brings the count back under. The issue is found by a hidden marker, so a hand-written issue with the same title is never touched, and a missing measurement never opens or closes it.

A2T evidence receipts get a nonterminal required-job attestation

When a pull-request head targeting Generous-Corp/pulp main adds or modifies the exact tracked evidence/receipt.json, the native macOS matrix child runs the A2T structural verifier after the ordinary build and test gate. It fetches the event-pinned head and the receipt's exact source revision, then runs the verifier from a detached checkout whose live HEAD is exactly that source revision. The PR-head receipt is supplied as a separate sibling input rather than overlaid into the source checkout. Before loading or executing Python, the issuer authenticates the issuer's schema validator and all four verifier dependencies as byte-identical S/E Git blobs, rejects any working-file mismatch, bounds verifier stdout and stderr, and uploads one a2t-structural-verification-<head> attestation. A changed receipt cannot be silently skipped: verifier failure or noncanonical output fails macos. The gate authenticates the event's exact base and head trees and compares only that receipt path. An unrelated or later tool-only PR that inherits an unchanged historical receipt skips verification. A receipt add/modify in a merge group or direct protected-main push reruns the same structural check but does not issue or upload PR attestation authority. Missing exact history, ambiguous path state, deletion, and other indeterminate relevant changes fail closed; shallow checkouts hydrate the two event commits by exact SHA first. The producer validates the artifact against the closed a2t-structural-verifier-attestation-v1.schema.json contract before writing it. The schema, issuer, verifier, and every dynamically loaded dependency are exact S bindings and must remain byte-identical through E; the adjacent golden fixture is the stable example for planning-side and other read-only consumers. That fixture is generated by python3 tools/scripts/a2t_structural_verification_ci.py --write-golden, and the producer test byte-compares it with canonical output so command-derived fields cannot drift independently.

The closed attestation also binds that exact protected target repository and ref. A develop/** or foreign-repository target may run verify-only but cannot issue or upload this protected-main artifact.

This artifact is deliberately not authority by itself. It records only facts available during that PR job: clean source and evidence revisions, exact file blobs and SHA-256s, trace digest, semantic command, workflow revision, run and attempt, job key, step identity, and zero-error result. It does not predict the future protected merge, report its own Actions artifact metadata, copy a final job conclusion, or claim terminal acceptance. The planning-side validator later authenticates the completed check/job/step and artifact metadata through GitHub, derives the protected merge identity, and matches those live facts to the downloaded bytes. Linux and Windows stay advisory and never issue this attestation.

The macOS PR head runs the pr-fast tier; the full suite runs in the queue

On pull_request, build.yml skips Test (non-Windows) and instead runs Test fast deterministic tier (pull request head): ctest -L '^pr-fast$' --no-tests=error over the static-contract checks listed in test/cmake/pr_fast_tests.cmake (lint, drift, registry-completeness and generated-manifest checks; about 115 tests, roughly 15 s). The full suite still runs on merge_group and push. The tier exists because a forgotten regeneration fails those checks deterministically, and once the full suite left the PR head it failed in the merge queue instead, ejecting every PR batched with it.

A member must be deterministic, finish in seconds on a loaded gate VM, and assert nothing about wall-clock time or host load; the timing-sensitive tests that motivated moving the full suite to the queue stay out. A listed name that is not registered in a configuration is reported at configure time (some members are platform-conditional), and pr-fast-tier-contract fails when the label selects fewer than 50 tests or drops one of its pinned members. The step writes ctest-pr-fast.junit.xml rather than ctest.junit.xml: the tier is not the merge's test evidence and must never satisfy a protected-validation receipt.

macOS on a pull request waits for approval

Off by default. Everything in this section is behind the repository variable PULP_MACOS_PR_GATING. Do not set this variable until Shipyard #929 (queue-arm-guard refuses a head whose required macos check is tier deferred) is merged AND verified live, by arming a deferred head and seeing the refusal. Unless it is exactly true, every pull request takes the full lane and macos-ci-label-reset.yml does nothing: today's behaviour. Switch it on only after the arm side refuses a deferred green durably (a Shipyard queue-arm-guard change, tracked separately). The tartci interim sweep already refuses it. Turning it off again is the rollback.

With gating on, a pull request's macOS leg runs in one of two lanes:

• Contract lane (every head the fleet App has not approved). The leg configures exactly as the gate does and runs the pr-fast tier (minus the members labelled needs-build) plus the pr-guard tier, then stops. No build, no iOS compile, no affected-tests tier, no full suite.
• Full lane (a head the fleet App labelled macos-ci). Build, pr-fast, pr-guard, the affected-tests tier and everything else the PR head ran before.

Linux runs on every push either way. merge_group, push and workflow_dispatch are unchanged: the merge queue runs the full macOS suite (or reuses an exact-tree receipt, as before) whatever the label says. The lane is a matrix field (matrix.lane); resolve-provider gets it from tools/ci/macos_pr_lane.py, which holds the whole rule and its tests.

Why: the fleet has a hard ceiling of eight macOS VMs (two per Mac, four Macs), and it was saturated, with PR-head macOS jobs waiting p50 8 min and p90 46 min. Over 12 hours on 2026-10-24/25 the macos job's time split 57% pull request (61 runs, 26% of them failed), 33% merge group and 7% dispatch. Those 61 runs covered 88 distinct heads, of which about 12 to 15 pull requests merged. Most PR macOS time went on heads that never landed.

Why the contract lane exists, and what it costs

Under the ALLGREEN merge queue a PR-caused macOS failure found in a merge group ejects the whole batch. The ones that did so most often are fast and need no build: script-test-inputs-drift, the consumption-census-* drift and negative-contract checks, and the A2T scope-history headroom guard. Deferring them to approval would move them into batches, so the contract lane keeps them on every head.

They need only a configured tree. The census facts, the compiled-test evidence script-test-inputs-drift compares, and the Scene3D boundary paths are all written at configure time. A Catch2 suite that is not built registers as a <target>_NOT_BUILT-<hash> placeholder (tools/cmake/PulpCatch.cmake), and the inventory readers skip those. The one member that runs a compiled target, project-package-mutation-control, carries needs-build and waits for the full lane. The A2T guard lives in gpu-trace-overhead-acceptance-selftest, and gpu-trace-overhead-verifier-selftest drives the same tooling; each takes about a minute, too slow for pr-fast, so both carry the pr-guard label (test/cmake/pr_fast_tests.cmake) and run in parallel.

Step times from the 12 most recent green PR macos jobs (2026-10-24), job time only, without VM clone and boot:

	Median	Range
Whole job today (full lane)	23.4 min	14.6 to 34.4
Steps the contract lane keeps (setup, configure, pr-fast, A2T issuer, uploads)	2.4 min	2.0 to 3.1
pr-guard (new, both lanes; two tests of about 68 s each, run in parallel)	about 1.2 min
Contract lane, expected	about 3.5 min
Deferred until approval (Build 5.9, affected tier 10.3, reuse record 2.1, SDK contract 2.6 when it applies, and others)	20.9 min	12.1 to 31.4

The full lane gets about a minute longer, for pr-guard. A member that turns out to need the build fails the contract lane by name ("Could not find executable", or its own message), and the fix is to add it to PULP_PR_FAST_NEEDS_BUILD.

The macos-ci contract

• Name. Exactly macos-ci.
• Only the fleet App's label counts. The App (shipyard-local[bot]) applies it as part of arming, after a trusted verdict comment whose last line is verdict: APPROVE head:<sha> names the current head. On a labeled event the lane checks the event's sender. On opened or reopened it reads who applied the label last from the issue's events. Anyone else's label, or an events listing that cannot be read, counts as no label.
• What starts a full-lane run. A workflow_dispatch of build.yml on the pull request's branch made by the fleet App, or an opened or reopened event on a pull request whose label the App applied. A label event starts nothing.
• build.yml does not listen for labeled. A label event's run would be a check suite newer than the head's real run with no macos check run in it, and the merge queue reads the required macos context from the newest suite: it refuses to enqueue ("Required status check "macos" is expected") even though the earlier run's macos passed, and re-running a job in the good run does not clear it. Shipyard adds several labels to every new pull request, so every head was shadowed this way (#11034, #11040). The github.event.action == 'labeled' conditions still in build.yml are unreachable.
• A new head needs approval again. On synchronize the lane is the contract lane whatever the labels say, because the event payload still lists the label the previous head earned. macos-ci-label-reset.yml (pull_request_target, checks out nothing) then does two things:
• It disables auto-merge. Otherwise an armed pull request would auto-merge on the new head's contract-lane green. A pull request already in the merge queue leaves the queue on a push anyway.
• It removes macos-ci, so the label never vouches for a head it was not applied to.

Re-arming needs a fresh verdict for the new head, the label, and a full-lane success. - A pull request armed before its checks ran is refused too, in three layers. 1. At the arm itself: macos-ci-arm-refusal.yml fires on auto_merge_enabled and disarms unless the head's latest macos check run from this repository's Actions workflows, whose workflow run is for a head in this repository, is a completed success with an explicit full (a dispatch's full suite), fast (full lane) or not-required (skip-safe) tier. It fails closed: a read that fails, or a check run it cannot place, is a refusal ("could not verify …"). Only a check run proven to be for a head in another repository is passed over, and it never falls back to an older run on uncertainty. It judges the pull request's current head, read live, not the event's (an arm event can arrive after a push); an unreadable current head is a refusal. It reads every page of the head's check runs and sorts them itself. Its comment says what it found about the macos run; it does not judge approval. 2. At the contract lane's conclusion: its macos job's last step, Refuse a head armed before approval (contract lane), fails when this exact head is armed for auto-merge or already in the merge queue (read over GraphQL with the read-only token; REST auto_merge reads null for a queued pull request): "armed before approval and a full macOS run: remove auto-merge, get a trusted APPROVE; the fleet App applies macos-ci for the full lane". An arm state it cannot read fails closed. Only the runner's post-steps follow it. 3. After the run: macos-ci-deferred-disarm.yml (workflow_run on a completed Build and Test run, no checkout, same-repository runs and pull requests only) disables auto-merge on a pull request still armed at a head whose macos is tier deferred, and dequeues it if it is already in the merge queue.

All three are gated on the variable. - Removing the label starts nothing: build.yml does not listen for unlabeled, and a run already in flight finishes. - Escape hatch. Dispatch build.yml on the branch: workflow_dispatch always runs the full lane. A hand-applied label no longer forces anything. Re-running a contract-lane run does not help either, because a re-run replays the original event.

The full-lane dispatch tests the merge ref (pr_number)

A plain workflow_dispatch of build.yml tests the bare branch head. A head that is behind main then fails freshness checks such as agent-capability-manifest-check, and the Linux leg asks for the operator's local runner (Linux (x64) [local]), which nothing serves on that path. The fleet App's full-lane dispatch therefore sets the pr_number input:

gh workflow run build.yml --ref <pr-head-branch> -f pr_number=<n>

With pr_number set, the run does the following: - Binds first, and resolves the merge commit once. classify's first step requires the pull request to be open, in this repository, with its head branch and commit equal to the dispatch's github.ref_name and github.sha. It then takes the pull request's merge commit (merge_commit_sha, which refs/pull/<n>/merge names) and requires exactly two parents: a base, then this head. Anything else fails closed, and so does macos. That covers a conflicting pull request, a merge GitHub has not computed yet, or a moved head. Without the binding, a dispatch on one branch could post another pull request's merge result on its head. - Tests that exact commit everywhere. classify, and every build leg through classify's output, check out the bound merge sha, never the moving ref. Build re-checks that HEAD is that sha and that its second parent is the run's head. - Records what it tested. The tested_ref, the tested_sha (the merge commit) and the merged_base (its first parent) are classify outputs and step outputs of Verify the tested merge ref, and appear in the job summary. The next step is named Tested <ref> at <sha> onto <base>, so a reader of the jobs API sees them without reading logs. The merged base is also the capability base and the base classify diffs against, as a pull request's pinned base.sha is. - Writes nothing a later run reads. The run executes a pull request's code. A dispatch writes caches in its ref's scope, so on this path the bind refuses the default branch, any release/* branch, and any branch with classic protection or a ruleset, or whose protection cannot be read. Every cache step only restores, and setup-node saves no npm cache. - Uploads no consumable artifacts. The iOS gate digest, the GPU-audio SDK, the reuse replay record and the plugin bundles are skipped. Only the ctest logs and the Rust fallback receipt upload, as diagnostics. The protected receipt and the A2T attestation are pull_request-only. Every artifact consumer (ios_gate_digest, reuse_replay_collect, binary_identity_shadow, base_poison_detector, protected_merge_receipt) reads only merge_group, pull_request or push-to-main runs, and no workflow_run consumer of Build and Test reads artifacts. - Routes every leg as a pull_request run does. macOS takes the PR-head class (pulp-build-pr-head), so it gets the same host caches and warm build dir (build-<key>) as the pull request's own runs. Linux goes to GitHub-hosted runners, and there is no Windows leg. None of the operator routing inputs applies: the provider and the Linux, Windows and macOS selectors. - Isolated from the pull request's other runs. Its concurrency group keys on github.ref, which is refs/heads/<branch>, so it cannot cancel the pull request's refs/pull/<n>/merge run or a merge group. It uses no secret and no environment beyond what a pull_request run gets.

Everything else is still a dispatch: the run's checks land on the head, and it runs the full suite. A dispatch without pr_number behaves as before. tools/ci/test_build_pr_dispatch.py pins the checkout, the binding, the recorded ref, and the routing both ways.

Approve first, then the full lane, then arm

A reviewer may give verdict: APPROVE head:<sha> before the head has had a full macOS run; the verdict is about the change, and the contract lane has already run the fast PR-caused checks. Approval is not arming:

1. The arm sweep sees the trusted APPROVE for the current head, applies macos-ci and dispatches build.yml on the branch with pr_number set (see below). The full lane runs.
2. The sweep arms only once the head's latest macos check run is a completed full-lane success: its shipyard-test-tier annotation is not deferred. An arm made earlier (absent, running, failed, deferred, or no full/fast/not-required tier) is refused the moment it is made: macos-ci-arm-refusal.yml (pull_request_target on auto_merge_enabled, no checkout, gated on the variable) disables auto-merge and comments with the reason.
3. If the full lane fails, the pull request needs a fix. The fix is a new head, which disarms the pull request, drops the label and needs a new verdict, and the cycle repeats.

A head that is already armed when the full lane reaches Decide pull-request test suite (and still merges onto the tip of main) also runs the full suite there and issues an exact-tree receipt, as before (see "Only a ready-to-land PR head issues a receipt" in the CI skill).

The tier record the arm guard reads

Shipyard's queue-arm-guard does not read shipyard-test-tier annotations, since any step, including the code under test, can print one. It reads a tier record instead (Shipyard docs/ghapp-guards.md, "A tiered required check must prove its full lane"). That record is a check run named shipyard-test-tier/macos on the head, whose summary is {"schema":"shipyard-test-tier/v1","check":"macos","tier":...,"run_id":...,"run_attempt":...,"trigger":...,"actor":...,"selector":...,"tested_ref":...,"tested_sha":...,"merged_base":...,"full_lane_workflow":"build.yml","full_suite_runs_in":"merge_group"}. tested_ref, tested_sha and merged_base bind it to the merge commit the full lane tested (null when the run tested none). trigger and actor name the event and account that started the run; the guard reports them but does not trust them. full_lane_workflow is what its fix line says to dispatch. The record carries no label. The guard counts only records created by the App named in [queue] tier_record_app, never github-actions: any job holding checks: write could rewrite those. It refuses a head with no counted record bound to its macos job's run and attempt, or with a deferred one.

.github/workflows/macos-tier-record.yml publishes the record:

• Trigger: workflow_run when a Build and Test run completes. It runs from main, with no checkout, for same-repository pull_request and workflow_dispatch runs only.
• The tiered job's own token: the guard also reads the macos job's GITHUB_TOKEN Permissions log group and refuses if Checks is write. build.yml grants checks: read explicitly, which satisfies that.
• Token: its own GITHUB_TOKEN only reads. The record is created with a token for a dedicated App, minted in the job from TIER_RECORD_APP_ID and TIER_RECORD_APP_KEY. Those are secrets of the tier-record GitHub Environment, whose deployment-branch policy allows main only. A workflow_run job runs on the default branch and qualifies. A job on a pull request branch is refused them, as repository secrets would not be. Only this job references the environment or those secrets; a test checks that. With the secrets absent the job fails visibly and publishes nothing.
• Tier: the more conservative of two signals, since neither proves enough alone. A pull request's own build.yml decides its step names, so steps alone cannot grant a full tier, and who started the run does not show the suite ran.
• The run. It must be a workflow_dispatch whose triggering actor is the fleet App (shipyard-local[bot]). The App moves an approved head to the full lane that way, since label events no longer start Build and Test runs. The App's latest verdict on the head, at or before the run's created_at, must be a trusted verdict: APPROVE head:<sha> with a reviewer: line. The run must also be a pr_number dispatch of that pull request, which tests its merge commit (see "The full-lane dispatch tests the merge ref"). Its macos job carries one succeeded step named Tested refs/pull/<n>/merge at <sha> onto <base>, and that commit has exactly the parents [<base>, <head>] on GitHub. Otherwise the tier is deferred: a pull_request run, a dispatch or rerun by anyone else, a plain dispatch, a missing or non-APPROVE verdict, or a head without exactly one same-repository pull request.

• Steps. The conclusions the runner recorded for that attempt's macos job:

  macos job	Tier
  Build ran, Test (non-Windows) ran	full
  Build ran	fast (full lane)
  Build skipped (the contract lane)	deferred
  bootstrap step succeeded (skip-safe)	not-required
  bootstrap step failed closed	deferred

A run with no macos job, two of them, a job for another head, neither step, or an unreadable listing publishes no record, and the guard refuses that head. So do fork heads, which land through an in-repository branch.

Because the run signal applies to every head, only the fleet App's dispatch of an approved head can be anything but deferred. Every pull_request run is deferred, even on the full lane and even when PULP_MACOS_PR_GATING is off. The rule has no exemptions, since one would reopen the forgery surface. Turn on tiered_required_checks only after the App exists and the arm sweep dispatches build.yml for every approved head, skip-safe and release ones included, with gating on. That key and [queue] tier_record_app are the last change, and neither is set yet.

Trust boundary. A same-repository pull request runs its own build.yml, so it can rename steps, or add a step that grants itself checks: write. The tests that pin step names and checks writers are in the same pull request and can be edited too, so they stop only an honest mistake. That is the trust level of every required context here (see "Required checks trust the pull request's own workflow YAML"). Three things stand behind it: - the run signal (event, triggering actor and the App's verdict), which no pull request controls; - the guard counting only the dedicated App's records; - the merge group's full macOS suite.

The repository's default workflow token permission is read. That covers only jobs that declare no permissions; a pull request can still declare checks: write explicitly. That is harmless here, because the guard counts only the dedicated App's records, which no pull request job can rewrite.

Why the contract lane's macos check is green

macos is a required context and the contract lane's job is the one named macos. When its tiers pass it concludes success. That passes the required check exactly as a full-lane success does, and the alternatives are no better: skipped and neutral satisfy a required check the same way (and an Actions job cannot conclude neutral), and a forced failure would turn every unapproved head red for a defect it does not have. A contract-lane failure is a real failure and stays red.

So GitHub cannot tell a contract-lane green from a full one, and the arming side has to. The pr-fast step's log carries ::notice title=shipyard-test-tier::{"schema":"shipyard-test-tier/v1","tier":"deferred",...} on the contract lane (and "tier":"fast" on the full lane), and the job summary says that nothing was built. tools/scripts/gate_suite_executed.py reads both lanes as tier-only, so use the annotation, or the skipped Build step, to tell them apart. That is why gating stays off until the arm side refuses deferred durably. If a head were armed on a contract-lane green anyway, nothing unsafe would land, because the merge group still runs macOS; the cost is merge-group exposure to failures only a build reveals.

Decisions this has to keep

• Decision 17, strict up-to-date protection stays on. Untouched: no ruleset or protection setting changes, and macos stays a required context on main. What keeps main green is the merge group's full macOS run of main plus the entry, and that run is unchanged.
• Decision 11, the live ALLGREEN merge queue. Unchanged in shape (ruleset, grouping, MERGE method). What ALLGREEN multiplies is a PR-caused failure first found in a batch. The contract lane keeps the fast PR-caused checks on every head, and the arming rule keeps a full PR-head macOS success before any head enters the queue, so batch exposure stays where it is today. It worsens only if a head is armed on a deferred green, which the default-off switch and the disarm-on-push exist to prevent.
• Decision 5, auto-merge rides on required checks. macos stays required, and the contract lane reports it, so auto-merge keeps working. Because a deferred green also satisfies it, the arming rule and the disarm-on-push are what keep auto-merge from admitting a head with no full macOS run.
• Decision 2, the fenced carrier's merge-update of a BEHIND armed PR. With gating on, that update is a synchronize: it disarms the pull request, drops macos-ci, and the next run is the contract lane. Update-branch already disarms auto-merge (see "The arm is not armed until you read it back" in the CI skill). Re-arming takes the full path: a verdict for the new head, the label, a full-lane success.
• Decision 23, the iOS compile gate. The contract lane builds nothing, so it neither runs the iOS gate nor records a digest. Those come from full-lane and merge-group runs, as the receipts already require.

Interactions

• Skip-safe pull requests (no native build input, including the release-bot and release-please version-bump fast path in classify) never had a macOS leg, and still report the skip-safe green from the macos alias. The lanes apply only when classify says a native build is required.
• A version-bump or release PR that the fast path cannot prove gets the contract lane like any other head. If the release path arms through a guard that enforces the rule above, that path must have the App apply macos-ci too. Otherwise its merge group runs macOS, as it does today.
• A2T evidence pull requests still issue their structural attestation on every head. The issuer is a Python step the contract lane keeps.

Rollback

Unset PULP_MACOS_PR_GATING, or set it to anything but true. Every pull request then takes the full lane and the reset workflow does nothing. pr-guard keeps running on the full lane.

Windows is gated by the merge queue, not by the PR head

Windows is advisory and runs entirely on GitHub-hosted runners, and a single run carries four Windows jobs: the Windows (x64) matrix leg plus the windows-msvc-release-gate, windows-midi2-gate, and windows-ble-gate compile gates. The repository draws all of those from one fixed pool of concurrent GitHub-hosted jobs, shared with every other workflow.

That pool is the scarce resource, and Windows is by far its largest consumer. With a handful of PRs open at once, advisory Windows work fills nearly every slot and the required hosted check — Build + prove + (owner-gated) deploy, on ubuntu-latest — cannot get a runner. The merge queue then holds its entry in AWAITING_CHECKS until the ruleset's check-response timeout expires, evicts it, and nothing lands at all. macOS is never implicated: it runs on the self-hosted Macs, which sit outside the hosted pool.

So Windows runs where it supplies independent value without blocking every merge:

• nightly cross-platform validation — catches Windows regressions as follow-up work without consuming the merge queue's hosted slots.
• push: main — publishes the Windows ccache.
• workflow_dispatch — explicit reruns when you want Windows early.

A PR head keeps macOS on the self-hosted Macs (with PULP_MACOS_PR_GATING on, the configure-only contract lane until approved; see "macOS on a pull request waits for approval") and Linux on GitHub-hosted Linux for fast signal. The Linux leg therefore does consume hosted capacity; the security boundary above deliberately prevents automatic PRs from reaching the private Mac Pro VMs. The advisory windows alias job short-circuits to green on pull_request — without that it would fail closed looking for a matrix leg that deliberately did not run.

The trade is later Windows feedback. Dispatch build.yml manually against the branch when a Windows-touching change needs proof before merge.

tools/scripts/test_windows_runner_policy.py locks this in: it executes resolve-provider's matrix resolver for each event and asserts hosted Linux and Windows are absent on merge_group, Windows remains reachable through workflow_dispatch, and macOS plus Linux still run on the PR head.

The native macOS job publishes the required context directly

Branch protection requires the literal macos context. On pull requests, Shipyard manual dispatches, and merge groups, the native macOS matrix child publishes that context directly. It becomes terminal with the macOS work and does not wait for the combined matrix, a reporter runner, or the jobs API. Advisory Linux and Windows legs may therefore continue after queue admission.

Because one job name carries the gate across all three events, that job also configures the same way on all three. It assembles its CMake arguments once — -DCMAKE_BUILD_TYPE=Release -DPULP_BUILD_EXAMPLES=OFF — with no github.event_name branch adding to the list. A per-event flag would publish a differently configured build under the gate's name, and the flag that used to sit here, -DPULP_ENABLE_GPU=OFF on workflow_dispatch, showed exactly what that costs: PULP_TEXT_SHAPING follows PULP_ENABLE_GPU, so Skia went with it, render_to_rgba returned an empty buffer, and capture-based view tests failed for the configuration rather than for the change under test. Since Shipyard's PR validation arrives through workflow_dispatch, that was the routine path, not a fringe one. tools/scripts/test_workflow_build_dirs.py pins the single assembly line and the absence of any cmake_args+= append.

Event-specific bootstrap jobs own macos only when classification intentionally omits native work or provider/classifier resolution fails closed. When a native matrix child exists, the corresponding bootstrap is inactive and uses an -unused display name so it cannot collide with or satisfy branch protection. tools/scripts/test_required_macos_alias.py and tools/scripts/test_windows_runner_policy.py pin both ownership paths.

The merge-group bootstrap's verdict lives in tools/ci/macos_merge_group_bootstrap.sh. A provider or classifier result of failure, skipped or none fails macos closed. A cancelled classify does not: GitHub cancels a job it never assigned a runner to, and failing closed on that turned a hosted-runner outage into a red required check, so the queue ejected the PR, re-batched, and met the same wait (2026-10-05, during a hosted-runner assignment incident). The bootstrap already has a runner, so it classifies the merge group itself with the classify job's scripts and proceeds on that answer: nothing native to build passes skip-safe; a group that needs the native build fails closed, because the native leg and the receipt reuse both wait on classify; a failing in-job classifier fails closed. tools/ci/test_macos_merge_group_bootstrap.py runs the script against stub results.

On a merge group, the classify job first runs the protected base branch's copy of tools/ci/merge_group_hold_check.py (fetched from merge_group.base_ref; the token never reaches a queue entry's code). It reads the group's pull requests from the first-parent chain of queue merge commits (each lands its second parent), and fails when one's latest trusted verdict on that head is verdict: HOLD head:<sha> or a CHANGES_REQUESTED review. An APPROVE counts only from the fleet App itself, a Bot (GraphQL names it shipyard-local, a login anyone can register as a user, so a user of that name never counts; every agent posts verdicts as the App through ghapp), never from a human account, so a human APPROVE posted after the App's HOLD cannot clear it: an agent holding the owner's personal gh login posted such approvals. A HOLD counts from the fleet App, an OWNER, MEMBER or COLLABORATOR, or the pull request's own author. The chain is read one commit at a time and comments and reviews are paginated, so a large group or a long pull request is read in full. The failure names the pull request, the HOLD's URL and the fix, and a failed classify fails macos closed through the bootstrap above, before any VM. It is the backstop for a HOLD on a head that was already armed or queued, which Shipyard's verdict-hold-hook otherwise dequeues or disarms when the HOLD posts. It fails closed: anything it cannot read (a commit on the chain, a pull request's comments or reviews, a missing token, a failed fetch of the base branch) fails the group with an error naming the pull request or group, the error and the fix (re-run the classify job, or re-queue). A stalled queue is the deliberate lesser harm. Only a base branch that does not carry the check yet passes, with a notice.

After the HOLD check, the classify job runs the merge-queue circuit breaker, the protected base branch's copy of tools/ci/queue_breaker.py check-group. While the PULP_QUEUE_BREAKER repository variable holds {"test": "<ctest name>", "fix_pr": <N>, "until": "<UTC ISO-8601>"}, it fails every merge group that carries neither fix_pr nor a queue-unblock label last applied by the fleet App or a repository admin or maintainer, before any VM, through the same classify-fails-macos-closed path as the HOLD check. It fails the group rather than skipping it: a skipped required job reads as green, which would merge untested code. The breaker is inactive when the variable is unset, once until passes, once fix_pr has merged, and when until is more than three hours ahead (ignored, so a forgotten variable cannot hold the queue for long); deleting the variable lifts it at once. Anything it cannot read lets the group proceed, since the breaker only saves VMs. Up to two groups may already be past classify when the variable is set; they run and fail as before. The separate queue-breaker-notify job, which checks out only the protected base branch and holds the only pull-requests: write token, comments the reason on each blocked pull request once per breaker state. On pull_request events classify judges PULP_QUEUE_BREAKER_CONTROL the same way but report-only, against the pull request itself, so a run log proves the variable reaches the check. Shipyard's steward must not re-arm pull requests the breaker ejects while it is open; until it honours the breaker, a re-armed pull request costs a classify run, not a VM. Tests: tools/ci/test_queue_breaker.py (ctest queue-breaker-selftest).

Who may sign the fleet App's approvals is set in .shipyard/config.toml: [auto_merge] reviewer_signers = ["pulp-review", "infra-review2"]. Every agent posts verdicts as the same App, so Shipyard's guards count an App verdict: APPROVE only with a reviewer: <name> line naming one of these, and never one ghapp stamped with an unattributed session. Shipyard's server-side review-verdict reads the list only from this file and refuses every App approval when it is missing, so a new reviewer is added here first.

[auto_merge] reviewer_stamp_sessions = ["fleet-pass-reviewer-*"] further restricts which sessions may approve: an App APPROVE counts only when the session in its ghapp stamp (the part after the host) matches one of these globs, so a reviewer tab that is not the scheduled standing pass does not count unless it is listed. A HOLD from any session still counts. The key is read by the Shipyard commit pinned in .github/workflows/review-verdict.yml, so bump that pin with the setting.

The preamble can run from a checkout below /Volumes/Workshop. Inline Python started with python3 - resolves the current directory before executing its stdin script, so a wedged checkout volume can freeze the routing probe even though the API response is available. RUNNER_TEMP is not a safe boundary here: on self-hosted Studios it can also live below /Volumes/Workshop. Those three helpers first cd /tmp (/private/tmp on macOS's system volume); the routing helper then uses GITHUB_WORKSPACE only as the absolute resolver-script argument. Keep new inline Python in a PULP_PREAMBLE_RUNS_ON_JSON job behind the same stable-cwd boundary. tools/scripts/test_preamble_python_stable_cwd.py enforces the complete set.

Why there is no PR gate settle window

A superseding push cancels a pull request's previous run through the build-<ref> concurrency group. There is deliberately no wait before the native matrix to give that cancellation a head start: pushes that land within about five minutes already cancel the run before its macOS leg claims a gate VM, and the pushes that cancelled a claimed VM in a 140-run sample arrived 4 to 27 minutes apart, so a window long enough to catch them would add that latency to every native pull-request run to save a few VM-minutes a day. tools/scripts/test_build_workflow.py (NoPrGateSettleTest) keeps it out.

The preamble and alias lanes run GitHub-hosted, and a persistent runner may not own them

PULP_PREAMBLE_RUNS_ON_JSON and PULP_ALIAS_RUNS_ON_JSON both contract to ubuntu-latest. Neither may be pointed at a label whose only provisioning is a statically named persistent runner, however idle and local that runner looks.

The reason is a measured outage, not a preference. Both variables used to name [self-hosted, macOS, ARM64, pulp-preamble], served by two persistent runners, pulp-preamble-m3 and pulp-preamble-m5. They failed independently and silently:

• pulp-preamble-m3 was stopped cleanly on 2026-09-01 (Runner listener exit with 0 return code, stop the service, no retry needed) and its LaunchAgent was never reloaded. Its on-disk configuration stayed valid and correct for the post-org-move URL, so nothing on the host looked wrong.
• pulp-preamble-m5 kept serving alone until 2026-09-12T19:16:47Z, when its server-side registration was deleted. The runner did exactly what it is designed to do — The runner no longer exists on the server. Cleaning up local configuration. — and erased its own .runner, .credentials, and .credentials_rsaparams. Its KeepAlive LaunchAgent then respawned a listener that could only exit Not configured, thousands of times, which reads on the host as "the service is loaded" and in launchctl list as a - in the PID column.

From that moment the label had zero runners. repos/<owner>/<repo>/actions/runners reported total_count: 0 while the org scope still listed unrelated runners, so the census was not lying and nothing was offline-but-recoverable — the registrations were gone. GitHub does not reject a runs-on it cannot satisfy; it queues the job. So every build.yml run stacked up behind resolve-provider/classify with no error anywhere, the required macos check never reported, and nothing merged for more than ten hours.

The alias lane makes that worse than a slow check. It is the last job in a run and does one terminal jobs-API read, so a starved alias means the run never reaches a terminal state, holds its ref's concurrency group, and leaves the next push's run at pending with zero jobs — a wedge that survives re-pushes and clears only by cancelling the older run by hand. Cancel the run on the stale head, the one that still has jobs: a plain POST /actions/runs/<id>/cancel returns an empty {} either way and will not move a run whose jobs were never assigned, so use .../force-cancel and read the run status back instead of trusting the response.

This is the failure mode decision 4 of .agents/contract.toml already names: self-hosted runner names are EPHEMERAL, never static; a runnerless required lane is HELD, never a retry storm. A persistent runner has a static name by construction, so a lane that gates a required check must not depend on one. The contract rows for these two lanes previously declared provisioning: persistent with literal hosts entries, which contradicted that decision; they now declare github-hosted with no hosts.

Why not the self-hosted Linux pool instead. It was the obvious substitute and it does not qualify today. Measured on 2026-09-13, the macpro x86_64 pool had exactly one runner left (pulp-auto-ephemeral-200), registered "Ephemeral": "True" — good for a single job and then gone — while its provisioner pulp-ephemeral-pool@2 sat in an exit-75 governor-refusal loop at restart counter 4040, having created no VM since 2026-09-10 because two post-job husks (VMs 201 and 202) pin 16 GB the reaper declines to reclaim (SKIP 201 — post-job clone lacks a host generation). A required-gating lane aimed at one non-renewing slot is a black hole with a delay on it. build.yml's own resolve-provider comment states the standing precondition: move this lane to a self-hosted selector only once that pool is confirmed always-on rather than on-demand, or the required gate just starves on a different pool.

What hosted costs, honestly. Moving off the shared hosted pool was originally meant to stop hosted queue saturation from starving the required gate, and that pressure is real. It is a latency risk. A label no runner carries is a certainty. The preamble and alias jobs are cheap, platform-agnostic shell plus one API read — they contain no macOS-only tooling — and hosted minutes are free on a public repository, so this is the correct default until a pool exists that is both always-on and not statically named. Restoring a self-hosted selector is a contract edit here plus a variable edit, reviewed together, never a variable edit alone.

Whether the gate has a GPU is observed, not assumed

Every GPU case in the suite skips when no adapter is present, which is the right behavior on a developer laptop and on the GitHub-hosted runners that carry no representative GPU. It also means a green macos check reads the same whether the self-hosted runner has a working adapter or quietly lost one: the skipped cases are the only difference, and nothing fails.

PULP_REQUIRE_GPU_ADAPTER is how a lane states that it does have one. It is a policy switch, not a device probe. The single case that reads it, A lane that requires a GPU adapter has one, skips when the variable is unset and asserts when it is set: the surface is created, initialized, and its adapter must report available, must not be Dawn's Null backend (which validates API calls and composites nothing), and must not be a CPU adapter. Set it locally to turn a silent GPU skip into a real failure:

PULP_REQUIRE_GPU_ADAPTER=1 ctest --test-dir build --output-on-failure \
  -R '^A lane that requires a GPU adapter has one$'

build.yml sets it on the self-hosted macOS leg in a step marked continue-on-error, so the runner's adapter state shows up in the log without the required check depending on an answer nobody has measured yet. Once the runs establish that the lane really does render, promote the step by deleting that line; if they establish that it does not, the honest fix is to stop calling it GPU coverage rather than to keep the skips.

Routing contract (checked)

Every *_RUNS_ON_JSON repo variable is a lane: it names the labels a class of jobs is dispatched to. The intended lane→label mapping lives in tools/scripts/runner_topology.json, and tools/scripts/runner_topology_check.py reconciles it against the live repo variables and the live registered runners.

The contract is the source of truth for lane→label. Label values quoted inline elsewhere in this guide are illustrative and can lag; the contract plus its checker are authoritative, because they are the only pair that is verified.

The required macOS variable is intentionally the pre-dispatch selector. For same-repository automatic work, build.yml removes the legacy pulp-gate-fast label and adds exactly one mutually exclusive event class: pulp-build-merge-group (provider-derived lease priority 110) or pulp-build-pr-head (priority 100). The TartCI source profile is authoritative for what a host can serve: its Pulp lane uses assignment_mode = "event-class-v2", declares both tier rows with their exact workflow and repository runner-group ID, and declares no fixed lane priority. The contract records that transformation separately from the repo variable so neither representation is mistaken for the other.

The transformation removes any existing managed event-class labels before adding the current event's class. This matters because the shared repository selector normally carries the PR-head class; a merge-group job must request only pulp-build-merge-group. GitHub treats labels as an all-of set, so a job requesting both classes is unschedulable by the event-class-v2 fleet, whose ephemeral runners advertise one class at a time.

If a merge-group run is manually removed while recovering this routing defect, re-admission must use a new reviewed head. Shipyard intentionally refuses to re-arm an unchanged head after a guarded manual removal, because doing so could re-enqueue a known-bad batch state. Make the smallest relevant follow-up correction, push it, obtain a fresh exact-head review, and then re-run the normal arm path. Do not bypass the queue-arm guard or use an empty commit. If the VM reaches the guest but its hydration SSH process remains stuck before Connected to GitHub, terminate only that recorded process and let TartCI discard the VM; then wait for GitHub to finish cancelling the ghost run before re-admitting the new head. A merge-group run that reaches the runner and then exits because hydration was terminated is a failed-check batch, so it also requires a new reviewed head before admission; do not force-arm the unchanged head.

The failure this prevents

GitHub does not validate runs-on. A job that asks for a label no runner carries is not an error — it is queued, forever. There is no warning, no annotation, no failed check. The only symptom is jobs piling up while the pool looks saturated, which is indistinguishable from "we're just busy".

That makes a mis-pointed routing variable silent. A relief valve routed into a black hole is worse than no relief valve: it reports healthy and relieves nothing, and the queue it was supposed to drain grows behind it.

The same class of bug already bit the busy probe in build.yml: reading actions/runners needs Administration: Read, which the default GITHUB_TOKEN lacks, so the probe 403s and falls back to BUSY=0 — silently disabling overflow. Nothing about either failure is visible without asking.

What the checker asserts

Check	Failure it catches
drift	A variable was edited without updating the contract (or vice versa). The variable is a reviewed artifact, not a blind edit.
black-hole	The lane's labels are satisfiable by no runner.
queue-stalled	The lane has no live runner and its oldest queued job has waited past the provisioning budget. Work is arriving and nothing is answering it — strictly stronger evidence than black-hole, which only says nothing has arrived.
visibility-incomplete	The lane's labels matched no runner, and a runner scope refused the census — so the check never looked everywhere it needed to. Reported at the lane's normal level (an error for a required lane), never below it: a genuinely dead org-scoped lane is indistinguishable from an unreadable one.
degraded	The only matching runners are offline — the host may just be asleep. A warning, not an error: a different failure from a label nobody owns.
undeclared	A live routing variable with no lane in the contract.
hosted-unknown	A runs-on value that is not self-hosted and not a known GitHub image — i.e. a typo, which queues forever.
must-unset	A paid Namespace overflow variable is set (cost guard).
event-class-contract	The variable/base transformation is internally incomplete or contradictory.
profile-contract-drift	A supplied TartCI source profile does not serve the contracted event classes, scope, or post-transform labels, or incorrectly fixes one priority for both classes.
profile-receipt-drift	A supplied installed-profile receipt does not bind to the exact supplied source-profile digest.
source-manifest-drift	A supplied private desired-fleet manifest disagrees with the Pulp contract or source profile, including its declared tart_home.
host-silent	A declared fleet LANE completed no build.yml job for silence_hours while at least demand_min_jobs of the work it serves completed elsewhere, counted across two independent witnesses: jobs carrying a label set it is observed to serve, plus distinct pieces of work for a job name only it is observed to run. Reported at the level in hosts.severity, which ships as info (see the per-lane census below).
host-map-broken	Self-hosted jobs ran in the window and not one runner name matched any declared host prefix. A lane rename, not a silent fleet. Silence is not evaluated while the map cannot identify a host.
host-silence-degraded	A jobs read failed or the run walk was cut short, so the window was not fully read and every silence verdict is suppressed rather than reported on evidence that does not support it.
host-unobserved, host-last-served, host-serving-inflight, host-idle, host-unmeasurable, host-map-unmapped, host-lane-census	Census state for the step summary, always info. They record what the sweep saw so a would-be verdict can be counted against real traffic before anything pages.

Label matching is subset containment: GitHub dispatches to a runner only if it carries every label in the array. A lane requesting [self-hosted, macOS, ARM64, pulp-build, pulp-build-studio] is not served by a runner carrying only pulp-preamble, however much the labels overlap.

Three runner states, not two

online / offline is not the whole story. Tart runners register JIT and ephemeral (tools/ci/tart-runner.sh, tart-runner-linux.sh): they exist only while a job runs and vanish when idle. For those lanes an empty registry proves nothing — the provisioner may simply have nothing to do.

So ephemeral lanes are judged on service history instead: has any job been dispatched to this exact label set inside the lookback window? A label set with no runner and no recent service has nothing provisioning it, and that is a black hole. This distinction is load-bearing — without it the release lanes, which are idle between releases, would be flagged as broken every sweep.

Service history is gathered from the workflows that consume the lane, found by scanning .github/workflows for the variable. A repo-wide "last N runs" sweep is not a time window: on a busy repo the newest 100 runs were measured covering well under an hour, so any lane used less often than that — every release lane — would be condemned on every sweep. Scoping to the consuming workflow makes 20 runs reach back months for a handful of API calls. The scan is also lazy: a lane with a live runner costs zero API calls.

Service history cannot expire — so read the queue too

Service history is a claim about the past, and it has no staleness notion: a lane that served jobs for weeks and whose provisioner died three hours ago still answers "yes, recently served". That is not a hypothetical. The release lane reported the provisioner is alive and idle while two releases sat queued behind it, because every signal the checker had was historical and every one of them looked healthy.

The queue is the only surface where a dead provisioner is visible while it is happening. So a lane with no live runner is also judged on the age of the oldest job currently queued for its exact label set, and crossing service_evidence.queued_stall_seconds (default 1800) reports queue-stalled.

The signal is queue age, never queue presence. On a JIT lane the healthy sequence is job-queues-first, then the provisioner notices and boots — so "queued job and no live runner" is the ordinary transient, and a presence-based check would fire on every burst and be switched off within a week. The budget is derived from the fleet's own trigger (min_queued_age_seconds is 0 on m3/m5 and 600 on m1) with wide margin for VM boot and runner registration, so only a genuine stall crosses it.

Like the service scan this is lazy — reached only when a lane has no live runner — and it is additive: the provider defaults to empty, so a lane with an empty queue keeps exactly the verdict it had before.

Honest limits. This check proves a lane can be served; it does not prove jobs are being served well. It will not catch a runner that is online but wedged, a lane that is slow rather than dead, or a black hole in a runs-on hard-coded in a workflow rather than driven by a variable. A capacity shortfall (labels resolve, queue still grows) is caught only once the queue crosses the stall budget above, and only when no runner is live — a lane with one wedged runner online and a growing queue still reads healthy. The per-host census below watches the other half of that gap from the other side: not "can these labels be served" but "is this HOST still serving". An ephemeral lane whose consuming workflow has not run inside the lookback window yields no evidence and is reported as a black hole — a false positive that is deliberately biased loud, on the grounds that a silent relief valve is what caused this in the first place.

Looked and found nothing vs. was not allowed to look. Those two are not the same claim, and the checker no longer conflates them:

• No service evidence within the lookback window → black-hole. The checker read every scope it needs and found nothing behind the labels. The loud bias above stays exactly as it is: a lane reported dead on thin evidence costs an operator a minute, a lane reported healthy on none costs a queue nobody can explain.
• A refused observation → visibility-incomplete, still an error. Reading actions/runners at org scope needs Administration: Read. Without it the org query 403s, every org-group runner is invisible, and "no runner carries these labels" becomes a claim the check never established — the exact misreading that had two sessions declare the live Mac Pro Linux lane dead on 2026-08-16. So the verdict is renamed to what the evidence supports, and its severity is unchanged: the workflow branches only on the exit code, and its tracking issue auto-closes on a clean sweep, so demoting this would let a persistent token-scope regression report green hourly and hide a real dead lane behind a permissions bug. Fix the token scope, or verify the lane's provisioner by hand — do not read it as either verdict.

A lane can read healthy while a host serves nothing

Every check above is about a lane: can this label set be served. None of them can answer is this host still serving, and those two came apart on m5. Its disk filled, its runner stayed registered and online, every lane it backs kept resolving to a live runner, and m5 quietly served nothing for days. Each lane check was correct and the fleet was down a third of its capacity.

The hosts block in runner_topology.json closes that half. It is a per-host census, not a new monitor and not a heartbeat: it reads the same jobs API the lane checks already read, and keeps the two fields those throw away, runner_name and completed_at. Host identity is the runner-name prefix, because tartci fixes an ephemeral runner's name as <lane>-<supervisor pid>-<boot index>, so the lane prefix is stable across boots while the full name is not. That prefix map is the one reviewed datum; everything else the census computes from observation.

silent(lane) := observed(lane)
                AND (now - last_served(lane)) > silence_hours
                AND demand(lane, silence_hours) >= demand_min_jobs

demand(lane, w) := jobs completed elsewhere within w carrying a label set
                   this lane is observed to serve
                 + distinct (job name, run) pairs completed elsewhere
                   within w for a job name only this lane is observed to run

Demand is the clause that makes this survivable. Silence on its own fires every quiet night and gets muted inside a week. Silence beside completed sibling jobs carrying a label set this lane is observed to serve is a lane that could have taken work and did not. sibling_demand counts only label sets the lane itself completed jobs for in the window, so m5's pulp-preamble traffic, which no other lane serves, is never demand against m1 or m3.

A lane with no sibling is measured against the work instead. That last sentence cuts both ways: if no other declared lane serves any label set this one serves, then sibling demand is not low, it is unmeasurable, and the clause scores zero however hard the fleet is working. That is not a corner case. It is the shape of the lane that stopped on 2026-09-12, which had been the busiest on the fleet. For a lane like that the checker asks the same question against the work: did the jobs this lane is observed to run keep completing somewhere else? The walk is scoped to one workflow, so a job name identifies a job definition rather than a category, and only names this lane owns count. A name another declared lane also ran in the window is not this lane's signature, and counting it would manufacture displacement out of unrelated work. Ownership is judged among declared self-hosted lanes only, so a hosted runner cannot disown a name: rerouting a broken lane's jobs to a hosted runner is the most common way its silence gets hidden, and it is precisely what this is built to see.

Two witnesses, one question, and neither gates the other. Label set and job name are independent readings of the same question, and each survives a case the other cannot. A reroute changes the label set by definition, so only the name witness sees one. A singleton lane has no sibling at all, so only the name witness exists for it. They are summed into one demand count, and the finding reports the split so the reader can see which one carried it. Gating either on the other is the defect that hid the motivating incident: an earlier build only consulted the name witness when the lane had no peer, and "peer" spanned the whole 72h observation window while demand is its 6h subset, so a single stale 40h completion on a lane that had itself since died was enough to switch the name witness off for a lane whose work was visibly being taken elsewhere.

Three smaller rules keep the count honest:

• Only a completion disowns a name. A job still running on a sibling has not established that the name is shared. Counting it would hand any concurrent job a veto over the verdict for as long as it runs.
• The threshold counts distinct work, not rows. Five reruns of one job are five rows and one job's worth of demand, so displaced work is keyed by (job name, run), falling back to the runner name when a payload carries no run id. The run rather than the runner, because a hosted runner is named GitHub Actions <id> with a fresh id per job: on exactly the runners displaced work lands on, a runner-name key collapses nothing. The name rather than the run alone, because one run holds many job definitions, and a whole workflow rerouted off a lane at once is as many pieces of demand as it has jobs.
• A runner outside the prefix map is named off-fleet. It is where the work went, so it is the half of the proof that matters most, and it has no declared host by construction. Calling it hosted would be a guess, and the wrong one exactly when a lane rename has left a self-hosted runner unmapped.

Five cases are deliberately not verdicts:

• Bootstrap. A host with no mapped job at all in the observation window is host-unobserved, never silent. A host that has never reported cannot page, and a host decommissioned for a month falls out of observation on its own rather than needing to be un-declared.
• In flight. A host with a job still running has not stopped serving. A job wedged long enough to matter is a different failure with a different owner (the stale-run reaper), so the census reports the state and declines the verdict.
• Idle fleet. Silence under the demand threshold is host-idle: nothing this lane serves was being served anywhere else either. A lane with no sibling reports host-idle too when not enough of its own work completed elsewhere, since the name witness is a real measurement that came back low.
• A lane neither witness can reach. If no other declared lane serves a label set this one serves, and every job name it ran was also run by a declared lane, then neither witness exists and nothing about this lane can be measured at all. That is host-unmeasurable, and it says so in those words. Reporting a structural blind spot in the same language as a quiet fleet is how the blind spot reads as a clean bill of health.
• An unread window. If any read failed, or the walk hit its run cap before covering the window, every silence verdict is suppressed and the sweep reports host-silence-degraded. This is the same fail-closed discipline the checker already applies to an unreadable runners API.

The identity is the lane, and the host is inventory. m5 declares two prefixes: the ephemeral gate lane m5- and the persistent pulp-preamble-m5 runner. A host-scoped predicate read a completion on either one as the host serving, so the cheap always-up lane vouched for the expensive gate lane that had stopped, which is the shape of the incident the rule exists for. Each lane therefore carries its own verdict, keyed host/prefix on a host that declares more than one. A host's always-up lane cannot vouch for a lane beside it that stopped, and evidence stays with the lane it describes: an online registration on the preamble runner is not an alibi for the gate lane next to it. Any host declaring more than one prefix that has any mapped job in the window also reports host-lane-census: last-served and job count per prefix, side by side.

Read the {n} job(s) counts as counts down to where the walk stopped, not counts over the window. The walk exits as soon as every declared prefix is proven, so a healthy lane's count says how many of its jobs were seen before that exit, and a busier lane can report fewer jobs than a quieter one simply because the exit came sooner. The last-served age beside it is the load-bearing number; the count is only there to show the age rests on real traffic.

It is instrumentation, not a verdict. It is the number that decides whether the host stays the unit of identity when the rule is promoted, or whether the predicate has to move down to the lane.

Two knobs bound the cost. service_evidence.lookback_hours is 720h, which is right for a lane that fires per release and wrong here: build.yml alone holds over 16,000 runs, so a 720h per-job walk would cost thousands of API calls every hour. The census uses its own observation_hours (72h, past a weekend and far past silence_hours) with a server-side created>= filter and pagination instead of the 20-run lane cap, plus a max_runs ceiling. It walks newest first and stops as soon as it is past the silence window AND every declared prefix is already proven to have served inside it. The verdict is per host, but the bound is per lane, because proving m5 over the union of its prefixes would let the always-up preamble runner stop the walk and leave the gate lane's real last completion unread, making the census wrong about exactly the host it was added to instrument. Being proven is the bound that matters: runs are ordered by creation, the rule is about completion, and the two come apart. A long-queued job, or a rerun (which keeps its run's original creation time), can complete hours after its run was created, so an older run can still carry a host's newest completion. Stopping at the first completion seen would read such a host as silent while it served minutes ago, which is a false fire in the exact direction this rule exists to avoid. A lane that has not served inside the window therefore pays the full walk on every sweep: that is both the one case where the full walk is the evidence and the one case where an older run can still change the answer. max_runs is sized well clear of the live window for that reason, and a truncated walk reports degraded rather than guessing.

It ships in shadow mode. hosts.severity is info, so the census reports to the step summary and nothing else: no issue, no assignee, no red run. That is deliberate, so one week of hourly sweeps can count would-be fires against real traffic before anything pages. Promoting it is a reviewed edit of that one value to error plus a per-host issue step in runner-topology-check.yml, and it must not happen before the operator notification path has been confirmed to reach a human.

The census does not diagnose and does not remediate. It does not read on-host disk receipts, does not ssh anywhere, and adds no host-side agent. It reports the persistent-registration state for a silent host as the discriminator between "powered off" and "up but not serving", which is the m5 shape, and leaves the call to an operator.

Where it runs, and why

• runner-topology-check.yml — hourly cron on ubuntu-latest, opening and auto-closing a tracking issue. The invariant is about live fleet state, so it can break with no commit at all: a runner is decommissioned, a host renamed, a variable edited in the web UI. A PR gate would never see any of that. It runs GitHub-hosted deliberately — a check that queues behind the saturated pool it is auditing is no check.
• runner-topology-selftest (ctest) — the diff-shaped half: contract well-formedness and the reconciliation logic. No network, so it runs on every PR for free and never adds an API call to the required macOS gate.
• fleet-snapshot-selftest (ctest) — snapshot regeneration and --check against a fake tartci checkout: a new profile file is picked up by glob, a removed one fails --check by name.
• runner-topology-static-selftest (ctest) — --mode=static on the real contract, workflows, and snapshot: the required gate must read REACHABLE, and a typo'd label must read UNSERVED.

The checker exits 2 when live state cannot be read, distinct from pass (0) and violation (1), so a missing token scope fails loudly instead of reporting a false green.

Changing a lane

A lane that omits unset_fallback is declaring that the consuming workflow has no route when the variable is unset, which the checker escalates to an error. Read the consuming workflow's whole resolution chain before believing that: an unset variable is frequently a deliberate state rather than a stalled lane, and the fallback is not always hosted. release-cli.yml resolves PULP_RELEASE_MACOS_RUNS_ON_JSON through PULP_LOCAL_MACOS_RUNS_ON_JSON before it ever reaches macos-15, so leaving it unset routes releases to the gate's pre-dispatch label set, which no registration serves. Reading only the last element of such a chain produces a confident and wrong conclusion in both directions. Record the real terminus as unset_fallback so the report keeps naming only the lanes that would genuinely have nowhere to run.

Edit the variable and its lane in runner_topology.json in the same change — the drift check exists to make that atomic. Then:

# Reconcile against the live fleet (uses ghapp locally — the App token bucket).
python3 tools/scripts/runner_topology_check.py --mode=report

# Advisory (never fails), useful while iterating.
python3 tools/scripts/runner_topology_check.py --mode=hint

# Optional read-only cross-repo evidence (all inputs are fixtures; no host query):
python3 tools/scripts/runner_topology_check.py --mode=report \
  --fleet-profile /path/to/profiles/m3-macos-fleet.toml \
  --fleet-receipt /path/to/installed-receipt.json \
  --fleet-source-manifest /path/to/fleet/local-macos-desired.json

The optional evidence flags are repeatable where appropriate and never install, reload, enable, or inspect a runner. They compare the checked-in TartCI profile, its installation receipt, and the private desired-fleet manifest. Keep repo-specific labels and host declarations in those Pulp/private inputs; generic Shipyard and TartCI code must not grow a second Pulp host table.

Offline reachability (--mode=static)

--mode=static answers the diff-shaped half of "can this lane be served" with no network: every lane's contracted value (and its unset_fallback) is judged against tools/scripts/fleet_advertised_labels.json, a snapshot of what each runner registration advertises, generated from the fleet's checked-in macOS profiles (schema tartci.advertised-labels/v1). A job is reachable only when all three hold: the registration advertises every requested label, it mints runners for the job's workflow name:, and the repository matches.

python3 tools/scripts/runner_topology_check.py --mode=static          # table, exit 1 on a required UNSERVED
python3 tools/scripts/runner_topology_check.py --mode=static --json

Verdicts: REACHABLE (names the host/lane/class that serves it), UNSERVED (names the labels no registration advertises, or the workflow mismatch), HOSTED, SENTINEL, UNKNOWN (workflow name not statically knowable, no consuming workflow, or a supervisor the snapshot does not cover, such as the Proxmox Linux pool), and UNDECLARED (a *_RUNS_ON_JSON variable a workflow reads with no lane, info only).

Read REACHABLE as declared supply, never as live service. A host whose installed profile has drifted from its checked-in one reads reachable here and serves nothing; that is what --mode=report exists for. The required macOS gate is judged on its dispatched label set (build.yml's event-class rewrite, via the same _event_projection the live checker uses), not the raw variable, which still carries a label the event-class registrations deliberately omit. Nothing in Pulp names a machine: the snapshot is tartci's published fleet/advertised-labels.json (or, in a tartci that predates it, tartci's own generator run over every profiles/*-macos-fleet.toml, discovered by glob), so adding or removing a host is a profile change in tartci plus a regeneration here.

python3 tools/scripts/fleet_snapshot.py --tartci /path/to/tartci --write   # regenerate
python3 tools/scripts/fleet_snapshot.py --tartci /path/to/tartci --check   # 1 = stale, 2 = unreadable

--check compares registrations only (provenance metadata is expected to move) and names every host, lane, or registration added, removed, or changed. runner-topology-check.yml runs it hourly against a fresh shallow clone of the public tartci repo on Python 3.12 (tomllib), and a stale snapshot joins the same finding and tracking issue as a live routing violation; a failed clone is reported as unreadable state (exit 2), never as green. Pulp reads only labels, workflows, and repository from the snapshot, never tartci internals.

Egress-relay contract for the protected macOS gate

The gate VMs reach the internet only through an egress relay whose allowlist tartci publishes as profiles/pulp-protected-macos-bootstrap-hosts.toml (literal_hosts plus transitive_hosts). A download from any other host fails inside every gate VM at once: a pip install added to build.yml before the relay admitted PyPI failed every m5 gate job for a day. Pulp keeps a copy, tools/scripts/relay_contract_hosts.toml, and the relay-contract-hosts ctest fails when the gate needs a host the copy lacks, naming the host, the step or file that needs it, and the tartci file to update:

python3 tools/scripts/relay_contract_check.py                                  # gate check (the ctest)
python3 tools/scripts/relay_contract_check.py --tartci /path/to/tartci --check  # copy vs tartci: 1 = drift, 2 = unreadable
python3 tools/scripts/relay_contract_check.py --tartci /path/to/tartci --write  # refresh the copy

The required hosts are derived, not listed: literal http(s):// hosts in run: scripts of build.yml jobs that can run on self-hosted macOS (steps confined to Linux or Windows by if: are skipped), the hosts of the package managers those scripts invoke (pip ⇒ pypi.org and files.pythonhosted.org, npm ⇒ registry.npmjs.org, brew ⇒ ghcr.io and formulae.brew.sh), and a short CORPUS_HOSTS list for downloads a registered test makes itself, each pinned to the source text that performs it. Adding a host is a tartci change first: land the relay entry, then --write the copy in the Pulp PR that needs it. The hourly runner-topology sweep runs --check against the same fresh tartci clone as the snapshot check and reports drift in the same finding.

decisions_contract.py --mode probe --live needs Shipyard >= 0.208.0, the first release with shipyard landing; an older binary is reported as shipyard landing unavailable (need >= 0.208.0; found ...) with exit 2, never as a contract failure. --landing-json FILE needs no Shipyard at all.

Fact-checking declared supply

Three layers, each answering a different question, so a disagreement names which side drifted. None of them lists a host: every one is derived from tartci's profiles or from observed runner names.

1. Declared (git). What the checked-in fleet profiles say every runner registration advertises. tartci publishes it as fleet/advertised-labels.json; Pulp keeps a copy and checks it:

python3 tools/scripts/fleet_snapshot.py --tartci /path/to/tartci --check
python3 tools/scripts/fleet_snapshot.py --source-url --check   # tartci's published file on main
python3 tools/scripts/runner_topology_check.py --mode=static    # every lane vs declared supply

With --tartci, the published file is used when present, otherwise tartci's generator runs over every profiles/*-macos-fleet.toml. The hourly topology sweep runs the --check against a fresh clone. 2. Installed (machines). Whether each host actually runs the profile it declares. This lives on the hosts, in tartci: tartci fleet-macos verify-supply (installed vs declared) and tartci fleet-macos profile-drift. Pulp cannot see it; a host whose installed profile drifted reads REACHABLE in layer 1 and serves nothing. 3. Observed (GitHub). Whether the declared registrations served jobs. runner_topology_check.py --mode=report reads the jobs its host census already fetched (no second crawl) and reports, per snapshot registration serving a required lane, OBSERVED (with last seen), NOT_OBSERVED (jobs it could have served ran elsewhere), or IDLE (no such demand in the window), plus UNDECLARED_OBSERVED for a runner prefix whose jobs match no registration by name (<host_id>-<lane>-...) and labels. tartci's scripts/supply_observed.py answers the same question from its side.

python3 tools/scripts/runner_topology_check.py --mode=report   # live; observed-supply lines

These findings are INFO/WARN only and never a ctest: the census covers one workflow and stops early once every host is proven, so IDLE is a prompt to look, not a fault. Jobs are matched to a registration by GitHub's own rule (job labels a subset of the runner's), not by the registration's minting workflow list: release-cli's darwin legs were observed running on pulp-gate runners.

Routing overrides

A lane whose contracted value is a deliberate temporary state cites an entry in runner_topology.json's overrides array by override_id. Each override records subject (the variable), value, owner, reason, revert_condition, since, and expires. Every mode lists active overrides with their age, and fails on an expired override, an override whose subject is not a declared lane or routing control, an override whose value no longer matches its lane, or a lane citing an override that does not exist. The hourly live sweep enforces expiry against the real clock; the ctest pins its clock so an expiry date never reddens unrelated PRs. Renewing is an edit of expires with the reason in the commit.

An unset variable is not automatically a gap

Four lanes read as broken in the contract while behaving exactly as intended, because "unset" and "hosted" each mean something specific per lane. Read the consuming workflow before calling one of these a black hole:

• PULP_RELEASE_MACOS_RUNS_ON_JSON is set to the gate's base labels, not unset. Unsetting it falls through to PULP_LOCAL_MACOS_RUNS_ON_JSON, whose pulp-gate-fast label no registration advertises, so releases would queue forever. The set value routes releases onto idle gate runners, which is why hosted starvation (2026-05-18, 2026-06-09) no longer blocks publishing; see "The macOS release VM lane and cross-lane priority" above.
• PULP_OVERFLOW_BUILD_MACOS_RUNS_ON_JSON is currently OFF, by contract. The lane is designed to be hosted — overflow exists to add capacity when the local pool is saturated, so pointing it at the same local labels is a no-op under the exact condition it must relieve. But hosted macOS overflow is disabled for cost control, so the live value is the local-only sentinel: a documented off-switch, not a label set. runner_topology.json contracts that sentinel as this lane's expect, so the intended off state reads as compliant rather than as drift. It previously contracted ["macos-15"] while the variable held the sentinel, which parked a severity: required lane at a permanent ERROR [drift] for its own intended state — a standing red trains readers to skim the report, and a genuine drift then hides in the noise. Do not "fix" this by unsetting the variable: unset_fallback is ["macos-15"], so unsetting re-enables the hosted overflow the sentinel exists to disable. The variable must stay explicitly set.
• PULP_INTEL_RELEASE_MACOS_RUNS_ON_JSON is hosted because darwin-x64 is cross-compiled on Apple Silicon, not built on the native Intel image. The Mac mini native-Intel lane (PULP_NATIVE_INTEL_RUNS_ON_JSON) is a separate advisory lane and is not a substitute for this release leg.
• PULP_RELEASE_CONTROL_LINUX_RUNS_ON_JSON coordinates a release rather than building one. It resolves to PULP_LOCAL_LINUX_RUNS_ON_JSON next, but reaching the Mac Pro pool is not a selector decision alone: those runners sit in a restricted org runner group, so the group's workflow-ref restriction governs which workflows may use them.

The general rule: a lane is only "missing" once you have read the fallback chain in its consuming workflow. A variable, a supervisor process, a runner-group row, and an idle VM are each individually consistent with a lane that works and with one that does not.

macOS overflow routing (Plan B)

Namespace is OFF (cost). We build macOS on local Macs + GitHub-hosted only. PULP_NAMESPACE_BUILD_MACOS_RUNS_ON_JSON is kept UNSET, so the Namespace overflow described here never fires — it's a documented break-glass option, not the active path. The required gate is the clean-per-job JIT VM pool (PULP_LOCAL_MACOS_RUNS_ON_JSON). Its repo variable carries the reviewed legacy base selector, but build.yml replaces pulp-gate-fast with the exact PR-head or merge-group class before assignment; profiles serving the gate must advertise both classes and let provider code derive their priorities. Do not repurpose the Namespace var to point at self-hosted runners (see CLAUDE.md "Runner priority").

Read the live variable, not this page's defaults. A routing var describes reality; build.yml's || fallback is only what happens when the var is unset. The reviewed contract currently expects GitHub-hosted macos-15 overflow; local-only remains the explicit disable sentinel. Confirm before reasoning about a route:

gh variable list -R Generous-Corp/pulp | grep RUNS_ON_JSON

Live routing state

Do not maintain another selector table here. The reviewed source is tools/scripts/runner_topology.json; the live source is the GitHub repo variables. Reconcile the two with:

PATH="$HOME/.config/tartci/ghapp-shim:$PATH" \
  python3 tools/scripts/runner_topology_check.py --mode=report

The required macOS contract records the pre-dispatch base and the event-class-v2 transformation separately, overflow is contracted to GitHub-hosted capacity, and Namespace variables remain unset. Exact repo labels and declared profile bindings belong in the JSON contract and private fleet/profile inputs.

When the local self-hosted Mac runner is saturated, build.yml's resolve-provider job can route new PR or workflow-dispatch macOS legs to the configured generic overflow target. Live policy sets that target to the local-only sentinel, so overflow is disabled. Namespace remains an explicit, paid break-glass option and is never selected automatically.

Precedence (highest first, resolved per dispatch):

1. Operator override — gh workflow run build.yml --field macos_runner_selector_json='"<label>"'. Always wins.
2. Overflow — for PR and workflow-dispatch events, an idle registered local runner keeps the local route. Otherwise BUSY >= PULP_LOCAL_MAC_OVERFLOW_THRESHOLD (default 2) selects PULP_OVERFLOW_BUILD_MACOS_RUNS_ON_JSON. The local-only sentinel disables this branch; an unset variable restores the hosted macos-15 fallback.
3. Local default — PULP_LOCAL_MACOS_RUNS_ON_JSON. For the current label set, read the lane in tools/scripts/runner_topology.json — that file is checked against the live fleet, so it cannot drift the way a value quoted here can.

Tuning knobs (repo variables):

Variable	Default	Purpose
PULP_LOCAL_MAC_OVERFLOW_THRESHOLD	2	BUSY count that triggers overflow. Raise when Plan A's 2nd local runner lands.
PULP_LOCAL_MAC_RUNNER_LABEL	pulp-gate-fast	Label the busy probe looks for in a macOS job's labels array. It must name a label the gate actually dispatches, and today it does not: build.yml strips pulp-gate-fast and appends one event-class label, so a dispatched macos job never carries this value and the probe always counts zero. Inert while overflow is the local-only sentinel; re-tune it before re-enabling overflow rather than reading the pinned value as proven.
PULP_OVERFLOW_BUILD_MACOS_RUNS_ON_JSON	["macos-15"] when unset	Generic overflow selector JSON, or the bare sentinel local-only to keep work local.

Disabling overflow (the live state):

gh variable set PULP_OVERFLOW_BUILD_MACOS_RUNS_ON_JSON \
  --body local-only --repo Generous-Corp/pulp

The next PR's resolve-provider keeps the macOS leg on the local selector.

Inspecting routing decisions: resolve-provider's stderr prints a one-line summary, e.g. resolve-provider: macOS route = overflow (BUSY=2 >= 2); selector = "namespace-profile-generouscorp-macos". Find it via the GitHub Actions UI under the resolve-provider job's log, or:

gh run view <run-id> --log --repo Generous-Corp/pulp | grep "macOS route"

Manual overflow / rescue is still available via shipyard rescue <PR> and remains useful for in-flight PRs that queued before the overflow logic kicked in (until Shipyard's rescue re-runs pull_request runs, it dispatches a branch-tip workflow_dispatch replacement, which is the wrong proof for a required check under strict protection; for a required pull_request run, ghapp run cancel <run> if it is still running, then ghapp run rerun <run> --failed). With Plan B in place, manual rescue should be needed much less frequently.

pulp overflow — operator surface

tools/cli/cmd_overflow.cpp wraps the three repo variables behind a discoverable CLI:

# Show current routing state (local target, overflow target, threshold,
# plus self-hosted runner registration if visible to the default token):
pulp overflow status

# Turn overflow on (defaults to free GH-hosted "macos-15"):
pulp overflow enable
pulp overflow enable --to '"macos-15"'

# Turn overflow off — every macOS leg goes to the local target.
# In-flight cloud jobs continue to completion; only new dispatches change.
pulp overflow disable

# Read / set the BUSY threshold (default 2; set to 1 for single-runner setups):
pulp overflow threshold
pulp overflow threshold 1

pulp overflow disable writes the local-only sentinel; deleting the variable would restore the hosted default. It does not cancel in-flight cloud runs. To The protected-main retarget workflow does not currently move a PR back to the local pool; see the fail-closed boundary below.

Per-PR macOS retargeting (pulp macos)

For the case where automatic overflow picked the "wrong" cloud pool — e.g. you want to push a specific PR to Namespace for paid-fast turnaround or move it to GitHub-hosted — use the build-macos.yml workflow + the pulp macos CLI:

# Local is intentionally refused until the two-account Tart class is proven:
pulp macos retarget --pr 1910 --to local  # fails closed

# Pay to skip the queue (Namespace billable, fast parallel):
pulp macos retarget --pr 1910 --to namespace

# Force GH-hosted macos-15 (free, slower):
pulp macos retarget --pr 1910 --to github-hosted

# See where the current macOS check is routed:
pulp macos status --pr 1910

pulp macos retarget cancels any in-flight macOS-bearing workflow_run for the PR and fires a fresh build-macos.yml dispatch on the chosen runner. A checks-write-only controller creates one in-progress check run on the resolved exact PR head before the build and completes that same check afterward, so branch protection can accept the newest macos check without re-running Linux/Windows.

build-macos.yml is independent of build.yml's matrix — they share check names but not workflow_runs. The matrix workflow continues running Linux/Windows as usual; only the macOS leg is replaced.

The retarget lane consumes the same reduced required-gate CTest labels and the same pinned, checksum-verified Chrome as build.yml. The CLI dispatches the workflow definition from protected main, never from the PR branch. Before claiming a macOS runner, that trusted control path resolves one open internal PR, validates that PR still targets Generous-Corp/pulp:main, and pins both its exact head and the immutable base SHA recorded on the PR. The build runner first checks out the trusted workflow SHA with Git credentials disabled, then fetches and verifies the exact PR objects without materializing them. The first PR-head checkout happens only after the isolated clone belongs to nobody. That build job has only contents-read permission, no Actions/Namespace cache action, no persistent ccache, and run-unique build, FetchContent, Skia, and Chrome paths removed at teardown. Because a protected-main Actions job also carries implicit runtime cache credentials, every PR-controlled setup/CMake/build/test command executes as the separate nobody uid through an empty, explicit environment. The PR source is a disposable, non-hardlinked clone created by trusted control before ownership transfers to that uid; no ACTIONS_*, GITHUB_*, token, credential, or Actions command-file variable crosses the account boundary. Proxy variables are omitted too because proxy URLs may carry userinfo credentials. The checks-write token exists only in the pending/final controller jobs, which never check out or execute PR code and revalidate the complete PR identity (open state, base repository/ref/SHA, and head repository/ref/SHA) before posting. Before the pending check is created, the trusted controller uploads an immutable one-day recovery identity. A separate source-free workflow_run reconciler on protected main uses that identity to terminalize the exact check if cancellation prevents the normal completer from running; it never checks out PR code. That hardened checkout clones full history (fetch-depth: 0) rather than the shallow default. The GPU provenance selftests read real per-path Git history, and the remedy the other lanes use — tools/scripts/hydrate_gpu_provenance_commits.py, which reconnects a shallow clone by fetching the event ref — cannot run here: this checkout sets persist-credentials: false, so no credential remains for a fetch, and it pins ref: ${{ github.sha }}, so GITHUB_REF no longer names the checked-out commit. Cloning in full is the only remedy compatible with both hardening choices, and it is why this lane's checkout looks different from build.yml's.

The local route always fails closed: today's JIT Tart guest is disposable, but its Actions runner and PR code share the administrative guest account, so PR code could still reach protected-main runtime/cache credentials during the job. Re-enable local retarget only after a separate hardened two-account image/class proves that the runner-controller account is unreachable from the build account. Namespace must likewise equal the approved namespace-profile-generouscorp-macos selector. Capability-history checks compare against the event-pinned base, not newer live main; the native merge queue validates the eventual combined tree separately. Keep these contracts mirrored: a provider reroute must not turn the required gate into a full benchmark lane, expose protected cache/write authority to PR code, validate a different commit pair, or depend on stale runner checkout history.

Automated zero-job recovery sets recovery=true and supplies expected_head_sha, source_run_id, and source_run_attempt. Before publishing a pending check, the protected resolver rejects a moved PR or any source that is no longer the exact queued pull_request attempt for build.yml with an exhaustive zero-job census. Manual operator retargets leave recovery false, omit the source identity, and continue to resolve the current live head. The untrusted wrapper changes to its isolated home before dropping privileges and explicitly forwards only the run-unique source path required by setup/build/test; it never inherits the protected Actions checkout as its working directory.

That isolated home, and every other run-unique path the lane creates, live under a run-unique root beneath /private/tmp rather than under $RUNNER_TEMP. $RUNNER_TEMP sits inside the runner account's home directory, which is mode 700, so the nobody uid cannot traverse into it: the wrapper changes into the isolated home as the trusted user and then drops privileges, and every untrusted command inherits a working directory it cannot resolve, failing in getcwd before the command itself runs. /private/tmp and its ancestors are world traversable, and the root is created 0711 so nobody can traverse into its own paths without enumerating anything beside them. Because that base is world-writable and sticky, the root is created without mkdir -p: a path that already exists belongs to someone else and the run fails closed instead of adopting it. The if: always() teardown step is the only thing that removes the root, and its safety guard matches the same /private/tmp/pulp-retarget-* prefix; the root literal and that guard must move together or cleanup refuses and leaks the untrusted tree.

Workflow inputs (visible in gh workflow run build-macos.yml --help):

Input	Default	Effect
pr_number	inferred from target_ref	Identifies the open internal PR and its immutable base/head pair
runner	github-hosted	Routes to "macos-15"
runner=local	—	Fails closed pending a proven two-account Tart runner class
runner=namespace	—	Routes to PULP_NAMESPACE_BUILD_MACOS_RUNS_ON_JSON
runner=github-hosted	—	Routes to "macos-15" (free GH-hosted)
target_ref	(workflow's ref name)	Exact internal PR head branch to validate and build
expected_head_sha	empty	Optional immutable head binding for automated recovery; mismatch fails closed before build
source_run_id	empty	Exact zero-job Build and Test run; accepted only with recovery=true
source_run_attempt	empty	Positive exact attempt for source_run_id; accepted only with recovery=true
recovery	false	Revalidate the exact queued source run and exhaustive zero-job census before publishing macos

Opportunistic reroute daemon

tools/scripts/macos_reroute_watcher.py is a long-running watcher (intended as a launchd agent on the self-hosted Mac) that automates the "when local frees up, claw back queued GH-hosted jobs" pattern. It polls every 30 seconds:

1. Is the local Mac runner idle? (process-based detection via ps; no admin token needed.)
2. Is there a queued Build-and-Test workflow_run whose macOS job has macos-15 (or nscloud-* / namespace-profile-*) labels — i.e., dispatched to cloud but not yet picked up?

The watcher remains installed-capable but its local handoff is intentionally inert while local retarget is fail-closed. Do not enable it as an automatic reroute authority until the two-account Tart prerequisite above is proven.

Install (one-time per host):

# Copy the template into LaunchAgents, substituting your Pulp checkout path:
sed "s|\$PULP_REPO|$PWD|g" \
  tools/launchd/pulp-macos-reroute-watcher.plist.template \
  > ~/Library/LaunchAgents/com.danielraffel.pulp.macos-reroute-watcher.plist

launchctl load ~/Library/LaunchAgents/com.danielraffel.pulp.macos-reroute-watcher.plist

# Logs:
tail -F ~/Library/Logs/pulp/macos-reroute-watcher.log

Run by hand for testing:

python3 tools/scripts/macos_reroute_watcher.py --interval 30 --log-level DEBUG

Stop:

launchctl unload ~/Library/LaunchAgents/com.danielraffel.pulp.macos-reroute-watcher.plist

The watcher is safe to run alongside the overflow probe in build.yml — they cooperate. The probe decides where to dispatch initially; the watcher opportunistically reroutes dispatches that landed on cloud while local was busy, once local frees up before the cloud runner has picked up the job. If the cloud runner has already started, the watcher takes no action.

Self-hosted runner operations: prevent, recover, maintain

Shipyard v0.55.0+ is the minimum pin for the full self-hosted-runner operational toolkit, and Pulp pins v0.56.2+ so rescue, update, and shipyard wait pr have REST fallback paths when GraphQL is rate-limited. The commands are discoverable from shipyard --help and replace the legacy planning/scripts/runner-watchdog.sh + manual reinstall workflow.

# Recover one PR whose required macOS check is wedged or stale
shipyard rescue <PR>                      # cancel queued runs + redispatch to github-hosted
shipyard rescue <PR> --rerun-failed       # also dispatch branch-tip replacements of cancelled/failed runs
                                          # (not reruns: for a required pull_request run, use
                                          # `ghapp run rerun <run> --failed` instead)
shipyard rescue <PR> --dry-run            # preview without acting
shipyard rescue --all-stuck               # repo-wide stuck-run sweep
shipyard rescue <PR> --to github-hosted   # explicit destination provider

# Prevent future wedges on a self-hosted runner host
shipyard runner watch --kill-hung-workers # implies --fix; pair with launchd/systemd

# Keep the installed Shipyard CLI current
shipyard update --check --json            # report installed vs available
shipyard update                           # apply latest stable
shipyard update --to v0.56.2              # pin or roll back to Pulp's minimum
shipyard update --dry-run                 # plan only

# Wait after handoff/rescue without depending solely on GraphQL
shipyard wait pr <PR> --state green       # REST fallback as of v0.56.2

Frequent safety nets do not run at their cron cadence

GitHub delays schedule events under load and drops the ones that pile up. On this repository every hourly-or-faster cron fires roughly once every five hours, whatever it says: over 2026-10-01 00:00Z to 2026-10-02 17:00Z each */15 and */30 workflow got 7 or 8 scheduled runs (82 to 164 expected) and each hourly one got 6. Daily crons are the control: they fire every day, five to seven hours late. Run numbers stay contiguous, so the runs are never created; this is GitHub-side, not a disabled job or a deleted run.

.github/schedule-backstop.json lists the frequent safety nets (the watchdogs in this section, the release reconciler and cadence check, version-at-land, and the rest). An external dispatcher, tartci's schedule-backstop agent, runs on one fleet host every five minutes and calls workflow_dispatch on a listed workflow when the newest main run of that workflow, from any event, is older than its cadence_minutes and none is queued or running. It only dispatches; the workflows still run on GitHub-hosted runners. The cron stays as the backstop's own backstop, so with the agent off or its host down the behaviour is exactly the throttled cron. Because the freshness test counts every event, a workflow that an event trigger already keeps fresh is never dispatched.

tools/scripts/schedule_backstop_check.py (in workflow-lint.yml and gates.sh) holds every listed workflow to what the dispatcher assumes: workflow_dispatch with no required input, a cadence equal to its cron, a top-level concurrency group, and no self-hosted runner label. A new hourly-or-faster cron must be listed or added to excluded with a reason.

To judge it, count runs per workflow per day against 1440 / cadence_minutes:

ghapp api "repos/Generous-Corp/pulp/actions/workflows/merge-stall-check.yml/runs?per_page=1&created=>=2026-10-03T00:00:00Z" --jq .total_count

Off-fleet queue-age watchdog (runner-health-check.yml)

.github/workflows/runner-health-check.yml sweeps every 30 minutes and opens a tracking issue when dispatch or a runner lane has stopped serving work. It runs on ubuntu-latest on purpose: a guard that lives on the fleet dies with the fleet, so the one outage it exists to report would be the outage that silences it. It is the symptom-level backstop under the recovery tooling above — shipyard rescue and runner watch fix a wedge you already know about; this tells you a wedge exists.

Why queue age, and not a runner-label check. The macOS lanes are JIT/ephemeral: a runner registers with GitHub only while it serves a job. So "zero runners carry label pulp-studio-01" is both the healthy-idle state and the dead-lane state, and nothing on GitHub's side can tell them apart. A label-satisfiability probe therefore false-alarms every idle night and gets muted within a week. Queue age is the observable that separates alive from dead, and it is cause-agnostic — it catches causes nobody has enumerated yet, not just the one that happened last time.

Why it stays quiet on a busy afternoon. A deep queue on a healthy pool is normal: the measured baseline on this repo under normal load is a median queue age of 5 min, an oldest of 31 min, and 3 runs past 30 min. A naive "queued > 30 min" rule alarms on that. So an alarm requires two independent conditions:

1. Age — the job has waited past alarm_minutes (default 45, roughly 1.5x the observed healthy maximum).
2. Liveness — its lane shows no sign of life: nothing with comparable labels is in_progress, and nothing with comparable labels has started since the job queued.

The liveness condition carries the false-alarm load, which is what lets the age threshold stay tight enough to detect a dead lane within 45–75 minutes. A saturated pool keeps its runners visibly busy, so it stays quiet at any queue depth; one runner grinding on a 90-minute job is alive, not dead; an idle fleet has nothing queued and so says nothing. Only "work piling up with nothing serving it" alarms. Findings between 30 and 45 minutes appear in the run summary only, never on the issue.

Before job expansion. A pull-request Build and Test run from .github/workflows/build.yml can remain pending or queued with an empty jobs array. At that point GitHub has not emitted a job or requested labels, so runner-liveness evidence cannot see it. The watchdog records the run only when the jobs API reports both total_count: 0 and jobs: [], then an exact-run reread confirms the same status, head, event, and workflow. It applies the same age thresholds. Push, merge-group, workflow-dispatch, and unrelated workflow runs are excluded: they have separate concurrency/merge-stall semantics. A zero-job finding points first to an older non-terminal run on the same ref holding the workflow concurrency group, not to Tart capacity. This alert is the off-fleet backstop when a current-head run is stranded.

Any failed API read or truncated run listing makes the sweep degraded. A degraded sweep suppresses alarms and never closes the tracking issue; absence is not evidence of recovery until a complete sweep. There is one exception. On this repo the completed listing truncates on every sweep, and a truncated completed listing can hide only a comparable job that started after the stalled one queued and has already finished. It can't hide an in-progress job, because in-progress jobs come from their own listing. So when the completed listing's truncation is the sweep's only gap, a queued job alarms once it has waited past 2 x alarm_minutes (TRUNCATED_COMPLETED_ALARM_FACTOR) with no comparable job in progress. That alarm can open, update or reopen the tracker. It can't close it.

Before this exception existed, every sweep was degraded, so the watchdog could never alarm. A native Intel lane whose only runner had been deleted showed up as a warn row 322 minutes deep, night after night, and no tracker opened.

A deaf watchdog is itself an alarm. A sweep with a gap other than the completed-listing exception can't alarm, so it is blind. The sweep records that by running a Record blind sweep step, and the next sweep reads that step back from the jobs API, so no state is stored anywhere. When 4 sweeps in a row are blind (about two hours), the watchdog opens its own tracker, Queue watchdog blind: <reason>. The tracker names the gap, such as failing reads or truncated listings, along with the likely cause and the fix. The first sweep that isn't blind closes it. An unreadable history can delay this alarm but can't invent one.

Org runner census authentication. Label reconciliation reads orgs/<org>/actions/runners, which needs Administration: Read on the org. The workflow uses RELEASE_BOT_TOKEN and falls back to GITHUB_TOKEN. When the fallback is in effect, the org scope refuses the read, and every runner_census_blind row says so. The fix is a GitHub App installation token that has org self-hosted-runner read, minted in the job with actions/create-github-app-token. Adding that secret is an owner decision.

The issue is edited in place each sweep and closes automatically on recovery — the same open/update/auto-close contract as the release watchdogs (see release-watchdog.md). A runner-lane report names the labels the stalled jobs asked for; a pre-expansion report names the workflow run, ref, and status instead.

Thresholds and analysis live in tools/scripts/queue_age_watchdog.py, tested by tools/scripts/test_queue_age_watchdog.py — which pins the measured baseline above as a must-stay-quiet regression case, so a future threshold edit that would re-introduce afternoon false alarms fails at PR time.

# Tune or dry-run a sweep by hand
gh workflow run runner-health-check.yml -f dry_run=true
gh workflow run runner-health-check.yml -f alarm_minutes=60

# Replay a recorded snapshot offline (no API calls, verdict pinned to capture time)
python3 tools/scripts/queue_age_watchdog.py --snapshot snapshot.json

Label reconciliation: why a stalled job is not being picked up

Queue age answers is the lane alive. When the answer is no, it does not say why, and the reader is left guessing among a wedged worker, an expired token, an asleep host, and a label nothing serves. On 2026-09-21 it was the label: three queued macos jobs each requested pulp-build-merge-group while every online runner advertised pulp-build-pr-head. Those jobs were unschedulable from the moment they queued, the merge queue head sat in AWAITING_CHECKS behind them, and 5 h 30 min passed with zero merges and no alarm anywhere.

The same sweep now reconciles the two. For each distinct label set some job has already been queued on past alarm_minutes, it compares the request against the labels online self-hosted runners advertise — across both the repo and org scopes, because runners in an org runner group are invisible to the repo endpoint and this org keeps online ones.

finding	level	means
unschedulable_labels	alarm	No online runner, busy or idle, advertises the full requested set. The finding names the labels nothing carries.
runner_census_blind	warn	The census could not be completed (a scope refused, a listing came back short, or nothing was online), so no label can be called unserved this sweep. It reports the unconfirmed diff as a lead, never a verdict.

Three properties make this safe, and each is pinned by a test:

• Demand-gated, so an idle fleet is silent by construction. This is not the scheduled label-satisfiability census argued against above. That census asks "does anything advertise label X?" of the whole fleet on a timer, and because these runners are JIT — registered only while serving — a healthy lane answers "no" every idle night until the alarm is muted. This check has nothing to evaluate unless a real job is stalled on a real label set. The JIT objection is bounded rather than ignored: a healthy lane mints a runner in seconds to minutes, so a missing label has had 45 minutes to appear before anything fires.
• Saturation is not unschedulable. GitHub places a job on one runner that carries every requested label, so "schedulable" means some online runner's label set is a superset of the request. A busy superset is a deep queue on a working lane and stays silent at any age.
• Fails closed on blindness. A failed or empty runner read reports an evidence gap — never "unschedulable", and never a clean bill of health either. That gap is scoped to this finding: it deliberately does not use the sweep-wide degraded predicate, because neither a truncated run listing nor a failed jobs call can falsify a queued job's own requested labels, and sharing the predicate would leave the check permanently degraded on a repo busy enough to truncate.

The org-scoped runners API needs Administration: Read, which GITHUB_TOKEN does not carry, so the workflow passes secrets.RELEASE_BOT_TOKEN when it is configured (the same fallback runner-topology-check.yml uses). Without it the org scope refuses, the census records the refusal, and every reconciliation reports runner_census_blind instead of naming a label — honest, and disarmed.

Contribution: the host that goes quiet while the lane stays healthy

Queue age answers is the lane alive. It cannot answer is every host still in it, and those are different questions with different answers. On 2026-09-15 one macOS host stopped serving at 10:02Z and did not serve again for 7 h 06 min. Its two peers absorbed the load, so jobs kept being picked up, so every queue-age sweep in that window was correctly quiet. Six monitors read green; three of them had died of the same cause as the host they were watching.

The same sweep therefore also groups the required macos job's runner_name by host prefix over the last three hours and reports:

finding	level	means
host_stopped_contributing	alarm	an expected host served zero macos jobs in the window while the fleet served at least 3
unknown_fleet_host	alarm	a self-hosted runner served under a name no expected prefix matches — a rename drops a host out of coverage silently, so it is loud in both directions
sole_host_for_class	warn	only one host served a class label; the next silent-host alarm on it is an outage rather than a degradation
contribution_guard_unconfigured	warn	the expected-host list is empty, so nothing can alarm — the guard reporting its own disarmament
sweep_cadence	warn	the gap since the previous sweep, when it exceeds twice the promised interval

The demand floor is the whole design. Below three fleet-served jobs in the window there was not enough work to distinguish an idle host from a dead one, and the check stays silent. That is what keeps this off the runner-label census described above: a census reads zero on a healthy fleet whenever nothing asked for that class, so a census-based alarm fires every quiet night and is muted within a week.

It opens its own tracking issue rather than reusing the queue-stall tracker. A silent host is not a stalled queue, and naming it one sends the reader to audit a queue that was working the whole time.

Expected prefixes come from the repo variable PULP_FLEET_EXPECTED_MACOS_HOSTS (for example m1-,m5-,studio-, matching each host's TARTCI_RUNNER_NAME_PREFIX). Unset falls back to the built-in list rather than disarming the check; set-but-empty reports contribution_guard_unconfigured.

The window adapts to what the collector can actually see. MAX_RUNS_PER_STATUS caps each status listing at 60 runs, and on this repo the completed listing is always truncated: measured 2026-09-15, those 60 runs spanned 2.35 h. A fixed 3 h window over that evidence is a window that can never be filled, and the first draft of this check treated the truncation as an evidence gap — which made it permanently degraded, permanently unable to alarm, and permanently green. So the cutoff is max(requested window, oldest observed job), every finding reports the span it was actually computed over (window_hours, window_truncated_by_coverage), and a window shorter than 1.5 h produces no finding at all. Truncation now costs reach, not correctness. A failed jobs call is still disqualifying, because that one can hide a host that really was working.

How this guard fails, and how you would know.

• Its cadence. Detection latency is bounded by how often it actually runs, not by its cron expression: GitHub has been delivering this workflow's */30 as roughly one sweep every four hours, which multiplied every latency here by eight and reddened nothing. Each sweep now measures the gap since its predecessor and reports sweep_cadence, so the degradation is visible where the findings are. Measured live on 2026-09-15: 193 minutes.
• Why there is no second trigger. workflow_run on Build and Test is the obvious fix and is deliberately absent. One sweep costs 4 run listings plus one jobs call per observed run — up to ~245 calls and ~4 minutes — against GITHUB_TOKEN's 1000 req/hr/repo, so ~4 sweeps/hour is the ceiling. Firing per gate completion exceeds it, and the failure is silent: the sweep starts failing its own API calls, which it correctly reads as incomplete evidence and suppresses alarms on. A trigger that converts a detection guard into a quiet one is worse than a slow guard. Raising the cadence needs the per-run jobs fan-out reduced first.
• Total absence. A sweep that never runs cannot report its own absence. What is visible is that both tracking issues stop being updated and the workflow's run list goes quiet in public.
• A half-finished scan. If the scan step produces no counts, the step fails rather than reporting zero alarms.
• Calibration, openly unresolved. The demand floor is 3 fleet-served jobs in the window. Under a uniform-assignment model with three hosts, a healthy host drawing zero of 3 jobs has probability (2/3)³ ≈ 30%, so this floor alone is not a strong false-positive bound. Assignment is not uniform (JIT polling, lease priorities, m1's deliberate 10-minute delay), so the uniform model overstates the risk — but the true base rate has not been measured. The floor is exposed as --contribution-min-fleet-jobs and as a workflow_dispatch input so it can be raised without a code change once the base rate is known.

# What the sweep would say right now, without touching an issue
PULP_GH_BIN=ghapp python3 tools/scripts/queue_age_watchdog.py \
    --repo Generous-Corp/pulp --snapshot-out snapshot.json

# Replay it, or a widened window, offline
python3 tools/scripts/queue_age_watchdog.py --snapshot snapshot.json \
    --contribution-window-hours 6

Diagnosing a VM lane: idle looks exactly like dead

The macOS and Linux VM lanes are JIT — a runner registers with GitHub only while serving one job, then deregisters. A runner census therefore cannot tell a healthy idle lane from a dead one: "zero runners carry pulp-build-vm" is both states at once. Do not conclude a lane is dead from actions/runners, and do not build a label-satisfiability alarm on a JIT label — it would fire every idle night. Satisfiability is a valid check only for the persistent bare-metal Studios.

The signal that separates alive from dead on a JIT lane is queue age (not queue depth — 40 queued runs with a 5-minute median is healthy churn from many concurrent agents). Baseline measured on a healthy busy pool (2026-07-16): median queue age 5 min, oldest 31 min, 3 runs over 30 min. A naive ">30 min = broken" threshold alarms on that healthy pool; calibrate above it.

To check a lane host-side:

# Non-interactive ssh does NOT source .zprofile, so it lacks /opt/homebrew/bin
# and will falsely report "tart is not installed". Always use a login shell:
ssh <host> 'zsh -lc "launchctl list | grep -E \"tart-runner|qemu-runner\""'

Last-exit 0 means the supervisor is healthy and the lane is alive regardless of what the runner census says.

Runner agent crash-loops with exit 75 (the /usr/sbin PATH trap)

Symptom: a runner LaunchAgent shows last-exit 75 (EX_TEMPFAIL) and crash-loops under KeepAlive; its log says lease denied … rc=2; no VM ever boots; jobs queue on that lane forever.

Cause: tartci's host_profile.py shells bare sysctl — which lives at /usr/sbin/sysctl — to read hw.ncpu / hw.memsize. macOS launchd agents run with a minimal PATH, and the generated plist's PATH omits /usr/sbin:

/Users/<u>/.config/tartci/ghapp-shim:/opt/homebrew/bin:/usr/local/bin:/Users/<u>/.local/bin:/usr/bin:/bin

sysctl raises FileNotFoundError → host_profile.py exits 1 → the tartci lease governor cannot compute a memory budget → it denies every lease (failing closed, which is correct) → no VM ever starts.

Diagnose this FIRST — before suspecting tart, the network, or auth:

launchctl list | grep -E 'tart-runner|qemu-runner'   # last-exit 75 = this bug
# then inspect EnvironmentVariables:PATH in the agent's plist for /usr/sbin

Fix: append :/usr/sbin:/sbin to the plist's PATH and reload the agent. Exit goes 75 → 0, the log turns to lease acquired … cores=6 mem_mb=8192, a VM boots, and the queue drains. The failing set is exactly the /usr/sbin-missing set: agents that already carry it are exit 0.

TART_HOME is per-host by design

The repo does not own a host→path table. Every VM tool resolves the store through tools/ci/lib/tart-home.sh: explicit TART_HOME first, then vm_home from tartci host-profile --json, otherwise a loud error. That shared helper replaced the contradictory script-local defaults; do not revive a guessed $HOME/VMs or /Volumes/.../VMs fallback in prose or code. The repo holds the resolution rule; the TartCI host profile and its install receipt bind the value.

A default-store tart list is not active-work proof. If it reports no running VMs while tart run processes or guest setup are present, the store identity is unresolved and the result is unknown, not idle. Operational checks must resolve the receipt-bound profile first, query Tart with that exact TART_HOME, and corroborate with process/guest state before declaring an idle boundary. The Pulp topology checker only compares supplied profile/receipt/manifest fixtures; live store/process reconciliation belongs in TartCI.

These three traps share one shape, covered in the ci skill under "The unifying invariant — no name without a heartbeat": a name is trustworthy only if an automated process dereferences it on a schedule and alarms on failure.

Off-fleet merge-stall watchdog (merge-stall-check.yml)

.github/workflows/merge-stall-check.yml sweeps every 30 minutes and opens a tracking issue when PRs are merge-ready but not merging or the GitHub merge queue has stopped advancing. It runs on ubuntu-latest for the same reason as the queue-age watchdog: the wedge it catches lives in whatever presses the merge button (Shipyard's per-host queue-tick), so an on-fleet guard would die with the thing it watches.

The gap it closes — the opposite shape from the queue-age watchdog. The queue-age guard alarms on a dead runner lane: jobs sitting queued because runners died. This one alarms on the inverse: every required check green, nothing queued, and still nothing merging — the signature of an auto-merger silently held in reap-only mode. No job-level signal sees "everything is green and nobody is merging"; the only observable is a population of merge-ready PRs that stays merge-ready and unmerged. (Motivating incident: the repo went ~4 hours with 34 PRs open and nothing merging while every check was green.)

The alarm predicate. A PR trips only when ALL hold:

1. Required checks green — every check in the repo's REQUIRED set. That set is read from branch protection at runtime, not hardcoded; if the token cannot read protection rules it falls back to the complete documented main set: macos, Enforce version & skill sync, Build + prove + (owner-gated) deploy, Vellum trusted freeze, Vellum freeze, and drift-fast.
2. mergeStateStatus in {CLEAN, BEHIND} — GitHub's own merge verdict. DIRTY (conflicts), BLOCKED (a required check red/missing/review pending), and UNSTABLE (a non-required check still moving) are excluded — those wait on something real, not on the merger.
3. Auto-merge enabled — the signal that a machine, not a human, owns pressing merge. A green PR without it is waiting on a person and must not alarm.
4. Merge-ready longer than the threshold (default 45 min), measured from the completion time of the last required check to go green — a real duration, independent of the sweep cadence.

Why two consecutive sweeps. A single snapshot can misread — a per-PR REST poll of merge state gets rate-limited and returns false CLEAN/BEHIND readings under load, which is exactly how the incident state looked wrong. Collection therefore uses one GraphQL call for every open PR's mergeStateStatus (tools/scripts/merge_stall_watchdog.py), and on top of that a PR must satisfy the full predicate on two consecutive sweeps before it is issue-worthy: the first qualifying sweep records it as pending (run-summary only), the second promotes it to alarm. A normal in-flight PR that merges within a tick never reaches the second observation, so it never trips. The cross-sweep memory is the set of stuck PR numbers, persisted as a workflow artifact — crash-safe, held by GitHub independently of this repo or any host.

Merge-queue predicate. Once the queue is non-empty, the watchdog also reads its GraphQL MergeQueue.entries head and the latest merge_group Actions run. It alarms when the head has waited at least 30 minutes and no new merge-group batch has started in that window. The age window is already the anti-flap period, so this condition alarms on its first observed sweep. The report names the head PR, queue depth/state, last batch start, and any required check that is missing, queued, in progress, or red. This catches the incident where a required hosted alias waited behind advisory work while matching self-hosted build capacity was idle.

The issue is edited in place each sweep and closes automatically once no PR is stuck merge-ready — the same open/update/auto-close contract as the release watchdogs (see release-watchdog.md). A degraded API sweep never closes an existing tracker; only a complete snapshot can prove recovery.

Third condition — the outcome heartbeat. The two predicates above both infer health from a component: a PR's merge state, a queue head's age, a batch having been dispatched. Each can read healthy while the thing that matters has stopped. So a third condition asks the outcome directly: nothing has landed on main for throughput_threshold_minutes (default 90) while the merge queue is non-empty. The non-empty queue is the denominator — a quiet repository merging nothing is correct and never alarms, no matter how long it has been. Runner capacity is reported on the finding and shapes the diagnosis text, but is deliberately not a condition: requiring "capacity is online" would let a total fleet outage silence the outcome alarm, which is the substitution of a component for the outcome this condition exists to stop. Because it is cause-agnostic, it catches deadlocks nobody has enumerated — a routing typo, an exhausted pool, a label no runner advertises, a host that went down.

Fourth condition — the sweep's own blindness. A collection failure yields zero findings, which every consumer reads exactly like a clean bill of health. This guard once ran four hours into a total merge stall with both of its reads failed, printed "Merges are flowing", and finished green; twenty consecutive sweeps were degraded with the merge-queue read failing every time, so the queue alarm had been structurally unable to fire for days while reporting success every 30 minutes. A failed read is therefore now itself an alarm that names the stages that failed and the verdicts they silenced. Silence is evidence of health only when the instrument demonstrably ran — so when reading a quiet sweep, confirm its snapshot shows a non-zero open-PR count and queue depth rather than trusting the absence of findings.

Reads are kept inside the budget that completes: the open-PR query pages 25 at a time (50 asks GitHub for 50 check rollups at once and times out with HTTP 504 under normal load), and transient 502/503/504 responses get a bounded retry. Terminal failures are never retried.

Analysis and the predicate live in tools/scripts/merge_stall_watchdog.py, tested by tools/scripts/test_merge_stall_watchdog.py — which pins the must-stay-quiet cases (young PR, DIRTY, BLOCKED, no auto-merge, single-sweep blip) as regressions so a future edit that would make the guard cry wolf fails at PR time. Queue-specific tests pin empty/young/recent-batch cases quiet and an old head plus old batch as an immediate alarm. Throughput tests pin the empty queue and a recent merge quiet, and prove a dead fleet does not suppress the alarm; blindness tests replay the real degraded sweep and assert it can no longer render as calm.

# Dry-run a sweep by hand (log findings, do not touch the issue)
gh workflow run merge-stall-check.yml -f dry_run=true
gh workflow run merge-stall-check.yml -f threshold_minutes=60
gh workflow run merge-stall-check.yml -f queue_threshold_minutes=30
gh workflow run merge-stall-check.yml -f throughput_threshold_minutes=90

# Replay a recorded snapshot offline (no API calls, verdict pinned to capture time)
python3 tools/scripts/merge_stall_watchdog.py --snapshot snapshot.json --prev-state state.json

GraphQL quota fallback for PR sweeps

The gh pr ... --json and gh pr merge paths can consume or require GitHub's GraphQL quota. That quota is separate from the REST core quota and can hit zero while REST still has thousands of calls available.

When a broad PR sweep hits GraphQL exhaustion, switch the sweep to REST instead of waiting:

gh api rate_limit --jq '.resources | {core, graphql}'
gh api repos/OWNER/REPO/pulls/PR
gh api repos/OWNER/REPO/commits/SHA/check-runs?per_page=100
gh api repos/OWNER/REPO/actions/jobs/JOB_ID/logs

For a PR already verified green through REST, merge through the REST endpoint:

head_sha=$(gh api repos/OWNER/REPO/pulls/PR --jq '.head.sha')
gh api repos/OWNER/REPO/pulls/PR/merge \
  -X PUT \
  -f sha="$head_sha" \
  -f merge_method=squash \
  -f commit_title='subject (#PR)'

If the merge endpoint returns 405 Base branch was modified, refresh the PR state and check runs through REST, recompute head_sha, then retry once only if the refreshed head SHA and green status are still the values you intend to merge. This is a transport fallback, not a validation bypass: do not merge around real CI, coverage, sanitizer, or review failures.

Use shipyard rescue when a PR is otherwise ready but blocked by queued, cancelled, or failed runner contexts caused by a self-hosted-runner wedge. It is the PR-side recovery path and avoids the old failure mode where cancelling queued runs left required checks stuck as failure. For a required check, mind what it starts (until Shipyard's rescue re-runs pull_request runs, it dispatches a branch-tip workflow_dispatch replacement, which is the wrong proof for a required check under strict protection; for a required pull_request run, ghapp run cancel <run> if it is still running, then ghapp run rerun <run> --failed).

Use shipyard runner watch --kill-hung-workers on the runner host itself. It auto-cancels stale queued runs and kills hung Runner.Worker processes through Shipyard's safe recovery sequence: snapshot, SIGTERM, grace period, SIGKILL, child reaping, partial-build quarantine, Listener verification, and optional wait for GitHub status to flip. Its JSON output uses runner.watch envelopes with event=auto_kill_worker and phase values of attempt, killed, failed, or no-pid-found.

Use shipyard update instead of the old ad hoc curl install.sh | sh path once a machine already has Shipyard installed. Pulp still records the canonical repo pin in tools/shipyard.toml; shipyard update --check --json is the machine-local drift check, while shipyard pin bump --to vX.Y.Z is the repo pin-change workflow.

Required Merge Process (All Agents)

Every change to main must go through this workflow — no exceptions:

1. Branch — work on feature/* or fix/*, never directly on main
2. Ship — run shipyard pr to create and track the PR, validate on macOS + Ubuntu + Windows, and merge on green
3. GitHub Actions — PR also triggers build+test CI on all 3 platforms (redundant safety net)

The ci skill (.agents/skills/ci/SKILL.md) captures this as the authoritative trigger list — natural-language phrases like "ship this", "push a PR", "we're done", and "run CI" all route through shipyard pr.

Legacy: pulp ci-local

tools/local-ci/local_ci.py is the previous CI controller. It remains available as a fallback while the Shipyard path finishes replacing it, but it is scheduled for removal.

TL;DR

• pulp ci-local run queues the current HEAD in a machine-global queue shared by every worktree on that Mac.
• pulp ci-local run <branch> queues that branch tip's exact commit SHA, not the launching checkout's HEAD.
• pulp ci-local run --smoke queues a fast clean install/export preflight instead of a full test run.
• The queue serializes jobs, not targets. One CI job runs at a time, but its requested targets (mac, ubuntu, windows) run in parallel inside that job.
• Mac runs locally. Ubuntu and Windows run over SSH against repos you already cloned on those machines.
• Remote targets validate the exact queued git SHA, not "whatever the branch points to later". The runner uploads that SHA as a git bundle before validation, so full-matrix checks do not depend on the host already seeing your latest branch tip.
• pulp ci-local status shows the active runner, pending jobs, SSH/VM reachability, and live per-target state for the running job. pulp ci-local bump <job-id> high moves a pending job forward.
• queueing now prints the submission root, current cwd, config path/source, and per-target host preflight before a job is recorded
• queueing fails fast if you launched from the wrong git root or selected an SSH target that is currently unreachable with no fallback, unless you explicitly override that safety check
• While a job is running, pulp ci-local status also shows live per-target state such as mac=pass, ubuntu=pass, windows=running.
• Quiet long-running targets now emit runner heartbeats, so status can show heartbeat=..., idle=..., and liveness=quiet|stuck even when the underlying toolchain has not printed a new line recently.
• If you queue a newer SHA for the same branch, targets, and validation mode, older pending work is superseded automatically instead of sitting behind it forever.
• pulp ci-local logs <job-id> --target windows tails the saved per-target log from the machine-global CI state dir, so you do not need ad hoc SSH just to see whether a target is building or testing.
• pulp ci-local evidence [branch] shows the last-good exact-SHA target evidence already recorded for a branch, so you can keep earlier same-SHA passes instead of rerunning them blindly.
• pulp ci-local cleanup shows reclaimable local-CI disk usage without deleting anything; --apply is blocked while jobs are running.
• pulp ci-local cloud workflows lists the GitHub Actions workflows that the local CI control plane knows how to dispatch, plus which runner providers each one supports.
• pulp ci-local cloud run <workflow> [branch] dispatches a GitHub Actions workflow deliberately when workflow semantics or neutral-host confirmation matter more than the local queue.
• pulp ci-local cloud status shows the latest tracked GitHub Actions dispatches that this machine has launched; pulp ci-local status includes the same recent cloud summary alongside local queue state.
• Persistent local CI hosts now keep a prepared root per target + validation so a narrow same-SHA rerun can reuse earlier work instead of rematerializing from scratch.
• If a runner is interrupted, the queued job keeps its last-known per-target state so you can see what already passed before deciding whether to rerun everything or just the remaining target.
• Jobs submitted through pulp ci-local are globally queued, and validation itself now takes a per-host lock on macOS/Linux plus a Windows host mutex, so old validate-build.sh runs wait instead of colliding.
• SSH targets receive a per-job git bundle before validation. That keeps exact-SHA validation working even when the host validates from a stale local mirror instead of GitHub directly.
• Windows SSH jobs execute from short detached worktrees under C:\pulp-ci, and stale worktree metadata is pruned automatically before reruns.
• If a stale runner leaves behind an old Windows validator, the next drain pass now targets that specific remote validator PID for cleanup before starting new work, and status keeps the cleanup result visible.
• For Windows SSH validation, choose the configured target whose non-interactive PowerShell context resolves git, cmake, and ctest. Keep those host aliases local to your environment; shared repo docs should describe the selection rule, not your personal machine names.
• Reuse is a persistent-host feature for local macOS and SSH-backed/self-hosted hosts. Ephemeral cloud runners should keep the default clean path unless a later policy explicitly opts them in.
• Truly raw ad hoc ssh, cmake, or custom background processes still bypass coordination until they are stopped or migrated.

Why local instead of cloud

Pulp has GitHub Actions workflows for CI, but running them on every branch costs money. Local CI is free and faster for iterative development — you get results in minutes from machines you already own or have running locally. Cloud CI remains available for release branches, public PRs, and the narrow cases where you need workflow-level or neutral-host confirmation.

Cloud orchestration is now available through the same control plane:

• pulp ci-local cloud workflows
• pulp ci-local cloud run <workflow> [branch]
• pulp ci-local cloud status [dispatch-id|latest]

That cloud surface is intentionally separate from the local queue. run, check, ship, enqueue, and drain still operate on the exact-SHA local/SSH queue. cloud run dispatches GitHub Actions explicitly and tracks the result beside local CI state instead of pretending a hosted workflow is just another local target.

Namespace is now wired into the deliberate cloud companion path for both docs-check.yml and build.yml. The normal day-to-day default remains local-first: macOS runs locally, while deliberate cloud dispatches can route Linux/Windows through Namespace and keep macOS local unless you opt into a one-off cloud macOS selector.

How it works

When you run pulp ci-local, it:

1. Queues a job in a machine-global queue shared by every worktree on that Mac
2. Prints the exact queue intent first: submission root, cwd, config path/source, and remote-host preflight
3. Runs only one queue drain owner at a time, so separate agents do not stampede the same Mac and VMs
4. Validates locally on Mac via ./validate-build.sh --ref <sha>
5. For each SSH target in config.json: uploads a per-job git bundle, injects that exact SHA into the configured repo on the host, then validates it there
6. If an SSH target is unreachable, it tries to start the corresponding UTM VM, waits for it to boot, then retries the SSH connection
7. Drains queued work on login or wake if you install the launchd agent

Mac validation always runs. SSH targets are skipped if disabled in config.

GitHub Actions companion

Use the cloud subcommands when you want GitHub Actions as the orchestrator, not when you want another exact-SHA local queue job:

pulp ci-local cloud workflows
pulp ci-local cloud defaults
pulp ci-local cloud history
pulp ci-local cloud compare build
pulp ci-local cloud recommend build
pulp ci-local cloud run build feature/my-branch
pulp ci-local cloud run build feature/my-branch --provider namespace
pulp ci-local cloud run build feature/my-branch --provider namespace --macos-runner-selector-json '"namespace-profile-big-apple"'
pulp ci-local cloud run build feature/my-branch --provider namespace --macos-runner-selector-json '"nscloud-macos-tahoe-arm64-6x14"'
pulp ci-local cloud run docs-check feature/my-branch --provider namespace --wait
pulp ci-local cloud run docs-check feature/my-branch --provider namespace --runner-selector-json '"namespace-profile-big-apple"'
pulp ci-local cloud namespace doctor
pulp ci-local cloud namespace setup
pulp ci-local cloud status
pulp ci-local cloud status latest --refresh

Important constraints in the current phase:

• cloud run dispatches by branch name, not by a detached exact SHA
• cloud dispatch records are persisted under the same machine-global CI state directory as local results, but they do not enter queue.json
• local status remains fast and local-first; it shows the latest tracked cloud summaries without hitting GitHub unless you explicitly run cloud status --refresh
• cloud defaults shows the effective workflow/provider defaults plus where the current selector values came from (local config versus repo-variable fallback)
• cloud history shows recent tracked cloud runs with saved timing plus any configured estimated cost line items
• cloud compare <workflow> rolls up observed provider medians for a workflow from tracked run history
• cloud recommend <workflow> suggests a provider from recorded cloud history; it is intentionally conservative and uses observed medians instead of hardcoded guesses
• cloud status now reports Namespace runtime/machine-shape truth when the run was launched on Namespace and nsc can see the matching instances
• tracked cloud runs now persist queue-delay and elapsed-duration timing so the later comparison view can answer "how long did GitHub-hosted vs Namespace take?" from saved run history instead of rough notes
• estimated cost output is opt-in via local config; every estimate is labeled estimated; verify provider pricing
• if the provider CLI does not expose billing totals, Pulp keeps reporting runtime and machine shape instead of inventing invoice truth
• if a Namespace dispatch dies in resolve-provider before any matrix leg starts, inspect the GitHub run annotations first; provider billing or control-plane failures are a different problem from repo or workflow breakage
• build.yml now accepts runner_provider and routes Linux and Windows through the selected provider; macOS is omitted from the cloud build by default so it can stay local-first
• Default provider for PR checks is controlled by the GitHub repo variable PULP_DEFAULT_RUNNER_PROVIDER. Set it to namespace to route all PR checks through Namespace runners (faster, parallel). Set to github-hosted to use GitHub-hosted runners (free tier, queued). The workflow_dispatch input overrides this for manual runs. To change the default:

  # Switch to Namespace (recommended for faster CI)
  gh variable set PULP_DEFAULT_RUNNER_PROVIDER --body "namespace"
  
  # Switch back to GitHub-hosted
  gh variable set PULP_DEFAULT_RUNNER_PROVIDER --body "github-hosted"
  

• build also accepts one-off leg overrides: --linux-runner-selector-json, --windows-runner-selector-json, and --macos-runner-selector-json; that means you can keep the normal Linux/Windows Namespace + macOS local default and still do an explicit one-off macOS Namespace build without changing saved config
• those one-off selector overrides can be either: a Namespace profile label such as "namespace-profile-generouscorp-macos", or a direct Namespace machine label such as "nscloud-macos-tahoe-arm64-6x14"
• docs-check accepts an explicit --runner-selector-json override, for example "namespace-profile-default" or ["self-hosted","linux"]
• if no explicit selector is passed, docs-check falls back to github_actions.workflows.docs-check.providers.<provider>.runner_selector_json in local config when present, then to the repo variable PULP_NAMESPACE_DOCS_CHECK_RUNS_ON_JSON for the Namespace provider
• build can take Linux/Windows Namespace selectors from github_actions.workflows.build.providers.namespace.linux_runner_selector_json and .windows_runner_selector_json in local config, and the workflow also supports repo-variable fallbacks PULP_NAMESPACE_BUILD_LINUX_RUNS_ON_JSON and PULP_NAMESPACE_BUILD_WINDOWS_RUNS_ON_JSON
• macOS Namespace is an explicit validation path, not part of the default cloud build: if you want to test macOS on Namespace, provide --macos-runner-selector-json, or set github_actions.workflows.build.providers.namespace.macos_runner_selector_json in local config, or PULP_NAMESPACE_BUILD_MACOS_RUNS_ON_JSON
• make sure that selector points at a real macOS-capable Namespace profile: GitHub job names alone do not guarantee the underlying OS, and a Linux-backed profile can still satisfy the runs-on label while executing the leg on Linux instead of macOS
• if you want macOS to stay local-first by default, leave the macOS selector unset in shared config and repo variables, and pass --macos-runner-selector-json only for one-off validation runs
• for the Namespace path, install the nsc CLI and run nsc login first before trying to route work there; that is the recommended operator setup path for this pilot
• SSH/VM target topology and Namespace provider setup stay separate: targets.* still configures local/SSH validation hosts, while Namespace provider routing lives under the GitHub Actions workflow/provider config and the cloud namespace helper commands

Fast-CI vs full-CI (build.yml)

The Build and Test workflow has two test trajectories without forking the YAML:

• Fast-CI runs on pull_request events. The ctest invocation excludes BOTH the validation and slow CTest labels, dropping the longest-running tests so PR cycle time stays tight. Examples that carry LABELS slow today (defined in test/CMakeLists.txt):
• cmake-ios-auv3-configure — fresh-cache ~3 min iOS-leg configure
• cmake-pulp-add-binary-data-encoder / cmake-pulp-install-layout
• pulp-test-hot-reload, pulp-test-scripted-ui, pulp-test-scan-cache, pulp-test-scan-blacklist (filesystem-mtime sleep loops)
• pulp-test-sync, pulp-test-sync-race-hammer, pulp-test-events-timer-helpers (race + timer hammers; also covered under sanitizer.yml's TSan lane)

• agent-capability-installed-sdk (roughly 12 minutes to install the SDK and compile/run every exported capability and typed binding; label slow-affected). The classifier restores this exact test into the main ctest run (started first by its COST, ctest_gate_args.py --affected-slow) on the parallel macOS and Linux matrix legs for capability manifest, schema, history, registry/generator, vocabulary, CMake target/export, install-rule, and compile-test changes. A selected documentation-only surface still allocates the native job; unknown diffs run it fail-closed, so unrelated PRs get the speedup without weakening affected changes.

• Full-CI runs on push to main, the nightly schedule, and workflow_dispatch. Only the validation label is excluded — every slow-labelled test runs before code lands on the release lane.

Both paths satisfy the branch-protection-required direct macos context and the advisory linux / windows aliases. The macOS matrix child reports its own result; only the advisory aliases read matrix outcomes.

iOS library compile gate

The macOS matrix leg also configures Pulp with the Xcode generator and builds pulp-timebase, pulp-timeline, pulp-playback, pulp-sequence, and both SMF libraries (pulp-smf-interop and pulp-smf-interchange) as arm64 static libraries for both iphonesimulator and iphoneos. The gate uses the repository's iOS 16.3 libc++ floor, matching the AUv3 and host-app helpers. This is compile coverage only: it does not run iOS tests, build an app or AUv3, or pull core/host into the mobile graph. GPU rendering, examples, and tests stay disabled so the gate exercises the dependency boundary needed by the sequencer libraries.

Run the same gate locally on a Mac with both SDKs installed:

bash test/cmake/test_ios_compile_gate.sh "$PWD" "$PWD/build-ios-compile-gate"

The script uses Pulp's platform-wide FetchContent source cache, so fresh worktrees reuse dependency checkouts while keeping simulator and device build products separate. The two SDK legs configure with Ninja when it is on PATH (the Xcode generator otherwise): a Ninja configure takes about a minute where Xcode's takes seven to eight, and Ninja honours the compiler launcher, so the gate VM's persistent ccache serves a repeat build from cache. The GPU leg stays on the Xcode generator: its AUv3 .appex and host-app embedding use Xcode product types, and a Ninja configure of that tree fails in CMake's Swift compiler check for the iOS target. Because its Xcode configure is minutes of mostly serial try-compile work, the gate starts the GPU leg in the background and builds the SDK legs alongside it; it waits for the GPU leg and fails on its status before the Simulator phase, and stops it if an SDK leg fails first. tools/scripts/test_ios_compile_gate_legs.py (ctest ios-compile-gate-legs) pins that orchestration against stub toolchain executables. The existing test_ios_source_syntax.sh sweep runs after the real builds as the cheap, locally callable fallback for iOS-specific translation units.

Tagging a new test as slow

Add LABELS slow either to a single test's set_tests_properties, or to a Catch2 binary's catch_discover_tests(... PROPERTIES LABELS slow) so every discovered test inherits the label:

add_test(NAME my-expensive-cmake-smoke COMMAND ...)
set_tests_properties(my-expensive-cmake-smoke PROPERTIES
    LABELS "smoke;slow"
    TIMEOUT 600)

add_executable(pulp-test-my-suite test_my_suite.cpp)
target_link_libraries(pulp-test-my-suite PRIVATE pulp::view Catch2::Catch2WithMain)
catch_discover_tests(pulp-test-my-suite PROPERTIES LABELS slow)

Multi-label lists are preserved as lists by Pulp's Catch discovery wrapper:

catch_discover_tests(pulp-test-my-suite
    LABELS "audio;slow;quality-lab")

All three labels reach CTest, so ctest -L and ctest -LE selection does not depend on label order.

Rule of thumb for slow: a test consistently >5 sec on at least one platform, OR a sleep-bounded smoke (file-mtime, hammer race, message-loop bound) whose value lies in soak coverage rather than per-PR feedback. Anything covered by sanitizers.yml or another scheduled lane is a strong candidate.

Demoting a fast test to slow (or vice versa)

ctest --test-dir build -L slow -N lists every test currently tagged slow. To move a test in or out of the fast-CI surface, add or remove the slow label in test/CMakeLists.txt (or the appropriate subdir CMakeLists) and reconfigure. There's no separate registry to keep in sync.

Seeing which tests did not run

A CTest skip (SKIP_RETURN_CODE) is green, so on the required macos check a test that has never run once looks exactly like a test that runs and passes every time. build.yml's non-Windows test step therefore passes --output-junit, and an always() observation step summarizes notrun and disabled tests — name, skip reason, labels, and the skipping command's output — into the job summary, with ctest.junit.xml kept in the ctest-logs-<key> artifact even on green runs. It observes and never asserts: skipping is frequently the correct outcome (no GPU, no device, no vendor SDK), and the summary also prints the registered ctest -N population beside the report's declared tests= count so a gap created by label exclusions or --exclude-regex stays visible.

The same observer works locally on an explicit downloaded or local artifact. Use a baseline when the question is “what changed?” rather than merely “what did not run?”:

python3 tools/scripts/ctest_nonruns.py /absolute/path/ctest.junit.xml --json
python3 tools/scripts/ctest_nonruns.py /absolute/path/ctest.junit.xml --registered 20000
python3 tools/scripts/ctest_nonruns.py /tmp/current/ctest.junit.xml \
  --baseline /tmp/known-good/ctest.junit.xml --json

On a leg whose full suite was skipped (a pull-request head that ran only the fast tier, or a build that stopped before ctest) there is no ctest.junit.xml by design. The CI step then passes --fallback with the fast tier's ctest-pr-fast.junit.xml and --not-run-reason: it observes the fast tier's report when one exists and otherwise records observation: not_run with the reason and exits 0. A suite that ran and left no report still exits 2.

CI also writes ctest.nonruns.json beside ctest.junit.xml in each non-Windows ctest-logs-<key> artifact. Download two artifacts when investigating a change; the JSON is the ready-to-read single-run projection, while --baseline over the two retained XML files computes transitions. Unique CTest names are matched by the SHA-256 of the full name. Same-name duplicates are compared as status-count groups; the tool reports an ambiguous group instead of guessing per-case transitions. current_only and baseline_only mean only “present in one supplied artifact”: selection, configuration, and source changes can all cause that shape. They are leads, not proof that a test was added or removed.

--registered is optional caller-supplied context, not an inferred selection. The helper reads only that regular file (64 MiB maximum), requires CTest's testsuite dialect, and records its SHA-256 without claiming current-head provenance. Counts cover all entries; at most 100 non-run rows, 100 comparison rows total, and 100 issues are displayed, with omitted counts. Names/reasons/labels are bounded to 512 characters and the last output line to 160, so keep the original XML for full detail. When comparison rows exceed the shared budget, newly failing tests, new non-runs, and failing or non-running current-only groups are retained before recoveries; digest order breaks ties. pulp.ctest-nonruns.v2 JSON and the workflow's Markdown summary share one interpretation. --json-output <path> writes the same object to a regular, non-symlink file while retaining Markdown on stdout, which is how CI publishes the job summary and agent-readable artifact from one observation. Reports include bounded test output; treat them as potentially sensitive artifacts, not as instructions or safe-to-publish logs.

Exit 0 means the observation was readable, including failed or correctly skipped tests. Exit 2 means missing, malformed, empty, or inconsistent evidence—not a code failure. The workflow retains continue-on-error: true; the original CTest invocation still owns the test verdict. A missing or empty report never claims all tests ran. Filtered/configure-time absent tests remain outside this observer's view; Shipyard and the canonical CTest inventory retain selection and exact-head validation ownership. No new tool installation is required.

Nothing in CI provisions the pinned trace_processor_shell, so pulp-rust-gpu-trace-analysis-integration skips on every run and the GPU trace-analysis acceptance tests do not execute. That gap is now reported rather than hidden: the suite appears in the non-run table with SKIP_RETURN_CODE=77. Provisioning it is a separate decision, because one measured run showed the suite has a failure waiting behind the skip.

Switching a job's runner without a code change

Provider-switchable build, release, coverage, and sanitizer decisions use tools/scripts/resolve_runs_on.py. Their runner can be flipped between GitHub-hosted, Namespace, and local self-hosted by setting a repository variable — no workflow edit or PR.

General selector precedence

For each target handled by resolve_runs_on.py, the resolver checks:

1. A workflow_dispatch input (if present on the workflow) — one-off override.
2. The target's repository variable (the PULP_*_RUNS_ON_JSON values below).
3. For the build matrix only: PULP_DEFAULT_RUNNER_PROVIDER + the provider's selector var (PULP_NAMESPACE_* or PULP_LOCAL_*).
4. A hard-coded default label (e.g. macos-14, ubuntu-24.04, macos-15).

When a variable below is unset, the workflow resolves to that target's reviewed hard-coded default. Defaults may be updated when a hosted toolchain changes; for example, UBSan uses macos-26 to avoid the invalid-vptr diagnostics produced by the Xcode 16.4 image. Setting one variable moves one job. Nothing more.

Coverage is stricter than the build matrix. It reads explicit workflow_dispatch inputs and PULP_COVERAGE_*_RUNS_ON_JSON, not PULP_NAMESPACE_BUILD_*. If coverage moves local, use a dedicated ephemeral label such as pulp-coverage-vm-macos; do not point coverage at pulp-build, pulp-build-vm, or the warm macOS gate pool.

Global default (build.yml matrix only)

Variable	Effect	Example
PULP_DEFAULT_RUNNER_PROVIDER	Default provider for Linux and Windows legs of build.yml. One of github-hosted | namespace | local. Falls back to github-hosted when unset.	gh variable set PULP_DEFAULT_RUNNER_PROVIDER --body "namespace"

build.yml — Linux / Windows / macOS legs

Variable	Provider	Example
PULP_NAMESPACE_BUILD_LINUX_RUNS_ON_JSON	Namespace	gh variable set PULP_NAMESPACE_BUILD_LINUX_RUNS_ON_JSON --body '["namespace-profile-generouscorp"]'
PULP_NAMESPACE_BUILD_WINDOWS_RUNS_ON_JSON	Namespace	gh variable set PULP_NAMESPACE_BUILD_WINDOWS_RUNS_ON_JSON --body '["namespace-profile-generouscorp-windows"]'
PULP_NAMESPACE_BUILD_MACOS_RUNS_ON_JSON	Namespace (optional)	gh variable set PULP_NAMESPACE_BUILD_MACOS_RUNS_ON_JSON --body '"namespace-profile-generouscorp-macos"'
PULP_LOCAL_MACOS_RUNS_ON_JSON	Fast local macOS ARM64 JIT VM pool; see the live table under "macOS overflow routing"	gh variable set PULP_LOCAL_MACOS_RUNS_ON_JSON --body '["self-hosted","macOS","ARM64","pulp-build","pulp-build-vm","pulp-gate-fast"]'
PULP_OVERFLOW_BUILD_MACOS_RUNS_ON_JSON	Overflow is disabled live with local-only. Unset → build.yml falls back to GitHub-hosted ["macos-15"]; another reviewed selector re-enables overflow.	gh variable set PULP_OVERFLOW_BUILD_MACOS_RUNS_ON_JSON --body 'local-only'
PULP_LOCAL_LINUX_RUNS_ON_JSON	Dispatch and release-fallback Linux x86_64 Proxmox VM pool	gh variable set PULP_LOCAL_LINUX_RUNS_ON_JSON --body '["self-hosted","Linux","X64","pulp-build-linux-x64","pulp-host-macpro"]'
PULP_AUTO_LINUX_RUNS_ON_JSON	Reserved selector for a separately reviewed protected-event routing change; installing the provider roles does not enable it	Do not set during provider-only activation
PULP_LOCAL_WINDOWS_RUNS_ON_JSON	Local Windows ARM64 QEMU pool	gh variable set PULP_LOCAL_WINDOWS_RUNS_ON_JSON --body '["self-hosted","Windows","ARM64","pulp-build-windows","pulp-host-macstudio"]'

Protected Vellum trusted gate

PULP_VELLUM_TRUSTED_RUNS_ON_JSON optionally routes both jobs in vellum-trusted-gate.yml to the restricted ephemeral Mac Pro Linux lane:

gh variable set PULP_VELLUM_TRUSTED_RUNS_ON_JSON \
  --body '["self-hosted","Linux","X64","pulp-build-linux-x64","pulp-host-macpro"]'

When the variable is unset, the workflow deliberately falls back to ubuntu-latest. This keeps the required trusted gate routable when local Linux capacity is unavailable. Do not point it at an unrestricted or persistent runner: the workflow handles pull_request_target and mints the narrowly scoped Vellum reader credential only after checking out and binding literal protected main controls.

drift-fast and the wclap job

PULP_DRIFT_FAST_RUNS_ON_JSON (the required drift-fast job in drift-fast.yml) and PULP_WCLAP_RUNS_ON_JSON (the required "Build + prove + (owner-gated) deploy" job in wclap-cloudflare.yml) route those jobs away from GitHub-hosted runners. Unset, each falls back to ubuntu-latest, so nothing changes until one is set. They exist because a required check whose hosted job is never assigned a runner is cancelled after about fifteen minutes, and the merge queue ejects the group: on 2026-10-05, during a hosted-runner assignment incident, a group was ejected through drift-fast this way. Both are JSON and parsed, like every *_RUNS_ON_JSON selector. Before setting one to a self-hosted pool, check that pool's runner group admits the workflow: pulp-trusted-build lists its workflows (tools/ci/verify_linux_runner_group.py) and does not include drift-fast.yml, and it refuses wclap-cloudflare.yml outright because that workflow holds the Cloudflare deploy secret. A job routed to a group that does not admit its workflow is never assigned and waits forever.

The Linux and Windows label sets include a pulp-host-* label that pins the lane to one machine, so the supervisor serving them must carry it too — GitHub selects a runner only when it carries every requested label. Declare the machine once, in the LaunchAgent, via --host-tag / PULP_RUNNER_HOST_TAG (tools/launchd/pulp-{tart-runner-linux,qemu-runner-windows}.plist.template). A supervisor that cannot resolve one refuses to register rather than contribute a runner that is online, idle, and selectable by nothing. Declared tags live in tools/scripts/runner_topology.json.

For pull requests, this privileged workflow evaluates a locally constructed merge of the checked-out protected-main commit and the exact API-resolved PR head. It verifies the fetched refs/pull/N/head against that SHA and uses only trusted-base code to build and validate the two-parent candidate. Conflicts, missing commits, and provenance mismatches fail closed. The workflow does not use refs/pull/N/merge, because GitHub may leave that synthetic ref based on an older main commit while a PR is BEHIND.

Advisory macOS selectors

Variable	Precedence and behavior	Example
PULP_ADVISORY_MACOS_RUNS_ON_JSON	Repository variable, then hosted macos-15. No dispatch/provider override.	gh variable set PULP_ADVISORY_MACOS_RUNS_ON_JSON --body '["self-hosted","macOS","ARM64","pulp-advisory-macos"]'
PULP_ADVISORY_GPU_MACOS_RUNS_ON_JSON	Repository variable only; unset skips the proof. No dispatch/provider override.	gh variable set PULP_ADVISORY_GPU_MACOS_RUNS_ON_JSON --body '["self-hosted","macOS","ARM64","pulp-advisory-gpu"]'

These selectors are resolved by tools/scripts/resolve_advisory_macos_runner.py. The resolver fails closed if an operator points an advisory workflow at pulp-build* or pulp-preamble*, or if any configured self-hosted selector lacks an explicit pulp-advisory-* identity. Hosted strings are an explicit reviewed allowlist (macos-14, macos-15, macos-26, and macos-latest), so a typo fails during resolution instead of waiting forever for a nonexistent runner. Until a separately governed advisory supervisor is installed, leave the ordinary advisory selector unset (hosted macOS) and the GPU advisory selector unset (proof skipped). This repository does not use Orchard for placement; Shipyard, tartci, and GitHub runner labels are the complete control path.

.shipyard/ci-profiles/normal-local-fast.toml is the repo-local, read-only policy mirror. Its PR-only github.windows-x64-runtime target records the stable windows-2022 functional lane, while the shared github.windows-x64 target records the windows-latest coverage/scheduled lane. The current Shipyard profile planner does not apply these selectors to a dispatch; build.yml remains authoritative. Inspect the profile when reviewing policy, then verify the workflow input and repository-variable precedence before changing live routing.

Do not put ordered fallback chains directly into GitHub Actions. GitHub receives one runs-on selector per job; Shipyard/tartci must resolve "Mac Studio, then M5/blackbook, then GitHub" before dispatch or variable application.

Windows local QEMU is Windows ARM64. An x64 MSVC/Prism smoke can be useful, but it is not a replacement for the GitHub-hosted Intel/x64 functional gate. The required build.yml functional matrix is pinned to windows-2022 so its CRT and Visual Studio generation do not move underneath the complete runtime suite. The standalone MSVC release-path, MIDI 2, and BLE compile gates remain on windows-latest; release builds and the nightly Intel safety net also keep tracking the newest hosted image. The MSVC release-path configure intentionally enables PULP_ENABLE_INSPECTOR while keeping runtime inspector endpoints off. Starting at the release product matrix's inspector_sdk_floor, published SDKs promise the split inspector archive family, so the Windows compile gate must match the tagged release configuration or it can miss an expensive packaging failure. tools/scripts/test_windows_runner_policy.py enforces this split across the actual build, release, coverage, and nightly workflows plus the release runner resolver and Shipyard mirror. It runs in workflow-lint, including when the profile or the policy test itself changes, so these surfaces cannot drift while an isolated mirror test remains green.

Nightly GitHub Intel validation

.github/workflows/cross-platform-check.yml is the scheduled Linux/Windows Intel safety net for this profile. It runs GitHub-hosted ubuntu-latest and windows-latest, files or updates one deduped issue per broken platform, and auto-closes the tracker when the platform recovers. Do not add a duplicate nightly Intel workflow unless this one is deliberately retired.

debug-o0.yml — the clean -O0 full-suite lane

Decisions contract row #9 requires one unoptimised full-suite lane: an ODR violation between two TUs that compile different bodies for one header symbol is invisible at -O3 (each TU inlines its own copy) and red at -O0 (the linker keeps one body for both), and NDEBUG-gated assertions only fire unoptimised. .github/workflows/debug-o0.yml is that lane. It configures Debug with tests ON and examples/GPU OFF, runs the same CTest selection as the Shipyard macOS lane, and runs clean on a GitHub-hosted macos-15 runner (row #8 keeps ODR-prone lanes out of warm self-hosted build directories).

• Nightly on main. A scheduled failure opens, or comments on, one tracking issue titled "Debug -O0 nightly suite is failing on main".
• On demand against a PR head: ghapp workflow run debug-o0.yml --ref <branch>.
• PULP_DEBUG_O0_MACOS_RUNS_ON_JSON overrides the runner. Leave it unset; never point it at the self-hosted build-gate pool.

The Shipyard macOS host lane still builds Debug. Moving it to Release, so it shares the worktree's build/ configuration with agent builds, waits until this lane is green on main and the changed-surface inventory contract is re-pinned for a Release lane.

sanitizers.yml — per-sanitizer target selection

The automatic matrix runs on every relevant pull request and once nightly. It deliberately does not rerun after every push to main: that duplicated four hosted jobs after each merge and competed with the next merge group's required checks. The nightly schedule is the independent post-merge backstop; workflow_dispatch remains available for an immediate operator run.

Each sanitizer job resolves independently. Setting one variable moves exactly that sanitizer; the others stay on their defaults.

ASan, TSan, and UBSan configure through PULP_SANITIZER=<kind>, including ASan's example-bundle lifecycle build. Besides applying the compiler and linker flags, the named option marks sanitizer bundles as test-only instrumentation so relocatability validation permits the compiler-injected Xcode runtime. Installed-SDK consumer fixtures carry the matching instrumentation flags because instrumented static libraries retain runtime references. The strict shipping verifier remains unchanged and still rejects external compiler runtimes for ordinary release artifacts.

Variable	Default label when unset	Example (dedicated sanitizer VM label)
PULP_SANITIZER_ASAN_RUNS_ON_JSON	macos-14	gh variable set PULP_SANITIZER_ASAN_RUNS_ON_JSON --body '["self-hosted","macOS","ARM64","pulp-sanitizer-vm-macos"]'
PULP_SANITIZER_TSAN_RUNS_ON_JSON	macos-14	gh variable set PULP_SANITIZER_TSAN_RUNS_ON_JSON --body '["self-hosted","macOS","ARM64","pulp-sanitizer-vm-macos"]'
PULP_SANITIZER_UBSAN_RUNS_ON_JSON	macos-26	gh variable set PULP_SANITIZER_UBSAN_RUNS_ON_JSON --body '["self-hosted","macOS","ARM64","pulp-sanitizer-vm-macos"]'
PULP_SANITIZER_RTSAN_RUNS_ON_JSON	ubuntu-24.04	gh variable set PULP_SANITIZER_RTSAN_RUNS_ON_JSON --body '["self-hosted","linux","x64","sanitizer"]'

UBSan uses a RelWithDebInfo build with non-recovering undefined-behaviour instrumentation. This keeps symbols and the complete test matrix while running production-scale DSP certification renders with optimized code; a Debug/O0 build can turn seconds of offline spectral and stability analysis into per-test hang-guard timeouts without reporting undefined behaviour.

The three macOS sanitizers (ASan/TSan/UBSan) carry a --deny-labels pulp-build,pulp-build-vm guard in sanitizers.yml's resolver, so a sanitizer can never be misrouted onto the required-gate pool (the resolver hard-fails). They no longer read PULP_NAMESPACE_BUILD_MACOS_* either; the per-sanitizer variable is the single switch.

Capacity finding: localize at most one sanitizer, and only TSan. macOS allows only two running macOS guests per host (Apple's limit), and both belong to the required macos build gate. A local sanitizer VM is a third guest, so localizing is gated on the tartci idle-gate: the pulp-sanitizer-vm-macos lane shares TART_HOME with the gate (a real host-wide 2-guest semaphore) and yields its slot whenever the gate has queued/in-progress required work. A host profile such as M3's can serve both event-class-v2 Pulp classes, so an advisory lane sharing that host must retain the template's yield keys; the exact selector is self-hosted,macOS,ARM64,pulp-build,pulp-build-vm,pulp-build-merge-group,pulp-build-pr-head. It can never start while required Build and Test work is demanding the host, which protects the strict merge queue from the coverage-lane failure mode. Pick TSan: it is the longest sanitizer (scoped -j1 serial, ~45 min on the 3 vCPU macos-14) and the highest value for a real-time audio framework, and being single-core-bound it gains most from a local M-series runner. ASan stays on macos-15, UBSan stays on macos-26, and RTSan stays on Linux: the four run in parallel on GitHub but would serialize (~4×) on one cap=1 local lane, which is slower than hosted except during a hosted backlog. Full parallel local sanitizers would need a third macOS host. Roll out one sanitizer at a time, each behind a measured go/no-go (gate queue latency + matrix wall-clock).

One-off overrides via workflow_dispatch

sanitizers.yml accepts one *_runner_selector_json input per sanitizer. They win over the corresponding repo variable for a single manual run:

gh workflow run sanitizers.yml \
  -f tsan_runner_selector_json='["self-hosted","macOS","ARM64","pulp-sanitizer-vm-macos"]'

Prove TSan green via this dispatch (with the pulp-sanitizer-vm-macos LaunchAgent loaded and the tartci idle-gate present) before setting PULP_SANITIZER_TSAN_RUNS_ON_JSON. The lane template is tools/launchd/pulp-tart-runner-sanitizer-macos.plist.template (ships parked — see its header for the load/quiet-window preconditions).

A loaded LaunchAgent is not the proof. The supervisor can be running and still be unable to serve, in which case TSan queues forever with no error because GitHub does not reject an unsatisfiable label set. Before flipping the variable, check all three on the supervisor host, not just the process:

• the pulp-build-runner macOS golden is present in the agent's TART_HOME (tart list); without it the lane can never boot a guest;
• the supervisor's log shows queue scans succeeding, not repeated SCAN BLIND (gh queue scan failed), which means its GitHub auth is dead and it cannot see queued work at all;
• running_macos_vms is not already saturated by an unrelated guest. The counter is host-wide by design, so a Linux VM parked on the same host can pin a cap=1 lane at full and silently starve it.

PULP_SANITIZER_TSAN_RUNS_ON_JSON therefore stays at its hosted value until a dispatch proof records a real assignment. runner_topology.json contracts the hosted value so the checker fails loudly if the variable is flipped ahead of that proof.

coverage.yml accepts linux_runner_selector_json, macos_runner_selector_json, and windows_runner_selector_json inputs. The macOS Tart coverage proof path is:

gh workflow run coverage.yml \
  -f macos_runner_selector_json='["self-hosted","macOS","ARM64","pulp-coverage-vm-macos"]'

Only set PULP_COVERAGE_MACOS_RUNS_ON_JSON to that selector after the proof run uploads the os-macos Codecov flag. The coverage LaunchAgent uses --queue-match-labels so existing hosted Coverage jobs do not accidentally boot a local coverage VM.

build.yml has the equivalent linux_runner_selector_json, windows_runner_selector_json, and macos_runner_selector_json inputs. These are the same inputs already documented above; they are listed here for completeness alongside the repo-variable knobs.

For a trusted Linux-only Mac Pro proof during hosted saturation, disable the Windows leg explicitly:

gh workflow run build.yml --ref <trusted-branch> \
  -f linux_runner_selector_json='["self-hosted","Linux","X64","pulp-build-linux-x64","pulp-host-macpro"]' \
  -f run_windows=false

This switch affects only workflow_dispatch; automatic events keep their documented matrix policy.

GitHub-hosted Windows is opt-in. A push to main schedules no Windows job, and a build.yml dispatch runs its Windows (x64) leg on the self-hosted Windows x64 lane, not a hosted image. The leg and the three windows-latest compile gates run hosted only when one of these is set:

• -f allow_hosted_windows=true on the dispatch;
• the repository variable PULP_ALLOW_HOSTED_WINDOWS=1, which also lets a push run it.

-f windows_runner_selector_json=... pins the leg to a named runner instead.

resolve-provider prints resolve-provider: Windows leg omitted: GitHub-hosted Windows is opt-in ... when it leaves the leg out, and the advisory windows alias then passes with that line. android.yml runs its windows-x64 row on the self-hosted lane on the nightly and a dispatch, and leaves it out of a push unless PULP_ALLOW_HOSTED_WINDOWS=1; no required context reads it. windows-cli-compile.yml is the one listed exception: its push verdict on main is the release backstop auto-release.yml's pre-tag check reads. Schedules and pull requests keep their documented Windows policy. tools/scripts/test_windows_runner_policy.py (HostedWindowsOptInTests) scans every workflow and fails on a job that can start a hosted windows-* runner on a push, or a build.yml dispatch, without vars.PULP_ALLOW_HOSTED_WINDOWS == '1' in its condition.

Reverting

Unset the variable and the job falls back to the hard-coded default immediately on the next run:

gh variable delete PULP_SANITIZER_TSAN_RUNS_ON_JSON

No code change is needed to revert, either.

Registering a self-hosted Mac runner (appendix)

Flipping a job to "self-hosted" labels assumes those labels are advertised by a running GitHub Actions runner somewhere. Register one on the Mac you want the job to run on:

# 1. From repo Settings -> Actions -> Runners, click "New self-hosted runner"
#    to get a short-lived registration token. Then on the Mac:
mkdir -p ~/actions-runner && cd ~/actions-runner
curl -o actions-runner.tar.gz -L https://github.com/actions/runner/releases/latest/download/actions-runner-osx-arm64.tar.gz
tar xzf actions-runner.tar.gz

# 2. Register with labels that match the JSON you set in the repo var.
#    Example for the TSan / sanitizer lane:
./config.sh --url https://github.com/Generous-Corp/pulp \
            --token <REGISTRATION_TOKEN> \
            --name "$(hostname)-sanitizer" \
            --labels "self-hosted,macos,arm64,sanitizer" \
            --work _work

# 3. Install as a launchd service so it runs at login and survives reboots.
./svc.sh install
./svc.sh start
./svc.sh status

Operational note. Self-hosted runners execute arbitrary code from any branch that can trigger the workflow. Use them on dedicated hardware / VMs you control, not shared personal machines. Apple Silicon hosts should prefer arm64 labels so jobs don't try to match Intel-only labels.

Agents do NOT register runners. Treat these commands as a human ops task documented here for completeness.

Creating a Namespace macOS runner profile

Today, nsc can verify login/workspace state and inspect the instances created by GitHub Actions, but it does not create or edit GitHub Actions runner profiles from this workflow. Creating a new runner profile is currently a Namespace dashboard step.

Use this path in Namespace:

• GitHub Actions -> Profiles -> New Profile

Recommended fields for the first macOS validation profile:

• Name in the UI: generouscorp-macos
• OS & Architecture: macOS on Apple Silicon
• Resources: 6 vCPU, 14 GB RAM
• Base image: a recent Xcode/macOS image appropriate for your build
• Cache toggles: leave enabled unless you have a reason to turn them off

Important selector detail:

• the Namespace UI shows the profile name without the GitHub runner prefix
• the selector you pass to Pulp/GitHub Actions is the prefixed form
• example: UI profile generouscorp-macos becomes selector "namespace-profile-generouscorp-macos"
• for one-off experiments you can skip profile creation entirely and pass a direct machine label instead, for example: "nscloud-macos-tahoe-arm64-6x14"

After creating the profile, validate it with a one-off run:

pulp ci-local cloud run build feature/my-branch \
  --provider namespace \
  --macos-runner-selector-json '"namespace-profile-generouscorp-macos"'

Or use a direct machine label for an ad hoc run:

pulp ci-local cloud run build feature/my-branch \
  --provider namespace \
  --macos-runner-selector-json '"nscloud-macos-tahoe-arm64-6x14"'

Then confirm the backing instance shape with:

nsc instance history --all -o json --max_entries 10

For a real macOS runner, the matching entry should report:

• user_label.nsc.runner-profile-tag = "namespace-profile-generouscorp-macos"
• shape.os = "macos"
• shape.machine_arch = "arm64"

If it instead shows linux/amd64, the profile label is valid but the backing runner is not a real macOS machine yet.

Prerequisites

• UTM — free VM manager for macOS (Apple Silicon and Intel)
• SSH key access to your VMs (password auth is not supported)
• The Pulp repo cloned on each VM at the path specified in config.json

UTM is the simplest option, but any SSH-reachable host works: Proxmox, a cloud VM (Azure/AWS/GCP), or a physical machine on your network. Cloud VMs cost money to run but are otherwise fully supported.

Setup

1. Create your config

cp tools/local-ci/config.example.json tools/local-ci/config.json

Local CI now prefers a machine-global config at ~/Library/Application Support/Pulp/local-ci/config.json on macOS (or the platform-equivalent state_dir()/config.json) so every worktree on the same machine sees the same host topology. tools/local-ci/config.json remains the fallback if no shared config exists, and PULP_LOCAL_CI_CONFIG still overrides both when you need an explicit one-off config.

Create the initial file from the example, then copy it to the shared state location if you want all worktrees to reuse it:

mkdir -p ~/Library/Application\\ Support/Pulp/local-ci
cp tools/local-ci/config.example.json ~/Library/Application\\ Support/Pulp/local-ci/config.json

Edit the chosen config.json and fill in your SSH hostnames and repo paths. The host field is the primary SSH target. fallback_host, if present, is tried next. The utm_fallback block is optional and is only used if SSH targets are unreachable.

Keep those aliases environment-local. Shared skills and docs should not hardcode your personal hostnames or VM names; they should explain how to choose the right target and where that target is configured.

The optional github_actions.workflows.docs-check.providers.namespace.runner_selector_json value lets you set the default Namespace runs-on selector that cloud run docs-check should dispatch when you do not pass --runner-selector-json explicitly.

1b. Optional estimated billing config

If you want per-run and billing-period cost estimates in cloud status, cloud history, and cloud compare, fill in the telemetry.billing block in your local config.

These numbers are estimates only. Verify provider pricing.

Example:

{
  "telemetry": {
    "billing": {
      "enable_provider_reported_totals": false,
      "currency": "USD",
      "billing_period_start_day": 1,
      "github_hosted_job_os_rates_per_minute": {
        "linux": 0.008,
        "windows": 0.016,
        "macos": 0.08
      },
      "namespace_profile_tag_rates_per_hour": {
        "namespace-profile-generouscorp": 0.50,
        "namespace-profile-generouscorp-macos": 1.20
      }
    }
  }
}

Notes:

• GitHub-hosted estimates use per-job OS rates when Pulp can infer the runner OS
• Namespace estimates prefer a profile-tag hourly rate and fall back to a machine-shape rule if you configured one
• if no matching rate exists, the CLI prints cost: unavailable (...)
• enable_provider_reported_totals is off by default; turn it on only if you want Pulp to ask GitHub for repo-wide billing totals when that API is available
• provider-reported GitHub totals are shown separately from tracked-run estimates because they are repo-wide current-period figures, not per-run truth
• GitHub can still return unavailable here if the account/API path does not support the newer billing endpoints

1a. Recommended Namespace setup

If you want to use the Namespace runner-provider path, the easiest setup today is:

brew install namespace-so/tap/nsc   # or use the install method from Namespace docs
nsc login

That is the recommended operator path for this pilot. Pulp can dispatch the GitHub workflow without shelling out to nsc, but keeping nsc installed makes it much easier to verify your Namespace workspace, inspect the account, and later support thin pulp ci-local cloud namespace ... helper commands without re-implementing Namespace setup logic inside Pulp.

Once nsc is installed, Pulp's thin helper commands can verify the state for you:

pulp ci-local cloud namespace doctor
pulp ci-local cloud namespace setup

doctor checks that nsc exists, verifies login state, and prints the current workspace identity. setup stays deliberately thin: it runs nsc login when needed and then re-renders the same status.

{
  "targets": {
    "mac": {
      "type": "local",
      "enabled": true
    },
    "ubuntu": {
      "type": "ssh",
      "host": "ubuntu",
      "repo_path": "/home/yourname/Code/pulp-validate",
      "utm_fallback": {
        "vm_name": "Ubuntu 24.04",
        "boot_wait_secs": 30,
        "ssh_retry_secs": 60
      }
    },
    "windows": {
      "type": "ssh",
      "host": "win",
      "repo_path": "C:\\Users\\yourname\\pulp-validate",
      "cmake_generator": "Visual Studio 17 2022",
      "cmake_platform": "x64",
      "cmake_generator_instance": "",
      "fallback_host": "win2",
      "utm_fallback": {
        "vm_name": "Windows 11",
        "boot_wait_secs": 60,
        "ssh_retry_secs": 120
      }
    }
  }
}

SSH host aliases come from ~/.ssh/config. Set them up there rather than putting raw IPs in this file. This makes it easy to prefer a fast local VM as the primary target and keep a slower hardware-backed machine as the fallback when you only need it for edge cases.

Before trusting a Windows SSH target for CI, verify that its non-interactive PowerShell context resolves git, cmake, and ctest. An interactive shell that works is not sufficient proof for the SSH service context the runner actually uses.

If your Windows VM is Windows on ARM, you can either set cmake_platform to "ARM64" explicitly or leave it blank and let the runner infer ARM64 vs x64 from the remote host. If CMake keeps picking the wrong Visual Studio install, set cmake_generator_instance to the exact VS path, for example C:/Program Files/Microsoft Visual Studio/2022/Community. If you leave cmake_generator_instance blank, the runner prefers a full Visual Studio install over BuildTools when both are present. The pinned WebGPU dependency already has a Windows aarch64 prebuilt for this path, so ARM Windows smoke runs can stay on the normal GPU-enabled configuration. This is useful for fast smoke validation on a local UTM VM. Keep an x64 Windows machine for parity runs when you need the authoritative Windows architecture.

2. Set up SSH keys

Each VM needs your public key in its authorized_keys. The Linux path is straightforward; Windows requires extra steps because OpenSSH on Windows uses a separate file with strict ACLs for admin users.

Find your public key (on your Mac)

If your private key is ~/.ssh/id_ed25519, your public key is:

cat ~/.ssh/id_ed25519.pub

Copy the output — you'll paste it on each VM. If you're running the VM in UTM or another hypervisor and can't copy/paste between host and guest, install the guest tools for your hypervisor first (e.g. SPICE guest tools for UTM/QEMU, VMware Tools, VirtualBox Guest Additions).

Linux (Ubuntu)

If ssh-copy-id is available and you can already reach the VM by password:

ssh-copy-id ubuntu    # or whatever your host alias is

If you're setting up from scratch on a fresh VM, SSH into it (or open its console) and run:

1. Note the VM's IP address:

ip addr show

Look for the inet line under your active adapter (usually enp0s1 or eth0).

2. Install and enable the SSH server (if not already running):

sudo apt update && sudo apt install -y openssh-server
sudo systemctl enable --now ssh

3. Add your public key:

mkdir -p ~/.ssh && chmod 700 ~/.ssh
echo "ssh-ed25519 AAAA...your-key-here..." >> ~/.ssh/authorized_keys
chmod 600 ~/.ssh/authorized_keys

4. (Optional) Disable password auth for tighter security:

sudo sed -i 's/^#\?PasswordAuthentication.*/PasswordAuthentication no/' /etc/ssh/sshd_config
sudo systemctl restart ssh

Windows

On the Windows VM, open PowerShell as Administrator and run:

1. Note the VM's IP address (you'll need it for SSH config later):

ipconfig

Look for the IPv4 Address line under your active adapter.

2. Create the admin authorized_keys file and add your public key:

New-Item -Force -ItemType File -Path "C:\ProgramData\ssh\administrators_authorized_keys"
Add-Content -Path "C:\ProgramData\ssh\administrators_authorized_keys" -Value "ssh-ed25519 AAAA...your-key-here..."

3. Fix the ACL (OpenSSH ignores the file if permissions are wrong):

icacls "C:\ProgramData\ssh\administrators_authorized_keys" /inheritance:r /grant "SYSTEM:(F)" /grant "Administrators:(F)"

4. Make sure sshd is running and set to auto-start:

Set-Service -Name sshd -StartupType Automatic
Start-Service sshd

Why administrators_authorized_keys? Windows OpenSSH uses C:\ProgramData\ssh\administrators_authorized_keys for users in the Administrators group, not ~/.ssh/authorized_keys. The ACL step is required — without it, sshd silently skips the file and falls back to password auth.

Set up SSH config on your Mac

Add entries to ~/.ssh/config so you can type ssh win instead of remembering IPs and usernames:

Host win
  HostName 192.168.64.5
  User your-username
  IdentityFile ~/.ssh/id_ed25519
  IdentitiesOnly yes
  ConnectTimeout 5

Host ubuntu
  HostName 192.168.64.4
  User your-username
  IdentityFile ~/.ssh/id_ed25519
  IdentitiesOnly yes
  ConnectTimeout 5

Replace the HostName values with the actual IPs from ipconfig (Windows) or ip addr (Linux). The host aliases here (win, ubuntu) are what you'll use in hosts.local.json for CI targets.

Test passwordless login

ssh ubuntu exit && echo "ok"
ssh win exit && echo "ok"

3. Clone the repo on each VM

The runner does a git fetch + checkout on the target, so the repo must already exist at the configured repo_path.

# On each VM:
git clone https://github.com/your-org/pulp.git ~/Code/pulp-validate

4. (Optional) Install the launchd drain agent

To automatically drain the queue on login and every 30 minutes:

cp tools/local-ci/dev.pulp.local-ci.plist ~/Library/LaunchAgents/
launchctl load ~/Library/LaunchAgents/dev.pulp.local-ci.plist

Edit the plist first if your repo is at a different path. To remove:

launchctl unload ~/Library/LaunchAgents/dev.pulp.local-ci.plist
rm ~/Library/LaunchAgents/dev.pulp.local-ci.plist

Usage

# Enqueue the current HEAD and wait for completion
pulp ci-local run

# Queue even if your current cwd belongs to a different git root than the script checkout
pulp ci-local run --allow-root-mismatch

# Fast preflight: clean configure/build/install + installed-SDK smoke, no tests
pulp ci-local run --smoke

# Fast PR preflight with a comment that is clearly labeled as smoke-only
pulp ci-local check 56 --smoke

# Run Mac-only while iterating locally
pulp ci-local run --targets mac

# Queue background work with explicit priority
pulp ci-local enqueue --priority low

# Bump a pending job to the front of the queue
pulp ci-local bump <job-id> high

# Drain pending jobs if no other runner already owns the queue
pulp ci-local drain

# Show queue, active runner, recent results, live target state, and VM status
pulp ci-local status

# Tail a running or completed target log
pulp ci-local logs <job-id> --target windows

# Show accumulated exact-SHA target evidence for a branch
pulp ci-local evidence feature/my-branch --limit 3

# Show local-CI disk usage and reclaimable artifacts without deleting anything
pulp ci-local cleanup
pulp ci-local cleanup --dry-run

# Delete stale bundles/logs/results once no local CI job is running
pulp ci-local cleanup --apply

# Include prepared build/install caches too; later reruns will rebuild them
pulp ci-local cleanup --apply --include-prepared

pulp ci-local run is the most common command. It enqueues the current HEAD, joins the machine-global queue, and waits until that exact job finishes.

Develop branch workflow

For complex, multi-piece features that use a develop/* integration branch, PRs target the develop branch instead of main. The ship command supports this via --base:

# Ship a feature to the develop branch (not main)
pulp ci-local ship feature/pkg-registry --base develop/package-manager

# The develop branch itself ships to main at phase boundaries
pulp ci-local ship develop/package-manager

GitHub Actions CI triggers on PRs to both main and develop/** branches, so CI runs automatically regardless of the target.

If you pass a branch name explicitly, for example pulp ci-local run feature/my-branch, local CI resolves and records that branch tip's exact SHA immediately. This prevents a stale launching checkout from accidentally queuing its own HEAD while you intended to validate a different branch.

Before queueing, local CI now also records: - the worktree root that is actually being queued - the current cwd and its git root, if any - the config path and whether it came from PULP_LOCAL_CI_CONFIG, shared state, or the worktree fallback - the selected SSH host/transport intent for each remote target

If the current cwd belongs to a different git root than the local_ci.py checkout you are invoking, queueing fails fast by default. Pass --allow-root-mismatch only when that mismatch is intentional.

If a selected SSH target is down and no fallback host or UTM fallback is configured, queueing now fails fast instead of burning time on a doomed job. Pass --allow-unreachable-targets only when you deliberately want to queue past that preflight.

Use --smoke when you want a quicker preflight before a full matrix run. Smoke mode still validates a clean detached worktree and installed SDK export path, but it disables tests, examples, and GPU in that clean build and skips ctest. Queue summaries and PR comments label these jobs as validation=smoke so they are not mistaken for full validation.

When a rerun is narrow and stays on the exact same SHA, local CI can now reuse the prepared root for that target + validation on persistent hosts. Status output calls this out as prepared=reused or prepared=clean so reused proof is never mistaken for a fresh cold path.

While a job is still running, pulp ci-local status reports live per-target state for the active job when available, for example:

Runner: pid=12345 active=[abcd1234ef56] feature/my-branch

Running (1):
  [abcd1234ef56] feature/my-branch @ 0123456789ab priority=normal targets=mac,ubuntu,windows
    submission: root=/Users/me/Code/pulp-worktree config=/Users/me/Library/Application Support/Pulp/local-ci/config.json (shared-state)
    live targets: mac=pass, ubuntu=pass, windows=running
    windows: phase=test, output=2026-04-01T01:34:18+00:00, heartbeat=2026-04-01T01:34:33+00:00, idle=15s, liveness=quiet, log=windows.log
      37/1263 Test: OSC 4-byte alignment

If a run is interrupted after some targets have finished, the job is requeued but keeps its last known target state:

Pending (1):
  [abcd1234ef56] feature/my-branch @ 0123456789ab priority=normal targets=mac,ubuntu,windows
    last known targets: mac=pass, ubuntu=pass, windows=running

Results are written to the machine-global state directory:

• macOS: ~/Library/Application Support/Pulp/local-ci/results/
• Linux: ${XDG_STATE_HOME:-~/.local/state}/pulp/local-ci/results/

A non-zero exit means at least one target failed.

If a newer SHA is queued for the same branch, targets, and validation mode, older pending work is marked superseded and written to the results directory with a reference to the replacement job. If a runner dies and reconciliation finds a newer replacement already queued for that same scope, the stale running job is also superseded instead of being requeued.

Cleanup And Disk Usage

pulp ci-local status now includes a local footprint summary so retained CI state stops being invisible drift:

• bundles
• prepared build/install caches
• logs
• results
• tracked cloud-run records

Use pulp ci-local cleanup to inspect what can be reclaimed. The command is a dry run by default, and --dry-run is available explicitly when you want that spelled out in scripts or notes.

What is cleaned automatically after job completion:

• completed-job git bundles once no pending/running job still needs them
• orphaned logs outside retained queue history
• orphaned result files outside retained queue history

What is not cleaned automatically in this first pass:

• prepared build/install state under prepared/<target>/<mode>

Prepared state is an intentional reuse cache. If you include it in manual cleanup, later reruns will rebuild it from scratch.

Examples:

# Inspect reclaimable space
pulp ci-local cleanup

# Show the same dry-run plan explicitly
pulp ci-local cleanup --dry-run

# Delete stale bundles/logs/results
pulp ci-local cleanup --apply

# Also delete prepared caches
pulp ci-local cleanup --apply --include-prepared

Safety rules:

• cleanup --apply is blocked while local CI jobs are running
• prepared cleanup is destructive to cached build/install state
• logs/results tied to jobs still present in queue history are retained

If you need immediate manual cleanup outside the CLI, make sure no pulp ci-local job is active first.

Desktop automation

pulp ci-local desktop ... adds a GUI/session automation layer under the same local CI control plane. Use it when an agent needs to launch an app, inspect it, click on it, capture screenshots, or publish a local evidence gallery without logging into the target machine manually.

Current desktop commands:

# Prepare one target and record its contract/receipt
pulp ci-local desktop install mac
pulp ci-local desktop install ubuntu
pulp ci-local desktop install windows

# Health and capability reporting
pulp ci-local desktop doctor mac
pulp ci-local desktop status
pulp ci-local desktop recent mac --limit 3
pulp ci-local desktop proof windows --action inspect --source-mode exact-sha --sha <commit-sha>

# Configure artifact/publish settings
pulp ci-local desktop config show
pulp ci-local desktop config set artifact_root ~/Library/Application\\ Support/Pulp/desktop-automation/runs
pulp ci-local desktop config set publish_mode none

# Run GUI actions
pulp ci-local desktop smoke mac --bundle-id com.apple.TextEdit --label textedit-smoke
pulp ci-local desktop inspect mac --command '/path/to/pulp-ui-preview' --label ui-preview-inspect --pulp-app-automation
pulp ci-local desktop click mac --command '/path/to/pulp-ui-preview' --click-view-id bypass-toggle --capture-ui-snapshot --pulp-app-automation
pulp ci-local desktop inspect windows --command 'notepad.exe' --label notepad-inspect
pulp ci-local desktop click windows --command 'notepad.exe' --click 885,18 --label notepad-maximize

# Run against an exact prepared SHA instead of the live checkout
pulp ci-local desktop inspect mac \
  --command './build-desktop-automation/examples/ui-preview/pulp-ui-preview' \
  --source-mode exact-sha \
  --sha <commit-sha> \
  --prepare-command 'cmake -S . -B build-desktop-automation && cmake --build build-desktop-automation --target pulp-ui-preview' \
  --pulp-app-automation

# Publish or prune local bundles
pulp ci-local desktop publish mac --limit 5 --label mac-gallery
pulp ci-local desktop cleanup mac --older-than-days 14 --keep-last 10

Ubuntu prerequisite:

sudo apt-get update
sudo apt-get install -y git-lfs xvfb xauth xdotool imagemagick wmctrl x11-utils
git lfs install

Supported Ubuntu/Linux setup tiers:

• baseline deterministic backend: xvfb + xauth
• source/bootstrap prerequisite: git-lfs
• richer interaction/capture lane: xdotool, imagemagick, x11-utils, and wmctrl

xvfb-run is the supported deterministic backend for Ubuntu/Linux desktop automation. A visible :0 display socket alone is not enough for SSH-driven automation because X11 authorization is often unavailable inside the remote shell. For repeatable CI and agent-driven runs, use the package set above and keep xvfb-run as the documented default.

desktop doctor ubuntu is an aggregate report. If multiple prerequisites are missing, it reports the full missing set plus remediation commands in one run instead of stopping at the first failure.

desktop doctor ubuntu checks the non-interactive SSH environment, not your interactive shell. setup.sh now prepends the common ~/.local/bin path automatically before dependency checks, but if git-lfs still fails over SSH after that, add the real install location to the non-interactive login-shell PATH or install git-lfs system-wide so git lfs version succeeds without extra shell setup.

Exact-SHA source prep on Ubuntu/Linux uses the same non-interactive SSH environment. The controller now treats bundle-based checkout and LFS materialization as separate steps:

• prepend ~/.local/bin before any git-lfs-dependent command
• fetch and checkout with GIT_LFS_SKIP_SMUDGE=1
• attach the clone URL as origin
• then let setup.sh --deps-only --ci / git lfs pull materialize the SDK blobs

That split matters on fresh VMs because a bundle checkout alone does not carry an origin remote, and LFS smudge/pull fails if it cannot resolve the repository URL.

Fresh Ubuntu proof checklist:

1. Start from a fresh source root, PULP_HOME, and PULP_PROJECTS_DIR
2. Run ./setup.sh --deps-only --ci
3. Build pulp-cli
4. Run pulp create <ProjectName> --manufacturer "<Name>" --no-interactive
5. Run pulp build inside the generated project
6. Verify actual emitted artifacts, not just configure/test success

Current expected native outputs from that proof are:

• Linux: VST3 target output under build/VST3, CLAP, LV2, and the standalone binary
• macOS: build/VST3/<Name>.vst3, build/AU/<Name>.component, build/CLAP/<Name>.clap, and the standalone .app bundle
• Windows: build/VST3/Debug/<Name>.dll for the VST3 target, build/CLAP/Debug/<Name>.clap, and the standalone .exe

If web formats are required, make them explicit in the generated project format list; the default native create proof does not imply web artifact output.

Windows first-time setup checklist:

1. Install and enable OpenSSH Server.
2. Keep a normal desktop user logged in to the VM. The Windows session-agent runs inside that logged-in session; SSH by itself is not a GUI session.
3. Make sure winget is available. desktop install windows uses it to provision required remote tooling such as Git when the VM is still fresh.
4. Run pulp ci-local desktop install windows once. This bootstraps the scheduled task, installs required remote tooling when possible, and writes the target-side PowerShell agent under %LOCALAPPDATA%\\Pulp\\desktop-automation-agent.
5. Run pulp ci-local desktop doctor windows and make sure SSH, the scheduled-task contract, and the required git check are green before attempting live proofs.
6. For source builds on the Windows VM itself, use powershell -ExecutionPolicy Bypass -File .\setup.ps1. The wrapper imports the Visual Studio environment and uses a short temporary drive alias so first-time bootstrap does not fail on long nested dependency paths.

The short-path rule is not theoretical. Windows source builds can fail from long nested checkout roots and then pass once the same source tree is mapped through a temporary drive alias before the first configure/build. Treat setup.ps1 or an equivalent short-path wrapper as the supported bootstrap path for Windows source builds.

Remote tooling policy on Windows:

• required: git
• used by the exact-SHA bundle-sync and prepare flows
• desktop install windows will provision it via winget when possible
• optional: gh
• useful for remote GitHub workflows on the target
• not required for smoke/inspect/click proofs
• optional: gh auth
• advisory only; authenticate it only if you intentionally want GitHub CLI workflows on the Windows target

Remote repo bootstrap policy on Windows:

• first-time desktop install windows should not require GitHub credentials on the target VM
• the controller prefers a locally uploaded git bundle to materialize pulp-validate
• origin is still attached when available so later fetches remain truthful
• gh and stored Git credentials are optional unless you intentionally want GitHub workflows on the Windows machine itself

Useful host-side verification commands:

Get-Service sshd
Set-Service -Name sshd -StartupType Automatic
Start-Service sshd
Get-NetFirewallRule -Name *ssh*
where.exe winget
where.exe git
where.exe gh

Useful first-time remote installs if you want to pre-provision them manually:

winget install --id Git.Git -e --source winget --accept-package-agreements --accept-source-agreements --disable-interactivity
winget install --id GitHub.cli -e --source winget --accept-package-agreements --accept-source-agreements --disable-interactivity

Supported Windows v1 interaction tiers:

• generic window-capture lane:
• --command only
• works for normal desktop apps such as notepad.exe
• supports window screenshot capture and coordinate clicks
• Pulp-owned app automation lane:
• add --pulp-app-automation
• enables ViewInspector snapshots and view-target selectors such as --click-view-id

Artifact bundles are written outside the repo by default:

• macOS: ~/Library/Application Support/Pulp/desktop-automation/runs/
• Linux: ${XDG_STATE_HOME:-~/.local/state}/pulp/desktop-automation/runs/
• Windows: %LOCALAPPDATA%\\Pulp\\desktop-automation\\runs\\

Each bundle stores:

• manifest.json
• stdout.log / stderr.log
• prepare.log when exact-SHA mode runs a fresh prepare step
• ui-tree.json when a UI snapshot is available
• screenshots/window.png
• screenshots/before.png / screenshots/diff.png when an interaction captures before/after evidence

The artifact root also maintains rolling summaries for agents and status tooling:

• latest-run.json — newest observed run summary
• latest-proof.json — newest successful proof summary
• runs.jsonl — raw summary stream for recent desktop automation runs
• target-scoped copies under <artifact-root>/<target>/...
• _published/latest-report.json — newest staged local HTML/JSON gallery summary
• _published/reports.jsonl — raw summary stream for local published galleries

manifest.json now includes additive source provenance when desktop actions run through the controller:

• source.mode (live or exact-sha)
• source.branch
• source.sha
• source.prepare_command
• source.prepare_timeout_secs
• source.prepared_root
• source.launch_cwd

Desktop reporting surfaces are intentionally split:

• desktop recent = raw run history, including failed attempts
• desktop proof = successful proof summaries grouped by target + action + source.mode + source.sha
• desktop status = target config plus latest_run, latest_proof, and the newest local publish summary (latest_publish)

Use desktop proof when you need to answer questions like:

• “What live-host proof do we already have for Ubuntu on this SHA?”
• “Did Windows ever pass this exact-SHA inspect lane?”
• “What is the newest successful proof, even if the newest run failed?”

Exact-SHA desktop source mode

desktop smoke, desktop click, and desktop inspect all share a controller-owned source mode:

• --source-mode live|exact-sha
• --branch
• --sha
• --prepare-command
• --prepare-timeout

Behavior:

• live launches from the target's normal working copy behavior.
• exact-sha prepares a per-target source root for the requested SHA, launches from that prepared root, and records the prepared-root provenance in the run manifest.
• On Windows, --prepare-command executes inside a generated .cmd script under cmd.exe. Use double quotes for paths, generator names, and arguments. POSIX-style single-quoted tokens are treated as literal text and are rejected by the controller before the remote prepare step starts.
• When desktop_automation.targets.<target>.optional.webview_driver=true, desktop doctor probes the configured webdriver_url through the WebDriver /status endpoint and reports whether the driver is actually reachable and ready, not just whether the URL exists in config.

Preparation/cache semantics:

• Prepared roots are keyed by target + sha + prepare_command.
• A repeated identical request may reuse the prepared root instead of rebuilding it.
• prepare_command only runs when a fresh prepared root is created.

Launch behavior:

• Desktop actions switch their launch cwd to the prepared root in exact-SHA mode.
• Repo-local executable paths in the first command token are rewritten into the prepared root automatically.
• The current exact-SHA workflow is a --command lane. Do not assume --bundle-id participates in exact-SHA source preparation.

pulp-ui-preview is currently Apple-desktop-only, so the Linux and Windows source-build examples below use the cross-platform PulpGain standalone target.

Examples:

# macOS local exact-SHA inspect
pulp ci-local desktop inspect mac \
  --command './build-desktop-automation/examples/ui-preview/pulp-ui-preview' \
  --source-mode exact-sha \
  --sha <commit-sha> \
  --prepare-command 'cmake -S . -B build-desktop-automation && cmake --build build-desktop-automation --target pulp-ui-preview' \
  --pulp-app-automation

# Ubuntu xvfb exact-SHA smoke against a Linux-supported standalone
pulp ci-local desktop smoke ubuntu \
  --command './build-desktop-automation/examples/pulp-gain/PulpGain' \
  --source-mode exact-sha \
  --sha <commit-sha> \
  --prepare-command 'cmake -S . -B build-desktop-automation && cmake --build build-desktop-automation --target PulpGain_Standalone'

# Windows session-agent exact-SHA smoke
pulp ci-local desktop smoke windows \
  --command '.\\build-desktop-automation\\examples\\pulp-gain\\Debug\\PulpGain.exe' \
  --source-mode exact-sha \
  --sha <commit-sha> \
  --prepare-command 'cmake -S . -B build-desktop-automation -G \"Visual Studio 17 2022\"; cmake --build build-desktop-automation --target PulpGain_Standalone --config Debug'

# Windows generic live inspect
pulp ci-local desktop inspect windows \
  --command 'notepad.exe' \
  --label notepad-inspect

# Windows generic live click with before/after evidence
pulp ci-local desktop click windows \
  --command 'notepad.exe' \
  --click 885,18 \
  --label notepad-maximize

# Query the newest successful Windows proof for one SHA
pulp ci-local desktop proof windows \
  --action smoke \
  --source-mode exact-sha \
  --sha <commit-sha>

Desktop adapter truth

• macos-local
• runs directly on the local logged-in macOS session
• supports bundle launch via --bundle-id
• supports Pulp-owned app automation (--pulp-app-automation) for direct launch commands, including ViewInspector snapshots and view-target clicks
• linux-xvfb
• runs GUI smoke/inspect/click through xvfb-run
• currently supports --command only
• currently requires --pulp-app-automation for the click/inspect lane
• windows-session-agent
• bootstraps a scheduled task plus target-side PowerShell agent in the logged-in Windows desktop session
• requires a real logged-in desktop user; SSH alone is not enough
• currently supports --command only
• supports generic window-capture smoke/inspect/click for normal desktop apps
• supports coordinate clicks and before/after screenshot diffs without --pulp-app-automation
• supports ViewInspector snapshots and view-target selectors only with --pulp-app-automation
• uses the scheduled task plus target-side agent as the honest v1 Windows interaction lane; external UI automation tools are optional future adapters, not the core controller

Proof lookup

desktop proof is the first-class proof query surface for desktop automation:

• filters:
• target
• --action
• --source-mode live|exact-sha|legacy
• --sha
• --branch
• groups successful proofs by target/action/source.mode/source.sha
• ignores failed runs when computing proof summaries

Example:

pulp ci-local desktop proof ubuntu --action click --source-mode exact-sha --sha <commit-sha>

desktop status now reports both:

• latest_run: the newest run, even if it failed
• latest_proof: the newest successful proof summary for that target
• latest_publish: the newest local HTML/JSON gallery summary staged under _published/

Desktop config keys

tools/local-ci/config.json accepts a desktop_automation block:

{
  "desktop_automation": {
    "artifact_root": "",
    "publish_mode": "none",
    "publish_branch": "dev-artifacts",
    "retention_days": 14,
    "targets": {
      "mac": {
        "adapter": "macos-local",
        "bootstrap": "launchagent",
        "capability_tier": "v2",
        "optional": {
          "webview_driver": false,
          "webdriver_url": "",
          "debug_attach": false,
          "debugger_command": "lldb",
          "video_capture": false,
          "frame_stats": false
        }
      },
      "ubuntu": {
        "adapter": "linux-xvfb",
        "bootstrap": "xvfb-run",
        "capability_tier": "v2",
        "optional": {
          "webview_driver": false,
          "webdriver_url": "",
          "debug_attach": false,
          "debugger_command": "lldb",
          "video_capture": false,
          "frame_stats": false
        }
      },
      "windows": {
        "adapter": "windows-session-agent",
        "bootstrap": "scheduled-task",
        "capability_tier": "v2",
        "task_name": null,
        "remote_root": null,
        "optional": {
          "webview_driver": false,
          "webdriver_url": "",
          "debug_attach": false,
          "debugger_command": "",
          "video_capture": false,
          "frame_stats": false
        }
      }
    }
  }
}

For Windows:

• task_name is optional. If omitted, local CI uses PulpDesktopAutomationAgent-<target>.
• remote_root is optional. If omitted, the agent is installed under %LOCALAPPDATA%\Pulp\desktop-automation-agent.
• optional.webview_driver enables the future WebView/WebDriver capability vocabulary for that target. Pair it with optional.webdriver_url only when the app under test actually exposes a localhost WebDriver endpoint in debug/test mode.
• optional.debug_attach, optional.video_capture, and optional.frame_stats are opt-in groundwork flags. They make the target advertise and doctor those optional tiers; they do not magically make the adapter support them unless the required tooling is also present.

Convenience updates through the CLI:

pulp ci-local desktop config set target.mac.webview_driver true
pulp ci-local desktop config set target.mac.webdriver_url http://127.0.0.1:4444
pulp ci-local desktop config set target.mac.debug_attach true
pulp ci-local desktop config set target.mac.debugger_command lldb
pulp ci-local desktop config set target.mac.video_capture true
pulp ci-local desktop config set target.mac.frame_stats true

Recommended host-side remediation when desktop doctor windows reports SSH service reset during handshake:

Get-Service sshd
Set-Service -Name sshd -StartupType Automatic
Start-Service sshd
Get-NetFirewallRule -Name *ssh*

Treat that failure as a Windows host-side OpenSSH issue, not a desktop-agent contract failure.

Desktop publication

pulp ci-local desktop publish always stages a local HTML/JSON gallery from recent bundles. In the default publish_mode=none path, that is the whole feature. When publish_mode=branch, the same report is also mirrored to the configured publish branch under desktop-automation/latest/ and desktop-automation/reports/<report-id>/.

• index.html
• index.json
• copied screenshots and diffs
• source manifest/log references
• _published/latest-report.json and _published/reports.jsonl rollups for the newest/known local galleries

Use desktop config set publish_mode ... only when you intentionally want publication behavior. The default should stay none for normal development.

Branch publication notes:

• publish_mode=branch pushes the latest local report to publish_branch
• the branch stores desktop-automation/latest/ plus immutable desktop-automation/reports/<report-id>/ snapshots
• when the repo remote is GitHub, the publish report includes clickable branch/tree/blob URLs for the mirrored artifacts

Evidence Tracking

pulp ci-local evidence summarizes the last-good recorded results by exact SHA, target, and validation mode. This is the operator-facing answer to:

• what already passed on this branch?
• which exact SHA has Windows full proof?
• do we really need to rerun macOS again?

The compact evidence section in pulp ci-local status uses the same data so the current branch’s known-good results stay visible during active work.

Working A Failure

Do not wait for a whole matrix to finish before reacting. The fastest loop is:

1. start a run
2. watch pulp ci-local status
3. tail pulp ci-local logs <job-id> --target <name> on the first failing or suspicious target
4. begin the narrowest local repro or code inspection immediately
5. rerun only the truthful scope needed after the fix

In practice, that means:

• one process owns CI monitoring and host state
• one process or agent works the likely fix locally as soon as a failure becomes actionable
• user updates should be sent when a target changes state or the first actionable failure appears, not only when asked
• a target that already failed is enough to start debugging; do not burn time waiting for unrelated targets to finish unless their result changes the fix
• once a failure is actionable, start the fix track in parallel unless it would contend with the same host or invalidate the active run
• do not rerun a target that already passed on the exact same SHA unless that prior result is untrustworthy or the environment changed
• if only one or two targets are stale, rerun only those targets instead of the whole matrix
• once the failure surface is isolated, prefer the minimum sufficient proof instead of a symmetric rerun
• a direct exact-SHA validate on one target counts as valid evidence for that target; keep earlier same-SHA passes for the other targets unless something actually invalidated them
• on persistent hosts, narrow same-SHA reruns should prefer prepared-state reuse instead of paying again for clean worktree/setup/build work
• use --smoke first when the risk is install/export/build structure rather than runtime test behavior
• all targets on one SHA is a goal, not a reason to blindly rerun already-green same-SHA targets
• if a broader in-flight job is no longer informative, cut over to the narrower rerun instead of letting the queue drift

Priorities

Jobs are ordered by priority first, then FIFO within the same priority.

• low — background validation
• normal — default interactive work
• high — shipping, PR checks, or work you want to run first

You can set the initial priority with --priority and change a pending job later with:

pulp ci-local bump <job-id> high

pulp ci-local status prints the job ids you can bump.

Exact SHAs On Remote Targets

Remote targets validate the queued SHA, not the latest branch tip. That keeps queued jobs truthful, and the runner now uploads that exact SHA to SSH targets as a git bundle before validation.

If you queued work with an explicit branch name, the runner first resolves that branch name to a commit SHA and then treats the run exactly like any other exact-SHA validation.

That means this works even for a local-only commit:

pulp ci-local run --targets mac,ubuntu,windows

pulp ci-local ship still pushes first because it opens and validates a PR, but ordinary local validation no longer depends on the remote host already having your branch tip.

Running Mac-only

If you don't have VMs set up, disable the SSH targets in your active CI config:

"ubuntu": {
  "type": "ssh",
  "enabled": false,
  ...
}

Mac validation still runs. You get single-platform coverage, which is better than nothing for catching build breaks before pushing.

You can also keep the SSH targets enabled and request Mac-only while iterating:

pulp ci-local run --targets mac

Compiler coverage: which lanes use which compiler

Worth knowing before you trust a green PR, because the answer is not symmetric.

Native Linux workflows install their shared system prerequisites through .github/actions/install-linux-build-deps, backed by the portable tools/ci/install_linux_build_deps.py resolver and tools/ci/linux_build_deps.json. Profiles describe capabilities (native and native-webview); compiler versions, analysis tools, caches, and other lane-specific packages stay explicit at each call site. The workflow policy file enumerates adopters and reviewed exclusions, and workflow-lint rejects a new direct apt workflow that has no owner. Update the manifest once when a native dependency changes instead of copying the package into individual build, coverage, sanitizer, release, or portability lanes.

Every Linux lane in PR CI compiles with Clang — "Public headers compile standalone (Linux Clang)", "IWYU (Linux, Clang)", "RealtimeSanitizer (Linux x86_64, Clang 18)". macOS is Clang by definition. Windows is MSVC.

That left GCC compiled in exactly one place: release-path-pr-gate.yml, which is path-triggered on release files (Skia pins, tools/deps/manifest.json, tools/cmake/Pulp*.cmake, the top-level CMakeLists.txt). Most PRs never trigger it, so a GCC-only error inside core/ could sit on main indefinitely.

It did, and not once. core/host/src/signal_graph.cpp keeps acquiring two identical .custom_latency_for entries in one designated-initializer list — Clang accepts that and silently takes the last, so nothing on the Apple or Clang-Linux lanes notices. git log -S '.custom_latency_for' shows the same defect fixed four separate times:

4371eebce  fix(host): remove a duplicate binder designator that GCC rejects
1bdd0434a  fix(host): drop the duplicated custom-latency binder
402620df4  fix(host): drop a duplicate designator that breaks every non-Apple release build
077ffabda  build(host): drop a duplicate designator that breaks the MSVC build

Every one of those was caught late — by MSVC, by a non-Apple release build, or by the release-path gate firing on an unrelated PR. The binder list is long and sits where merges collide, so the duplicate keeps coming back; what was missing was a PR-time lane that says so immediately.

gcc-compile-gate.yml closes that hole. It runs on every PR and compiles the core libraries with g++ and nothing else:

Option	Value	Why
PULP_ENABLE_GPU	OFF	no Dawn/Skia fetch or build — this is what keeps the gate in minutes rather than a full release build
PULP_BUILD_TESTS	OFF	the gate asks "does core/ compile under GCC", not "does it work"
PULP_BUILD_EXAMPLES	OFF	same
PULP_ENABLE_DESIGN_IMPORT	OFF	authoring subsystem, not core portability
PULP_ENABLE_INSPECTOR	OFF	dev surface, not core portability

The discovered core-target list is built through tools/ci/governed-build.sh cmake --build build-gcc --target $(cat targets.txt), so the gate's target coverage stays dynamic while job selection remains under the shared governor.

Read a failure here literally. The lane runs no tests and touches no hardware, so it cannot flake on load or timing the way the GPU-perf lanes can. A red result is a real compiler divergence. Clang accepting the same code does not make it portable.

What it deliberately does not cover: GCC behavior. Nothing is executed, so a construct both compilers accept but implement differently is still only caught by the Clang test lanes. Widening this to run tests under GCC is a separate decision with a real time cost.

It also guards one option combination. The lane configures with PULP_ENABLE_DESIGN_IMPORT=OFF, which is the option's own documented "release/ship OFF" setting — and that configuration was once unlinkable, because tools/import-design was added unconditionally while the pulp::view design-IR sources it links sit behind that option. The discovery step now runs --assert-absent pulp-import-design against the codemodel the lane already produces, so a re-broken guard fails here immediately instead of surfacing as an undefined-reference wall in someone's release build. It costs no extra configure time. Because the guard lives in the top-level CMakeLists.txt, that file is one of the lane's path triggers alongside core/**.

For contributors

You don't need the same VM setup as the original developer. Options:

• Mac-only: Disable all SSH targets. Fast, free, covers the primary development platform.
• UTM VMs: Free. Requires ~40 GB of disk for both VMs. UTM images can be created from ISO or from the UTM gallery.
• Cloud VMs: Works with any SSH-accessible host. Costs money while running — stop them when not in use.
• Physical machines: A spare Linux box or Windows machine on your network works fine.

Local CI config is intentionally gitignored. Keep your host topology local, and prefer the machine-global config path so every worktree uses the same host map by default.

Branch refreshes are refused unless they fix something

.shipyard/config.toml sets [merge] refresh_branch = "only-if-conflicting". main lands through a merge queue that builds and validates the merge result itself, and build.yml cancels an in-flight run whenever the PR ref moves, so refreshing a merely BEHIND PR (update-branch, or merging origin/main into it) restarts the ~25-40 minute required macos gate for nothing. With this policy Shipyard's ghapp branch-refresh guard refuses an App-authenticated refresh when the PR is mergeable and no required check is failing; a conflicting PR, a red required check, or any state it cannot read is still refreshed. The guard only takes effect on hosts where shipyard guards status shows branch-refresh-guard current. A local git merge origin/main && git push bypasses it, so do not merge main into a PR unless it conflicts.

Steward auto-handoff stays off

.shipyard/config.toml sets [merge_steward] auto_handoff = false, and nothing reads a handoff receipt: Pulp removed its repository-wide steward workflow and recovery worker, and PRs land through the native merge queue. Do not pass shipyard pr --workstream-id; it opts one PR into the same unread handoff.

Troubleshooting

JSONDecodeError on Shipyard queue file

Shipyard's local job queue lives at ~/Library/Application Support/shipyard/queue/queue.json on macOS (~/AppData/Local/shipyard/queue/queue.json on Windows, ${XDG_STATE_HOME:-~/.local/state}/shipyard/queue/queue.json on Linux). On rare crashes Shipyard can truncate this file to zero bytes, which then breaks every subsequent invocation with a JSONDecodeError.

Recovery (run once):

echo '{"jobs": []}' > ~/Library/Application\ Support/shipyard/queue/queue.json

Re-running tools/install-shipyard.sh also performs this reset automatically. Tracked as #528.

macpro Proxmox host: upgrading across a major PVE release

macpro is a MacPro6,1 running Proxmox VE on 192.168.86.43 (vmbr0, sole physical port enp11s0). It is a standalone node — no cluster, no Ceph, no ZFS — with the root filesystem on pve-root (ext4 on LVM) and guest disks on the pve-data LVM-thin pool. Guest images therefore survive a root-filesystem disaster; the two are separate logical volumes.

Run pve8to9 --full (or the equivalent for the next hop) and drive it to 0 failures before switching any repo. On this host it flagged three things worth knowing again next time:

• systemd-boot meta-package installed but unused. It is a hard failure for the upgrade and safe to remove here: the host boots GRUB via \EFI\proxmox\shimx64.efi, bootctl is-installed reports no, and the ESP has no /EFI/systemd or /loader. Verify those before removing.
• The removable-media bootloader goes stale. GRUB only refreshes /EFI/proxmox/, leaving /EFI/BOOT/BOOTx64.efi frozen at its old build. On Apple firmware the removable path is the fallback that catches a wiped NVRAM entry, so let GRUB own it: echo 'grub-efi-amd64 grub2/force_efi_extra_removable boolean true' | debconf-set-selections -v -u then reinstall grub-efi-amd64. Both ESP paths should end up the same size and timestamp.
• LVM autoactivation on existing guest volumes is disabled from PVE 9 onward; /usr/share/pve-manager/migrations/pve-lvm-disable-autoactivation --assume-yes converts them. Without --assume-yes it prompts and defaults to no.

Reboot into the newest kernel of the old release before switching repos. That proves the bootloader work is sound while rollback is still trivial.

Do not assume the kernel version. PVE 9.2.x ships kernel 7.0, not the 6.14 series visible in older repo metadata. Assert on what the repo actually resolves to rather than a remembered number, and confirm the running kernel appears in grub.cfg before rebooting into it.

Expect the upgrade to be slow on this hardware and do not read slow as broken. The 8→9 dist-upgrade moved ~750 packages and took about five hours. dpkg is fsync-bound and macpro's SATA-attached Apple SSD serialises flushes badly under sustained write load — measured 1.24 s per flush mid-upgrade versus 8.4 ms idle on the same disk, with SMART clean and no ATA errors. The drive is healthy; the latency is load-induced queueing. Measure flush latency only on an idle system, or the number means nothing:

dd if=/dev/zero of=/root/.lat bs=4k count=50 oflag=dsync   # ~0.4s idle is normal here
dd if=/dev/zero of=/root/.lat2 bs=4k count=50              # control: no flush

force-unsafe-io does not help — it governs dpkg's own fsyncs, not the ext4 journal commits that actually stall. Raising the commit interval (mount -o remount,commit=60 /) gives a modest gain and reverts on reboot.

After the upgrade, re-check the apt sources. The PVE 9 migration to deb822 .sources re-enabled the enterprise repo, which 401s without a subscription and breaks every apt update including pve-daily-update.timer. The community post-pve-install.sh script does not disable the new-format file, and leaves the legacy pve-no-subscription.list alongside the new proxmox.sources, producing "configured multiple times" warnings. Fix both:

printf 'Enabled: false\n' >> /etc/apt/sources.list.d/pve-enterprise.sources
rm -f /etc/apt/sources.list.d/pve-no-subscription.list   # superseded by proxmox.sources
apt-get update    # must be clean: no 401, no "multiple times"

That script also installs the subscription-nag patch as a DPkg::Post-Invoke hook and disables pve-ha-lrm/pve-ha-crm/corosync. Both are intended and correct for a standalone node, but the apt hook fires after every dpkg run — move it aside for the duration of a major upgrade.

Nothing needs starting by hand for CI afterwards. Templates 9000–9005 stay templates, pulp-win-ci starts on demand, and pulp-ephemeral-pool@1.service clones a runner on its own. Confirm with a registered runner, not a green unit — see below.

Ephemeral Linux runner pool crash-loops: no free clone id

pulp-ephemeral-pool@N.service restarting every ~30s with status=1/FAILURE means a precondition check failed, not that a job died. Read the reason first:

journalctl -u 'pulp-ephemeral-pool@*' -n 20 --no-pager | grep ERROR

ERROR: no free clone id in 200..202 means every VMID in the pool's clone range is already allocated. The pool clones a golden template into the first free id; stopped clones left over from earlier jobs keep those ids taken. The reaper refuses to clear them:

SKIP 200 — stopped legacy clone lacks a host generation

It only deletes clones it can prove are orphaned, by reading a marker out of the clone's description:

host_generation = sed -n 's/^pulp-runner-generation=\([^;]*\).*/\1/p'
host_scope      = sed -n 's/.*;pulp-runner-scope=\([^;]*\).*/\1/p'

The usual root cause is a half-deployed supervisor. The supervisor stamps that description immediately after cloning:

qm set "$VMID" \
    --description "pulp-runner-generation=${RUNNER_NAME};pulp-runner-scope=${REGISTRATION_API}"

A host running a supervisor from before that change never writes it, so every clone it creates is permanently unreapable

This failure is silent, and that is the dangerous part. The reaper still runs on its timer, still exits 0, and still reports success — it simply classifies every clone as unreclaimable and moves on. "0 orphans reclaimed" is indistinguishable from "0 orphans existed", so the detector reads healthy precisely when it has stopped working. Nothing in its exit code, its timer state, or a systemctl status distinguishes the two.

Never accept the reaper's quiet run as evidence that no orphans exist. Pair it with a positive control that must return non-zero — count the clones it actually considered, not the ones it removed:

journalctl -u pulp-ephemeral-reap.service --since '-1h' --no-pager \
  | grep -cE 'SKIP|REAP|WOULD REAP'     # control: 0 here means it saw nothing at all
qm list | grep -c pulp-ci-ephemeral     # ground truth: how many clones exist

A reaper reporting success while qm list shows stopped clones accumulating is the signature of this bug, not of a healthy pool.

The reaper only ever reports these clones, the pool cannot allocate around them, and no amount of restarting resolves it. Deleting the stale clones by hand buys exactly one cycle; the next clone is born just as unreapable.

Confirm which side is stale before clearing anything:

# ground truth: a healthy clone has a non-empty description
for id in 200 201 202; do
  printf '%s desc=[%s]\n' "$id" "$(qm config $id 2>/dev/null | sed -n 's/^description: //p')"
done

# does the deployed supervisor even know how to stamp it?
grep -c 'pulp-runner-scope' /usr/local/sbin/pulp-ephemeral-runner.sh

All-empty descriptions plus a 0 from that grep confirm the deployed supervisor predates the marker.

Fix it by reinstalling the matched set, never one file. An earlier attempt copied only the supervisor and crash-looped the pool on verify_linux_runner_group.py: error: the following arguments are required: --profile, because the host's verifier and role wrappers were from another generation. proxmox-host-health.sh --install installs every file in its table together, with a backup, so that failure cannot recur:

systemctl mask --runtime pulp-ephemeral-pool@1     # drain: the current job finishes
# a checkout of main on the host, then:
tools/ci/proxmox-host-health.sh --install "$PWD"
systemctl unmask --runtime pulp-ephemeral-pool@1
systemctl reset-failed 'pulp-ephemeral-pool@*'; systemctl restart pulp-ephemeral-pool@1
qm config 200 | grep '^description: pulp-runner-generation='   # the new clone is reapable

Clones made by the old supervisor stay unreapable after the reinstall. Destroy those by hand once, after proving each is idle or finished, then let the pool make a new clone:

mkdir -p /root/ephemeral-configs-backup
cp /etc/pve/qemu-server/<id>.conf /root/ephemeral-configs-backup/
qm stop <id>; qm destroy <id> --purge --destroy-unreferenced-disks 1

A running clone with a registered, idle runner must be deregistered first (ghapp api --method DELETE repos/Generous-Corp/pulp/actions/runners/<id>) so GitHub cannot assign it a job while it stops.

An idle JIT orphan is reclaimed by deregistering it. JIT runner labels are read-only, so a label fence can never strip pulp-build-linux-x64 or pulp-host-macpro. An earlier reaper relied on one and logged dispatch fence could not be proved on every pass. The reaper and the supervisor's shutdown now fence by DELETE, which GitHub refuses for a busy runner. Each step is logged as FENCE <vmid> n/4 (reaper) or JIT fence n/4 (supervisor); see the reaper description above. A SKIP ... JIT fence 2/4: GitHub refused line means the runner was busy, and the next pass retries it.

The trusted role fails closed against the live runner group. verify_linux_runner_group.py --policy trusted requires group pulp-trusted-build to select build.yml, pr-safe-linux.yml, vellum-freeze-check.yml and version-skill-check.yml. The live group does not list pr-safe-linux.yml, so the verifier refuses with group must select exactly the authorized protected workflow set. Until the group or the policy changes, macpro runs only the repository-scoped generic slot. Check before enabling a protected role:

ghapp api orgs/Generous-Corp/actions/runner-groups/3 --jq .selected_workflows
python3 /usr/local/lib/pulp/verify_linux_runner_group.py --gh /usr/local/bin/ghapp \
  --repo Generous-Corp/pulp --group-id 3 --policy trusted

A green pool service is not a ready runner

systemctl is-active reporting active only means the script is executing. The runner is ready when a clone is actually up and registered:

qm list                                          # a clone in the pool range is running
journalctl -u pulp-ephemeral-pool@1.service -n 10 --no-pager
# JIT runner pulp-auto-ephemeral-NNN is visible to GitHub (online, busy=false)

A settled restart counter is likewise not proof. Check NRestarts has stopped climbing and that a clone is running:

systemctl show pulp-ephemeral-pool@1.service -p NRestarts --value

Pool preconditions are layered — fixing one reveals the next

The pool validates several preconditions before cloning and reports only the first that fails, so clearing one surfaces the next and reads like a regression. One host went through all three in sequence:

1. cannot verify runner group policy ... Temporary failure in name resolution — host DNS was down; nothing to do with the runner.
2. ERROR: no free clone id in 200..202 — stale clones, above.
3. ERROR: automatic Linux runners require the Proxmox firewall (... pending changes) — firewall mid-reload. This one clears itself; verify with pve-firewall status reporting enabled/running with no pending changes.

Diagnose from the newest ERROR line each time rather than assuming the previous fix failed.

The Shipyard macOS lane builds Debug — on purpose

.shipyard/config.toml configures the macOS validation lane with -DCMAKE_BUILD_TYPE=Debug. This contradicts CLAUDE.md ("Release is the default") and looks like config drift. It is deliberate, and flipping it to Release would remove the only lane in CI that can see a whole class of undefined behaviour.

On 2026-07-12 it caught a real ODR violation (#6081). snap_to_zero() is an inline function template defined in a header, its body gated by a build-time macro, and a test TU redefined that macro before including the header. Both translation units then emitted the same mangled symbol with different bodies:

build	what happens	result
-O3	each TU inlines its own copy, so each behaves per its own macro	the A/B test appears to work — Release is green, the bug is invisible by construction
-O0	nothing inlines; both TUs emit a weak symbol, the linker keeps exactly one, and both call it	the "disabled" reference silently ran the enabled code — Debug is red

The red test was the mild outcome. The linker's choice is arbitrary: had it kept the other definition, the assertions would have passed while exercising a no-op — a null test, asserting nothing, green forever.

The fix shape is not "delete the redefine". It is: give the variant its own binary, compiled consistently end to end, linking no default-built TU (see test/denormal_null_refgen.cpp). The class is now guarded by tools/scripts/test_odr_macro_gated_headers.py.

A perf gate failing there is a mis-calibrated gate, not a reason to flip the lane

Debug builds are much slower, and CLAUDE.md is right that Debug is the wrong default for most work. The answer is not "Debug everywhere" — it is keep one -O0 lane, and calibrate perf gates for the build they actually run in.

test/test_yoga_layout_bench.cpp is the worked example. Its timing threshold is 0.25 x a 60fps frame (4166.7us), sized at ~11x an M-series Release baseline (~380us) to tolerate a loaded CI box. But in the Debug lane the same 484-node pass takes ~4420us — about 11.6x slower, which eats the entire safety margin. The gate sat permanently at the edge (4421.8us vs 4166.7us, ~6% over) and load merely tipped it. It was never "flaky because the box was busy"; it was a Release-calibrated gate running unoptimized, where it measured the absence of the optimizer, not the cost of layout.

The timing assertion is now #ifdef NDEBUG-gated — the GitHub macOS lane configures Release, so it still runs with real coverage and the right calibration. The structural assertions (allocs_per_pass > 0, frees-match-allocs) still run in every build; they catch real regressions and do not care about the optimizer.

A false red is worse than no gate: it trains everyone to wave away red as "probably the box" — which is exactly how a real bug gets dismissed.

---

The Shipyard macOS lane writes its own reuse records

Live test reuse keys each executable against an earlier build's reuse record, and that record must come from a run on the same toolchain, so the local mac lane keeps its own. [validation.default.overrides.macos] sets reuse_record = true; Shipyard then exports a fresh SHIPYARD_REUSE_RECORD_DIR to each run's stages and files it in its host-local store afterwards, only when a parsing job.json was written (Shipyard's docs/changed-surface-selection.md, "Host-local reuse records"). The lane lists no GitHub credentials, so nothing is uploaded.

• The test stage runs ctest through tools/ci/lane_reuse_record.py test, which adds --output-junit into the record, keeps ctest's LastTest.log, and calls tools/ci/reuse_record.py write --run-kind lane with the run id taken from the directory's name and the same --link-members --object-deps --codemodel --inventory flags build.yml uses. A bounded changed-surface plan is recorded by run_changed_surface_tests.py, including the full run it falls back to when a plan is refused.
• Recording never changes a verdict. The stage exits with ctest's status; a recorder failure, or a record directory that cannot be written, is a warning (the tests still run, and a bounded plan still writes its result receipt); and Shipyard's run log ends with one === reuse-record: ... === line saying whether a record was kept.
• The lane configures with -DPULP_RECORD_LINK_MAPS=ON and with Ninja: the record reads per-object dependencies from Ninja's dependency log, which the Makefiles generator does not keep. tools/ci/require_build_generator.sh removes a build directory left on another generator before configure, since CMake will not switch in place; the first run on such a worktree is a cold build.
• Cost: writing a record on a Debug, examples-on tree takes about 20 s and 7 MB, most of it hashing test binaries and projecting registrations.

Keyed shadow runs measure reuse against those records

[targets.mac.changed_surface_selection.executable_reuse] turns a build-and-test plan on a shadow_compare host into a keyed shadow run (Shipyard's docs/changed-surface-selection.md, "Executable-keyed shadow runs"). It skips nothing: the configured build and full ctest run with their own verdict, and the result reports which tests reuse would have skipped and which of those failed.

• derivation_paths is the key code's import closure plus the adapter. Shipyard reads it only from the protected base and copies those files from there, so a PR that edits any of them, or drops one from the list, runs the full suite. A new import in that code must be added to the list in the same change, or every keyed run fails to derive on the bare copy. A file the key code does not import must stay out: its edits would make every base that moved it read as selector-policy drift. tools/ci/test_executable_keys.py (KeyCodeClosureTests) pins the list to exactly that closure plus the adapter. The test-name pattern the keys share with the per-test receipts shadow lives in tools/ci/always_run_names.py for that reason.
• base_record says where a record's job.json states platform, toolchain, completeness, cleanliness, the full suite and problems; Shipyard binds every qualifying record (up to eight) and the adapter picks the one whose toolchain matches its own configure.
• After the run Shipyard re-runs rederive over the run's copied inputs and writes rederivation-<result sha256>.json into the trial directory. A mismatch is refused, and the second refusal on a host sets PULP_REUSE_LIVE to off.
• Shipyard binds a record by a digest over its files in byte order of their relative paths; tools/ci/executable_keys.py's load_record must digest the same way or every derivation refuses its record.

"Can this PR actually land?" — the two-detector wedge check

shipyard status answers "did my validation pass". It does not answer "can the pull request merge", and on 2026-09-13 the gap between those two questions cost roughly six hours in which no PR in this repository could land. PULP_PREAMBLE_RUNS_ON_JSON named the runner label pulp-preamble; no runner in either registration scope carried it; every build.yml run — pull_request, workflow_dispatch and merge_group — queued forever at its first job. Throughout, Shipyard reported mac: local reachable=true and the blocked PR's ship-state as healthy, with the correct SHA and one attempt. Every word of that was true.

Two detectors now answer the second question, and they are deliberately different from each other in every way that matters.

1. The precondition detector — shipyard landability (and the ship preflight)

Runs on the fleet, as the Shipyard App, before work queues.

shipyard landability --repo Generous-Corp/pulp        # on demand

It reads the required contexts from branch protection, resolves each one to the jobs that render to it, walks those jobs' transitive needs closure, resolves every runs-on through the vars.*_RUNS_ON_JSON indirection, and checks the resulting label sets against the live runner census in both registration scopes.

The needs closure is the whole check. The required macos context is produced by an alias job and by a matrix leg, and every path to it passes through resolve-provider and classify. On 2026-09-13 the context's own job was routed through a variable that was fine and those two were not — so a check that looked only at the producing job would have returned a clean bill of health in the middle of the outage. Six lanes gate one context here.

shipyard ship / shipyard pr run the same check as a preflight and refuse with exit 7 (EXIT_LANE_UNSERVED) when a required context cannot be scheduled. Cost: four API calls cold, zero warm (a 300-second fact cache), plus a local git show of the workflow file.

[landability] workflows in .shipyard/config.toml

The check can only resolve a required context to a lane if it has read the workflow that produces it. Branch protection on main requires six contexts, and the tool's built-in default reads only build.yml — which left the others no_producer: not checked, and reported as a warning that reads identically to a clean result. .shipyard/config.toml therefore names all six producers explicitly:

required context	producing workflow
macos	build.yml
Enforce version & skill sync	version-skill-check.yml
Build + prove + (owner-gated) deploy	wclap-cloudflare.yml
Vellum freeze	vellum-freeze-check.yml
Vellum trusted freeze	vellum-trusted-gate.yml
drift-fast	drift-fast.yml

None of the six is path-filtered under pull_request — wclap-cloudflare.yml keeps its paths: under push on purpose, because a path-filtered required check leaves unrelated pull requests stuck on "Expected — Waiting for status" forever. Add a row here whenever a workflow starts producing a required context, or that context silently stops being checked.

1b. The trigger detector — will the gate ever be requested?

Exit 7 answers can the required contexts be scheduled. That presupposes a run will be requested, and on 2026-09-14 one was not: a pull request on a sibling repository was opened against a feature base, its gate declared on.pull_request.branches: [main], GitHub evaluated that trigger exactly as documented, and created no run. The pull request sat CLEAN with an empty check rollup for 2 h 48 m.

The same command now also classifies links (1)–(3) and (5) of the chain and refuses with exit 8 (EXIT_TRIGGER_UNREACHABLE) — deliberately not 7, because the remedies are disjoint: a 7 is fixed on the fleet, an 8 on the pull request or the workflow file by its author. --allow-unserved-lane does not wave an 8 through; the narrow escape is --allow-unreachable-trigger <workflow>. It costs zero additional API calls: the protection read is the one the lane check already makes, and workflows, base and diff are local.

shipyard landability --pr <N>     # adds up to 3 reads, the third only if needed

Of its eight verdicts only one — triggered with no run on the head yet — is ever resolved by waiting. The rest name the clause that refused and whose fix it is. Two are worth memorising because they are the ones people get wrong:

• base_excluded after a retarget. GitHub auto-retargets a stacked child pull request when its parent merges; that is an edited event, and a workflow without edited in on.pull_request.types never sees it. The fix is a push (synchronize), which the tool prints and never performs.
• wrong_evidence. A workflow_dispatch run checks out the branch tip, not refs/pull/N/merge, so it is not the run a strict: true requirement was written for. Dispatch is the right tool only for re-running an existing pull_request run (POST actions/runs/{id}/rerun), never for producing a missing one.

It refuses only on Unserved — a self-hosted label set that no runner advertises and no fresh host attestation declares. An unreadable census, an unparsable expression, a Starved lane or a missing attestation all warn and proceed: each of those is a statement about the instrument, and an instrument that cannot see must not be able to stop the fleet.

Unserved and Starved look identical and have opposite fixes:

verdict	what it means	fix
Unserved	no runner in either scope carries the labels, and no host declares the lane	restore the runner, or unset the routing variable so the job falls back to the workflow's own literal
Starved	an online, not-busy runner does carry the labels and work is queueing anyway	runner-group access, ephemeral consumption, or a workflows permission — not a runner restore

It never dispatches, re-dispatches, cancels or retries. Decisions contract [default] #4: a runnerless required lane is HELD, never a retry storm. On 2026-09-13 four blind re-dispatches helped nothing and created a second wedge by filling the concurrency group.

Per-run escape: --allow-unserved-lane <label> prints the diagnosis as a warning and proceeds. It exists for a human who has just restored a runner and is waiting for the census to catch up. Never set it from automation.

2. The outcome detector — .github/workflows/landing-watchdog.yml

Runs every 30 minutes on ubuntu-latest with GITHUB_TOKEN, and is redundant on four axes by construction:

axis	precondition detector	outcome detector
host	M1 / M3 / M5	ubuntu-latest
credential	Shipyard App installation token	GITHUB_TOKEN, no admin scope
method	resolve labels, census, attest	is a required context absent or unassigned past T
code	Shipyard, Rust	tools/scripts/landing_watchdog.py, no shared library

The method axis is the one that matters. A precondition model can be wrong about a mechanism it never modelled, and that session found four distinct wedges of which existing models described one. An outcome check cannot be wrong that way: it does not care why macos is missing from a head SHA two hours after the last push. The cost is that it cannot say why — which is what detector 1 is for, and the issue body says so.

Thresholds: a required context absent 45 minutes after the head last moved; a check run unassigned for 30 minutes; a workflow run pending with zero jobs for 15 minutes (the concurrency-holder signature, which reads exactly like runner saturation and is not). It opens, edits and closes one issue labelled ci-landing-wedge, and writes nothing else.

A required context counts as present when the head carries it as a check run or a commit status: Vellum trusted freeze reaches a pull request's head as a status posted by a pull_request_target run, so a check-run-only reader would call it absent on every pull request. The required set is [governance] required_status_checks, all six contexts.

Every tick first replays tools/scripts/fixtures/landing_watchdog_wedge.json against that set as a negative control; it must fire (absent, unassigned, zero_jobs) and must not flag the fixture's healthy pull request. The replay rewrites the fixture's REPLACE_OLD timestamps against the current clock; read raw, every age is zero and the control cannot fire.

Budget: 1 call plus 3 per open PR (check runs, combined status, workflow runs), plus one jobs probe per pending run (at most 3), every 30 minutes, on GITHUB_TOKEN's own per-repository bucket.

Both detectors report their own failure

This is the part that matters most, because the same session found five sensors on this fleet that had been dead for weeks to months — one crashing every tick for roughly three months — and not one of them reported its own death. A measurement aimed at the wrong target does not error; it succeeds and returns empty, and empty reads as a clean finding.

• shipyard landability assesses two synthetic control lanes on every run, against the same census and attestations: a label nothing can serve must come back Unserved, a hosted label must come back Served. If they fail to discriminate it exits non-zero instead of reporting a result it did not measure.
• The watchdog workflow replays a captured fixture of the 2026-09-13 wedge before every live scan and fails the job if the classifier does not fire on it — and equally if it flags the fixture's healthy control PR. It also refuses a scan with an empty required-context list, which would otherwise report every PR clean.
• The host attestation writer (below) probes a launchd label it knows exists and one it knows cannot, and records launchd_readable: false rather than a census of zero when the domain does not discriminate.

The host attestation

A runner census cannot tell an idle just-in-time pool apart from a persistent runner that crash-looped away, because both register nothing. Only the host knows. tartci_host_attestation.py runs every 300 seconds on each of M1, M3 and M5 under com.danielraffel.shipyard.host-attestation and writes ~/.tartci/state/host-attestation.json, which the landability check reads.

Its rule, and the reason it exists: no side may pass a check by delegation unless it names the artifact carrying the other side's verdict, and absence of that artifact is a fault rather than a pass. Before it, tartci's watchdog printed ✓ … runtime health is owned by Shipyard over a service in a crash loop with 3,684 launches and no registration file, while Shipyard knew nothing about that host at all.

Verify deployment on a host — and never assume it:

~/.local/share/pulp-landing/staging/install_host_attestation.sh --verify

The receipt prints the launchd state, the installed writer's SHA-256, the attestation's age, and the SHA-256 of the writer that actually produced the file. A mismatch is reported as SKEW, which is a distinct condition from stale and from dead — the distinction a prior incident collapsed when a behind-the-times checker reported every lane on a healthy host as missing its heartbeat.

Use launchctl print, never launchctl list, when checking any of this by hand: list renders a KeepAlive job in a crash loop as - 0, byte-identical to a healthy idle service, which is exactly how the M5 preamble runner stayed invisible through 3,684 respawns.

What the required macos check covers, per event

event	build	tests
pull_request	yes	no
merge_group	yes	yes
push / workflow_dispatch	yes	yes

The test phase is roughly forty percent of the gate, and every open pull request queues behind the same small pool of self-hosted macOS runners. Running the tests once, in the queue, against the commit that will actually land, keeps main fully protected and returns that time to the pull requests waiting for a slot.

So a green macos on a pull request means it built. Test results arrive when the queue validates it.

The required gate must not need a third-party service at run time

build.yml's Install visual-analysis Python dependencies step installs the declared set (tools/motion/visual/requirements.txt) so the non-skippable visual-python-deps-present ctest stays answerable. The step body lives in tools/ci/install_visual_python_deps.sh <build-dir>, which every job that runs a broad ctest calls after configuring, the sanitizer jobs included; those once ran ctest without it, so the check failed in every ASan, TSan, UBSan and RTSan run. The opt-in Shipyard mac lane calls it too, at the end of its macOS configure stage in .shipyard/config.toml, so the host interpreter that configure records gets the same set. It used to pass --upgrade, which asks PyPI for a newer wheel even when the requirement is already met — turning "pypi.org is reachable from this VM" into a precondition of the required macos check.

On 2026-09-23 that precondition failed. Ephemeral VMs on one host refused CONNECT to pypi.org (Tunnel connection failed: 403 Forbidden), so the step died at 21 of 41 and took out every merge_group batch that happened to land there, while the identical batch passed on a host whose VMs could reach it. The queue stopped merging and the cause looked like a PR defect, because the batch is named after one PR.

The step now checks with pip install --dry-run --no-index first and only reaches the network when the set is genuinely missing. Two consequences worth keeping in mind:

• Pre-provisioning the wheels into the golden image now works. Before this change it did not: --upgrade contacted the index regardless, so a fully provisioned VM still needed PyPI.
• A missing dependency is still fatal, just at the right place. The visual-python-deps-present ctest is the proof, and it names what is absent. Making the install non-fatal instead would produce the failure mode this check exists to prevent — seven ctests silently skipping, which reads as green.

When adding any step to a required gate, ask whether it can fail because a service outside this fleet is unreachable. If it can, that is a fleet-wide outage waiting on someone else's uptime.

Two more downloads followed the same pattern and are now fetched up front:

• pulp-rs crates. The Rust CLI is built by a CMake custom command and tested by cargo test ctests, and a gate VM starts with an empty cargo registry cache, so those commands pulled about 100 crates from crates.io in the middle of Build and Test. A VM that lost DNS for a couple of minutes (Could not resolve host: index.crates.io) then failed macos as a compile or test failure. The Fetch pulp-rs crates step runs cargo fetch on experimental/pulp-rs/Cargo.lock first, with three attempts spaced 20 s and 40 s apart (PULP_CARGO_FETCH_RETRY_DELAY_SECS). Exhausting them only warns, because the build's own cargo call still tries the registry. A repository variable PULP_CARGO_NET_OFFLINE=1 additionally exports CARGO_NET_OFFLINE=true after a successful fetch, so any later cargo download fails loudly instead of quietly reaching out; it is off by default.

• Chrome for Testing. The pinned download retried only on errors curl calls transient, which excludes a resolve failure (exit 6) and a receive reset (exit 56), the two that actually happened. It now passes --retry-all-errors and keeps the SHA-256 check.

• ccache (macOS). The gate VM golden bakes ccache, yet brew install ccache still reached the Homebrew API and one failed attempt failed the gate. The step now uses an installed ccache as is and only installs when it is absent, with three attempts spaced 20 s and 40 s apart (PULP_BREW_RETRY_DELAY_SECS) and a brew update before each retry.

• pip and git in the other required contexts. vellum-freeze-check.yml, vellum-trusted-gate.yml and version-skill-check.yml install PyYAML and fetch commits. Each such command runs through tools/ci/net-retry.sh (PULP_NET_RETRY_ATTEMPTS, default 3; PULP_NET_RETRY_DELAY_SECS, default 20, multiplied by the attempt number), which wraps only the network command and returns its exit status after the last attempt. The trusted merge-group job keeps an inline loop instead, because it runs the protected base commit's tools, which can predate a helper the workflow file names.

tools/scripts/test_build_fetch_resilience.py runs the cargo and ccache steps' real scripts against stub cargo and brew, drives net-retry.sh with a failing stub, checks the cargo step precedes every cargo consumer, rejects any curl in the build job that lacks --retry-all-errors, and rejects a pip install or git fetch/clone in those three required workflows that neither goes through net-retry.sh nor sits in a step with its own retry loop.

A deleted merge-queue branch is a notice, not a fetch failure. hydrate_gpu_provenance_commits.py fetches the event ref first. On a merge_group run that is gh-readonly-queue/..., which GitHub deletes when it drops or re-forms the group while the run keeps going, so git answers couldn't find remote ref. The script reports that as a ::notice::... merge group gone line and falls back to the event commit. Every other fetch error, including any other error on the queue branch, is still a bounded fetch failed warning, and unresolved or non-ancestor provenance still fails the step.

Protected-validation receipt reuse

A merge group may skip rebuilding and retesting macos/linux when the pull request's own run already validated the identical tree. The pull-request run issues a receipt (tools/scripts/protected_merge_receipt.py issue), and the merge group verifies it with the verifier from its protected base commit.

A receipt carries the evidence of the test run it stands in for, not a claimed verdict. It records the CTest selection (label and regex filters), the selected inventory, per-test results from the JUnit report, and the recorded exit status. It is only issued when the Test (non-Windows) step itself ran and succeeded. Verification refuses any receipt that:

• records a failed or unexecuted selection,
• selects a narrowed tier (an include label or regex, or a different excluded label set than the merge group runs), or
• covers less than 80% of the built test inventory.

A pull-request run that skips its tests therefore issues no receipt, and the merge group validates in full. Any receipt that is missing, stale, or rejected also falls back to full validation.

Capability changes that affect Pulp::gpu-audio also publish the exact tested ARM64 macOS install prefix as pulp-gpu-audio-sdk-<sha>-macos. The artifact is created by cmake --install from the Release build after the installed-SDK contract passes and includes a source-bound JSON file manifest. Downstream consumer validation must use that prefix and its source_sha; the build tree or an unrelated SDK directory is not equivalent evidence.

Artifact uploads cross the network after the tests have passed, and the artifact service has reset that connection (ECONNRESET) on gate VMs. The two build-job uploads are therefore treated differently. The ctest-logs-<key> upload is diagnostic and is continue-on-error, so it can never turn a green gate red; its absence on such a run means the upload failed, not that the tests did. The GPU-audio SDK upload is a contract downstream consumers rely on, so a failed first attempt is retried once after a 90-second pause with overwrite: true (the reset can land after the artifact record exists), and the job still fails if that retry fails. The plugins-<key> bundles on macOS and the ios-gate-ok-<digest> marker are also continue-on-error. Nothing reads the bundles; they are a download for people. A lost iOS marker only makes a later run build the iOS gate again. An "Upload progress stalled" on the plugin bundles once ejected a merge batch after every test passed. The protected-validation receipt and the A2T verification stay failing, because merge decisions read them.

The GPU provenance hydration step is bounded

Hydrate bounded GPU provenance commits in build.yml reconnects a shallow checkout with one git fetch --unshallow per candidate event ref. The script gives all of those fetches a shared 300 s budget and treats a fetch that runs past it like an unfetchable ref (a WARN, then its fail-closed ancestry checks decide). The step also carries timeout-minutes: 10. Before this, a stalled fetch held the required macos gate for 43 minutes with no output and no build.