CI Validation

Pulp validates branches on macOS (local), Ubuntu (SSH), and Windows (SSH) before merging.

Setting up a dedicated machine as a persistent CI runner? See self-hosted-runner.md for the walkthrough + first-run gotchas (git-lfs hook conflict, Xcode license, Apple Clang version skew).

Fork routing in YAML is defense in depth, not access control

The macOS runner is chosen by the resolver in build.yml, which normally honors PULP_LOCAL_MACOS_RUNS_ON_JSON and routes to the fast M3/M5 VM pool. For a pull request whose head branch lives in another repository, both self-hosted selectors are ignored and the leg falls through to the GitHub-hosted macos-15 label.

This checked-in routing is useful defense in depth, but it is not a security boundary: pull-request workflow YAML is contributor-controlled and can remove its own guard. The local Macs hold the Developer ID signing keychain and the notary key (~/.config/pulp/secrets/), and PULP_LOCAL_MACOS_RUNS_ON_JSON is a repo variable — variables, unlike secrets, do resolve for fork runs. Without the guard, one "Approve and run" click on a fork pull request could otherwise execute contributor code on the credentialed machines. The real boundary must be an organization runner group restricted to selected trusted workflow refs (or an equivalent trusted dispatcher). Until then, do not add private pools to automatic PR routing.

The leg is rerouted rather than skipped, so a fork contributor still gets a real macOS result on a clean throwaway runner. Note that the required macos check is posted by the local lane, so a fork PR still cannot merge on its own — the maintainer adopts the commits onto an in-repo contrib/* branch and ships that.

Same-repo pull requests, pushes, and workflow_dispatch runs are unaffected. Covered as defense in depth by tools/scripts/test_fork_pr_runner_routing.py (ctest: fork-pr-runner-routing), which runs the resolver the workflow actually embeds; that test does not prove the runners are inaccessible.

The physical Intel lane is advisory and isolated

The Intel Mac mini serves nightly-intel.yml through an ephemeral JIT supervisor, not the persistent required-gate pool. Its selector is exact:

self-hosted,macOS,X64,pulp-intel-native,pulp-host-macmini

PULP_NATIVE_INTEL_RUNS_ON_JSON is intentionally unset until the host passes tools/ci/native-intel-runner.sh --check and a manual dispatch proves a cold workspace can claim and finish the job. While unset, the native job uses macos-15-intel. To pilot without changing the variable, dispatch nightly-intel.yml with use_physical_intel enabled; that boolean maps internally to the exact selector above and cannot target another pool. To roll back after enabling it, unset the variable and redispatch any job already queued for the local labels; GitHub does not reroute an assigned job.

Before starting the supervisor, create a dedicated organization runner group for the Mac mini, restrict it to Generous-Corp/pulp and the protected default-branch .github/workflows/nightly-intel.yml, and set its numeric ID as PULP_NATIVE_INTEL_RUNNER_GROUP_ID in the LaunchAgent. The supervisor refuses the default group and an unset/non-numeric ID, then reads the organization runner-group API and requires the group to contain only this repository and only nightly-intel.yml@refs/heads/main. The GitHub credential therefore needs runner-group read access as well as repository runner administration. Prove that a workflow revision from a PR branch cannot target the group before enabling the repository selector; labels alone are not an access boundary.

The GitHub credential and controller must never share a uid with workflow jobs. The login account runs only the controller and holds gh/ghapp auth. Jobs run as the fixed hidden service identity pulp-ci (uid 499, primary group staff), which owns only its current disposable job root. Its directory-service home is the root-owned /var/empty, its login shell and authentication are disabled, and the worker supplies a new private HOME and TMPDIR for each job. It must not be an administrator and must not be able to write the controller checkout or worker shim. Do not copy the controller account's GitHub credential into it.

Creating that OS boundary is a one-time administrator operation:

1. Confirm uid 499 is unused, then create the fixed non-login identity. These are deliberate directory-service writes, so inspect the first command before continuing; do not choose another uid and weaken the fixed-identity check.

dscl . -list /Users UniqueID | awk '$2 == 499 { print; found=1 } END { exit found ? 1 : 0 }'
sudo dscl . -create /Users/pulp-ci
sudo dscl . -create /Users/pulp-ci RealName 'Pulp native Intel CI worker'
sudo dscl . -create /Users/pulp-ci UniqueID 499
sudo dscl . -create /Users/pulp-ci PrimaryGroupID 20
sudo dscl . -create /Users/pulp-ci NFSHomeDirectory /var/empty
sudo dscl . -create /Users/pulp-ci UserShell /usr/bin/false
sudo dscl . -create /Users/pulp-ci IsHidden 1
sudo dscl . -create /Users/pulp-ci AuthenticationAuthority ';DisabledUser;'
sudo dscl . -create /Users/pulp-ci Password '*'

Do not create /Users/pulp-ci, enable automatic login, or enable remote login for this identity. 2. Put the shared, Apple-signed Xcode at /Applications/Xcode.app and accept its license once. Put a verified, unconfigured GitHub Actions runner archive at /usr/local/share/pulp-native-intel/actions-runner-mini, verified CMake, Ninja, ccache, and Git LFS tools under /usr/local/share/pulp-native-intel, and a prewarmed cache at /usr/local/share/pulp-native-intel/ccache. The commands exposed in bin/ may be relative symlinks into the same trusted root (for example, into a complete CMake bundle); no link may escape it. Recursively set Xcode and this entire trusted root to root:wheel, then remove group/world write bits:

sudo chmod -RN /Applications/Xcode.app \
  /usr/local/share/pulp-native-intel
sudo chown -R root:wheel /Applications/Xcode.app \
  /usr/local/share/pulp-native-intel
sudo chmod -R go-w /Applications/Xcode.app \
  /usr/local/share/pulp-native-intel

The golden runner must never be configured or run in place. Jobs consume the warm ccache read-only with ccache depend mode explicitly disabled through CCACHE_NODEPEND=1 (decision 20); they use an ephemeral writable temp directory and cannot poison the cache. To refresh it, stop the LaunchAgent, build a new cache in a staging directory, install that directory as root:wheel without group/world write, run --check, and only then restart the controller. 3. Install the checked-in lifecycle shim immutably:

sudo install -d -o root -g wheel -m 0755 /usr/local/libexec
sudo install -o root -g wheel -m 0755 \
  tools/ci/native-intel-runner-worker.sh \
  /usr/local/libexec/pulp-native-intel-worker

1. Use sudo visudo -f /etc/sudoers.d/pulp-native-intel to install this narrow rule, replacing daniel only if the controller login is different:

daniel ALL=(root) NOPASSWD: /usr/local/libexec/pulp-native-intel-worker --check, /usr/local/libexec/pulp-native-intel-worker --clean, /usr/local/libexec/pulp-native-intel-worker --run

The root-owned shim accepts only those three fixed operations. Before each job it removes any job-installed crontab, kills leftover uid-499 processes, removes that uid's state from the host's mutable data roots (including macOS temp roots, /Users/Shared, and /Library/Caches), removes the fixed job root, and copies the immutable golden runner into it. The private per-job HOME, TMPDIR, ccache temp directory, runner, and workspace are all uid-owned only for the lifetime of that job. After the runner exits the shim kills leftovers and removes all of them again. Runner executables never survive into the next job; only the root-owned read-only warm ccache does. The worker also requires macOS's com.apple.atrun service to remain disabled and removes any uid-499 at/batch jobs before serving another workflow. JIT registration, runner-group verification, stale-registration removal, and all authenticated GitHub API calls remain in the controller. The shim receives only the ephemeral one-job JIT payload on standard input, drops to pulp-ci, runs run.sh with a clean environment, and contains no GitHub client or persistent credential. Both sides fail closed if the fixed account identity is absent or altered, the worker is mutable or has a write-granting ACL, passwordless delegation is absent, the shared Xcode/tools/cache have unsafe ownership, permissions, or escaping symlinks, or group verification fails. Xcode signature, Gatekeeper, license, xcodebuild, and clang probes run only after dropping to uid 499, so a root-only success cannot mask an unusable worker toolchain. A later identity, ownership, signature, or toolchain validation failure returns a terminal configuration status and leaves the controller alive but its lane offline; launchd therefore does not turn an integrity failure into a restart/retry storm. Restart the controller only after correcting the failed preflight.

The LaunchAgent template is tools/launchd/pulp-native-intel-runner.plist.template. RunAtLoad and KeepAlive restore the controller after the login account logs in. FileVault prevents the startup volume from mounting unattended after a cold power cycle, so the host is not available until a person unlocks it. Keep this lane advisory and do not weaken FileVault or give it any of the required ARM64 gate labels. The controller prefers ghapp when it is installed and otherwise uses its own authenticated rootless gh. The job account receives neither client nor token.

The dispatch-only Linux x64 lane runs on macpro (Proxmox)

Operator-dispatched build.yml runs may route the Linux (x64) leg via PULP_LOCAL_LINUX_RUNS_ON_JSON to ephemeral Proxmox VMs on macpro — a Late-2013 Mac Pro (Xeon E5-1650 v2, 6c/12t, 31 GB) repurposed as a Linux CI host. Automatic PR runs remain on GitHub-hosted Linux until an organization runner group provides the private-pool access boundary. The pool is native x86_64, which the job requires: the lane's earlier ARM64/Tart declaration would have changed its architecture rather than relocating it, silently deleting the only x64 Linux coverage.

That external boundary is a prerequisite, not a workflow TODO: create a dedicated organization runner group containing only the Mac Pro ephemeral runners, grant it to Generous-Corp/pulp only, and restrict workflow access to the protected default-branch copy of .github/workflows/build.yml. Then prove that a pull request changing its own workflow cannot target the group before enabling automatic PR or merge-group routing. Repository variables, event-name conditions, and tests in this repository are not substitutes for that control.

resolve-provider exposes the configured selector separately from the selector authorized for the current event. Its linux_route_reason output is one of explicit-dispatch, security-hosted, or unconfigured-hosted; the Linux matrix provider is derived from the selector that actually resolved. An operator dispatch with a configured selector fails instead of silently falling back to hosted Linux.

ssh macpro                       # 192.168.86.43, Proxmox VE 8.4
qm list                          # 9xxx = pulp-linux-golden* (templates)
systemctl status 'pulp-ephemeral-pool@*'
journalctl -u 'pulp-ephemeral-pool@1' -f

The supervisor and its systemd unit are versioned here as tools/ci/proxmox-ephemeral-runner-linux.sh and tools/ci/pulp-ephemeral-pool@.service; the host copies live at /usr/local/sbin/ and /etc/systemd/system/. The script's GOLDEN= names the template in use — read it rather than trusting a number written down here, since re-baking a warmer golden mints a new id.

Golden + disposable clone. The golden carries the dependency set, prebuilt Skia (external/skia-build/.../libskia.a), a warm ccache, the uncredentialed gh executable used by preamble/alias jobs, and the shared FetchContent source cache that setup.sh consults via PULP_SHARED_FETCHCONTENT_SOURCE_DIR. That last one is not optional: with it empty, every job re-clones three.js (~2.2 GB of history) before it can compile. Each job gets a linked clone (copy-on-write, ~28 s to boot), registers a --ephemeral runner, takes exactly one job, and the clone is destroyed. Nothing accumulates, so nothing needs cleaning — and the cache a job inherits cannot be poisoned by the job before it. This closes the reused-build-dir class outright, which matters because build.yml sets clean: false on self-hosted runners.

Two slots run via pulp-ephemeral-pool@{1,2}.service; systemd restarting a slot is what provisions the next clone. Add a slot by enabling @3 — but check the governor first.

The three clone VMIDs have deterministic network identities: 200..202 map to 192.168.86.251..253 and stable locally administered MAC addresses. Do not return to random clone MACs. Each short-lived MAC retains a DHCP lease after its VM is destroyed, and normal CI volume exhausted the LAN lease pool on 2026-08-02. The GitHub runner registration remains unique per invocation; stable network identity must not become a static Actions runner name.

Resource governance, mirroring the tiers in CLAUDE.md:

• Tier 0 — per-VM cores=4 cpulimit=4 cpuunits=50 balloon=0, hypervisor-enforced. cpuunits=50 is below the default so build VMs yield to the host; balloon=0 pins memory so a build is never squeezed mid-link.
• Tier 1 — /usr/local/sbin/macpro-governor.sh (status / can-start-new). Reserves 2 threads + 4 GB for the hypervisor. Memory is a hard limit; CPU allows 1.5x overcommit. That asymmetry is deliberate: an OOM mid-link yields a truncated object file that reads like a compiler bug, while CPU contention only costs time. Every clone is admitted through it, so nothing can oversubscribe the host.

Rollback: unset PULP_LOCAL_LINUX_RUNS_ON_JSON; operator-dispatched runs then use GitHub-hosted Linux. Automatic PR runs already use GitHub-hosted Linux. runs-on has no live fallback after a dispatched job is assigned, so if macpro is down or its pool is stopped, unset the variable and redispatch rather than waiting.

Registration uses a fine-grained PAT at /root/.config/pulp/secrets/gh-runner-pat (mode 600, root) with only Administration: read/write, minting a single-use registration token per job. That host credential never enters a guest. Jobs that call gh authenticate with the short-lived GITHUB_TOKEN injected by Actions; the golden must not contain a persistent gh login in any supported config or credential store.

Routing the Linux advisory lanes to macpro

Three advisory Linux lanes can run on the self-hosted x86_64 host instead of GitHub's pool. Measured cost on hosted runners, per PR:

Lane	Variable	Hosted wait	Hosted run
GCC compile (core, Linux)	PULP_LOCAL_GCC_RUNS_ON_JSON	63.2m	11.3m
IWYU (Linux, Clang)	PULP_LOCAL_IWYU_RUNS_ON_JSON	4.2m	5.2m
Public headers standalone	PULP_LOCAL_HEADERS_RUNS_ON_JSON	9.7m	2.9m

About 96 job-minutes of hosted load per PR. None is a required check, so a red result here never blocks a merge — which is why they are the right lanes to move first.

Each falls back to its GitHub-hosted label when the variable is unset, so the workflow change is inert until a variable is set. Flip them one at a time and watch a full cycle: runs-on has no automatic fallback once a variable is set, so a lane pointed at a stopped pool queues indefinitely rather than erroring. Rollback is unsetting the variable.

gh variable set PULP_LOCAL_IWYU_RUNS_ON_JSON \
  --repo Generous-Corp/pulp \
  --body '["self-hosted","Linux","X64","pulp-build-linux-x64","pulp-host-macpro"]'

Start with IWYU: it is the cheapest of the three, so a mistake costs the least. Check capacity first — ssh macpro /usr/local/sbin/macpro-governor.sh status — and remember the pool is two slots, so three routed lanes plus Linux (x64) will queue against each other before they queue against GitHub.

Windows runs nightly, not per merge

Windows is billed at 2x on GitHub-hosted runners and gates nothing — no Windows context appears in main's required checks, so the merge queue never waits for it. Measured across 12 runs it was roughly 90% of billable Actions spend, and with max_entries_to_build=2 each merge cycle ran it twice.

It now runs on schedule and workflow_dispatch only. Coverage did not move to nobody: cross-platform-check.yml already builds and tests Windows nightly, and its tracking-issues job find-or-creates a per-platform issue on failure, reopens a closed one, and auto-closes it on recovery. So a Windows regression is caught, filed as a work item, and picked up deliberately — instead of consuming queue capacity that the required checks are waiting behind.

Need Windows on a specific change before the nightly? Dispatch it:

ghapp workflow run build.yml --ref <branch>

This is a deliberate trade: up to ~24 h of latency on a Windows regression, in exchange for merge-queue capacity and spend. Revisit if Windows parity becomes an active workstream rather than a background one.

The FetchContent cache had to point at a real path

build.yml restored and saved three FetchContent paths and none of them ever populated — a different reason on each platform. On Linux the cached ~/.cache/Pulp/... did not match CMake's lowercase ~/.cache/pulp/...; on Windows the cached path carried an extra Cache/ segment versus $LOCALAPPDATA/Pulp/fc; and off Windows the sources never left <build>/_deps anyway, because pulp_configure_fetchcontent_base_dir returns early unless WIN32 (it is a MAX_PATH workaround for MSBuild, not a cache).

The fix caches <build>/_deps — where FetchContent already writes — rather than relocating it. Do not "improve" this by setting FETCHCONTENT_BASE_DIR to a path outside the build tree. PulpWclap.cmake and PulpWebUi.cmake resolve CHOC from <root>/build*/_deps/choc-src; moving it produces ERROR: configured CHOC source not found under build-macos/_deps and fails the required macOS gate. That was tried and reverted.

Worth it because three.js is a 2.2 GB git clone, fetched whenever PULP_BUILD_TESTS and PULP_ENABLE_GPU are both ON — the default on pull_request and merge_group. Measured on an ephemeral Linux runner with an otherwise identical tree: 414 s cold configure against 119 s warm.

If a dependency pin changes and a stale cache is suspected, the key includes hashFiles('setup.sh'); bump that or clear the Actions cache to force a refetch.

Primary: Shipyard

Shipyard is Pulp's primary CI tool. It delivers exact SHAs via git bundles, runs your build/test commands on each platform, and gates merges on per-SHA evidence.

The required macos gate runs the shipyard mac target (.shipyard/config.toml, [validation.default]). Its test step is ctest ... --repeat until-pass:2 --label-exclude "validation|slow|performance|bench|quality-lab" — it excludes the long slow tests, the example plugins' validation format-validators (reported by the path-filtered, currently advisory example-validation lane), and the relative-timing / CPU-budget / benchmark tests (performance|bench|quality-lab), and retries a single flake once so timing-flakes don't redden the gate. The perf/ratio tests are excluded (2026-07-21, mirrors build.yml) because they tolerate steady load but flake under the load variance of the Studio's 2 concurrent build VMs (cap=2) — a perf gate can't live on a cap=2 runner; it belongs in a dedicated cap=1 nightly/perf lane. The full lane model — what runs where, the label taxonomy, and how to route a new test — is docs/guides/test-lanes.md.

The same profile resolves its CMake interpreter through tools/ci/find_python311.py and passes the result as Python3_EXECUTABLE. Apple's command-line tools still expose Python 3.9, which can configure the project but cannot run the tomllib-based decisions-contract tests; the selector uses an installed 3.11+ interpreter or an existing uv 3.12 runtime and fails before the hour-long Debug build if neither exists.

./tools/install-shipyard.sh              # install pinned version
./tools/install-shipyard.sh --status     # compare installed vs pinned
shipyard run                              # validate current branch
shipyard pr                               # create, track, validate, and merge on green
shipyard cloud run build <branch>         # dispatch to Namespace
shipyard rescue <PR>                      # recover a wedged PR
shipyard runner watch --kill-hung-workers # prevent self-hosted runner wedges
shipyard update --check --json            # report installed vs latest

Runner timing metrics

Pulp does not store CI timing history in the Pulp CLI or MCP server. When a checkout uses Shipyard, and optionally tartci for disposable local VMs, Shipyard owns the timing database and query surface:

• Shipyard can import GitHub Actions job timings and local command evidence.
• tartci can optionally emit per-VM runtime records for macOS, Linux, and Windows VM lanes: boot/setup/run/cleanup durations, labels, host, provider, golden/cache hints, outcome, and failure class.
• Shipyard imports those tartci records into its local metrics store and exposes agent-readable summaries, slowest lanes, trend/drift checks, comparisons, and placement advice.

This is mainly for agents watching Pulp CI over time. It gives them enough history to answer "is this runner behaving normally?", "did boot/build time regress?", "which lane should I monitor next?", and "is this worth investigating or just within the usual range?" Humans can use the same commands for high-level platform comparisons, but no observability service is required.

The shipyard metrics commands require a Shipyard build that includes the metrics subcommand. Pulp's pin in tools/shipyard.toml is v0.81.4, which provides it, so no separate binary is needed.

# Enable VM runtime records on tartci hosts or LaunchAgents.
export TARTCI_RUNTIME_MEASURE=1
export TARTCI_RUNTIME_GH_ENRICH=1

# Inspect tartci's local VM timing records.
tartci runtime recent --repo Generous-Corp/pulp --limit 20 --json
tartci runtime summary --repo Generous-Corp/pulp --json

# Import both GitHub Actions and tartci VM timing into Shipyard's metrics store.
shipyard metrics import github --repo Generous-Corp/pulp --limit 50 --json
tartci runtime export --repo Generous-Corp/pulp --since-days 14 \
  | shipyard metrics import tartci --json

# Agent-friendly queries.
shipyard metrics summary --project pulp --json
shipyard metrics slowest --project pulp --limit 20 --json
shipyard metrics watch --project pulp --since 14d --json
shipyard metrics advise --project pulp --json

Use the Shipyard and tartci docs for setup details; this guide only records how Pulp expects agents and contributors to consume the optional integration. Without tartci, shipyard metrics import github and manual/command metrics still work for GitHub-hosted or SSH-backed CI lanes.

Pulp intentionally pins Shipyard in tools/shipyard.toml even if your daily global shipyard is newer. Use shipyard pin bump --to vX.Y.Z for pin updates instead of hand-editing the file; newer Rust Shipyard releases changed the macOS asset shape to a signed/notarized .dmg, and the bump command keeps the version and asset metadata in sync.

The public Pulp installer does not install Shipyard or GitHub CLI (gh). That is intentional: ordinary Pulp users do not need either tool to create, build, run, or upgrade projects. They are source-checkout contributor tools. pulp pr defaults to Shipyard and fails with install/switch guidance if Shipyard is missing; contributors who prefer their own PR flow can set pulp config set pr.workflow github or manual. The github workflow uses gh directly and requires it to be installed and authenticated. Run pulp status to see the effective workflow and local tool health.

Optional local VM routing

Core Pulp development can also use local, disposable VMs through tartci. This is optional: a normal contributor can open a PR and let GitHub Actions run on hosted runners. The value of the local VM setup is faster feedback on trusted Apple Silicon hardware while keeping every job clean-per-run.

The current Pulp routing policy is intentionally kept in parseable TOML at .shipyard/ci-profiles/normal-local-fast.toml instead of copied into this guide. It names the PR, release, coverage, scheduled, and issue-on-failure policies and maps stable target IDs to concrete GitHub runs-on selectors. Shipyard owns orchestration and profile selection; tartci owns the local VM providers, goldens, host caches, and per-host status. Use the upstream docs for details:

• Shipyard profiles explain how profiles and fallback resolution work.
• tartci explains the Tart/QEMU VM lanes, tartci status --json, and tartci profile explain|plan.
• mac-ci-host-setup.md is the Pulp-specific host setup guide for joining the macOS VM pool.

When pulp build, pulp dev, or pulp loop run on a tartci-governed host, the CLI asks tartci for a host-core lease and caps CMake parallelism to the leased job count. Lease-backed builds also run through a POSIX process-group watchdog by default. The watchdog terminates a build that stays over its CPU budget long enough to threaten the shared host; set PULP_TARTCI_WATCHDOG=monitor to log over-budget samples without killing, or PULP_TARTCI_WATCHDOG=0 to disable the wrapper. Operators can tune PULP_TARTCI_WATCHDOG_INTERVAL_SECS, PULP_TARTCI_WATCHDOG_SAMPLES, PULP_TARTCI_WATCHDOG_TERM_GRACE_SECS, PULP_TARTCI_WATCHDOG_CPU_PER_JOB, and PULP_TARTCI_WATCHDOG_PYTHON per host.

Shipping a PR: shipyard pr

shipyard pr is the single "ship this" orchestrator. Agents and humans should route every normal ship cycle through it rather than pairing gh pr create with shipyard ship manually. It:

1. Runs tools/scripts/skill_sync_check.py (hard-fails on missing SKILL.md updates).
2. Runs tools/scripts/version_bump_check.py --mode=apply to bump SDK / Claude plugin / marketplace versions consistently.
3. Commits the bump (if any) as chore: bump <surfaces>.
4. Pushes the branch, creates the PR, and records Shipyard tracking state.
5. Runs cross-platform validate + merge on green.
6. The auto-release workflow tags and publishes binaries on merge.

shipyard pr                              # primary ship path
shipyard pr --base develop/package-manager # ship to a develop branch
shipyard pr --title "..."                # override PR title
shipyard pr --dry-run                    # print the plan without executing

pulp pr is a compatibility wrapper that delegates to shipyard pr by default; it is valid, but guidance should name shipyard pr directly so humans and agents understand where PR tracking state lives. Its github and manual workflows are explicit local opt-outs and do not create Shipyard tracking state.

Direct gh pr create is an emergency/manual bypass only. If it is used, call out that the PR may not appear in Shipyard-managed state until it is reconciled or re-shipped through Shipyard.

Shipyard v0.3.0 workflow surface

Shipyard v0.3.0 adds stateful ship resume, SSH --resume-from staging, and incremental git bundles on top of the basic run / ship / cloud run surface above.

# Resume an interrupted ship
shipyard ship --resume                    # pick up where the last session left off
shipyard ship --no-resume                 # discard stale state and ship fresh

# Inspect in-flight ship state
shipyard ship-state list                  # self-describing inventory: PR, title, URL, tip SHA, dispatched run IDs
shipyard ship-state show <pr>             # full state for one PR
shipyard ship-state discard <pr>          # archive stale state

# Prune old ship state + evidence
shipyard cleanup --ship-state             # dry run — show what would be pruned
shipyard cleanup --ship-state --apply     # prune closed-PR state + aged records

# Fast test iteration on any target
shipyard run --resume-from build          # skip configure+setup, start at the build stage
shipyard run --resume-from test           # skip configure+build, run tests only

# The `windows` / `ubuntu` SSH targets are opt-in per machine and are NOT
# declared in .shipyard/config.toml — the commands below only work once you
# uncomment the matching block in .shipyard.local/config.toml (see
# .shipyard.local/config.toml.example). `shipyard targets list` shows what
# this machine actually has.
shipyard run --targets windows --smoke    # fast Windows-only preflight
shipyard run --targets windows --resume-from test   # ~2 min rerun vs ~15 min full

# Target and config inspection
shipyard targets                          # list configured targets with reachability
shipyard targets test windows             # probe a single target
shipyard config show                      # effective merged config
shipyard config profiles                  # list profiles plus the active one

Ship state lives at <state_dir>/ship/<pr>.json. Shipyard auto-resumes the next time you run shipyard ship on the same PR — it refuses to resume if the PR's head SHA or merge policy changed since the state was written, so a rebase or force-push deliberately forces a fresh ship.

--resume-from works on both local and SSH targets. On SSH targets, Shipyard probes the remote for a marker file proving the previous stage passed for the exact SHA, and skips earlier stages when it finds one.

Incremental bundles — SSH validation now sends only the git delta between the remote HEAD and the target SHA. Typical cycles drop from ~443 MB to a few KB. No configuration needed — Shipyard falls back to a full bundle automatically when the delta would be larger than the full pack.

Keeping fleet Macs on the Shipyard pin (optional)

tools/shipyard.toml pins the Shipyard version every checkout uses, and tools/install-shipyard.sh installs exactly that pin. On a machine that ships PRs every day the pin moves underneath you, and a machine that quietly falls behind — or, worse, drifts ahead after a stray shipyard update — runs a Shipyard that was never validated against Pulp's CI matrix and that disagrees with the SHIPYARD_VERSION every workflow declares.

tools/scripts/shipyard_autoupdate.py converges one machine onto the pin. Nothing about it is required: a public cloner runs install-shipyard.sh once and never thinks about this again. It exists for the local Macs.

v0.81.0 also gives the fleet watchdog an expected-host inventory independent of ephemeral runner names. Pulp declares the MacPro and Mac Mini active in .shipyard/config.toml; absence or insufficient online matches produces expected_host_unavailable. The planned MacBook Air is declared with active = false, so it remains visible without claiming capacity. Matching uses stable label subsets (pulp-host-macpro, pulp-host-macmini, and architecture), never a JIT runner identity. Inspect the combined view with:

shipyard runner fleet-status --repo Generous-Corp/pulp --json

The same report calls out Tart disk-floor and ccache-size admission failures and merge-group Linux jobs left on ubuntu-latest while online self-hosted Linux x64 capacity is idle.

# What would happen, without touching anything:
python3 tools/scripts/shipyard_autoupdate.py --check --json

# Converge now (no-op and silent if already at the pin):
python3 tools/scripts/shipyard_autoupdate.py

# Run it hourly, in the background, per machine:
tools/scripts/install_shipyard_autoupdate.sh
tools/scripts/install_shipyard_autoupdate.sh --status
tools/scripts/install_shipyard_autoupdate.sh --uninstall

Kill switch. Auto-update is on once installed, and off everywhere it is not installed. To stop it without uninstalling:

echo off > ~/.config/pulp/shipyard-autoupdate    # `on` resumes

PULP_SHIPYARD_AUTOUPDATE=0 does the same for a shell or a one-off run, and overrides the file. The file is the one that matters for the background agent: a launchd agent inherits no shell environment, so an env-only kill switch could not reach the thing it is meant to kill.

What it guarantees, and why each one is there:

Behaviour	Why
Converges to the pin, never to latest	The pin is the source of truth; a bare shipyard update tracks latest and strands the machine ahead of the pin (7 minors ahead on 2026-07-16).
Handles both directions	shipyard update refuses to go backwards — it reports update_available: false and exits 0 — so coming back from ahead of the pin goes through install-shipyard.sh.
Reads the pin from origin/main	A dev checkout is usually parked on a feature branch, which may carry an experimental pin. PULP_SHIPYARD_AUTOUPDATE_PIN_REF=worktree overrides.
Never updates mid-job	Swapping the binary under an in-flight ship could corrupt a run. It defers while a Pulp Runner.Worker or a validating shipyard subcommand is alive. The always-on shipyard daemon does not count as busy.
Fails closed	Any probe that cannot answer (ps fails, version unreadable, host offline) means "do not update". The working binary is left in place and the machine converges on a later tick — which is also how an intermittently-offline laptop is meant to behave.
Verifies the outcome	Exit 0 is not proof. The installed version is re-read and must equal the pin, so a declined update or a swallowed checksum failure reports as a failure instead of a false success.
One installer at a time	A hand-run converger and a background tick both writing ~/.local/bin/shipyard is exactly the half-installed binary to avoid; the install step is held under a machine-wide lock.
Silent when nothing changed	The steady state prints nothing. Every decision is still published to ~/.local/state/pulp/shipyard_autoupdate.json.

Host resource governance

Pulp's local Macs are shared: CI validation builds run alongside agent and developer builds on the same host. Two of them melted in July 2026 — one CPU-bound, one memory-bound/OOM — from unbounded builds oversubscribing the machine. A per-host build-resource governor now bounds every build path. It is tiered:

• Tier 0 — always, zero config. The pulp CLI bounds build parallelism to min(cores, RAM_budget / 1.5 GiB) on every build it emits (pulp build/dev/loop, the local-SDK build). No lease store required; override the RAM axis with PULP_BUILD_MEM_BUDGET_MB. tools/scripts/build_parallelism_guard.py rejects a bare --parallel/-j (unbounded) anywhere in the repo, and — on the shared-host surfaces agents copy from (CLAUDE.md, .shipyard/config.toml, .agents/skills/**) — also rejects an explicit but whole-machine count (-j$(nproc) / -j$(sysctl -n hw.ncpu) / --parallel $(getconf _NPROCESSORS_ONLN)): it has a count, so it is not unbounded, but on a shared Mac it claims every core, so N concurrent builds request N × cores and starve each other. The rule is a property of the host, not the command — so the guard fires only where a static scan can prove the surface is shared. It does NOT scan .github/workflows/**, and not because a workflow leg never shares a box: a workflow's runs-on is resolved dynamically (often ${{ fromJSON(matrix.runs_on_json) }} or a repo var) and can point at the shared self-hosted Studios — Pulp's own macOS matrix leg resolves to PULP_LOCAL_MACOS_RUNS_ON_JSON, the Studios that host the required macos gate. A file scan cannot resolve that, so in a workflow the bound is the author's responsibility: route a self-hosted macOS leg through tools/ci/governed-build.sh (as build.yml's intel-canary compile, examples-validation.yml, web-plugins.yml's gpu-audio-macos job, and format-baseline-diff.yml now do). The steer everywhere is pulp build / tools/ci/governed-build.sh, which take their -j from the governor.
• Tier 1 — tartci per-host lease governor. On a host running a tartci lease store, builds and VM runners acquire a weighted core+memory lease before starting; admission is min(core-budget, memory-budget), so a build that would exhaust RAM is refused even when CPU is free. Each host derives a role budget from tartci host-profile:
• dedicated-builder — a machine whose job is CI builds (largest core + memory budget).
• dev-overflow — a shared dev machine that also takes overflow CI, running its VM lane at non-gate priority so it never starves the required macos gate.
• light — a low-resource/travel host with a small budget.
• Tier 2 — Orchard fleet VM placement (shadow phase). Fleet-level placement, wired but placing nothing yet. See the tartci runbook's Orchard section.

The mac local lane is the one that historically escaped the CLI: Shipyard's local backend runs the .shipyard/config.toml build string directly on the host and does not pass through the pulp CLI. Every build stage in that config — default, parser, and smoke — is therefore wrapped by tools/ci/governed-build.sh, which acquires a tartci build lease sized from the host profile, exports the granted -j, runs the build as a child process, and releases the lease on exit. When tartci is absent (a build VM or a plain checkout) or the lease is denied (host saturated), it falls back to the Tier-0 bound — it never fails the build and never piles onto a saturated host. (The smoke lane previously used a raw --parallel $(getconf _NPROCESSORS_ONLN) and so ran whole-machine on the shared Mac while the required gate validated alongside it; it now takes a governed share like the other lanes.) The version-controlled overrides.windows recipes keep a fixed --parallel 4 instead: they run under PowerShell with no wrapper-path or $(…) assumptions, and unbounded MSBuild link parallelism trips LNK1104 on ARM64.

pulp status reports the active tier with a Build governance: Tier N (…) line. Host-side setup and the deeper lease/role/memory-axis mechanics live in the tartci repo (scripts/leases.py, scripts/host_profile.py, tartci host-profile / tartci leases).

Lane timeouts — and why a timeout looks like a broken PR

[targets.<name>] timeout_secs in .shipyard/config.toml bounds how long a validation lane may run. The mac lane is 7200s (2h) as of 2026-07-27, raised from 3600s.

The reason the value matters more than it looks: when a lane hits it, Shipyard reports

✗ Validation failed. PR #NNNN not merged.
    Target:  mac
    Error:   Validation timed out

with no per-target diagnostics. That is indistinguishable from a genuinely broken branch, and the natural response — re-push, or start debugging the diff — is wrong in both directions. Always read the lane log before believing the verdict:

tail -40 "~/Library/Application Support/shipyard/logs/<job-id>/mac.log"
grep -c "error:" "~/Library/Application Support/shipyard/logs/<job-id>/mac.log"

A log that ends mid-build at some percentage with zero error: lines was killed by the clock, not by your code.

Two properties worth knowing when reading a timeout:

• Queue wait is not charged against the budget. The clock starts when the lane starts, so a job that sat pending for 40 minutes still gets its full window. Check started_at vs completed_at in queue.json to tell queueing apart from a slow build.
• Warm build dirs are the difference between passing and timing out. On one loaded afternoon, a small change against a warm dir finished in 41 min and passed, while a broad core/view change was killed at 98% after 62 min and again at 67% after 113 min on a cold dir. Whether a branch lands should not depend on that, which is why the ceiling was raised rather than left to look like flakiness.

If you are running heavy work on the same machine — a VM, a parallel build — it competes with the lane directly. Shutting it down is a legitimate first move when a lane is timing out marginally.

Validation Profiles

Shipyard validates from a profile (shipyard run --pipeline <name>). Pulp's .shipyard/config.toml defines three:

Profile	When to use	What it runs
default	Most PRs. The lane every cross-platform target gates on.	Full setup → configure → build → test. Examples ON. Excludes the slow ctest label.
parser	PRs that only touch runtime-import parser code.	Same stages with PULP_BUILD_EXAMPLES=OFF; tests filter to --label-include parser-import. Skips plugin validators (auval / pluginval / clap-validator) and the broader format-adapter smoke surface.
smoke	Quick downstream-scaffold check after dependency or install-layout edits.	Configure + governed build only (both cmake --build steps go through tools/ci/governed-build.sh); runs the SDK-smoke export against a downstream scaffold.
gates	Version-bump / skill-sync gate scripts.	tools/scripts/skill_sync_check.py + tools/scripts/version_bump_check.py in report mode.

shipyard config profiles lists what is installed locally and which one is active.

Auto-selecting the parser profile

tools/scripts/validation_profile_select.py classifies the current diff and prints parser or default:

# Default: diff HEAD against origin/main
shipyard run --pipeline "$(python3 tools/scripts/validation_profile_select.py)"

# Explicit diff base
shipyard run --pipeline "$(python3 tools/scripts/validation_profile_select.py --base origin/develop)"

# Operate on a literal file list (e.g. piped from gh pr diff)
gh pr diff <PR> --name-only \
  | python3 tools/scripts/validation_profile_select.py --paths-from -

# JSON envelope (profile + matched + unmatched)
python3 tools/scripts/validation_profile_select.py --json

The script returns parser only when every changed path falls inside the explicit parser-only scope (the standalone tools/import-design tool, tools/import-validation scripts, the packages/pulp-import-ir package, test/fixtures/imports/**, the parser test files in test/, the core/view/.../design_import* family, and the import-runtime JS). Any path outside that set forces default — the safety bias is toward broad validation.

To opt out for an individual run, pass --pipeline default explicitly.

Cache-warming runs on main

build.yml triggers on push: branches: [main] in addition to pull_request / merge_group / workflow_dispatch. That run gates nothing — it exists solely to publish the GitHub-hosted Linux/Windows ccache and FetchContent caches that PR runs restore from.

It is needed because of how GitHub's cloud cache is scoped: a cache entry written by a PR run is visible only to that PR's own ref, so PR runs can never warm each other. Only a non-PR run on the default branch writes an entry every subsequent PR can read. Without the push trigger the Save … steps are unreachable and the matching Restore … steps are a permanent miss.

Each trigger runs a deliberately different slice of the matrix:

	PR run	merge_group run	push: main cache run
macOS matrix leg	yes	yes	no — omitted by resolve-provider
Linux matrix leg	yes	no — PR-head result is reused	yes (publishes the cache)
Windows matrix leg	no — see below	no — see below	yes (publishes the cache)
windows-{msvc-release,midi2,ble}-gate	no — see below	no — see below	no
required macos alias	yes	yes, after the macOS build completes	no
Writes to GitHub's cloud cache	no	no	Linux + Windows only

The macOS leg is dropped because macOS builds on the self-hosted Macs that serve the one required check in this repo, and those machines keep ccache and FetchContent on local disk between jobs. Scheduling a macOS leg on a push would put the required gate's runners under load to save a cache that is never uploaded — strictly a cost. For the same reason the two Save … steps are scoped runner.environment == 'github-hosted' && runner.os != 'macOS', which is narrower than the restore side on purpose.

Push runs are also exempt from cancel-in-progress: they share the refs/heads/main concurrency group, so cancelling a superseded one would kill its cache-save step exactly when main is busiest. PR runs still cancel.

The classify job diffs an event-dependent base (tools/scripts/resolve_classify_base.py): a PR diffs github.event.pull_request.base.sha, a push diffs github.event.before. On a push, origin/main resolves to HEAD itself and the diff is always empty — so a docs-only merge is indistinguishable from a core merge, and the run never skips. A docs-only merge to main now correctly skips the whole matrix.

Windows is gated by the merge queue, not by the PR head

Windows is advisory and runs entirely on GitHub-hosted runners, and a single run carries four Windows jobs: the Windows (x64) matrix leg plus the windows-msvc-release-gate, windows-midi2-gate, and windows-ble-gate compile gates. The repository draws all of those from one fixed pool of concurrent GitHub-hosted jobs, shared with every other workflow.

That pool is the scarce resource, and Windows is by far its largest consumer. With a handful of PRs open at once, advisory Windows work fills nearly every slot and the required hosted check — Build + prove + (owner-gated) deploy, on ubuntu-latest — cannot get a runner. The merge queue then holds its entry in AWAITING_CHECKS until the ruleset's check-response timeout expires, evicts it, and nothing lands at all. macOS is never implicated: it runs on the self-hosted Macs, which sit outside the hosted pool.

So Windows runs where it supplies independent value without blocking every merge:

• nightly cross-platform validation — catches Windows regressions as follow-up work without consuming the merge queue's hosted slots.
• push: main — publishes the Windows ccache.
• workflow_dispatch — explicit reruns when you want Windows early.

A PR head keeps macOS on the self-hosted Macs and Linux on GitHub-hosted Linux for fast signal. The Linux leg therefore does consume hosted capacity; the security boundary above deliberately prevents automatic PRs from reaching the private Mac Pro VMs. The advisory windows alias job short-circuits to green on pull_request — without that it would fail closed looking for a matrix leg that deliberately did not run.

The trade is later Windows feedback. Dispatch build.yml manually against the branch when a Windows-touching change needs proof before merge.

tools/scripts/test_windows_runner_policy.py locks this in: it executes resolve-provider's matrix resolver for each event and asserts hosted Linux and Windows are absent on merge_group, Windows remains reachable through workflow_dispatch, and macOS plus Linux still run on the PR head.

The required macOS alias never long-polls on the merge queue

Branch protection requires the stable macos alias so local/overflow provider changes cannot rename the gate. On merge_group, the build matrix contains only the real self-hosted macOS leg and the alias depends on that completed build. The hosted alias therefore runs only long enough to report the result; it does not occupy a hosted slot while polling through the native compile/test.

An alias is the last job in the run: it waits for the matrix leg and reports the outcome. Starve it of a runner and it never starts, so the run never reaches a terminal state. A run that never terminates holds its concurrency group, and build.yml sets cancel-in-progress on that group — so the next push's run sits at status pending with zero jobs and never dispatches. The pull request is then wedged: no checks, no failure, nothing to re-run. It survives further pushes and clears only by cancelling the older run by hand.

The tell is a Build and Test run whose status is pending and whose jobs array is empty, while the sibling workflows from the same trigger dispatched normally. If you hit it:

# find the older, non-terminal run on the same ref and cancel it
ghapp api -X POST repos/Generous-Corp/pulp/actions/runs/<old_run_id>/cancel

An advisory lane must never be able to strand a pull request, which is why these two moved off the hosted pool. tools/scripts/test_windows_runner_policy.py asserts all three aliases resolve through the toggle and that none is pinned to a bare ubuntu-latest.

The preamble can run from a checkout below /Volumes/Workshop. Inline Python started with python3 - resolves the current directory before executing its stdin script, so a wedged checkout volume can freeze the routing probe or the macos outcome poll even though the jobs API response is already available. RUNNER_TEMP is not a safe boundary here: on self-hosted Studios it can also live below /Volumes/Workshop. Those two helpers first cd /tmp (/private/tmp on macOS's system volume); the routing helper then uses GITHUB_WORKSPACE only as the absolute resolver-script argument. Keep new inline Python in a PULP_PREAMBLE_RUNS_ON_JSON job behind the same stable-cwd boundary. tools/scripts/test_preamble_python_stable_cwd.py enforces the complete set.

Routing contract (checked)

Every *_RUNS_ON_JSON repo variable is a lane: it names the labels a class of jobs is dispatched to. The intended lane→label mapping lives in tools/scripts/runner_topology.json, and tools/scripts/runner_topology_check.py reconciles it against the live repo variables and the live registered runners.

The contract is the source of truth for lane→label. Label values quoted inline elsewhere in this guide are illustrative and can lag; the contract plus its checker are authoritative, because they are the only pair that is verified.

The failure this prevents

GitHub does not validate runs-on. A job that asks for a label no runner carries is not an error — it is queued, forever. There is no warning, no annotation, no failed check. The only symptom is jobs piling up while the pool looks saturated, which is indistinguishable from "we're just busy".

That makes a mis-pointed routing variable silent. A relief valve routed into a black hole is worse than no relief valve: it reports healthy and relieves nothing, and the queue it was supposed to drain grows behind it.

The same class of bug already bit the busy probe in build.yml: reading actions/runners needs Administration: Read, which the default GITHUB_TOKEN lacks, so the probe 403s and falls back to BUSY=0 — silently disabling overflow. Nothing about either failure is visible without asking.

What the checker asserts

Check	Failure it catches
drift	A variable was edited without updating the contract (or vice versa). The variable is a reviewed artifact, not a blind edit.
black-hole	The lane's labels are satisfiable by no runner.
degraded	The only matching runners are offline — the host may just be asleep. A warning, not an error: a different failure from a label nobody owns.
undeclared	A live routing variable with no lane in the contract.
hosted-unknown	A runs-on value that is not self-hosted and not a known GitHub image — i.e. a typo, which queues forever.
must-unset	A paid Namespace overflow variable is set (cost guard).

Label matching is subset containment: GitHub dispatches to a runner only if it carries every label in the array. A lane requesting [self-hosted, macOS, ARM64, pulp-build, pulp-build-studio] is not served by a runner carrying only pulp-preamble, however much the labels overlap.

Three runner states, not two

online / offline is not the whole story. Tart runners register JIT and ephemeral (tools/ci/tart-runner.sh, tart-runner-linux.sh): they exist only while a job runs and vanish when idle. For those lanes an empty registry proves nothing — the provisioner may simply have nothing to do.

So ephemeral lanes are judged on service history instead: has any job been dispatched to this exact label set inside the lookback window? A label set with no runner and no recent service has nothing provisioning it, and that is a black hole. This distinction is load-bearing — without it the release lanes, which are idle between releases, would be flagged as broken every sweep.

Service history is gathered from the workflows that consume the lane, found by scanning .github/workflows for the variable. A repo-wide "last N runs" sweep is not a time window: on a busy repo the newest 100 runs were measured covering well under an hour, so any lane used less often than that — every release lane — would be condemned on every sweep. Scoping to the consuming workflow makes 20 runs reach back months for a handful of API calls. The scan is also lazy: a lane with a live runner costs zero API calls.

Honest limits. This check proves a lane can be served; it does not prove jobs are being served well. It will not catch a runner that is online but wedged, a lane that is slow rather than dead, a capacity shortfall (labels resolve, queue still grows), or a black hole in a runs-on hard-coded in a workflow rather than driven by a variable. An ephemeral lane whose consuming workflow has not run inside the lookback window yields no evidence and is reported as a black hole — a false positive that is deliberately biased loud, on the grounds that a silent relief valve is what caused this in the first place.

Where it runs, and why

• runner-topology-check.yml — hourly cron on ubuntu-latest, opening and auto-closing a tracking issue. The invariant is about live fleet state, so it can break with no commit at all: a runner is decommissioned, a host renamed, a variable edited in the web UI. A PR gate would never see any of that. It runs GitHub-hosted deliberately — a check that queues behind the saturated pool it is auditing is no check.
• runner-topology-selftest (ctest) — the diff-shaped half: contract well-formedness and the reconciliation logic. No network, so it runs on every PR for free and never adds an API call to the required macOS gate.

The checker exits 2 when live state cannot be read, distinct from pass (0) and violation (1), so a missing token scope fails loudly instead of reporting a false green.

Changing a lane

Edit the variable and its lane in runner_topology.json in the same change — the drift check exists to make that atomic. Then:

# Reconcile against the live fleet (uses ghapp locally — the App token bucket).
python3 tools/scripts/runner_topology_check.py --mode=report

# Advisory (never fails), useful while iterating.
python3 tools/scripts/runner_topology_check.py --mode=hint

macOS overflow routing (Plan B)

Namespace is OFF (cost). We build macOS on local Macs + GitHub-hosted only. PULP_NAMESPACE_BUILD_MACOS_RUNS_ON_JSON is kept UNSET, so the Namespace overflow described here never fires — it's a documented break-glass option, not the active path. The required gate is the clean-per-job JIT VM pool on M3 + M5 (PULP_LOCAL_MACOS_RUNS_ON_JSON), selected by the pulp-gate-fast label. M1 keeps the generic pulp-build-vm label as a rollback/non-required lane but cannot randomly win the serial required gate. Do not repurpose the Namespace var to point at self-hosted runners (see CLAUDE.md "Runner priority").

Read the live variable, not this page's defaults. A routing var describes reality; build.yml's || fallback is only what happens when the var is unset. The two disagree: build.yml defaults macOS overflow to GitHub-hosted ["macos-15"], while the live variable is the local-only sentinel and disables overflow. Confirm before reasoning about a route:

gh variable list -R Generous-Corp/pulp | grep RUNS_ON_JSON

Live routing state

Do not maintain another selector table here. The reviewed source is tools/scripts/runner_topology.json; the live source is the GitHub repo variables. Reconcile the two with:

PATH="$HOME/.config/tartci/ghapp-shim:$PATH" \
  python3 tools/scripts/runner_topology_check.py --mode=report

As of 2026-07-31, the required macOS contract selects the fast M3/M5 JIT class, overflow is contracted to the local-only sentinel, and Namespace variables remain unset. Exact labels and hosts belong only in the JSON contract.

When the local self-hosted Mac runner is saturated, build.yml's resolve-provider job can route new PR or workflow-dispatch macOS legs to the configured generic overflow target. Live policy sets that target to the local-only sentinel, so overflow is disabled. Namespace remains an explicit, paid break-glass option and is never selected automatically.

Precedence (highest first, resolved per dispatch):

1. Operator override — gh workflow run build.yml --field macos_runner_selector_json='"<label>"'. Always wins.
2. Overflow — for PR and workflow-dispatch events, an idle registered local runner keeps the local route. Otherwise BUSY >= PULP_LOCAL_MAC_OVERFLOW_THRESHOLD (default 2) selects PULP_OVERFLOW_BUILD_MACOS_RUNS_ON_JSON. The local-only sentinel disables this branch; an unset variable restores the hosted macos-15 fallback.
3. Local default — PULP_LOCAL_MACOS_RUNS_ON_JSON. For the current label set, read the lane in tools/scripts/runner_topology.json — that file is checked against the live fleet, so it cannot drift the way a value quoted here can.

Tuning knobs (repo variables):

Variable	Default	Purpose
PULP_LOCAL_MAC_OVERFLOW_THRESHOLD	2	BUSY count that triggers overflow. Raise when Plan A's 2nd local runner lands.
PULP_LOCAL_MAC_RUNNER_LABEL	pulp-gate-fast	Label the busy probe filters runners by. It must match the required gate class so rollback-only M1 capacity cannot suppress overflow.
PULP_OVERFLOW_BUILD_MACOS_RUNS_ON_JSON	["macos-15"] when unset	Generic overflow selector JSON, or the bare sentinel local-only to keep work local.

Disabling overflow (the live state):

gh variable set PULP_OVERFLOW_BUILD_MACOS_RUNS_ON_JSON \
  --body local-only --repo Generous-Corp/pulp

The next PR's resolve-provider keeps the macOS leg on the local selector.

Inspecting routing decisions: resolve-provider's stderr prints a one-line summary, e.g. resolve-provider: macOS route = overflow (BUSY=2 >= 2); selector = "namespace-profile-generouscorp-macos". Find it via the GitHub Actions UI under the resolve-provider job's log, or:

gh run view <run-id> --log --repo Generous-Corp/pulp | grep "macOS route"

Manual overflow / rescue is still available via shipyard rescue <PR> and remains useful for in-flight PRs that queued before the overflow logic kicked in. With Plan B in place, manual rescue should be needed much less frequently.

pulp overflow — operator surface

tools/cli/cmd_overflow.cpp wraps the three repo variables behind a discoverable CLI:

# Show current routing state (local target, overflow target, threshold,
# plus self-hosted runner registration if visible to the default token):
pulp overflow status

# Turn overflow on (defaults to free GH-hosted "macos-15"):
pulp overflow enable
pulp overflow enable --to '"macos-15"'

# Turn overflow off — every macOS leg goes to the local target.
# In-flight cloud jobs continue to completion; only new dispatches change.
pulp overflow disable

# Read / set the BUSY threshold (default 2; set to 1 for single-runner setups):
pulp overflow threshold
pulp overflow threshold 1

pulp overflow disable writes the local-only sentinel; deleting the variable would restore the hosted default. It does not cancel in-flight cloud runs. To force a currently-routed-to-cloud PR back to local, use pulp macos retarget --pr N --to local (see "Per-PR macOS retargeting" below).

Per-PR macOS retargeting (pulp macos)

For the case where automatic overflow picked the "wrong" pool — e.g. you want to push a specific PR to Namespace for paid-fast turnaround, or pull a queued GH-hosted job back to the local Mac because local just freed up — use the build-macos.yml workflow + the pulp macos CLI:

# Switch PR's macOS leg to the local self-hosted Mac, freeing the GH-hosted slot:
pulp macos retarget --pr 1910 --to local

# Pay to skip the queue (Namespace billable, fast parallel):
pulp macos retarget --pr 1910 --to namespace

# Force GH-hosted macos-15 (free, slower):
pulp macos retarget --pr 1910 --to github-hosted

# See where the current macOS check is routed:
pulp macos status --pr 1910

pulp macos retarget cancels any in-flight macOS-bearing workflow_run for the PR and fires a fresh build-macos.yml dispatch on the chosen runner. Branch protection's required macos check is satisfied by whichever workflow most recently produced that check name, so retargeting supersedes the previous macOS leg without re-running Linux/Windows.

build-macos.yml is independent of build.yml's matrix — they share check names but not workflow_runs. The matrix workflow continues running Linux/Windows as usual; only the macOS leg is replaced.

Workflow inputs (visible in gh workflow run build-macos.yml --help):

Input	Default	Effect
runner	local	Routes to PULP_LOCAL_MACOS_RUNS_ON_JSON
runner=namespace	—	Routes to PULP_NAMESPACE_BUILD_MACOS_RUNS_ON_JSON
runner=github-hosted	—	Routes to "macos-15" (free GH-hosted)
target_ref	(workflow's ref)	Branch / SHA to build

Opportunistic reroute daemon

tools/scripts/macos_reroute_watcher.py is a long-running watcher (intended as a launchd agent on the self-hosted Mac) that automates the "when local frees up, claw back queued GH-hosted jobs" pattern. It polls every 30 seconds:

1. Is the local Mac runner idle? (process-based detection via ps; no admin token needed.)
2. Is there a queued Build-and-Test workflow_run whose macOS job has macos-15 (or nscloud-* / namespace-profile-*) labels — i.e., dispatched to cloud but not yet picked up?

When both conditions hold, the watcher invokes pulp macos retarget --pr N --to local to cancel the cloud dispatch and rerun on local with warm caches. A 5-minute flap-guard prevents repeatedly bouncing the same PR.

Install (one-time per host):

# Copy the template into LaunchAgents, substituting your Pulp checkout path:
sed "s|\$PULP_REPO|$PWD|g" \
  tools/launchd/pulp-macos-reroute-watcher.plist.template \
  > ~/Library/LaunchAgents/com.danielraffel.pulp.macos-reroute-watcher.plist

launchctl load ~/Library/LaunchAgents/com.danielraffel.pulp.macos-reroute-watcher.plist

# Logs:
tail -F ~/Library/Logs/pulp/macos-reroute-watcher.log

Run by hand for testing:

python3 tools/scripts/macos_reroute_watcher.py --interval 30 --log-level DEBUG

Stop:

launchctl unload ~/Library/LaunchAgents/com.danielraffel.pulp.macos-reroute-watcher.plist

The watcher is safe to run alongside the overflow probe in build.yml — they cooperate. The probe decides where to dispatch initially; the watcher opportunistically reroutes dispatches that landed on cloud while local was busy, once local frees up before the cloud runner has picked up the job. If the cloud runner has already started, the watcher takes no action.

Self-hosted runner operations: prevent, recover, maintain

Shipyard v0.55.0+ is the minimum pin for the full self-hosted-runner operational toolkit, and Pulp pins v0.56.2+ so rescue, update, and shipyard wait pr have REST fallback paths when GraphQL is rate-limited. The commands are discoverable from shipyard --help and replace the legacy planning/scripts/runner-watchdog.sh + manual reinstall workflow.

# Recover one PR whose required macOS check is wedged or stale
shipyard rescue <PR>                      # cancel queued runs + redispatch to github-hosted
shipyard rescue <PR> --rerun-failed       # also re-arm cancelled/failed runs
shipyard rescue <PR> --dry-run            # preview without acting
shipyard rescue --all-stuck               # repo-wide stuck-run sweep
shipyard rescue <PR> --to github-hosted   # explicit destination provider

# Prevent future wedges on a self-hosted runner host
shipyard runner watch --kill-hung-workers # implies --fix; pair with launchd/systemd

# Keep the installed Shipyard CLI current
shipyard update --check --json            # report installed vs available
shipyard update                           # apply latest stable
shipyard update --to v0.56.2              # pin or roll back to Pulp's minimum
shipyard update --dry-run                 # plan only

# Wait after handoff/rescue without depending solely on GraphQL
shipyard wait pr <PR> --state green       # REST fallback as of v0.56.2

Off-fleet queue-age watchdog (runner-health-check.yml)

.github/workflows/runner-health-check.yml sweeps every 30 minutes and opens a tracking issue when a lane has stopped serving work. It runs on ubuntu-latest on purpose: a guard that lives on the fleet dies with the fleet, so the one outage it exists to report would be the outage that silences it. It is the symptom-level backstop under the recovery tooling above — shipyard rescue and runner watch fix a wedge you already know about; this tells you a wedge exists.

Why queue age, and not a runner-label check. The macOS lanes are JIT/ephemeral: a runner registers with GitHub only while it serves a job. So "zero runners carry label pulp-studio-01" is both the healthy-idle state and the dead-lane state, and nothing on GitHub's side can tell them apart. A label-satisfiability probe therefore false-alarms every idle night and gets muted within a week. Queue age is the observable that separates alive from dead, and it is cause-agnostic — it catches causes nobody has enumerated yet, not just the one that happened last time.

Why it stays quiet on a busy afternoon. A deep queue on a healthy pool is normal: the measured baseline on this repo under normal load is a median queue age of 5 min, an oldest of 31 min, and 3 runs past 30 min. A naive "queued > 30 min" rule alarms on that. So an alarm requires two independent conditions:

1. Age — the job has waited past alarm_minutes (default 45, roughly 1.5x the observed healthy maximum).
2. Liveness — its lane shows no sign of life: nothing with comparable labels is in_progress, and nothing with comparable labels has started since the job queued.

The liveness condition carries the false-alarm load, which is what lets the age threshold stay tight enough to detect a dead lane within 45–75 minutes. A saturated pool keeps its runners visibly busy, so it stays quiet at any queue depth; one runner grinding on a 90-minute job is alive, not dead; an idle fleet has nothing queued and so says nothing. Only "work piling up with nothing serving it" alarms. Findings between 30 and 45 minutes appear in the run summary only, never on the issue.

The issue is edited in place each sweep and closes automatically on recovery — the same open/update/auto-close contract as the release watchdogs (see release-watchdog.md). The report names the labels the stalled jobs asked for, so a human sees which lane is sick.

Thresholds and analysis live in tools/scripts/queue_age_watchdog.py, tested by tools/scripts/test_queue_age_watchdog.py — which pins the measured baseline above as a must-stay-quiet regression case, so a future threshold edit that would re-introduce afternoon false alarms fails at PR time.

# Tune or dry-run a sweep by hand
gh workflow run runner-health-check.yml -f dry_run=true
gh workflow run runner-health-check.yml -f alarm_minutes=60

# Replay a recorded snapshot offline (no API calls, verdict pinned to capture time)
python3 tools/scripts/queue_age_watchdog.py --snapshot snapshot.json

Diagnosing a VM lane: idle looks exactly like dead

The macOS and Linux VM lanes are JIT — a runner registers with GitHub only while serving one job, then deregisters. A runner census therefore cannot tell a healthy idle lane from a dead one: "zero runners carry pulp-build-vm" is both states at once. Do not conclude a lane is dead from actions/runners, and do not build a label-satisfiability alarm on a JIT label — it would fire every idle night. Satisfiability is a valid check only for the persistent bare-metal Studios.

The signal that separates alive from dead on a JIT lane is queue age (not queue depth — 40 queued runs with a 5-minute median is healthy churn from many concurrent agents). Baseline measured on a healthy busy pool (2026-07-16): median queue age 5 min, oldest 31 min, 3 runs over 30 min. A naive ">30 min = broken" threshold alarms on that healthy pool; calibrate above it.

To check a lane host-side:

# Non-interactive ssh does NOT source .zprofile, so it lacks /opt/homebrew/bin
# and will falsely report "tart is not installed". Always use a login shell:
ssh <host> 'zsh -lc "launchctl list | grep -E \"tart-runner|qemu-runner\""'

Last-exit 0 means the supervisor is healthy and the lane is alive regardless of what the runner census says.

Runner agent crash-loops with exit 75 (the /usr/sbin PATH trap)

Symptom: a runner LaunchAgent shows last-exit 75 (EX_TEMPFAIL) and crash-loops under KeepAlive; its log says lease denied … rc=2; no VM ever boots; jobs queue on that lane forever.

Cause: tartci's host_profile.py shells bare sysctl — which lives at /usr/sbin/sysctl — to read hw.ncpu / hw.memsize. macOS launchd agents run with a minimal PATH, and the generated plist's PATH omits /usr/sbin:

/Users/<u>/.config/tartci/ghapp-shim:/opt/homebrew/bin:/usr/local/bin:/Users/<u>/.local/bin:/usr/bin:/bin

sysctl raises FileNotFoundError → host_profile.py exits 1 → the tartci lease governor cannot compute a memory budget → it denies every lease (failing closed, which is correct) → no VM ever starts.

Diagnose this FIRST — before suspecting tart, the network, or auth:

launchctl list | grep -E 'tart-runner|qemu-runner'   # last-exit 75 = this bug
# then inspect EnvironmentVariables:PATH in the agent's plist for /usr/sbin

Fix: append :/usr/sbin:/sbin to the plist's PATH and reload the agent. Exit goes 75 → 0, the log turns to lease acquired … cores=6 mem_mb=8192, a VM boots, and the queue drains. The failing set is exactly the /usr/sbin-missing set: agents that already carry it are exit 0.

TART_HOME is per-host by design

VM homes differ intentionally per machine:

Host	TART_HOME	Storage
m3 (Studios)	/Volumes/Workshop/VMs	external SSD
m1, m5	~/VMs	internal SSD

Always set TART_HOME explicitly when invoking tools/ci/*.sh on a VM host. Those scripts carry contradictory hardcoded defaults — most assume /Volumes/Workshop/VMs, while reap-stray-vms.sh and setup-ci-host.sh assume $HOME/VMs. The worst failure mode is silent: on m3, reap-stray-vms.sh defaults to $HOME/VMs, which is empty on that host, so the stray-VM reaper inspects the wrong universe, reaps nothing, and exits 0 reporting success — a permanent no-op that looks like a pass.

The principle: a default is an undeclared name wearing a trench coat. On a VM host an unset TART_HOME should be a loud hard error naming the fix, never a guess. The repo holds RULES; the host holds VALUES — per-host truth belongs in the tartci host profile, not a repo constant.

These three traps share one shape, covered in the ci skill under "The unifying invariant — no name without a heartbeat": a name is trustworthy only if an automated process dereferences it on a schedule and alarms on failure.

Off-fleet merge-stall watchdog (merge-stall-check.yml)

.github/workflows/merge-stall-check.yml sweeps every 30 minutes and opens a tracking issue when PRs are merge-ready but not merging or the GitHub merge queue has stopped advancing. It runs on ubuntu-latest for the same reason as the queue-age watchdog: the wedge it catches lives in whatever presses the merge button (Shipyard's per-host queue-tick), so an on-fleet guard would die with the thing it watches.

The gap it closes — the opposite shape from the queue-age watchdog. The queue-age guard alarms on a dead runner lane: jobs sitting queued because runners died. This one alarms on the inverse: every required check green, nothing queued, and still nothing merging — the signature of an auto-merger silently held in reap-only mode. No job-level signal sees "everything is green and nobody is merging"; the only observable is a population of merge-ready PRs that stays merge-ready and unmerged. (Motivating incident: the repo went ~4 hours with 34 PRs open and nothing merging while every check was green.)

The alarm predicate. A PR trips only when ALL hold:

1. Required checks green — every check in the repo's REQUIRED set. That set is read from branch protection at runtime, not hardcoded; if the token cannot read protection rules it falls back to the complete documented main set: macos, Enforce version & skill sync, Build + prove + (owner-gated) deploy, Vellum trusted freeze, and Vellum freeze.
2. mergeStateStatus in {CLEAN, BEHIND} — GitHub's own merge verdict. DIRTY (conflicts), BLOCKED (a required check red/missing/review pending), and UNSTABLE (a non-required check still moving) are excluded — those wait on something real, not on the merger.
3. Auto-merge enabled — the signal that a machine, not a human, owns pressing merge. A green PR without it is waiting on a person and must not alarm.
4. Merge-ready longer than the threshold (default 45 min), measured from the completion time of the last required check to go green — a real duration, independent of the sweep cadence.

Why two consecutive sweeps. A single snapshot can misread — a per-PR REST poll of merge state gets rate-limited and returns false CLEAN/BEHIND readings under load, which is exactly how the incident state looked wrong. Collection therefore uses one GraphQL call for every open PR's mergeStateStatus (tools/scripts/merge_stall_watchdog.py), and on top of that a PR must satisfy the full predicate on two consecutive sweeps before it is issue-worthy: the first qualifying sweep records it as pending (run-summary only), the second promotes it to alarm. A normal in-flight PR that merges within a tick never reaches the second observation, so it never trips. The cross-sweep memory is the set of stuck PR numbers, persisted as a workflow artifact — crash-safe, held by GitHub independently of this repo or any host.

Merge-queue predicate. Once the queue is non-empty, the watchdog also reads its GraphQL MergeQueue.entries head and the latest merge_group Actions run. It alarms when the head has waited at least 30 minutes and no new merge-group batch has started in that window. The age window is already the anti-flap period, so this condition alarms on its first observed sweep. The report names the head PR, queue depth/state, last batch start, and any required check that is missing, queued, in progress, or red. This catches the incident where a required hosted alias waited behind advisory work while matching self-hosted build capacity was idle.

The issue is edited in place each sweep and closes automatically once no PR is stuck merge-ready — the same open/update/auto-close contract as the release watchdogs (see release-watchdog.md). A degraded API sweep never closes an existing tracker; only a complete snapshot can prove recovery.

Analysis and the predicate live in tools/scripts/merge_stall_watchdog.py, tested by tools/scripts/test_merge_stall_watchdog.py — which pins the must-stay-quiet cases (young PR, DIRTY, BLOCKED, no auto-merge, single-sweep blip) as regressions so a future edit that would make the guard cry wolf fails at PR time. Queue-specific tests pin empty/young/recent-batch cases quiet and an old head plus old batch as an immediate alarm.

# Dry-run a sweep by hand (log findings, do not touch the issue)
gh workflow run merge-stall-check.yml -f dry_run=true
gh workflow run merge-stall-check.yml -f threshold_minutes=60
gh workflow run merge-stall-check.yml -f queue_threshold_minutes=30

# Replay a recorded snapshot offline (no API calls, verdict pinned to capture time)
python3 tools/scripts/merge_stall_watchdog.py --snapshot snapshot.json --prev-state state.json

GraphQL quota fallback for PR sweeps

The gh pr ... --json and gh pr merge paths can consume or require GitHub's GraphQL quota. That quota is separate from the REST core quota and can hit zero while REST still has thousands of calls available.

When a broad PR sweep hits GraphQL exhaustion, switch the sweep to REST instead of waiting:

gh api rate_limit --jq '.resources | {core, graphql}'
gh api repos/OWNER/REPO/pulls/PR
gh api repos/OWNER/REPO/commits/SHA/check-runs?per_page=100
gh api repos/OWNER/REPO/actions/jobs/JOB_ID/logs

For a PR already verified green through REST, merge through the REST endpoint:

head_sha=$(gh api repos/OWNER/REPO/pulls/PR --jq '.head.sha')
gh api repos/OWNER/REPO/pulls/PR/merge \
  -X PUT \
  -f sha="$head_sha" \
  -f merge_method=squash \
  -f commit_title='subject (#PR)'

If the merge endpoint returns 405 Base branch was modified, refresh the PR state and check runs through REST, recompute head_sha, then retry once only if the refreshed head SHA and green status are still the values you intend to merge. This is a transport fallback, not a validation bypass: do not merge around real CI, coverage, sanitizer, or review failures.

Use shipyard rescue when a PR is otherwise ready but blocked by queued, cancelled, or failed runner contexts caused by a self-hosted-runner wedge. It is the PR-side recovery path and avoids the old failure mode where cancelling queued runs left required checks stuck as failure.

Use shipyard runner watch --kill-hung-workers on the runner host itself. It auto-cancels stale queued runs and kills hung Runner.Worker processes through Shipyard's safe recovery sequence: snapshot, SIGTERM, grace period, SIGKILL, child reaping, partial-build quarantine, Listener verification, and optional wait for GitHub status to flip. Its JSON output uses runner.watch envelopes with event=auto_kill_worker and phase values of attempt, killed, failed, or no-pid-found.

Use shipyard update instead of the old ad hoc curl install.sh | sh path once a machine already has Shipyard installed. Pulp still records the canonical repo pin in tools/shipyard.toml; shipyard update --check --json is the machine-local drift check, while shipyard pin bump --to vX.Y.Z is the repo pin-change workflow.

Required Merge Process (All Agents)

Every change to main must go through this workflow — no exceptions:

1. Branch — work on feature/* or fix/*, never directly on main
2. Ship — run shipyard pr to create and track the PR, validate on macOS + Ubuntu + Windows, and merge on green
3. GitHub Actions — PR also triggers build+test CI on all 3 platforms (redundant safety net)

The ci skill (.agents/skills/ci/SKILL.md) captures this as the authoritative trigger list — natural-language phrases like "ship this", "push a PR", "we're done", and "run CI" all route through shipyard pr.

Legacy: pulp ci-local

tools/local-ci/local_ci.py is the previous CI controller. It remains available as a fallback while the Shipyard path finishes replacing it, but it is scheduled for removal.

TL;DR

• pulp ci-local run queues the current HEAD in a machine-global queue shared by every worktree on that Mac.
• pulp ci-local run <branch> queues that branch tip's exact commit SHA, not the launching checkout's HEAD.
• pulp ci-local run --smoke queues a fast clean install/export preflight instead of a full test run.
• The queue serializes jobs, not targets. One CI job runs at a time, but its requested targets (mac, ubuntu, windows) run in parallel inside that job.
• Mac runs locally. Ubuntu and Windows run over SSH against repos you already cloned on those machines.
• Remote targets validate the exact queued git SHA, not "whatever the branch points to later". The runner uploads that SHA as a git bundle before validation, so full-matrix checks do not depend on the host already seeing your latest branch tip.
• pulp ci-local status shows the active runner, pending jobs, SSH/VM reachability, and live per-target state for the running job. pulp ci-local bump <job-id> high moves a pending job forward.
• queueing now prints the submission root, current cwd, config path/source, and per-target host preflight before a job is recorded
• queueing fails fast if you launched from the wrong git root or selected an SSH target that is currently unreachable with no fallback, unless you explicitly override that safety check
• While a job is running, pulp ci-local status also shows live per-target state such as mac=pass, ubuntu=pass, windows=running.
• Quiet long-running targets now emit runner heartbeats, so status can show heartbeat=..., idle=..., and liveness=quiet|stuck even when the underlying toolchain has not printed a new line recently.
• If you queue a newer SHA for the same branch, targets, and validation mode, older pending work is superseded automatically instead of sitting behind it forever.
• pulp ci-local logs <job-id> --target windows tails the saved per-target log from the machine-global CI state dir, so you do not need ad hoc SSH just to see whether a target is building or testing.
• pulp ci-local evidence [branch] shows the last-good exact-SHA target evidence already recorded for a branch, so you can keep earlier same-SHA passes instead of rerunning them blindly.
• pulp ci-local cleanup shows reclaimable local-CI disk usage without deleting anything; --apply is blocked while jobs are running.
• pulp ci-local cloud workflows lists the GitHub Actions workflows that the local CI control plane knows how to dispatch, plus which runner providers each one supports.
• pulp ci-local cloud run <workflow> [branch] dispatches a GitHub Actions workflow deliberately when workflow semantics or neutral-host confirmation matter more than the local queue.
• pulp ci-local cloud status shows the latest tracked GitHub Actions dispatches that this machine has launched; pulp ci-local status includes the same recent cloud summary alongside local queue state.
• Persistent local CI hosts now keep a prepared root per target + validation so a narrow same-SHA rerun can reuse earlier work instead of rematerializing from scratch.
• If a runner is interrupted, the queued job keeps its last-known per-target state so you can see what already passed before deciding whether to rerun everything or just the remaining target.
• Jobs submitted through pulp ci-local are globally queued, and validation itself now takes a per-host lock on macOS/Linux plus a Windows host mutex, so old validate-build.sh runs wait instead of colliding.
• SSH targets receive a per-job git bundle before validation. That keeps exact-SHA validation working even when the host validates from a stale local mirror instead of GitHub directly.
• Windows SSH jobs execute from short detached worktrees under C:\pulp-ci, and stale worktree metadata is pruned automatically before reruns.
• If a stale runner leaves behind an old Windows validator, the next drain pass now targets that specific remote validator PID for cleanup before starting new work, and status keeps the cleanup result visible.
• For Windows SSH validation, choose the configured target whose non-interactive PowerShell context resolves git, cmake, and ctest. Keep those host aliases local to your environment; shared repo docs should describe the selection rule, not your personal machine names.
• Reuse is a persistent-host feature for local macOS and SSH-backed/self-hosted hosts. Ephemeral cloud runners should keep the default clean path unless a later policy explicitly opts them in.
• Truly raw ad hoc ssh, cmake, or custom background processes still bypass coordination until they are stopped or migrated.

Why local instead of cloud

Pulp has GitHub Actions workflows for CI, but running them on every branch costs money. Local CI is free and faster for iterative development — you get results in minutes from machines you already own or have running locally. Cloud CI remains available for release branches, public PRs, and the narrow cases where you need workflow-level or neutral-host confirmation.

Cloud orchestration is now available through the same control plane:

• pulp ci-local cloud workflows
• pulp ci-local cloud run <workflow> [branch]
• pulp ci-local cloud status [dispatch-id|latest]

That cloud surface is intentionally separate from the local queue. run, check, ship, enqueue, and drain still operate on the exact-SHA local/SSH queue. cloud run dispatches GitHub Actions explicitly and tracks the result beside local CI state instead of pretending a hosted workflow is just another local target.

Namespace is now wired into the deliberate cloud companion path for both docs-check.yml and build.yml. The normal day-to-day default remains local-first: macOS runs locally, while deliberate cloud dispatches can route Linux/Windows through Namespace and keep macOS local unless you opt into a one-off cloud macOS selector.

How it works

When you run pulp ci-local, it:

1. Queues a job in a machine-global queue shared by every worktree on that Mac
2. Prints the exact queue intent first: submission root, cwd, config path/source, and remote-host preflight
3. Runs only one queue drain owner at a time, so separate agents do not stampede the same Mac and VMs
4. Validates locally on Mac via ./validate-build.sh --ref <sha>
5. For each SSH target in config.json: uploads a per-job git bundle, injects that exact SHA into the configured repo on the host, then validates it there
6. If an SSH target is unreachable, it tries to start the corresponding UTM VM, waits for it to boot, then retries the SSH connection
7. Drains queued work on login or wake if you install the launchd agent

Mac validation always runs. SSH targets are skipped if disabled in config.

GitHub Actions companion

Use the cloud subcommands when you want GitHub Actions as the orchestrator, not when you want another exact-SHA local queue job:

pulp ci-local cloud workflows
pulp ci-local cloud defaults
pulp ci-local cloud history
pulp ci-local cloud compare build
pulp ci-local cloud recommend build
pulp ci-local cloud run build feature/my-branch
pulp ci-local cloud run build feature/my-branch --provider namespace
pulp ci-local cloud run build feature/my-branch --provider namespace --macos-runner-selector-json '"namespace-profile-big-apple"'
pulp ci-local cloud run build feature/my-branch --provider namespace --macos-runner-selector-json '"nscloud-macos-tahoe-arm64-6x14"'
pulp ci-local cloud run docs-check feature/my-branch --provider namespace --wait
pulp ci-local cloud run docs-check feature/my-branch --provider namespace --runner-selector-json '"namespace-profile-big-apple"'
pulp ci-local cloud namespace doctor
pulp ci-local cloud namespace setup
pulp ci-local cloud status
pulp ci-local cloud status latest --refresh

Important constraints in the current phase:

• cloud run dispatches by branch name, not by a detached exact SHA
• cloud dispatch records are persisted under the same machine-global CI state directory as local results, but they do not enter queue.json
• local status remains fast and local-first; it shows the latest tracked cloud summaries without hitting GitHub unless you explicitly run cloud status --refresh
• cloud defaults shows the effective workflow/provider defaults plus where the current selector values came from (local config versus repo-variable fallback)
• cloud history shows recent tracked cloud runs with saved timing plus any configured estimated cost line items
• cloud compare <workflow> rolls up observed provider medians for a workflow from tracked run history
• cloud recommend <workflow> suggests a provider from recorded cloud history; it is intentionally conservative and uses observed medians instead of hardcoded guesses
• cloud status now reports Namespace runtime/machine-shape truth when the run was launched on Namespace and nsc can see the matching instances
• tracked cloud runs now persist queue-delay and elapsed-duration timing so the later comparison view can answer "how long did GitHub-hosted vs Namespace take?" from saved run history instead of rough notes
• estimated cost output is opt-in via local config; every estimate is labeled estimated; verify provider pricing
• if the provider CLI does not expose billing totals, Pulp keeps reporting runtime and machine shape instead of inventing invoice truth
• if a Namespace dispatch dies in resolve-provider before any matrix leg starts, inspect the GitHub run annotations first; provider billing or control-plane failures are a different problem from repo or workflow breakage
• build.yml now accepts runner_provider and routes Linux and Windows through the selected provider; macOS is omitted from the cloud build by default so it can stay local-first
• Default provider for PR checks is controlled by the GitHub repo variable PULP_DEFAULT_RUNNER_PROVIDER. Set it to namespace to route all PR checks through Namespace runners (faster, parallel). Set to github-hosted to use GitHub-hosted runners (free tier, queued). The workflow_dispatch input overrides this for manual runs. To change the default:

  # Switch to Namespace (recommended for faster CI)
  gh variable set PULP_DEFAULT_RUNNER_PROVIDER --body "namespace"
  
  # Switch back to GitHub-hosted
  gh variable set PULP_DEFAULT_RUNNER_PROVIDER --body "github-hosted"
  

• build also accepts one-off leg overrides: --linux-runner-selector-json, --windows-runner-selector-json, and --macos-runner-selector-json; that means you can keep the normal Linux/Windows Namespace + macOS local default and still do an explicit one-off macOS Namespace build without changing saved config
• those one-off selector overrides can be either: a Namespace profile label such as "namespace-profile-generouscorp-macos", or a direct Namespace machine label such as "nscloud-macos-tahoe-arm64-6x14"
• docs-check accepts an explicit --runner-selector-json override, for example "namespace-profile-default" or ["self-hosted","linux"]
• if no explicit selector is passed, docs-check falls back to github_actions.workflows.docs-check.providers.<provider>.runner_selector_json in local config when present, then to the repo variable PULP_NAMESPACE_DOCS_CHECK_RUNS_ON_JSON for the Namespace provider
• build can take Linux/Windows Namespace selectors from github_actions.workflows.build.providers.namespace.linux_runner_selector_json and .windows_runner_selector_json in local config, and the workflow also supports repo-variable fallbacks PULP_NAMESPACE_BUILD_LINUX_RUNS_ON_JSON and PULP_NAMESPACE_BUILD_WINDOWS_RUNS_ON_JSON
• macOS Namespace is an explicit validation path, not part of the default cloud build: if you want to test macOS on Namespace, provide --macos-runner-selector-json, or set github_actions.workflows.build.providers.namespace.macos_runner_selector_json in local config, or PULP_NAMESPACE_BUILD_MACOS_RUNS_ON_JSON
• make sure that selector points at a real macOS-capable Namespace profile: GitHub job names alone do not guarantee the underlying OS, and a Linux-backed profile can still satisfy the runs-on label while executing the leg on Linux instead of macOS
• if you want macOS to stay local-first by default, leave the macOS selector unset in shared config and repo variables, and pass --macos-runner-selector-json only for one-off validation runs
• for the Namespace path, install the nsc CLI and run nsc login first before trying to route work there; that is the recommended operator setup path for this pilot
• SSH/VM target topology and Namespace provider setup stay separate: targets.* still configures local/SSH validation hosts, while Namespace provider routing lives under the GitHub Actions workflow/provider config and the cloud namespace helper commands

Fast-CI vs full-CI (build.yml)

The Build and Test workflow has two test trajectories without forking the YAML:

• Fast-CI runs on pull_request events. The ctest invocation excludes BOTH the validation and slow CTest labels, dropping the longest-running tests so PR cycle time stays tight. Examples that carry LABELS slow today (defined in test/CMakeLists.txt):
• cmake-ios-auv3-configure — fresh-cache ~3 min iOS-leg configure
• cmake-pulp-add-binary-data-encoder / cmake-pulp-install-layout
• pulp-test-hot-reload, pulp-test-scripted-ui, pulp-test-scan-cache, pulp-test-scan-blacklist (filesystem-mtime sleep loops)

• pulp-test-sync, pulp-test-sync-race-hammer, pulp-test-events-timer-helpers (race + timer hammers; also covered under sanitizer.yml's TSan lane)

• Full-CI runs on push to main, the nightly schedule, and workflow_dispatch. Only the validation label is excluded — every slow-labelled test runs before code lands on the release lane.

Both paths satisfy the branch-protection-required macos / advisory linux / advisory windows alias gates because the alias jobs read each matrix leg's outcome via the GitHub API.

Tagging a new test as slow

Add LABELS slow either to a single test's set_tests_properties, or to a Catch2 binary's catch_discover_tests(... PROPERTIES LABELS slow) so every discovered test inherits the label:

add_test(NAME my-expensive-cmake-smoke COMMAND ...)
set_tests_properties(my-expensive-cmake-smoke PROPERTIES
    LABELS "smoke;slow"
    TIMEOUT 600)

add_executable(pulp-test-my-suite test_my_suite.cpp)
target_link_libraries(pulp-test-my-suite PRIVATE pulp::view Catch2::Catch2WithMain)
catch_discover_tests(pulp-test-my-suite PROPERTIES LABELS slow)

Multi-label lists are preserved as lists by Pulp's Catch discovery wrapper:

catch_discover_tests(pulp-test-my-suite
    LABELS "audio;slow;quality-lab")

All three labels reach CTest, so ctest -L and ctest -LE selection does not depend on label order.

Rule of thumb for slow: a test consistently >5 sec on at least one platform, OR a sleep-bounded smoke (file-mtime, hammer race, message-loop bound) whose value lies in soak coverage rather than per-PR feedback. Anything covered by sanitizers.yml or another scheduled lane is a strong candidate.

Demoting a fast test to slow (or vice versa)

ctest --test-dir build -L slow -N lists every test currently tagged slow. To move a test in or out of the fast-CI surface, add or remove the slow label in test/CMakeLists.txt (or the appropriate subdir CMakeLists) and reconfigure. There's no separate registry to keep in sync.

Switching a job's runner without a code change

Provider-switchable build, release, coverage, and sanitizer decisions use tools/scripts/resolve_runs_on.py. Their runner can be flipped between GitHub-hosted, Namespace, and local self-hosted by setting a repository variable — no workflow edit or PR.

General selector precedence

For each target handled by resolve_runs_on.py, the resolver checks:

1. A workflow_dispatch input (if present on the workflow) — one-off override.
2. The target's repository variable (the PULP_*_RUNS_ON_JSON values below).
3. For the build matrix only: PULP_DEFAULT_RUNNER_PROVIDER + the provider's selector var (PULP_NAMESPACE_* or PULP_LOCAL_*).
4. A hard-coded default label (e.g. macos-14, ubuntu-24.04, macos-15).

When a variable below is unset, the workflow resolves to that target's reviewed hard-coded default. Defaults may be updated when a hosted toolchain changes; for example, UBSan uses macos-26 to avoid the invalid-vptr diagnostics produced by the Xcode 16.4 image. Setting one variable moves one job. Nothing more.

Coverage is stricter than the build matrix. It reads explicit workflow_dispatch inputs and PULP_COVERAGE_*_RUNS_ON_JSON, not PULP_NAMESPACE_BUILD_*. If coverage moves local, use a dedicated ephemeral label such as pulp-coverage-vm-macos; do not point coverage at pulp-build, pulp-build-vm, or the warm macOS gate pool.

Global default (build.yml matrix only)

Variable	Effect	Example
PULP_DEFAULT_RUNNER_PROVIDER	Default provider for Linux and Windows legs of build.yml. One of github-hosted | namespace | local. Falls back to github-hosted when unset.	gh variable set PULP_DEFAULT_RUNNER_PROVIDER --body "namespace"

build.yml — Linux / Windows / macOS legs

Variable	Provider	Example
PULP_NAMESPACE_BUILD_LINUX_RUNS_ON_JSON	Namespace	gh variable set PULP_NAMESPACE_BUILD_LINUX_RUNS_ON_JSON --body '["namespace-profile-generouscorp"]'
PULP_NAMESPACE_BUILD_WINDOWS_RUNS_ON_JSON	Namespace	gh variable set PULP_NAMESPACE_BUILD_WINDOWS_RUNS_ON_JSON --body '["namespace-profile-generouscorp-windows"]'
PULP_NAMESPACE_BUILD_MACOS_RUNS_ON_JSON	Namespace (optional)	gh variable set PULP_NAMESPACE_BUILD_MACOS_RUNS_ON_JSON --body '"namespace-profile-generouscorp-macos"'
PULP_LOCAL_MACOS_RUNS_ON_JSON	Fast local macOS ARM64 JIT VM pool; see the live table under "macOS overflow routing"	gh variable set PULP_LOCAL_MACOS_RUNS_ON_JSON --body '["self-hosted","macOS","ARM64","pulp-build","pulp-build-vm","pulp-gate-fast"]'
PULP_OVERFLOW_BUILD_MACOS_RUNS_ON_JSON	Overflow is disabled live with local-only. Unset → build.yml falls back to GitHub-hosted ["macos-15"]; another reviewed selector re-enables overflow.	gh variable set PULP_OVERFLOW_BUILD_MACOS_RUNS_ON_JSON --body 'local-only'
PULP_LOCAL_LINUX_RUNS_ON_JSON	Dispatch-only Linux x86_64 Proxmox VM pool; automatic PR routing requires the external runner-group boundary above	gh variable set PULP_LOCAL_LINUX_RUNS_ON_JSON --body '["self-hosted","Linux","X64","pulp-build-linux-x64","pulp-host-macpro"]'
PULP_LOCAL_WINDOWS_RUNS_ON_JSON	Local Windows ARM64 QEMU pool	gh variable set PULP_LOCAL_WINDOWS_RUNS_ON_JSON --body '["self-hosted","Windows","ARM64","pulp-build-windows","pulp-host-macstudio"]'

The Linux and Windows label sets include a pulp-host-* label that pins the lane to one machine, so the supervisor serving them must carry it too — GitHub selects a runner only when it carries every requested label. Declare the machine once, in the LaunchAgent, via --host-tag / PULP_RUNNER_HOST_TAG (tools/launchd/pulp-{tart-runner-linux,qemu-runner-windows}.plist.template). A supervisor that cannot resolve one refuses to register rather than contribute a runner that is online, idle, and selectable by nothing. Declared tags live in tools/scripts/runner_topology.json.

Advisory macOS selectors

Variable	Precedence and behavior	Example
PULP_ADVISORY_MACOS_RUNS_ON_JSON	Repository variable, then hosted macos-15. No dispatch/provider override.	gh variable set PULP_ADVISORY_MACOS_RUNS_ON_JSON --body '["self-hosted","macOS","ARM64","pulp-advisory-macos"]'
PULP_ADVISORY_GPU_MACOS_RUNS_ON_JSON	Repository variable only; unset skips the proof. No dispatch/provider override.	gh variable set PULP_ADVISORY_GPU_MACOS_RUNS_ON_JSON --body '["self-hosted","macOS","ARM64","pulp-advisory-gpu"]'

These selectors are resolved by tools/scripts/resolve_advisory_macos_runner.py. The resolver fails closed if an operator points an advisory workflow at pulp-build* or pulp-preamble*, or if any configured self-hosted selector lacks an explicit pulp-advisory-* identity. Hosted strings are an explicit reviewed allowlist (macos-14, macos-15, macos-26, and macos-latest), so a typo fails during resolution instead of waiting forever for a nonexistent runner. Until a separately governed advisory supervisor is installed, leave the ordinary advisory selector unset (hosted macOS) and the GPU advisory selector unset (proof skipped). This repository does not use Orchard for placement; Shipyard, tartci, and GitHub runner labels are the complete control path.

.shipyard/ci-profiles/normal-local-fast.toml is the repo-local, read-only policy mirror. Its PR-only github.windows-x64-runtime target records the stable windows-2022 functional lane, while the shared github.windows-x64 target records the windows-latest coverage/scheduled lane. The current Shipyard profile planner does not apply these selectors to a dispatch; build.yml remains authoritative. Inspect the profile when reviewing policy, then verify the workflow input and repository-variable precedence before changing live routing.

Do not put ordered fallback chains directly into GitHub Actions. GitHub receives one runs-on selector per job; Shipyard/tartci must resolve "Mac Studio, then M5/blackbook, then GitHub" before dispatch or variable application.

Windows local QEMU is Windows ARM64. An x64 MSVC/Prism smoke can be useful, but it is not a replacement for the GitHub-hosted Intel/x64 functional gate. The required build.yml functional matrix is pinned to windows-2022 so its CRT and Visual Studio generation do not move underneath the complete runtime suite. The standalone MSVC release-path, MIDI 2, and BLE compile gates remain on windows-latest; release builds and the nightly Intel safety net also keep tracking the newest hosted image. The MSVC release-path configure intentionally enables PULP_ENABLE_INSPECTOR while keeping runtime inspector endpoints off. Starting at the release product matrix's inspector_sdk_floor, published SDKs promise the split inspector archive family, so the Windows compile gate must match the tagged release configuration or it can miss an expensive packaging failure. tools/scripts/test_windows_runner_policy.py enforces this split across the actual build, release, coverage, and nightly workflows plus the release runner resolver and Shipyard mirror. It runs in workflow-lint, including when the profile or the policy test itself changes, so these surfaces cannot drift while an isolated mirror test remains green.

Nightly GitHub Intel validation

.github/workflows/cross-platform-check.yml is the scheduled Linux/Windows Intel safety net for this profile. It runs GitHub-hosted ubuntu-latest and windows-latest, files or updates one deduped issue per broken platform, and auto-closes the tracker when the platform recovers. Do not add a duplicate nightly Intel workflow unless this one is deliberately retired.

sanitizers.yml — per-sanitizer target selection

The automatic matrix runs on every relevant pull request and once nightly. It deliberately does not rerun after every push to main: that duplicated four hosted jobs after each merge and competed with the next merge group's required checks. The nightly schedule is the independent post-merge backstop; workflow_dispatch remains available for an immediate operator run.

Each sanitizer job resolves independently. Setting one variable moves exactly that sanitizer; the others stay on their defaults.

ASan, TSan, and UBSan configure through PULP_SANITIZER=<kind>, including ASan's example-bundle lifecycle build. Besides applying the compiler and linker flags, the named option marks sanitizer bundles as test-only instrumentation so relocatability validation permits the compiler-injected Xcode runtime. Installed-SDK consumer fixtures carry the matching instrumentation flags because instrumented static libraries retain runtime references. The strict shipping verifier remains unchanged and still rejects external compiler runtimes for ordinary release artifacts.

Variable	Default label when unset	Example (dedicated sanitizer VM label)
PULP_SANITIZER_ASAN_RUNS_ON_JSON	macos-14	gh variable set PULP_SANITIZER_ASAN_RUNS_ON_JSON --body '["self-hosted","macOS","ARM64","pulp-sanitizer-vm-macos"]'
PULP_SANITIZER_TSAN_RUNS_ON_JSON	macos-14	gh variable set PULP_SANITIZER_TSAN_RUNS_ON_JSON --body '["self-hosted","macOS","ARM64","pulp-sanitizer-vm-macos"]'
PULP_SANITIZER_UBSAN_RUNS_ON_JSON	macos-26	gh variable set PULP_SANITIZER_UBSAN_RUNS_ON_JSON --body '["self-hosted","macOS","ARM64","pulp-sanitizer-vm-macos"]'
PULP_SANITIZER_RTSAN_RUNS_ON_JSON	ubuntu-24.04	gh variable set PULP_SANITIZER_RTSAN_RUNS_ON_JSON --body '["self-hosted","linux","x64","sanitizer"]'

UBSan uses a RelWithDebInfo build with non-recovering undefined-behaviour instrumentation. This keeps symbols and the complete test matrix while running production-scale DSP certification renders with optimized code; a Debug/O0 build can turn seconds of offline spectral and stability analysis into per-test hang-guard timeouts without reporting undefined behaviour.

The three macOS sanitizers (ASan/TSan/UBSan) carry a --deny-labels pulp-build,pulp-build-vm guard in sanitizers.yml's resolver, so a sanitizer can never be misrouted onto the required-gate pool (the resolver hard-fails). They no longer read PULP_NAMESPACE_BUILD_MACOS_* either; the per-sanitizer variable is the single switch.

Capacity finding: localize at most one sanitizer, and only TSan. macOS allows only two running macOS guests per host (Apple's limit), and both belong to the required macos build gate. A local sanitizer VM is a third guest, so localizing is gated on the tartci idle-gate: the pulp-sanitizer-vm-macos lane shares TART_HOME with the gate (a real host-wide 2-guest semaphore) and yields its slot whenever the gate has queued/in-progress work — it can never starve the required check the way the coverage lane did. Pick TSan: it is the longest sanitizer (scoped -j1 serial, ~45 min on the 3 vCPU macos-14) and the highest value for a real-time audio framework, and being single-core-bound it gains most from a local M-series runner. ASan stays on macos-15, UBSan stays on macos-26, and RTSan stays on Linux: the four run in parallel on GitHub but would serialize (~4×) on one cap=1 local lane, which is slower than hosted except during a hosted backlog. Full parallel local sanitizers would need a third macOS host. Roll out one sanitizer at a time, each behind a measured go/no-go (gate queue latency + matrix wall-clock).

One-off overrides via workflow_dispatch

sanitizers.yml accepts one *_runner_selector_json input per sanitizer. They win over the corresponding repo variable for a single manual run:

gh workflow run sanitizers.yml \
  -f tsan_runner_selector_json='["self-hosted","macOS","ARM64","pulp-sanitizer-vm-macos"]'

Prove TSan green via this dispatch (with the pulp-sanitizer-vm-macos LaunchAgent loaded and the tartci idle-gate present) before setting PULP_SANITIZER_TSAN_RUNS_ON_JSON. The lane template is tools/launchd/pulp-tart-runner-sanitizer-macos.plist.template (ships parked — see its header for the load/quiet-window preconditions).

coverage.yml accepts linux_runner_selector_json, macos_runner_selector_json, and windows_runner_selector_json inputs. The macOS Tart coverage proof path is:

gh workflow run coverage.yml \
  -f macos_runner_selector_json='["self-hosted","macOS","ARM64","pulp-coverage-vm-macos"]'

Only set PULP_COVERAGE_MACOS_RUNS_ON_JSON to that selector after the proof run uploads the os-macos Codecov flag. The coverage LaunchAgent uses --queue-match-labels so existing hosted Coverage jobs do not accidentally boot a local coverage VM.

build.yml has the equivalent linux_runner_selector_json, windows_runner_selector_json, and macos_runner_selector_json inputs. These are the same inputs already documented above; they are listed here for completeness alongside the repo-variable knobs.

For a trusted Linux-only Mac Pro proof during hosted saturation, disable the otherwise-default hosted Windows leg explicitly:

gh workflow run build.yml --ref <trusted-branch> \
  -f linux_runner_selector_json='["self-hosted","Linux","X64","pulp-build-linux-x64","pulp-host-macpro"]' \
  -f run_windows=false

This switch affects only workflow_dispatch; automatic events keep their documented matrix policy, and ordinary manual dispatches still include Windows.

Reverting

Unset the variable and the job falls back to the hard-coded default immediately on the next run:

gh variable delete PULP_SANITIZER_TSAN_RUNS_ON_JSON

No code change is needed to revert, either.

Registering a self-hosted Mac runner (appendix)

Flipping a job to "self-hosted" labels assumes those labels are advertised by a running GitHub Actions runner somewhere. Register one on the Mac you want the job to run on:

# 1. From repo Settings -> Actions -> Runners, click "New self-hosted runner"
#    to get a short-lived registration token. Then on the Mac:
mkdir -p ~/actions-runner && cd ~/actions-runner
curl -o actions-runner.tar.gz -L https://github.com/actions/runner/releases/latest/download/actions-runner-osx-arm64.tar.gz
tar xzf actions-runner.tar.gz

# 2. Register with labels that match the JSON you set in the repo var.
#    Example for the TSan / sanitizer lane:
./config.sh --url https://github.com/Generous-Corp/pulp \
            --token <REGISTRATION_TOKEN> \
            --name "$(hostname)-sanitizer" \
            --labels "self-hosted,macos,arm64,sanitizer" \
            --work _work

# 3. Install as a launchd service so it runs at login and survives reboots.
./svc.sh install
./svc.sh start
./svc.sh status

Operational note. Self-hosted runners execute arbitrary code from any branch that can trigger the workflow. Use them on dedicated hardware / VMs you control, not shared personal machines. Apple Silicon hosts should prefer arm64 labels so jobs don't try to match Intel-only labels.

Agents do NOT register runners. Treat these commands as a human ops task documented here for completeness.

Creating a Namespace macOS runner profile

Today, nsc can verify login/workspace state and inspect the instances created by GitHub Actions, but it does not create or edit GitHub Actions runner profiles from this workflow. Creating a new runner profile is currently a Namespace dashboard step.

Use this path in Namespace:

• GitHub Actions -> Profiles -> New Profile

Recommended fields for the first macOS validation profile:

• Name in the UI: generouscorp-macos
• OS & Architecture: macOS on Apple Silicon
• Resources: 6 vCPU, 14 GB RAM
• Base image: a recent Xcode/macOS image appropriate for your build
• Cache toggles: leave enabled unless you have a reason to turn them off

Important selector detail:

• the Namespace UI shows the profile name without the GitHub runner prefix
• the selector you pass to Pulp/GitHub Actions is the prefixed form
• example: UI profile generouscorp-macos becomes selector "namespace-profile-generouscorp-macos"
• for one-off experiments you can skip profile creation entirely and pass a direct machine label instead, for example: "nscloud-macos-tahoe-arm64-6x14"

After creating the profile, validate it with a one-off run:

pulp ci-local cloud run build feature/my-branch \
  --provider namespace \
  --macos-runner-selector-json '"namespace-profile-generouscorp-macos"'

Or use a direct machine label for an ad hoc run:

pulp ci-local cloud run build feature/my-branch \
  --provider namespace \
  --macos-runner-selector-json '"nscloud-macos-tahoe-arm64-6x14"'

Then confirm the backing instance shape with:

nsc instance history --all -o json --max_entries 10

For a real macOS runner, the matching entry should report:

• user_label.nsc.runner-profile-tag = "namespace-profile-generouscorp-macos"
• shape.os = "macos"
• shape.machine_arch = "arm64"

If it instead shows linux/amd64, the profile label is valid but the backing runner is not a real macOS machine yet.

Prerequisites

• UTM — free VM manager for macOS (Apple Silicon and Intel)
• SSH key access to your VMs (password auth is not supported)
• The Pulp repo cloned on each VM at the path specified in config.json

UTM is the simplest option, but any SSH-reachable host works: Proxmox, a cloud VM (Azure/AWS/GCP), or a physical machine on your network. Cloud VMs cost money to run but are otherwise fully supported.

Setup

1. Create your config

cp tools/local-ci/config.example.json tools/local-ci/config.json

Local CI now prefers a machine-global config at ~/Library/Application Support/Pulp/local-ci/config.json on macOS (or the platform-equivalent state_dir()/config.json) so every worktree on the same machine sees the same host topology. tools/local-ci/config.json remains the fallback if no shared config exists, and PULP_LOCAL_CI_CONFIG still overrides both when you need an explicit one-off config.

Create the initial file from the example, then copy it to the shared state location if you want all worktrees to reuse it:

mkdir -p ~/Library/Application\\ Support/Pulp/local-ci
cp tools/local-ci/config.example.json ~/Library/Application\\ Support/Pulp/local-ci/config.json

Edit the chosen config.json and fill in your SSH hostnames and repo paths. The host field is the primary SSH target. fallback_host, if present, is tried next. The utm_fallback block is optional and is only used if SSH targets are unreachable.

Keep those aliases environment-local. Shared skills and docs should not hardcode your personal hostnames or VM names; they should explain how to choose the right target and where that target is configured.

The optional github_actions.workflows.docs-check.providers.namespace.runner_selector_json value lets you set the default Namespace runs-on selector that cloud run docs-check should dispatch when you do not pass --runner-selector-json explicitly.

1b. Optional estimated billing config

If you want per-run and billing-period cost estimates in cloud status, cloud history, and cloud compare, fill in the telemetry.billing block in your local config.

These numbers are estimates only. Verify provider pricing.

Example:

{
  "telemetry": {
    "billing": {
      "enable_provider_reported_totals": false,
      "currency": "USD",
      "billing_period_start_day": 1,
      "github_hosted_job_os_rates_per_minute": {
        "linux": 0.008,
        "windows": 0.016,
        "macos": 0.08
      },
      "namespace_profile_tag_rates_per_hour": {
        "namespace-profile-generouscorp": 0.50,
        "namespace-profile-generouscorp-macos": 1.20
      }
    }
  }
}

Notes:

• GitHub-hosted estimates use per-job OS rates when Pulp can infer the runner OS
• Namespace estimates prefer a profile-tag hourly rate and fall back to a machine-shape rule if you configured one
• if no matching rate exists, the CLI prints cost: unavailable (...)
• enable_provider_reported_totals is off by default; turn it on only if you want Pulp to ask GitHub for repo-wide billing totals when that API is available
• provider-reported GitHub totals are shown separately from tracked-run estimates because they are repo-wide current-period figures, not per-run truth
• GitHub can still return unavailable here if the account/API path does not support the newer billing endpoints

1a. Recommended Namespace setup

If you want to use the Namespace runner-provider path, the easiest setup today is:

brew install namespace-so/tap/nsc   # or use the install method from Namespace docs
nsc login

That is the recommended operator path for this pilot. Pulp can dispatch the GitHub workflow without shelling out to nsc, but keeping nsc installed makes it much easier to verify your Namespace workspace, inspect the account, and later support thin pulp ci-local cloud namespace ... helper commands without re-implementing Namespace setup logic inside Pulp.

Once nsc is installed, Pulp's thin helper commands can verify the state for you:

pulp ci-local cloud namespace doctor
pulp ci-local cloud namespace setup

doctor checks that nsc exists, verifies login state, and prints the current workspace identity. setup stays deliberately thin: it runs nsc login when needed and then re-renders the same status.

{
  "targets": {
    "mac": {
      "type": "local",
      "enabled": true
    },
    "ubuntu": {
      "type": "ssh",
      "host": "ubuntu",
      "repo_path": "/home/yourname/Code/pulp-validate",
      "utm_fallback": {
        "vm_name": "Ubuntu 24.04",
        "boot_wait_secs": 30,
        "ssh_retry_secs": 60
      }
    },
    "windows": {
      "type": "ssh",
      "host": "win",
      "repo_path": "C:\\Users\\yourname\\pulp-validate",
      "cmake_generator": "Visual Studio 17 2022",
      "cmake_platform": "x64",
      "cmake_generator_instance": "",
      "fallback_host": "win2",
      "utm_fallback": {
        "vm_name": "Windows 11",
        "boot_wait_secs": 60,
        "ssh_retry_secs": 120
      }
    }
  }
}

SSH host aliases come from ~/.ssh/config. Set them up there rather than putting raw IPs in this file. This makes it easy to prefer a fast local VM as the primary target and keep a slower hardware-backed machine as the fallback when you only need it for edge cases.

Before trusting a Windows SSH target for CI, verify that its non-interactive PowerShell context resolves git, cmake, and ctest. An interactive shell that works is not sufficient proof for the SSH service context the runner actually uses.

If your Windows VM is Windows on ARM, you can either set cmake_platform to "ARM64" explicitly or leave it blank and let the runner infer ARM64 vs x64 from the remote host. If CMake keeps picking the wrong Visual Studio install, set cmake_generator_instance to the exact VS path, for example C:/Program Files/Microsoft Visual Studio/2022/Community. If you leave cmake_generator_instance blank, the runner prefers a full Visual Studio install over BuildTools when both are present. The pinned WebGPU dependency already has a Windows aarch64 prebuilt for this path, so ARM Windows smoke runs can stay on the normal GPU-enabled configuration. This is useful for fast smoke validation on a local UTM VM. Keep an x64 Windows machine for parity runs when you need the authoritative Windows architecture.

2. Set up SSH keys

Each VM needs your public key in its authorized_keys. The Linux path is straightforward; Windows requires extra steps because OpenSSH on Windows uses a separate file with strict ACLs for admin users.

Find your public key (on your Mac)

If your private key is ~/.ssh/id_ed25519, your public key is:

cat ~/.ssh/id_ed25519.pub

Copy the output — you'll paste it on each VM. If you're running the VM in UTM or another hypervisor and can't copy/paste between host and guest, install the guest tools for your hypervisor first (e.g. SPICE guest tools for UTM/QEMU, VMware Tools, VirtualBox Guest Additions).

Linux (Ubuntu)

If ssh-copy-id is available and you can already reach the VM by password:

ssh-copy-id ubuntu    # or whatever your host alias is

If you're setting up from scratch on a fresh VM, SSH into it (or open its console) and run:

1. Note the VM's IP address:

ip addr show

Look for the inet line under your active adapter (usually enp0s1 or eth0).

2. Install and enable the SSH server (if not already running):

sudo apt update && sudo apt install -y openssh-server
sudo systemctl enable --now ssh

3. Add your public key:

mkdir -p ~/.ssh && chmod 700 ~/.ssh
echo "ssh-ed25519 AAAA...your-key-here..." >> ~/.ssh/authorized_keys
chmod 600 ~/.ssh/authorized_keys

4. (Optional) Disable password auth for tighter security:

sudo sed -i 's/^#\?PasswordAuthentication.*/PasswordAuthentication no/' /etc/ssh/sshd_config
sudo systemctl restart ssh

Windows

On the Windows VM, open PowerShell as Administrator and run:

1. Note the VM's IP address (you'll need it for SSH config later):

ipconfig

Look for the IPv4 Address line under your active adapter.

2. Create the admin authorized_keys file and add your public key:

New-Item -Force -ItemType File -Path "C:\ProgramData\ssh\administrators_authorized_keys"
Add-Content -Path "C:\ProgramData\ssh\administrators_authorized_keys" -Value "ssh-ed25519 AAAA...your-key-here..."

3. Fix the ACL (OpenSSH ignores the file if permissions are wrong):

icacls "C:\ProgramData\ssh\administrators_authorized_keys" /inheritance:r /grant "SYSTEM:(F)" /grant "Administrators:(F)"

4. Make sure sshd is running and set to auto-start:

Set-Service -Name sshd -StartupType Automatic
Start-Service sshd

Why administrators_authorized_keys? Windows OpenSSH uses C:\ProgramData\ssh\administrators_authorized_keys for users in the Administrators group, not ~/.ssh/authorized_keys. The ACL step is required — without it, sshd silently skips the file and falls back to password auth.

Set up SSH config on your Mac

Add entries to ~/.ssh/config so you can type ssh win instead of remembering IPs and usernames:

Host win
  HostName 192.168.64.5
  User your-username
  IdentityFile ~/.ssh/id_ed25519
  IdentitiesOnly yes
  ConnectTimeout 5

Host ubuntu
  HostName 192.168.64.4
  User your-username
  IdentityFile ~/.ssh/id_ed25519
  IdentitiesOnly yes
  ConnectTimeout 5

Replace the HostName values with the actual IPs from ipconfig (Windows) or ip addr (Linux). The host aliases here (win, ubuntu) are what you'll use in hosts.local.json for CI targets.

Test passwordless login

ssh ubuntu exit && echo "ok"
ssh win exit && echo "ok"

3. Clone the repo on each VM

The runner does a git fetch + checkout on the target, so the repo must already exist at the configured repo_path.

# On each VM:
git clone https://github.com/your-org/pulp.git ~/Code/pulp-validate

4. (Optional) Install the launchd drain agent

To automatically drain the queue on login and every 30 minutes:

cp tools/local-ci/dev.pulp.local-ci.plist ~/Library/LaunchAgents/
launchctl load ~/Library/LaunchAgents/dev.pulp.local-ci.plist

Edit the plist first if your repo is at a different path. To remove:

launchctl unload ~/Library/LaunchAgents/dev.pulp.local-ci.plist
rm ~/Library/LaunchAgents/dev.pulp.local-ci.plist

Usage

# Enqueue the current HEAD and wait for completion
pulp ci-local run

# Queue even if your current cwd belongs to a different git root than the script checkout
pulp ci-local run --allow-root-mismatch

# Fast preflight: clean configure/build/install + installed-SDK smoke, no tests
pulp ci-local run --smoke

# Fast PR preflight with a comment that is clearly labeled as smoke-only
pulp ci-local check 56 --smoke

# Run Mac-only while iterating locally
pulp ci-local run --targets mac

# Queue background work with explicit priority
pulp ci-local enqueue --priority low

# Bump a pending job to the front of the queue
pulp ci-local bump <job-id> high

# Drain pending jobs if no other runner already owns the queue
pulp ci-local drain

# Show queue, active runner, recent results, live target state, and VM status
pulp ci-local status

# Tail a running or completed target log
pulp ci-local logs <job-id> --target windows

# Show accumulated exact-SHA target evidence for a branch
pulp ci-local evidence feature/my-branch --limit 3

# Show local-CI disk usage and reclaimable artifacts without deleting anything
pulp ci-local cleanup
pulp ci-local cleanup --dry-run

# Delete stale bundles/logs/results once no local CI job is running
pulp ci-local cleanup --apply

# Include prepared build/install caches too; later reruns will rebuild them
pulp ci-local cleanup --apply --include-prepared

pulp ci-local run is the most common command. It enqueues the current HEAD, joins the machine-global queue, and waits until that exact job finishes.

Develop branch workflow

For complex, multi-piece features that use a develop/* integration branch, PRs target the develop branch instead of main. The ship command supports this via --base:

# Ship a feature to the develop branch (not main)
pulp ci-local ship feature/pkg-registry --base develop/package-manager

# The develop branch itself ships to main at phase boundaries
pulp ci-local ship develop/package-manager

GitHub Actions CI triggers on PRs to both main and develop/** branches, so CI runs automatically regardless of the target.

If you pass a branch name explicitly, for example pulp ci-local run feature/my-branch, local CI resolves and records that branch tip's exact SHA immediately. This prevents a stale launching checkout from accidentally queuing its own HEAD while you intended to validate a different branch.

Before queueing, local CI now also records: - the worktree root that is actually being queued - the current cwd and its git root, if any - the config path and whether it came from PULP_LOCAL_CI_CONFIG, shared state, or the worktree fallback - the selected SSH host/transport intent for each remote target

If the current cwd belongs to a different git root than the local_ci.py checkout you are invoking, queueing fails fast by default. Pass --allow-root-mismatch only when that mismatch is intentional.

If a selected SSH target is down and no fallback host or UTM fallback is configured, queueing now fails fast instead of burning time on a doomed job. Pass --allow-unreachable-targets only when you deliberately want to queue past that preflight.

Use --smoke when you want a quicker preflight before a full matrix run. Smoke mode still validates a clean detached worktree and installed SDK export path, but it disables tests, examples, and GPU in that clean build and skips ctest. Queue summaries and PR comments label these jobs as validation=smoke so they are not mistaken for full validation.

When a rerun is narrow and stays on the exact same SHA, local CI can now reuse the prepared root for that target + validation on persistent hosts. Status output calls this out as prepared=reused or prepared=clean so reused proof is never mistaken for a fresh cold path.

While a job is still running, pulp ci-local status reports live per-target state for the active job when available, for example:

Runner: pid=12345 active=[abcd1234ef56] feature/my-branch

Running (1):
  [abcd1234ef56] feature/my-branch @ 0123456789ab priority=normal targets=mac,ubuntu,windows
    submission: root=/Users/me/Code/pulp-worktree config=/Users/me/Library/Application Support/Pulp/local-ci/config.json (shared-state)
    live targets: mac=pass, ubuntu=pass, windows=running
    windows: phase=test, output=2026-04-01T01:34:18+00:00, heartbeat=2026-04-01T01:34:33+00:00, idle=15s, liveness=quiet, log=windows.log
      37/1263 Test: OSC 4-byte alignment

If a run is interrupted after some targets have finished, the job is requeued but keeps its last known target state:

Pending (1):
  [abcd1234ef56] feature/my-branch @ 0123456789ab priority=normal targets=mac,ubuntu,windows
    last known targets: mac=pass, ubuntu=pass, windows=running

Results are written to the machine-global state directory:

• macOS: ~/Library/Application Support/Pulp/local-ci/results/
• Linux: ${XDG_STATE_HOME:-~/.local/state}/pulp/local-ci/results/

A non-zero exit means at least one target failed.

If a newer SHA is queued for the same branch, targets, and validation mode, older pending work is marked superseded and written to the results directory with a reference to the replacement job. If a runner dies and reconciliation finds a newer replacement already queued for that same scope, the stale running job is also superseded instead of being requeued.

Cleanup And Disk Usage

pulp ci-local status now includes a local footprint summary so retained CI state stops being invisible drift:

• bundles
• prepared build/install caches
• logs
• results
• tracked cloud-run records

Use pulp ci-local cleanup to inspect what can be reclaimed. The command is a dry run by default, and --dry-run is available explicitly when you want that spelled out in scripts or notes.

What is cleaned automatically after job completion:

• completed-job git bundles once no pending/running job still needs them
• orphaned logs outside retained queue history
• orphaned result files outside retained queue history

What is not cleaned automatically in this first pass:

• prepared build/install state under prepared/<target>/<mode>

Prepared state is an intentional reuse cache. If you include it in manual cleanup, later reruns will rebuild it from scratch.

Examples:

# Inspect reclaimable space
pulp ci-local cleanup

# Show the same dry-run plan explicitly
pulp ci-local cleanup --dry-run

# Delete stale bundles/logs/results
pulp ci-local cleanup --apply

# Also delete prepared caches
pulp ci-local cleanup --apply --include-prepared

Safety rules:

• cleanup --apply is blocked while local CI jobs are running
• prepared cleanup is destructive to cached build/install state
• logs/results tied to jobs still present in queue history are retained

If you need immediate manual cleanup outside the CLI, make sure no pulp ci-local job is active first.

Desktop automation

pulp ci-local desktop ... adds a GUI/session automation layer under the same local CI control plane. Use it when an agent needs to launch an app, inspect it, click on it, capture screenshots, or publish a local evidence gallery without logging into the target machine manually.

Current desktop commands:

# Prepare one target and record its contract/receipt
pulp ci-local desktop install mac
pulp ci-local desktop install ubuntu
pulp ci-local desktop install windows

# Health and capability reporting
pulp ci-local desktop doctor mac
pulp ci-local desktop status
pulp ci-local desktop recent mac --limit 3
pulp ci-local desktop proof windows --action inspect --source-mode exact-sha --sha <commit-sha>

# Configure artifact/publish settings
pulp ci-local desktop config show
pulp ci-local desktop config set artifact_root ~/Library/Application\\ Support/Pulp/desktop-automation/runs
pulp ci-local desktop config set publish_mode none

# Run GUI actions
pulp ci-local desktop smoke mac --bundle-id com.apple.TextEdit --label textedit-smoke
pulp ci-local desktop inspect mac --command '/path/to/pulp-ui-preview' --label ui-preview-inspect --pulp-app-automation
pulp ci-local desktop click mac --command '/path/to/pulp-ui-preview' --click-view-id bypass-toggle --capture-ui-snapshot --pulp-app-automation
pulp ci-local desktop inspect windows --command 'notepad.exe' --label notepad-inspect
pulp ci-local desktop click windows --command 'notepad.exe' --click 885,18 --label notepad-maximize

# Run against an exact prepared SHA instead of the live checkout
pulp ci-local desktop inspect mac \
  --command './build-desktop-automation/examples/ui-preview/pulp-ui-preview' \
  --source-mode exact-sha \
  --sha <commit-sha> \
  --prepare-command 'cmake -S . -B build-desktop-automation && cmake --build build-desktop-automation --target pulp-ui-preview' \
  --pulp-app-automation

# Publish or prune local bundles
pulp ci-local desktop publish mac --limit 5 --label mac-gallery
pulp ci-local desktop cleanup mac --older-than-days 14 --keep-last 10

Ubuntu prerequisite:

sudo apt-get update
sudo apt-get install -y git-lfs xvfb xauth xdotool imagemagick wmctrl x11-utils
git lfs install

Supported Ubuntu/Linux setup tiers:

• baseline deterministic backend: xvfb + xauth
• source/bootstrap prerequisite: git-lfs
• richer interaction/capture lane: xdotool, imagemagick, x11-utils, and wmctrl

xvfb-run is the supported deterministic backend for Ubuntu/Linux desktop automation. A visible :0 display socket alone is not enough for SSH-driven automation because X11 authorization is often unavailable inside the remote shell. For repeatable CI and agent-driven runs, use the package set above and keep xvfb-run as the documented default.

desktop doctor ubuntu is an aggregate report. If multiple prerequisites are missing, it reports the full missing set plus remediation commands in one run instead of stopping at the first failure.

desktop doctor ubuntu checks the non-interactive SSH environment, not your interactive shell. setup.sh now prepends the common ~/.local/bin path automatically before dependency checks, but if git-lfs still fails over SSH after that, add the real install location to the non-interactive login-shell PATH or install git-lfs system-wide so git lfs version succeeds without extra shell setup.

Exact-SHA source prep on Ubuntu/Linux uses the same non-interactive SSH environment. The controller now treats bundle-based checkout and LFS materialization as separate steps:

• prepend ~/.local/bin before any git-lfs-dependent command
• fetch and checkout with GIT_LFS_SKIP_SMUDGE=1
• attach the clone URL as origin
• then let setup.sh --deps-only --ci / git lfs pull materialize the SDK blobs

That split matters on fresh VMs because a bundle checkout alone does not carry an origin remote, and LFS smudge/pull fails if it cannot resolve the repository URL.

Fresh Ubuntu proof checklist:

1. Start from a fresh source root, PULP_HOME, and PULP_PROJECTS_DIR
2. Run ./setup.sh --deps-only --ci
3. Build pulp-cli
4. Run pulp create <ProjectName> --manufacturer "<Name>" --no-interactive
5. Run pulp build inside the generated project
6. Verify actual emitted artifacts, not just configure/test success

Current expected native outputs from that proof are:

• Linux: VST3 target output under build/VST3, CLAP, LV2, and the standalone binary
• macOS: build/VST3/<Name>.vst3, build/AU/<Name>.component, build/CLAP/<Name>.clap, and the standalone .app bundle
• Windows: build/VST3/Debug/<Name>.dll for the VST3 target, build/CLAP/Debug/<Name>.clap, and the standalone .exe

If web formats are required, make them explicit in the generated project format list; the default native create proof does not imply web artifact output.

Windows first-time setup checklist:

1. Install and enable OpenSSH Server.
2. Keep a normal desktop user logged in to the VM. The Windows session-agent runs inside that logged-in session; SSH by itself is not a GUI session.
3. Make sure winget is available. desktop install windows uses it to provision required remote tooling such as Git when the VM is still fresh.
4. Run pulp ci-local desktop install windows once. This bootstraps the scheduled task, installs required remote tooling when possible, and writes the target-side PowerShell agent under %LOCALAPPDATA%\\Pulp\\desktop-automation-agent.
5. Run pulp ci-local desktop doctor windows and make sure SSH, the scheduled-task contract, and the required git check are green before attempting live proofs.
6. For source builds on the Windows VM itself, use powershell -ExecutionPolicy Bypass -File .\setup.ps1. The wrapper imports the Visual Studio environment and uses a short temporary drive alias so first-time bootstrap does not fail on long nested dependency paths.

The short-path rule is not theoretical. Windows source builds can fail from long nested checkout roots and then pass once the same source tree is mapped through a temporary drive alias before the first configure/build. Treat setup.ps1 or an equivalent short-path wrapper as the supported bootstrap path for Windows source builds.

Remote tooling policy on Windows:

• required: git
• used by the exact-SHA bundle-sync and prepare flows
• desktop install windows will provision it via winget when possible
• optional: gh
• useful for remote GitHub workflows on the target
• not required for smoke/inspect/click proofs
• optional: gh auth
• advisory only; authenticate it only if you intentionally want GitHub CLI workflows on the Windows target

Remote repo bootstrap policy on Windows:

• first-time desktop install windows should not require GitHub credentials on the target VM
• the controller prefers a locally uploaded git bundle to materialize pulp-validate
• origin is still attached when available so later fetches remain truthful
• gh and stored Git credentials are optional unless you intentionally want GitHub workflows on the Windows machine itself

Useful host-side verification commands:

Get-Service sshd
Set-Service -Name sshd -StartupType Automatic
Start-Service sshd
Get-NetFirewallRule -Name *ssh*
where.exe winget
where.exe git
where.exe gh

Useful first-time remote installs if you want to pre-provision them manually:

winget install --id Git.Git -e --source winget --accept-package-agreements --accept-source-agreements --disable-interactivity
winget install --id GitHub.cli -e --source winget --accept-package-agreements --accept-source-agreements --disable-interactivity

Supported Windows v1 interaction tiers:

• generic window-capture lane:
• --command only
• works for normal desktop apps such as notepad.exe
• supports window screenshot capture and coordinate clicks
• Pulp-owned app automation lane:
• add --pulp-app-automation
• enables ViewInspector snapshots and view-target selectors such as --click-view-id

Artifact bundles are written outside the repo by default:

• macOS: ~/Library/Application Support/Pulp/desktop-automation/runs/
• Linux: ${XDG_STATE_HOME:-~/.local/state}/pulp/desktop-automation/runs/
• Windows: %LOCALAPPDATA%\\Pulp\\desktop-automation\\runs\\

Each bundle stores:

• manifest.json
• stdout.log / stderr.log
• prepare.log when exact-SHA mode runs a fresh prepare step
• ui-tree.json when a UI snapshot is available
• screenshots/window.png
• screenshots/before.png / screenshots/diff.png when an interaction captures before/after evidence

The artifact root also maintains rolling summaries for agents and status tooling:

• latest-run.json — newest observed run summary
• latest-proof.json — newest successful proof summary
• runs.jsonl — raw summary stream for recent desktop automation runs
• target-scoped copies under <artifact-root>/<target>/...
• _published/latest-report.json — newest staged local HTML/JSON gallery summary
• _published/reports.jsonl — raw summary stream for local published galleries

manifest.json now includes additive source provenance when desktop actions run through the controller:

• source.mode (live or exact-sha)
• source.branch
• source.sha
• source.prepare_command
• source.prepare_timeout_secs
• source.prepared_root
• source.launch_cwd

Desktop reporting surfaces are intentionally split:

• desktop recent = raw run history, including failed attempts
• desktop proof = successful proof summaries grouped by target + action + source.mode + source.sha
• desktop status = target config plus latest_run, latest_proof, and the newest local publish summary (latest_publish)

Use desktop proof when you need to answer questions like:

• “What live-host proof do we already have for Ubuntu on this SHA?”
• “Did Windows ever pass this exact-SHA inspect lane?”
• “What is the newest successful proof, even if the newest run failed?”

Exact-SHA desktop source mode

desktop smoke, desktop click, and desktop inspect all share a controller-owned source mode:

• --source-mode live|exact-sha
• --branch
• --sha
• --prepare-command
• --prepare-timeout

Behavior:

• live launches from the target's normal working copy behavior.
• exact-sha prepares a per-target source root for the requested SHA, launches from that prepared root, and records the prepared-root provenance in the run manifest.
• On Windows, --prepare-command executes inside a generated .cmd script under cmd.exe. Use double quotes for paths, generator names, and arguments. POSIX-style single-quoted tokens are treated as literal text and are rejected by the controller before the remote prepare step starts.
• When desktop_automation.targets.<target>.optional.webview_driver=true, desktop doctor probes the configured webdriver_url through the WebDriver /status endpoint and reports whether the driver is actually reachable and ready, not just whether the URL exists in config.

Preparation/cache semantics:

• Prepared roots are keyed by target + sha + prepare_command.
• A repeated identical request may reuse the prepared root instead of rebuilding it.
• prepare_command only runs when a fresh prepared root is created.

Launch behavior:

• Desktop actions switch their launch cwd to the prepared root in exact-SHA mode.
• Repo-local executable paths in the first command token are rewritten into the prepared root automatically.
• The current exact-SHA workflow is a --command lane. Do not assume --bundle-id participates in exact-SHA source preparation.

pulp-ui-preview is currently Apple-desktop-only, so the Linux and Windows source-build examples below use the cross-platform PulpGain standalone target.

Examples:

# macOS local exact-SHA inspect
pulp ci-local desktop inspect mac \
  --command './build-desktop-automation/examples/ui-preview/pulp-ui-preview' \
  --source-mode exact-sha \
  --sha <commit-sha> \
  --prepare-command 'cmake -S . -B build-desktop-automation && cmake --build build-desktop-automation --target pulp-ui-preview' \
  --pulp-app-automation

# Ubuntu xvfb exact-SHA smoke against a Linux-supported standalone
pulp ci-local desktop smoke ubuntu \
  --command './build-desktop-automation/examples/pulp-gain/PulpGain' \
  --source-mode exact-sha \
  --sha <commit-sha> \
  --prepare-command 'cmake -S . -B build-desktop-automation && cmake --build build-desktop-automation --target PulpGain_Standalone'

# Windows session-agent exact-SHA smoke
pulp ci-local desktop smoke windows \
  --command '.\\build-desktop-automation\\examples\\pulp-gain\\Debug\\PulpGain.exe' \
  --source-mode exact-sha \
  --sha <commit-sha> \
  --prepare-command 'cmake -S . -B build-desktop-automation -G \"Visual Studio 17 2022\"; cmake --build build-desktop-automation --target PulpGain_Standalone --config Debug'

# Windows generic live inspect
pulp ci-local desktop inspect windows \
  --command 'notepad.exe' \
  --label notepad-inspect

# Windows generic live click with before/after evidence
pulp ci-local desktop click windows \
  --command 'notepad.exe' \
  --click 885,18 \
  --label notepad-maximize

# Query the newest successful Windows proof for one SHA
pulp ci-local desktop proof windows \
  --action smoke \
  --source-mode exact-sha \
  --sha <commit-sha>

Desktop adapter truth

• macos-local
• runs directly on the local logged-in macOS session
• supports bundle launch via --bundle-id
• supports Pulp-owned app automation (--pulp-app-automation) for direct launch commands, including ViewInspector snapshots and view-target clicks
• linux-xvfb
• runs GUI smoke/inspect/click through xvfb-run
• currently supports --command only
• currently requires --pulp-app-automation for the click/inspect lane
• windows-session-agent
• bootstraps a scheduled task plus target-side PowerShell agent in the logged-in Windows desktop session
• requires a real logged-in desktop user; SSH alone is not enough
• currently supports --command only
• supports generic window-capture smoke/inspect/click for normal desktop apps
• supports coordinate clicks and before/after screenshot diffs without --pulp-app-automation
• supports ViewInspector snapshots and view-target selectors only with --pulp-app-automation
• uses the scheduled task plus target-side agent as the honest v1 Windows interaction lane; external UI automation tools are optional future adapters, not the core controller

Proof lookup

desktop proof is the first-class proof query surface for desktop automation:

• filters:
• target
• --action
• --source-mode live|exact-sha|legacy
• --sha
• --branch
• groups successful proofs by target/action/source.mode/source.sha
• ignores failed runs when computing proof summaries

Example:

pulp ci-local desktop proof ubuntu --action click --source-mode exact-sha --sha <commit-sha>

desktop status now reports both:

• latest_run: the newest run, even if it failed
• latest_proof: the newest successful proof summary for that target
• latest_publish: the newest local HTML/JSON gallery summary staged under _published/

Desktop config keys

tools/local-ci/config.json accepts a desktop_automation block:

{
  "desktop_automation": {
    "artifact_root": "",
    "publish_mode": "none",
    "publish_branch": "dev-artifacts",
    "retention_days": 14,
    "targets": {
      "mac": {
        "adapter": "macos-local",
        "bootstrap": "launchagent",
        "capability_tier": "v2",
        "optional": {
          "webview_driver": false,
          "webdriver_url": "",
          "debug_attach": false,
          "debugger_command": "lldb",
          "video_capture": false,
          "frame_stats": false
        }
      },
      "ubuntu": {
        "adapter": "linux-xvfb",
        "bootstrap": "xvfb-run",
        "capability_tier": "v2",
        "optional": {
          "webview_driver": false,
          "webdriver_url": "",
          "debug_attach": false,
          "debugger_command": "lldb",
          "video_capture": false,
          "frame_stats": false
        }
      },
      "windows": {
        "adapter": "windows-session-agent",
        "bootstrap": "scheduled-task",
        "capability_tier": "v2",
        "task_name": null,
        "remote_root": null,
        "optional": {
          "webview_driver": false,
          "webdriver_url": "",
          "debug_attach": false,
          "debugger_command": "",
          "video_capture": false,
          "frame_stats": false
        }
      }
    }
  }
}

For Windows:

• task_name is optional. If omitted, local CI uses PulpDesktopAutomationAgent-<target>.
• remote_root is optional. If omitted, the agent is installed under %LOCALAPPDATA%\Pulp\desktop-automation-agent.
• optional.webview_driver enables the future WebView/WebDriver capability vocabulary for that target. Pair it with optional.webdriver_url only when the app under test actually exposes a localhost WebDriver endpoint in debug/test mode.
• optional.debug_attach, optional.video_capture, and optional.frame_stats are opt-in groundwork flags. They make the target advertise and doctor those optional tiers; they do not magically make the adapter support them unless the required tooling is also present.

Convenience updates through the CLI:

pulp ci-local desktop config set target.mac.webview_driver true
pulp ci-local desktop config set target.mac.webdriver_url http://127.0.0.1:4444
pulp ci-local desktop config set target.mac.debug_attach true
pulp ci-local desktop config set target.mac.debugger_command lldb
pulp ci-local desktop config set target.mac.video_capture true
pulp ci-local desktop config set target.mac.frame_stats true

Recommended host-side remediation when desktop doctor windows reports SSH service reset during handshake:

Get-Service sshd
Set-Service -Name sshd -StartupType Automatic
Start-Service sshd
Get-NetFirewallRule -Name *ssh*

Treat that failure as a Windows host-side OpenSSH issue, not a desktop-agent contract failure.

Desktop publication

pulp ci-local desktop publish always stages a local HTML/JSON gallery from recent bundles. In the default publish_mode=none path, that is the whole feature. When publish_mode=branch, the same report is also mirrored to the configured publish branch under desktop-automation/latest/ and desktop-automation/reports/<report-id>/.

• index.html
• index.json
• copied screenshots and diffs
• source manifest/log references
• _published/latest-report.json and _published/reports.jsonl rollups for the newest/known local galleries

Use desktop config set publish_mode ... only when you intentionally want publication behavior. The default should stay none for normal development.

Branch publication notes:

• publish_mode=branch pushes the latest local report to publish_branch
• the branch stores desktop-automation/latest/ plus immutable desktop-automation/reports/<report-id>/ snapshots
• when the repo remote is GitHub, the publish report includes clickable branch/tree/blob URLs for the mirrored artifacts

Evidence Tracking

pulp ci-local evidence summarizes the last-good recorded results by exact SHA, target, and validation mode. This is the operator-facing answer to:

• what already passed on this branch?
• which exact SHA has Windows full proof?
• do we really need to rerun macOS again?

The compact evidence section in pulp ci-local status uses the same data so the current branch’s known-good results stay visible during active work.

Working A Failure

Do not wait for a whole matrix to finish before reacting. The fastest loop is:

1. start a run
2. watch pulp ci-local status
3. tail pulp ci-local logs <job-id> --target <name> on the first failing or suspicious target
4. begin the narrowest local repro or code inspection immediately
5. rerun only the truthful scope needed after the fix

In practice, that means:

• one process owns CI monitoring and host state
• one process or agent works the likely fix locally as soon as a failure becomes actionable
• user updates should be sent when a target changes state or the first actionable failure appears, not only when asked
• a target that already failed is enough to start debugging; do not burn time waiting for unrelated targets to finish unless their result changes the fix
• once a failure is actionable, start the fix track in parallel unless it would contend with the same host or invalidate the active run
• do not rerun a target that already passed on the exact same SHA unless that prior result is untrustworthy or the environment changed
• if only one or two targets are stale, rerun only those targets instead of the whole matrix
• once the failure surface is isolated, prefer the minimum sufficient proof instead of a symmetric rerun
• a direct exact-SHA validate on one target counts as valid evidence for that target; keep earlier same-SHA passes for the other targets unless something actually invalidated them
• on persistent hosts, narrow same-SHA reruns should prefer prepared-state reuse instead of paying again for clean worktree/setup/build work
• use --smoke first when the risk is install/export/build structure rather than runtime test behavior
• all targets on one SHA is a goal, not a reason to blindly rerun already-green same-SHA targets
• if a broader in-flight job is no longer informative, cut over to the narrower rerun instead of letting the queue drift

Priorities

Jobs are ordered by priority first, then FIFO within the same priority.

• low — background validation
• normal — default interactive work
• high — shipping, PR checks, or work you want to run first

You can set the initial priority with --priority and change a pending job later with:

pulp ci-local bump <job-id> high

pulp ci-local status prints the job ids you can bump.

Exact SHAs On Remote Targets

Remote targets validate the queued SHA, not the latest branch tip. That keeps queued jobs truthful, and the runner now uploads that exact SHA to SSH targets as a git bundle before validation.

If you queued work with an explicit branch name, the runner first resolves that branch name to a commit SHA and then treats the run exactly like any other exact-SHA validation.

That means this works even for a local-only commit:

pulp ci-local run --targets mac,ubuntu,windows

pulp ci-local ship still pushes first because it opens and validates a PR, but ordinary local validation no longer depends on the remote host already having your branch tip.

Running Mac-only

If you don't have VMs set up, disable the SSH targets in your active CI config:

"ubuntu": {
  "type": "ssh",
  "enabled": false,
  ...
}

Mac validation still runs. You get single-platform coverage, which is better than nothing for catching build breaks before pushing.

You can also keep the SSH targets enabled and request Mac-only while iterating:

pulp ci-local run --targets mac

Compiler coverage: which lanes use which compiler

Worth knowing before you trust a green PR, because the answer is not symmetric.

Native Linux workflows install their shared system prerequisites through .github/actions/install-linux-build-deps, backed by the portable tools/ci/install_linux_build_deps.py resolver and tools/ci/linux_build_deps.json. Profiles describe capabilities (native and native-webview); compiler versions, analysis tools, caches, and other lane-specific packages stay explicit at each call site. The workflow policy file enumerates adopters and reviewed exclusions, and workflow-lint rejects a new direct apt workflow that has no owner. Update the manifest once when a native dependency changes instead of copying the package into individual build, coverage, sanitizer, release, or portability lanes.

Every Linux lane in PR CI compiles with Clang — "Public headers compile standalone (Linux Clang)", "IWYU (Linux, Clang)", "RealtimeSanitizer (Linux x86_64, Clang 18)". macOS is Clang by definition. Windows is MSVC.

That left GCC compiled in exactly one place: release-path-pr-gate.yml, which is path-triggered on release files (Skia pins, tools/deps/manifest.json, tools/cmake/Pulp*.cmake, the top-level CMakeLists.txt). Most PRs never trigger it, so a GCC-only error inside core/ could sit on main indefinitely.

It did, and not once. core/host/src/signal_graph.cpp keeps acquiring two identical .custom_latency_for entries in one designated-initializer list — Clang accepts that and silently takes the last, so nothing on the Apple or Clang-Linux lanes notices. git log -S '.custom_latency_for' shows the same defect fixed four separate times:

4371eebce  fix(host): remove a duplicate binder designator that GCC rejects
1bdd0434a  fix(host): drop the duplicated custom-latency binder
402620df4  fix(host): drop a duplicate designator that breaks every non-Apple release build
077ffabda  build(host): drop a duplicate designator that breaks the MSVC build

Every one of those was caught late — by MSVC, by a non-Apple release build, or by the release-path gate firing on an unrelated PR. The binder list is long and sits where merges collide, so the duplicate keeps coming back; what was missing was a PR-time lane that says so immediately.

gcc-compile-gate.yml closes that hole. It runs on every PR and compiles the core libraries with g++ and nothing else:

Option	Value	Why
PULP_ENABLE_GPU	OFF	no Dawn/Skia fetch or build — this is what keeps the gate in minutes rather than a full release build
PULP_BUILD_TESTS	OFF	the gate asks "does core/ compile under GCC", not "does it work"
PULP_BUILD_EXAMPLES	OFF	same
PULP_ENABLE_DESIGN_IMPORT	OFF	authoring subsystem, not core portability
PULP_ENABLE_INSPECTOR	OFF	dev surface, not core portability

Read a failure here literally. The lane runs no tests and touches no hardware, so it cannot flake on load or timing the way the GPU-perf lanes can. A red result is a real compiler divergence. Clang accepting the same code does not make it portable.

What it deliberately does not cover: GCC behavior. Nothing is executed, so a construct both compilers accept but implement differently is still only caught by the Clang test lanes. Widening this to run tests under GCC is a separate decision with a real time cost.

It also guards one option combination. The lane configures with PULP_ENABLE_DESIGN_IMPORT=OFF, which is the option's own documented "release/ship OFF" setting — and that configuration was once unlinkable, because tools/import-design was added unconditionally while the pulp::view design-IR sources it links sit behind that option. The discovery step now runs --assert-absent pulp-import-design against the codemodel the lane already produces, so a re-broken guard fails here immediately instead of surfacing as an undefined-reference wall in someone's release build. It costs no extra configure time. Because the guard lives in the top-level CMakeLists.txt, that file is one of the lane's path triggers alongside core/**.

For contributors

You don't need the same VM setup as the original developer. Options:

• Mac-only: Disable all SSH targets. Fast, free, covers the primary development platform.
• UTM VMs: Free. Requires ~40 GB of disk for both VMs. UTM images can be created from ISO or from the UTM gallery.
• Cloud VMs: Works with any SSH-accessible host. Costs money while running — stop them when not in use.
• Physical machines: A spare Linux box or Windows machine on your network works fine.

Local CI config is intentionally gitignored. Keep your host topology local, and prefer the machine-global config path so every worktree uses the same host map by default.

Troubleshooting

JSONDecodeError on Shipyard queue file

Shipyard's local job queue lives at ~/Library/Application Support/shipyard/queue/queue.json on macOS (~/AppData/Local/shipyard/queue/queue.json on Windows, ${XDG_STATE_HOME:-~/.local/state}/shipyard/queue/queue.json on Linux). On rare crashes Shipyard can truncate this file to zero bytes, which then breaks every subsequent invocation with a JSONDecodeError.

Recovery (run once):

echo '{"jobs": []}' > ~/Library/Application\ Support/shipyard/queue/queue.json

Re-running tools/install-shipyard.sh also performs this reset automatically. Tracked as #528.

The Shipyard macOS lane builds Debug — on purpose

.shipyard/config.toml configures the macOS validation lane with -DCMAKE_BUILD_TYPE=Debug. This contradicts CLAUDE.md ("Release is the default") and looks like config drift. It is deliberate, and flipping it to Release would remove the only lane in CI that can see a whole class of undefined behaviour.

On 2026-07-12 it caught a real ODR violation (#6081). snap_to_zero() is an inline function template defined in a header, its body gated by a build-time macro, and a test TU redefined that macro before including the header. Both translation units then emitted the same mangled symbol with different bodies:

build	what happens	result
-O3	each TU inlines its own copy, so each behaves per its own macro	the A/B test appears to work — Release is green, the bug is invisible by construction
-O0	nothing inlines; both TUs emit a weak symbol, the linker keeps exactly one, and both call it	the "disabled" reference silently ran the enabled code — Debug is red

The red test was the mild outcome. The linker's choice is arbitrary: had it kept the other definition, the assertions would have passed while exercising a no-op — a null test, asserting nothing, green forever.

The fix shape is not "delete the redefine". It is: give the variant its own binary, compiled consistently end to end, linking no default-built TU (see test/denormal_null_refgen.cpp). The class is now guarded by tools/scripts/test_odr_macro_gated_headers.py.

A perf gate failing there is a mis-calibrated gate, not a reason to flip the lane

Debug builds are much slower, and CLAUDE.md is right that Debug is the wrong default for most work. The answer is not "Debug everywhere" — it is keep one -O0 lane, and calibrate perf gates for the build they actually run in.

test/test_yoga_layout_bench.cpp is the worked example. Its timing threshold is 0.25 x a 60fps frame (4166.7us), sized at ~11x an M-series Release baseline (~380us) to tolerate a loaded CI box. But in the Debug lane the same 484-node pass takes ~4420us — about 11.6x slower, which eats the entire safety margin. The gate sat permanently at the edge (4421.8us vs 4166.7us, ~6% over) and load merely tipped it. It was never "flaky because the box was busy"; it was a Release-calibrated gate running unoptimized, where it measured the absence of the optimizer, not the cost of layout.

The timing assertion is now #ifdef NDEBUG-gated — the GitHub macOS lane configures Release, so it still runs with real coverage and the right calibration. The structural assertions (allocs_per_pass > 0, frees-match-allocs) still run in every build; they catch real regressions and do not care about the optimizer.

A false red is worse than no gate: it trains everyone to wave away red as "probably the box" — which is exactly how a real bug gets dismissed.