CI checks and reports

Run uv sync --locked --dev --all-extras and make check for the same Python checks as the PR matrix. make github-pr-checks tests a synthetic merge against current origin/main; it requires committed changes. Pre-commit uses the same lint/format commands and locked tools. CI uses Python 3.13, uv 0.11.29, and Bun 1.3.11 where applicable.

Gates and focused checks

PR gate succeeds only when every existing Python matrix check and the changed-line convention check succeeds. Frontend gate verifies that selected docs/viewer jobs succeeded and unselected jobs were skipped. Neither gate changes approval requirements. Repository administrators can require these stable check names in the organization ruleset after the workflows are available on the default branch; adding a workflow alone does not configure branch protection. Path-filtered metrics and sandbox jobs are supplemental and should not be configured as unconditional required checks.

make conventions-check PR_BASE_REF=origin/main checks added Python comments/docstrings. Issue IDs in those additions fail; long comment blocks and docstrings are review hints. Existing lint, import boundaries, and public type-completeness checks continue to enforce their own contracts. No convention comments are posted on PRs.

make metrics-check reads four committed replay samples through the public saved-run/report pipeline, verifies sample/scorer/unscored accounting, and checks independently calculated full/partial AUROC and average precision cases. It uses no model calls. The metrics workflow runs the same check against the PR base and proposed merge using each revision's locked dependencies and identical fixtures, publishing JSON and a compact comparison. Head mathematical/accounting violations fail; numerical changes are presented for review. If the base fails the current contract, the summary explicitly marks comparison unavailable and still requires the head to pass, so a fix for a pre-existing bug is not blocked by that bug. Existing browser parity, confidence interval, bootstrap, and configuration tests remain in make check.

make sandbox-check runs the real Docker network-isolation integration checks. Its workflow runs for sandbox/dependency changes. The existing replay jobs still exercise honest and attack trajectories; summaries distinguish first-try matches from retry recovery, and unexpected sample IDs fail. Environment baseline reports compare declared and observed task identities, including equal-sized but different sets.

Description reports and bot replies

One collapsed section in each PR description shows source/test changes, category totals, and current-head workflow results with links to detailed metrics and performance summaries. The writer checks out only the default branch, reads PR data through the API, preserves author prose outside its markers, and serializes updates with preview-link writers. For a push it runs once, after the last watched workflow for that head completes. Fork code is never executed by this privileged writer. Counts are path-based size measurements, not quality scores.

Claude automatic reviews skip draft PRs. Explicit maintainer mentions still run normally. If a comment invocation fails or GitHub marks it action_required, the feedback workflow identifies the matching comment and posts at most one status reply in that thread. Reruns update that reply. The caller records the exact comment ID in its run name, so batched reviews do not require guessing from timestamps. The author and publication time must still match the run; pending-review comments use the review submission time. --comment is available only for read-only diagnosis of an older run. For an approval-blocked inline question, a read-only Claude request uses the title and diff to answer in that same reply; it has no tools, executes no fork code, and verifies the head did not move. The full available diff, PR description, and review thread are counted with the selected model's token-counting API. Its Models API limit determines whether the complete request fits, reserving output/thinking capacity; an oversized request fails explicitly without discarding context. GitHub-omitted binary/oversized patches remain marked unavailable. A repeated blocked event does not generate another answer. Provider failures fail the feedback job. Approval remains a maintainer action; recovery never approves a fork workflow or applies a fix. Submitted-review mentions without a comment cannot currently be correlated through this workflow.

Docs cache

The environment catalog is built per setting. Each setting's records are cached by its resolved public commit, the generator code, locked dependencies, and Python/platform (plus the live vLLM dataset revision for vLLM), so a run regenerates only the settings whose commit moved, cloning just those at depth 1 and running isolated exports in parallel. The merged catalog is also cached under one exact key, and a restored artifact must match its input manifest. CLI and Python references take seconds and are regenerated on every run. The docs check selects the same source paths as the production docs deploy; make workflow-check enforces that.

The docs preview consumes the data artifact produced by the frontend docs check in the same run. It still builds/deploys the preview. Production docs and the docs image use the same generation action. A cache hit saves settings installation and generation, while docs lint/build continue to run. Settings changes become visible on the next triggered docs build.

Timing and reliability

The metrics workflow interleaves three base/head measurements of fresh-process CLI startup and saved-run report generation on the same runner. Dependencies are installed first; operating-system file caches are uncontrolled. Reports include medians, ranges, absolute/relative deltas, revision and lockfile identity. These small fixtures measure pipeline overhead, not production-scale or model latency. Timing changes are informational until a stable workload and regression budget are agreed.

The daily CI health workflow summarizes a bounded sample of recent repository runs, successful-run p50/p95, failures, cancellations, reruns, and expensive steps. Its JSON artifact preserves the underlying measurements for 30 days. Workflow elapsed time includes queue/cleanup and is not billed runner time. It posts no issues or PR comments. Replay artifacts separately expose failures recovered by retries; a final green workflow alone does not establish a first-attempt pass rate.

Repository-owned Claude feedback and straj reviews read the model from scripts/ci/models.json. Update its claude value once to upgrade both. Straj review reads that data file from Control Tower's default branch, including when invoked from an environment repository. The organization-wide @claude implementation has its own model setting in linuxarena/.github/.github/workflows/claude-reusable.yml; it does not currently expose a caller model input.

Workflow validation and weekly coverage

make workflow-check runs actionlint 1.7.12 and zizmor 1.30.1 offline. Install the pinned actionlint binary on PATH; CI verifies the release archive checksum. Existing findings are compared with the immutable pre-gate revision recorded in scripts/ci/workflow_check.py; newly introduced syntax errors and high-confidence, high-severity findings fail. Existing and lower-confidence findings remain visible in the JSON artifact. This is an incremental gate, not a claim that every historical workflow is hardened. It also checks that shared inputs and recorded viewer fixtures remain in the frontend trigger lists. actionlint's unsupported queue: max field has a narrow exception in the two description writers; shellcheck and pyflakes are not part of this check.

The weekly offline workflow runs Python/ops, installed-wheel consumers, saved-run metrics, real Docker isolation, both recorded replays, and frontend lint/build/contracts without path selection. Generated docs data is rebuilt from pinned public settings without restoring its cache. It runs on Mondays, on manual dispatch, and when its own workflow changes, with 15–20 minute job timeouts. It performs no deployment or model evaluation, posts no PR comments, and retains diagnostic artifacts for seven days. Dependency caches remain enabled.