fix: allow graceful runner shutdown during stack maintenance

The host resolves the tailnet through MagicDNS on tailscale0 and everything
else through Cloudflare (systemd-resolved split DNS). MagicDNS has no
upstream for names outside the tailnet and answers them SERVFAIL, so
handing 100.100.100.100 to job containers as their first resolver made
every public lookup wait on a failure before falling back: image pulls
stalled and the e2e lane's *.localtest.me fetches blew a 5s poll.

A CoreDNS forwarder at a fixed address on the runner network now answers
both: ts.net is forwarded to MagicDNS, the rest to 1.1.1.1/1.0.0.1.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018j9YG8YFjcsVLnqzNz7LmA

Corefile

@@ -0,0 +1,19 @@
+# Split-horizon resolver for the runner's Docker daemon and every job
+# container it starts.
+#
+# The host resolves the tailnet through MagicDNS on tailscale0 and everything
+# else through Cloudflare on the default link (systemd-resolved split DNS).
+# MagicDNS itself has no upstream configured for names outside the tailnet and
+# answers them with SERVFAIL, so handing 100.100.100.100 to job containers as
+# their first resolver made every public lookup (image pulls, *.localtest.me
+# in the e2e lanes) wait on a failure before falling back -- long enough to
+# time out a 5s Playwright poll. Only ts.net goes to MagicDNS here.
+ts.net:53 {
+    errors
+    forward . 100.100.100.100
+}
+.:53 {
+    errors
+    cache 30
+    forward . 1.1.1.1 1.0.0.1
+}

README.md

@@ -52,19 +52,58 @@ It uses `actions-rust-lang/setup-rust-toolchain` which caches `~/.cargo` and
 
 ## Manage
 
-```bash
+The wrapper operates only this project's Compose stack:
-docker compose up -d      # apply config changes
-docker compose down       # stop (registration persists in volume)
-```
-
-⚠️ **Check for running jobs before recreating the runner.** Recreating it orphans
-any job in flight — Gitea marks them failed and you have to re-run:
 
 ```bash
-docker exec gitea-runner-dind docker ps --format '{{.Names}}' | grep -v buildkit
+./runner-stack.sh up -d   # start the stack
+./runner-stack.sh stop    # stop runner before DinD; retain containers and volumes
+./runner-stack.sh down    # remove containers; retain registration and caches
 ```
 
-Wait until that prints nothing (`buildx_buildkit_builder-*` is a helper, not a job).
+To apply an update on another existing runner, first wait for jobs to finish:
+
+```bash
+docker exec gitea-runner-dind docker ps --format '{{.Names}}'
+```
+
+Only persistent BuildKit helpers should remain. Then run:
+
+```bash
+git pull --ff-only
+./runner-stack.sh stop
+./runner-stack.sh up -d --force-recreate
+```
+
+Recreation loads changes in the bind-mounted runner YAML as well as Compose
+settings. It retains registration and cache volumes. Keep that machine's `.env`
+and intentional local capacity/label settings; resolve any pull conflicts while
+retaining `runner.shutdown_timeout: 5m`. A new installation still requires the
+per-machine registration setup above. Running the wrapper without arguments
+prints usage and does not change the stack.
+
+The runner waits up to five minutes for jobs to finish before cancelling them.
+Compose allows six minutes before SIGKILL, leaving additional time for cleanup.
+This ordering applies to Compose operations, not power loss or independently
+stopping DinD. Any other clients sharing the daemon must be stopped separately
+before stack maintenance.
+
+For changes only to `config.yaml`, restart the runner to load the file:
+
+```bash
+./runner-stack.sh restart runner
+```
+
+### Network allocation
+
+DinD allocates /24s from `10.210.0.0/16`: 256 subnet slots, including one for
+its default bridge. Keep this setting when updating the daemon, and verify
+that this range does not overlap routes in another machine's environment.
+Host Docker has separate allocation state.
+
+The larger pool provides headroom; interrupted jobs can still leave networks
+behind. Workflow-created networks need their own teardown on success, failure
+and cancellation. Inspect leftover networks inside DinD during idle maintenance;
+do not schedule blanket pruning while jobs are creating their stacks.
 
 ### Housekeeping
 

config.yaml

@@ -5,6 +5,8 @@ runner:
   # Run multiple jobs concurrently. Overridden by env at compose level too.
   capacity: 4
   timeout: 3h
+  # Let jobs finish before cancellation during planned maintenance.
+  shutdown_timeout: 5m
   # Keep the runner alive; jobs are ephemeral containers.
   insecure: false
   # Env injected into every job. Speeds up Rust builds.

docker-compose.yml

@@ -1,4 +1,17 @@
 services:
+  # Split DNS for the dind daemon and its job containers: ts.net -> MagicDNS,
+  # everything else -> Cloudflare. See Corefile for why MagicDNS alone fails.
+  dns:
+    image: coredns/coredns@sha256:af8c8d35a5d184b386c4a6d1a012c8b218d40d1376474c7d071bb6c07201f47d
+    container_name: gitea-runner-dns
+    restart: unless-stopped
+    command: ["-conf", "/Corefile"]
+    volumes:
+      - ./Corefile:/Corefile:ro
+    networks:
+      runner:
+        ipv4_address: 172.28.0.53
+
   # Isolated Docker daemon for the runner's job containers.
   # Keeps the host Docker clean and gives reliable cache networking.
   dind:
@@ -14,9 +27,35 @@ services:
     # as an env var. A bare "tcp://..." isn't recognized and silently falls back
     # to /var/run/docker.sock, which doesn't exist in the runner container -- so
     # this shared socket is what makes `docker build` etc. work inside jobs.
-    command: ["dockerd", "--host=tcp://0.0.0.0:2375", "--host=unix:///var/run/docker.sock", "--host=unix:///dind-sock/docker.sock", "--tls=false"]
+    # Two DNS settings, for two different resolvers, and both are load-bearing.
+    #
+    #   dns: (below)   the dind CONTAINER's resolv.conf, which is what dockerd
+    #                  itself uses to pull images. Without an explicit list it
+    #                  inherits Docker's embedded resolver, which forwards to
+    #                  the host's systemd-resolved; with Tailscale DNS enabled
+    #                  there, public names come back SERVFAIL and every job
+    #                  dies at "docker pull" of the runner image itself. This
+    #                  is what took the runner down on 2026-09-05.
+    #
+    #   --dns (here)   what dockerd hands to the JOB containers it creates.
+    #                  The tailnet resolver first, so NOMAD_ADDR and the
+    #                  staging Postgres resolve inside a job (they are
+    #                  MagicDNS names); a public resolver second, because
+    #                  100.100.100.100 answers only for the tailnet and
+    #                  SERVFAILs everything else -- getaddrinfo falls through.
+    #
+    # Allocate 256 /24 subnets instead of Docker's large default /16 and /20
+    # subnets. Jobs and their sibling test stacks share this daemon's pool.
+    # The default bridge uses one slot. Interrupted cleanup can retain others;
+    # a larger pool provides headroom but does not replace network cleanup.
+    command: ["dockerd", "--host=tcp://0.0.0.0:2375", "--host=unix:///var/run/docker.sock", "--host=unix:///dind-sock/docker.sock", "--tls=false", "--dns=172.28.0.53", "--dns=1.1.1.1", "--default-address-pool", "base=10.210.0.0/16,size=24"]
     environment:
       DOCKER_TLS_CERTDIR: ""
+    dns:
+      - 172.28.0.53
+      - 1.1.1.1
+    depends_on:
+      - dns
     networks:
       - runner
     volumes:
@@ -38,6 +77,8 @@ services:
     image: docker.io/gitea/act_runner@sha256:7940221bcfc9df16f3401ddafb2231bf98bba5517bc381224c7a2c504b558974
     container_name: gitea-runner-wsl
     restart: unless-stopped
+    # Longer than runner.shutdown_timeout so cancellation can clean up too.
+    stop_grace_period: 6m
     depends_on:
       dind:
         condition: service_healthy
@@ -66,6 +107,9 @@ services:
 
 networks:
   runner:
+    ipam:
+      config:
+        - subnet: 172.28.0.0/24
 
 volumes:
   dind-storage:

runner-stack.sh

@@ -0,0 +1,10 @@
+#!/usr/bin/env bash
+# Operate only this repository's runner stack, from any working directory.
+set -euo pipefail
+cd -- "$(dirname -- "${BASH_SOURCE[0]}")"
+if [ "$#" -eq 0 ]; then
+  printf 'Usage: %s <compose arguments>\n' "$0" >&2
+  printf 'Apply while idle: stop, then up -d --force-recreate\n' >&2
+  exit 2
+fi
+exec docker compose -f docker-compose.yml "$@"