fix: allow graceful runner shutdown during stack maintenance

This commit is contained in:
2026-09-08 17:52:37 +07:00
parent 9f21d349f6
commit 9b6f5c426e
4 changed files with 66 additions and 15 deletions
+48 -9
View File
@@ -52,19 +52,58 @@ It uses `actions-rust-lang/setup-rust-toolchain` which caches `~/.cargo` and
## Manage ## Manage
```bash The wrapper operates only this project's Compose stack:
docker compose up -d # apply config changes
docker compose down # stop (registration persists in volume)
```
⚠️ **Check for running jobs before recreating the runner.** Recreating it orphans
any job in flight — Gitea marks them failed and you have to re-run:
```bash ```bash
docker exec gitea-runner-dind docker ps --format '{{.Names}}' | grep -v buildkit ./runner-stack.sh up -d # start the stack
./runner-stack.sh stop # stop runner before DinD; retain containers and volumes
./runner-stack.sh down # remove containers; retain registration and caches
``` ```
Wait until that prints nothing (`buildx_buildkit_builder-*` is a helper, not a job). To apply an update on another existing runner, first wait for jobs to finish:
```bash
docker exec gitea-runner-dind docker ps --format '{{.Names}}'
```
Only persistent BuildKit helpers should remain. Then run:
```bash
git pull --ff-only
./runner-stack.sh stop
./runner-stack.sh up -d --force-recreate
```
Recreation loads changes in the bind-mounted runner YAML as well as Compose
settings. It retains registration and cache volumes. Keep that machine's `.env`
and intentional local capacity/label settings; resolve any pull conflicts while
retaining `runner.shutdown_timeout: 5m`. A new installation still requires the
per-machine registration setup above. Running the wrapper without arguments
prints usage and does not change the stack.
The runner waits up to five minutes for jobs to finish before cancelling them.
Compose allows six minutes before SIGKILL, leaving additional time for cleanup.
This ordering applies to Compose operations, not power loss or independently
stopping DinD. Any other clients sharing the daemon must be stopped separately
before stack maintenance.
For changes only to `config.yaml`, restart the runner to load the file:
```bash
./runner-stack.sh restart runner
```
### Network allocation
DinD allocates /24s from `10.210.0.0/16`: 256 subnet slots, including one for
its default bridge. Keep this setting when updating the daemon, and verify
that this range does not overlap routes in another machine's environment.
Host Docker has separate allocation state.
The larger pool provides headroom; interrupted jobs can still leave networks
behind. Workflow-created networks need their own teardown on success, failure
and cancellation. Inspect leftover networks inside DinD during idle maintenance;
do not schedule blanket pruning while jobs are creating their stacks.
### Housekeeping ### Housekeeping
+2
View File
@@ -5,6 +5,8 @@ runner:
# Run multiple jobs concurrently. Overridden by env at compose level too. # Run multiple jobs concurrently. Overridden by env at compose level too.
capacity: 4 capacity: 4
timeout: 3h timeout: 3h
# Let jobs finish before cancellation during planned maintenance.
shutdown_timeout: 5m
# Keep the runner alive; jobs are ephemeral containers. # Keep the runner alive; jobs are ephemeral containers.
insecure: false insecure: false
# Env injected into every job. Speeds up Rust builds. # Env injected into every job. Speeds up Rust builds.
+6 -6
View File
@@ -44,12 +44,10 @@ services:
# 100.100.100.100 answers only for the tailnet and # 100.100.100.100 answers only for the tailnet and
# SERVFAILs everything else -- getaddrinfo falls through. # SERVFAILs everything else -- getaddrinfo falls through.
# #
# --default-address-pool replaces the daemon default of fifteen /16s in # Allocate 256 /24 subnets instead of Docker's large default /16 and /20
# 172.17-172.31, which a CI run exhausts ("all predefined address pools # subnets. Jobs and their sibling test stacks share this daemon's pool.
# have been fully subnetted"): the host holds three, each lane creates its # The default bridge uses one slot. Interrupted cleanup can retain others;
# own network, and a killed job leaves one behind. 10.210/16 in /24s is 256 # a larger pool provides headroom but does not replace network cleanup.
# networks and collides with nothing here (no 10.x route on the host; the
# tailnet is 100.64/10).
command: ["dockerd", "--host=tcp://0.0.0.0:2375", "--host=unix:///var/run/docker.sock", "--host=unix:///dind-sock/docker.sock", "--tls=false", "--dns=172.28.0.53", "--dns=1.1.1.1", "--default-address-pool", "base=10.210.0.0/16,size=24"] command: ["dockerd", "--host=tcp://0.0.0.0:2375", "--host=unix:///var/run/docker.sock", "--host=unix:///dind-sock/docker.sock", "--tls=false", "--dns=172.28.0.53", "--dns=1.1.1.1", "--default-address-pool", "base=10.210.0.0/16,size=24"]
environment: environment:
DOCKER_TLS_CERTDIR: "" DOCKER_TLS_CERTDIR: ""
@@ -79,6 +77,8 @@ services:
image: docker.io/gitea/act_runner@sha256:7940221bcfc9df16f3401ddafb2231bf98bba5517bc381224c7a2c504b558974 image: docker.io/gitea/act_runner@sha256:7940221bcfc9df16f3401ddafb2231bf98bba5517bc381224c7a2c504b558974
container_name: gitea-runner-wsl container_name: gitea-runner-wsl
restart: unless-stopped restart: unless-stopped
# Longer than runner.shutdown_timeout so cancellation can clean up too.
stop_grace_period: 6m
depends_on: depends_on:
dind: dind:
condition: service_healthy condition: service_healthy
+10
View File
@@ -0,0 +1,10 @@
#!/usr/bin/env bash
# Operate only this repository's runner stack, from any working directory.
set -euo pipefail
cd -- "$(dirname -- "${BASH_SOURCE[0]}")"
if [ "$#" -eq 0 ]; then
printf 'Usage: %s <compose arguments>\n' "$0" >&2
printf 'Apply while idle: stop, then up -d --force-recreate\n' >&2
exit 2
fi
exec docker compose -f docker-compose.yml "$@"