fix: allow graceful runner shutdown during stack maintenance

This commit is contained in:
2026-09-08 17:52:37 +07:00
parent 9f21d349f6
commit 9b6f5c426e
4 changed files with 66 additions and 15 deletions
+48 -9
View File
@@ -52,19 +52,58 @@ It uses `actions-rust-lang/setup-rust-toolchain` which caches `~/.cargo` and
## Manage
```bash
docker compose up -d # apply config changes
docker compose down # stop (registration persists in volume)
```
⚠️ **Check for running jobs before recreating the runner.** Recreating it orphans
any job in flight — Gitea marks them failed and you have to re-run:
The wrapper operates only this project's Compose stack:
```bash
docker exec gitea-runner-dind docker ps --format '{{.Names}}' | grep -v buildkit
./runner-stack.sh up -d # start the stack
./runner-stack.sh stop # stop runner before DinD; retain containers and volumes
./runner-stack.sh down # remove containers; retain registration and caches
```
Wait until that prints nothing (`buildx_buildkit_builder-*` is a helper, not a job).
To apply an update on another existing runner, first wait for jobs to finish:
```bash
docker exec gitea-runner-dind docker ps --format '{{.Names}}'
```
Only persistent BuildKit helpers should remain. Then run:
```bash
git pull --ff-only
./runner-stack.sh stop
./runner-stack.sh up -d --force-recreate
```
Recreation loads changes in the bind-mounted runner YAML as well as Compose
settings. It retains registration and cache volumes. Keep that machine's `.env`
and intentional local capacity/label settings; resolve any pull conflicts while
retaining `runner.shutdown_timeout: 5m`. A new installation still requires the
per-machine registration setup above. Running the wrapper without arguments
prints usage and does not change the stack.
The runner waits up to five minutes for jobs to finish before cancelling them.
Compose allows six minutes before SIGKILL, leaving additional time for cleanup.
This ordering applies to Compose operations, not power loss or independently
stopping DinD. Any other clients sharing the daemon must be stopped separately
before stack maintenance.
For changes only to `config.yaml`, restart the runner to load the file:
```bash
./runner-stack.sh restart runner
```
### Network allocation
DinD allocates /24s from `10.210.0.0/16`: 256 subnet slots, including one for
its default bridge. Keep this setting when updating the daemon, and verify
that this range does not overlap routes in another machine's environment.
Host Docker has separate allocation state.
The larger pool provides headroom; interrupted jobs can still leave networks
behind. Workflow-created networks need their own teardown on success, failure
and cancellation. Inspect leftover networks inside DinD during idle maintenance;
do not schedule blanket pruning while jobs are creating their stacks.
### Housekeeping
+2
View File
@@ -5,6 +5,8 @@ runner:
# Run multiple jobs concurrently. Overridden by env at compose level too.
capacity: 4
timeout: 3h
# Let jobs finish before cancellation during planned maintenance.
shutdown_timeout: 5m
# Keep the runner alive; jobs are ephemeral containers.
insecure: false
# Env injected into every job. Speeds up Rust builds.
+6 -6
View File
@@ -44,12 +44,10 @@ services:
# 100.100.100.100 answers only for the tailnet and
# SERVFAILs everything else -- getaddrinfo falls through.
#
# --default-address-pool replaces the daemon default of fifteen /16s in
# 172.17-172.31, which a CI run exhausts ("all predefined address pools
# have been fully subnetted"): the host holds three, each lane creates its
# own network, and a killed job leaves one behind. 10.210/16 in /24s is 256
# networks and collides with nothing here (no 10.x route on the host; the
# tailnet is 100.64/10).
# Allocate 256 /24 subnets instead of Docker's large default /16 and /20
# subnets. Jobs and their sibling test stacks share this daemon's pool.
# The default bridge uses one slot. Interrupted cleanup can retain others;
# a larger pool provides headroom but does not replace network cleanup.
command: ["dockerd", "--host=tcp://0.0.0.0:2375", "--host=unix:///var/run/docker.sock", "--host=unix:///dind-sock/docker.sock", "--tls=false", "--dns=172.28.0.53", "--dns=1.1.1.1", "--default-address-pool", "base=10.210.0.0/16,size=24"]
environment:
DOCKER_TLS_CERTDIR: ""
@@ -79,6 +77,8 @@ services:
image: docker.io/gitea/act_runner@sha256:7940221bcfc9df16f3401ddafb2231bf98bba5517bc381224c7a2c504b558974
container_name: gitea-runner-wsl
restart: unless-stopped
# Longer than runner.shutdown_timeout so cancellation can clean up too.
stop_grace_period: 6m
depends_on:
dind:
condition: service_healthy
+10
View File
@@ -0,0 +1,10 @@
#!/usr/bin/env bash
# Operate only this repository's runner stack, from any working directory.
set -euo pipefail
cd -- "$(dirname -- "${BASH_SOURCE[0]}")"
if [ "$#" -eq 0 ]; then
printf 'Usage: %s <compose arguments>\n' "$0" >&2
printf 'Apply while idle: stop, then up -d --force-recreate\n' >&2
exit 2
fi
exec docker compose -f docker-compose.yml "$@"