fix: allow graceful runner shutdown during stack maintenance
This commit is contained in:
@@ -52,19 +52,58 @@ It uses `actions-rust-lang/setup-rust-toolchain` which caches `~/.cargo` and
|
||||
|
||||
## Manage
|
||||
|
||||
```bash
|
||||
docker compose up -d # apply config changes
|
||||
docker compose down # stop (registration persists in volume)
|
||||
```
|
||||
|
||||
⚠️ **Check for running jobs before recreating the runner.** Recreating it orphans
|
||||
any job in flight — Gitea marks them failed and you have to re-run:
|
||||
The wrapper operates only this project's Compose stack:
|
||||
|
||||
```bash
|
||||
docker exec gitea-runner-dind docker ps --format '{{.Names}}' | grep -v buildkit
|
||||
./runner-stack.sh up -d # start the stack
|
||||
./runner-stack.sh stop # stop runner before DinD; retain containers and volumes
|
||||
./runner-stack.sh down # remove containers; retain registration and caches
|
||||
```
|
||||
|
||||
Wait until that prints nothing (`buildx_buildkit_builder-*` is a helper, not a job).
|
||||
To apply an update on another existing runner, first wait for jobs to finish:
|
||||
|
||||
```bash
|
||||
docker exec gitea-runner-dind docker ps --format '{{.Names}}'
|
||||
```
|
||||
|
||||
Only persistent BuildKit helpers should remain. Then run:
|
||||
|
||||
```bash
|
||||
git pull --ff-only
|
||||
./runner-stack.sh stop
|
||||
./runner-stack.sh up -d --force-recreate
|
||||
```
|
||||
|
||||
Recreation loads changes in the bind-mounted runner YAML as well as Compose
|
||||
settings. It retains registration and cache volumes. Keep that machine's `.env`
|
||||
and intentional local capacity/label settings; resolve any pull conflicts while
|
||||
retaining `runner.shutdown_timeout: 5m`. A new installation still requires the
|
||||
per-machine registration setup above. Running the wrapper without arguments
|
||||
prints usage and does not change the stack.
|
||||
|
||||
The runner waits up to five minutes for jobs to finish before cancelling them.
|
||||
Compose allows six minutes before SIGKILL, leaving additional time for cleanup.
|
||||
This ordering applies to Compose operations, not power loss or independently
|
||||
stopping DinD. Any other clients sharing the daemon must be stopped separately
|
||||
before stack maintenance.
|
||||
|
||||
For changes only to `config.yaml`, restart the runner to load the file:
|
||||
|
||||
```bash
|
||||
./runner-stack.sh restart runner
|
||||
```
|
||||
|
||||
### Network allocation
|
||||
|
||||
DinD allocates /24s from `10.210.0.0/16`: 256 subnet slots, including one for
|
||||
its default bridge. Keep this setting when updating the daemon, and verify
|
||||
that this range does not overlap routes in another machine's environment.
|
||||
Host Docker has separate allocation state.
|
||||
|
||||
The larger pool provides headroom; interrupted jobs can still leave networks
|
||||
behind. Workflow-created networks need their own teardown on success, failure
|
||||
and cancellation. Inspect leftover networks inside DinD during idle maintenance;
|
||||
do not schedule blanket pruning while jobs are creating their stacks.
|
||||
|
||||
### Housekeeping
|
||||
|
||||
|
||||
@@ -5,6 +5,8 @@ runner:
|
||||
# Run multiple jobs concurrently. Overridden by env at compose level too.
|
||||
capacity: 4
|
||||
timeout: 3h
|
||||
# Let jobs finish before cancellation during planned maintenance.
|
||||
shutdown_timeout: 5m
|
||||
# Keep the runner alive; jobs are ephemeral containers.
|
||||
insecure: false
|
||||
# Env injected into every job. Speeds up Rust builds.
|
||||
|
||||
+6
-6
@@ -44,12 +44,10 @@ services:
|
||||
# 100.100.100.100 answers only for the tailnet and
|
||||
# SERVFAILs everything else -- getaddrinfo falls through.
|
||||
#
|
||||
# --default-address-pool replaces the daemon default of fifteen /16s in
|
||||
# 172.17-172.31, which a CI run exhausts ("all predefined address pools
|
||||
# have been fully subnetted"): the host holds three, each lane creates its
|
||||
# own network, and a killed job leaves one behind. 10.210/16 in /24s is 256
|
||||
# networks and collides with nothing here (no 10.x route on the host; the
|
||||
# tailnet is 100.64/10).
|
||||
# Allocate 256 /24 subnets instead of Docker's large default /16 and /20
|
||||
# subnets. Jobs and their sibling test stacks share this daemon's pool.
|
||||
# The default bridge uses one slot. Interrupted cleanup can retain others;
|
||||
# a larger pool provides headroom but does not replace network cleanup.
|
||||
command: ["dockerd", "--host=tcp://0.0.0.0:2375", "--host=unix:///var/run/docker.sock", "--host=unix:///dind-sock/docker.sock", "--tls=false", "--dns=172.28.0.53", "--dns=1.1.1.1", "--default-address-pool", "base=10.210.0.0/16,size=24"]
|
||||
environment:
|
||||
DOCKER_TLS_CERTDIR: ""
|
||||
@@ -79,6 +77,8 @@ services:
|
||||
image: docker.io/gitea/act_runner@sha256:7940221bcfc9df16f3401ddafb2231bf98bba5517bc381224c7a2c504b558974
|
||||
container_name: gitea-runner-wsl
|
||||
restart: unless-stopped
|
||||
# Longer than runner.shutdown_timeout so cancellation can clean up too.
|
||||
stop_grace_period: 6m
|
||||
depends_on:
|
||||
dind:
|
||||
condition: service_healthy
|
||||
|
||||
Executable
+10
@@ -0,0 +1,10 @@
|
||||
#!/usr/bin/env bash
|
||||
# Operate only this repository's runner stack, from any working directory.
|
||||
set -euo pipefail
|
||||
cd -- "$(dirname -- "${BASH_SOURCE[0]}")"
|
||||
if [ "$#" -eq 0 ]; then
|
||||
printf 'Usage: %s <compose arguments>\n' "$0" >&2
|
||||
printf 'Apply while idle: stop, then up -d --force-recreate\n' >&2
|
||||
exit 2
|
||||
fi
|
||||
exec docker compose -f docker-compose.yml "$@"
|
||||
Reference in New Issue
Block a user