fix: allow graceful runner shutdown during stack maintenance
This commit is contained in:
@@ -52,19 +52,58 @@ It uses `actions-rust-lang/setup-rust-toolchain` which caches `~/.cargo` and
|
|||||||
|
|
||||||
## Manage
|
## Manage
|
||||||
|
|
||||||
```bash
|
The wrapper operates only this project's Compose stack:
|
||||||
docker compose up -d # apply config changes
|
|
||||||
docker compose down # stop (registration persists in volume)
|
|
||||||
```
|
|
||||||
|
|
||||||
⚠️ **Check for running jobs before recreating the runner.** Recreating it orphans
|
|
||||||
any job in flight — Gitea marks them failed and you have to re-run:
|
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
docker exec gitea-runner-dind docker ps --format '{{.Names}}' | grep -v buildkit
|
./runner-stack.sh up -d # start the stack
|
||||||
|
./runner-stack.sh stop # stop runner before DinD; retain containers and volumes
|
||||||
|
./runner-stack.sh down # remove containers; retain registration and caches
|
||||||
```
|
```
|
||||||
|
|
||||||
Wait until that prints nothing (`buildx_buildkit_builder-*` is a helper, not a job).
|
To apply an update on another existing runner, first wait for jobs to finish:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
docker exec gitea-runner-dind docker ps --format '{{.Names}}'
|
||||||
|
```
|
||||||
|
|
||||||
|
Only persistent BuildKit helpers should remain. Then run:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git pull --ff-only
|
||||||
|
./runner-stack.sh stop
|
||||||
|
./runner-stack.sh up -d --force-recreate
|
||||||
|
```
|
||||||
|
|
||||||
|
Recreation loads changes in the bind-mounted runner YAML as well as Compose
|
||||||
|
settings. It retains registration and cache volumes. Keep that machine's `.env`
|
||||||
|
and intentional local capacity/label settings; resolve any pull conflicts while
|
||||||
|
retaining `runner.shutdown_timeout: 5m`. A new installation still requires the
|
||||||
|
per-machine registration setup above. Running the wrapper without arguments
|
||||||
|
prints usage and does not change the stack.
|
||||||
|
|
||||||
|
The runner waits up to five minutes for jobs to finish before cancelling them.
|
||||||
|
Compose allows six minutes before SIGKILL, leaving additional time for cleanup.
|
||||||
|
This ordering applies to Compose operations, not power loss or independently
|
||||||
|
stopping DinD. Any other clients sharing the daemon must be stopped separately
|
||||||
|
before stack maintenance.
|
||||||
|
|
||||||
|
For changes only to `config.yaml`, restart the runner to load the file:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
./runner-stack.sh restart runner
|
||||||
|
```
|
||||||
|
|
||||||
|
### Network allocation
|
||||||
|
|
||||||
|
DinD allocates /24s from `10.210.0.0/16`: 256 subnet slots, including one for
|
||||||
|
its default bridge. Keep this setting when updating the daemon, and verify
|
||||||
|
that this range does not overlap routes in another machine's environment.
|
||||||
|
Host Docker has separate allocation state.
|
||||||
|
|
||||||
|
The larger pool provides headroom; interrupted jobs can still leave networks
|
||||||
|
behind. Workflow-created networks need their own teardown on success, failure
|
||||||
|
and cancellation. Inspect leftover networks inside DinD during idle maintenance;
|
||||||
|
do not schedule blanket pruning while jobs are creating their stacks.
|
||||||
|
|
||||||
### Housekeeping
|
### Housekeeping
|
||||||
|
|
||||||
|
|||||||
@@ -5,6 +5,8 @@ runner:
|
|||||||
# Run multiple jobs concurrently. Overridden by env at compose level too.
|
# Run multiple jobs concurrently. Overridden by env at compose level too.
|
||||||
capacity: 4
|
capacity: 4
|
||||||
timeout: 3h
|
timeout: 3h
|
||||||
|
# Let jobs finish before cancellation during planned maintenance.
|
||||||
|
shutdown_timeout: 5m
|
||||||
# Keep the runner alive; jobs are ephemeral containers.
|
# Keep the runner alive; jobs are ephemeral containers.
|
||||||
insecure: false
|
insecure: false
|
||||||
# Env injected into every job. Speeds up Rust builds.
|
# Env injected into every job. Speeds up Rust builds.
|
||||||
|
|||||||
+6
-6
@@ -44,12 +44,10 @@ services:
|
|||||||
# 100.100.100.100 answers only for the tailnet and
|
# 100.100.100.100 answers only for the tailnet and
|
||||||
# SERVFAILs everything else -- getaddrinfo falls through.
|
# SERVFAILs everything else -- getaddrinfo falls through.
|
||||||
#
|
#
|
||||||
# --default-address-pool replaces the daemon default of fifteen /16s in
|
# Allocate 256 /24 subnets instead of Docker's large default /16 and /20
|
||||||
# 172.17-172.31, which a CI run exhausts ("all predefined address pools
|
# subnets. Jobs and their sibling test stacks share this daemon's pool.
|
||||||
# have been fully subnetted"): the host holds three, each lane creates its
|
# The default bridge uses one slot. Interrupted cleanup can retain others;
|
||||||
# own network, and a killed job leaves one behind. 10.210/16 in /24s is 256
|
# a larger pool provides headroom but does not replace network cleanup.
|
||||||
# networks and collides with nothing here (no 10.x route on the host; the
|
|
||||||
# tailnet is 100.64/10).
|
|
||||||
command: ["dockerd", "--host=tcp://0.0.0.0:2375", "--host=unix:///var/run/docker.sock", "--host=unix:///dind-sock/docker.sock", "--tls=false", "--dns=172.28.0.53", "--dns=1.1.1.1", "--default-address-pool", "base=10.210.0.0/16,size=24"]
|
command: ["dockerd", "--host=tcp://0.0.0.0:2375", "--host=unix:///var/run/docker.sock", "--host=unix:///dind-sock/docker.sock", "--tls=false", "--dns=172.28.0.53", "--dns=1.1.1.1", "--default-address-pool", "base=10.210.0.0/16,size=24"]
|
||||||
environment:
|
environment:
|
||||||
DOCKER_TLS_CERTDIR: ""
|
DOCKER_TLS_CERTDIR: ""
|
||||||
@@ -79,6 +77,8 @@ services:
|
|||||||
image: docker.io/gitea/act_runner@sha256:7940221bcfc9df16f3401ddafb2231bf98bba5517bc381224c7a2c504b558974
|
image: docker.io/gitea/act_runner@sha256:7940221bcfc9df16f3401ddafb2231bf98bba5517bc381224c7a2c504b558974
|
||||||
container_name: gitea-runner-wsl
|
container_name: gitea-runner-wsl
|
||||||
restart: unless-stopped
|
restart: unless-stopped
|
||||||
|
# Longer than runner.shutdown_timeout so cancellation can clean up too.
|
||||||
|
stop_grace_period: 6m
|
||||||
depends_on:
|
depends_on:
|
||||||
dind:
|
dind:
|
||||||
condition: service_healthy
|
condition: service_healthy
|
||||||
|
|||||||
Executable
+10
@@ -0,0 +1,10 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# Operate only this repository's runner stack, from any working directory.
|
||||||
|
set -euo pipefail
|
||||||
|
cd -- "$(dirname -- "${BASH_SOURCE[0]}")"
|
||||||
|
if [ "$#" -eq 0 ]; then
|
||||||
|
printf 'Usage: %s <compose arguments>\n' "$0" >&2
|
||||||
|
printf 'Apply while idle: stop, then up -d --force-recreate\n' >&2
|
||||||
|
exit 2
|
||||||
|
fi
|
||||||
|
exec docker compose -f docker-compose.yml "$@"
|
||||||
Reference in New Issue
Block a user