Root cause found for the cascading task losses — and it corrects my previous post.
I set "verbose = 2" in the shipped job.toml and captured a recurrence in full. It is not the wrapper's container-state handling as I suggested last time.
The podman SSH connection to BOINC's VM drops intermittently, and docker_wrapper treats a single failed query as proof the container is gone.
The captured moment
A task running healthily for six minutes at 11.73% CPU and 310 MB, then:
running docker command: stats --no-stream --format "{{.CPUPerc}}\ {{.MemUsage}}" boinc__..._0
command output:
11.73% 310.1MB / 6.182GB
running docker command: ps --all -f "name=boinc__..._0"
program: ".../Run_Podman" /opt/homebrew/bin/podman
Cannot connect to Podman. Please verify your connection to the Linux system using
`podman system connection list`, or try `podman machine init` and `podman machine start`
Error: unable to connect to Podman socket: failed to connect: ssh: handshake failed: EOF
command output:
running docker command: logs boinc__..._0 -> same error, empty output
running docker command: container rm boinc__..._0 -> same error
running docker command: image rm boinc__... -> same error
2026-09-16 07:24:18 (34251): called boinc_finish(1)
Why a blip becomes a lost workunit
Podman reports the connection error on
stderr, which the wrapper does not capture — so as far as the wrapper is concerned the "ps" command returned
nothing. Empty output is indistinguishable from "the container no longer exists", so it concludes the job has ended and finishes with an error. Every following command fails the same way, so it cannot even kill or remove the container. Meanwhile the container is still running perfectly. A single retry would have saved the task.
Why it takes out everything at once
The connection is shared across the VM, so every running task's next poll fails within seconds of the others. In the 07:24 cascade on 16 Sep all
11 running tasks logged the error, with uptimes of 5, 6, 7, 8, 8, 9, 9, 17 and 36 minutes — healthy work well into its computation. The VM itself never stopped; it stayed up at 6144 MB throughout.
Corrections to what I posted before
"kills tasks ~1s after container start" wrong - these ran 5-36 minutes
"startup race / POLL_PERIOD" wrong - and read from the wrong wrapper source
"triggered by host load" wrong - a cascade fired at load 1.71
"triggered by concurrency" wrong - fires at 4, absent at 8
"cache-hit builds / shared image IDs" wrong - these happen constantly while healthy
"same bug as the ssh handshake cascades" correct - it is exactly that
I also earlier suggested capping concurrency as a mitigation for this. That does not help — concurrency is not the driver. The app_config.xml cap is still worth having for the 2 GiB VM memory problem, but not for this.
What would fix it
Two separate things:
1. docker_wrapper should retry a failed status query instead of treating one empty result as terminal, and should tell "the podman command failed" apart from "the container is gone".
2. The per-command connection model makes it far more likely to bite: every Run_Podman call opens a fresh SSH connection, and the wrapper polls ps plus stats per task every few seconds, so a long task accumulates thousands of chances to hit the drop.
Why the SSH connection drops at all is a podman/gvproxy-on-macOS question, outside BOINC's control. But the wrapper's reaction to it is what converts a transient blip into lost work.
Reported at
https://github.com/lfield/lhcathome/issues/21
Where this host stands
With the /cvmfs, VM memory and output-permission workarounds in place: 16 Sep saw
122 uploads against 9 losses, about 93% success. The remaining failures are this connection bug and the missing aarch64 generator builds.
If you are on macOS and see tasks die after running normally for several minutes, with "absent" output and nothing useful in the task log, check for "ssh: handshake failed: EOF". Setting verbose = 2 in job.toml makes it visible — note that keeping the byte count identical (verbose = 1 to verbose = 2) avoids BOINC re-downloading the file over its size check.
