I should have said it before.

I am running it on VMware workstation.

And since there is no way to deactivate docker tasks in the project preferences, I mainly get docker tasks and few VBox so it keeps failing again and again... and they don't fail immediately but CPU cycles are wasted each time (between 1 and 8 mn), so this is definitively not good, I'm forced to completely deactivated theory now...

Root cause found for the cascading task losses — and it corrects my previous post.

I set "verbose = 2" in the shipped job.toml and captured a recurrence in full. It is not the wrapper's container-state handling as I suggested last time.

The podman SSH connection to BOINC's VM drops intermittently, and docker_wrapper treats a single failed query as proof the container is gone.

The captured moment

A task running healthily for six minutes at 11.73% CPU and 310 MB, then:

running docker command: stats --no-stream --format "{{.CPUPerc}}\ {{.MemUsage}}" boinc__..._0 command output: 11.73% 310.1MB / 6.182GB running docker command: ps --all -f "name=boinc__..._0" program: ".../Run_Podman" /opt/homebrew/bin/podman Cannot connect to Podman. Please verify your connection to the Linux system using `podman system connection list`, or try `podman machine init` and `podman machine start` Error: unable to connect to Podman socket: failed to connect: ssh: handshake failed: EOF command output: running docker command: logs boinc__..._0 -> same error, empty output running docker command: container rm boinc__..._0 -> same error running docker command: image rm boinc__... -> same error 2026-09-16 07:24:18 (34251): called boinc_finish(1)

Why a blip becomes a lost workunit

Podman reports the connection error on stderr, which the wrapper does not capture — so as far as the wrapper is concerned the "ps" command returned nothing. Empty output is indistinguishable from "the container no longer exists", so it concludes the job has ended and finishes with an error. Every following command fails the same way, so it cannot even kill or remove the container. Meanwhile the container is still running perfectly. A single retry would have saved the task.

Why it takes out everything at once

The connection is shared across the VM, so every running task's next poll fails within seconds of the others. In the 07:24 cascade on 16 Sep all 11 running tasks logged the error, with uptimes of 5, 6, 7, 8, 8, 9, 9, 17 and 36 minutes — healthy work well into its computation. The VM itself never stopped; it stayed up at 6144 MB throughout.

Corrections to what I posted before

"kills tasks ~1s after container start" wrong - these ran 5-36 minutes "startup race / POLL_PERIOD" wrong - and read from the wrong wrapper source "triggered by host load" wrong - a cascade fired at load 1.71 "triggered by concurrency" wrong - fires at 4, absent at 8 "cache-hit builds / shared image IDs" wrong - these happen constantly while healthy "same bug as the ssh handshake cascades" correct - it is exactly that

I also earlier suggested capping concurrency as a mitigation for this. That does not help — concurrency is not the driver. The app_config.xml cap is still worth having for the 2 GiB VM memory problem, but not for this.

What would fix it

Two separate things:

1. docker_wrapper should retry a failed status query instead of treating one empty result as terminal, and should tell "the podman command failed" apart from "the container is gone".
2. The per-command connection model makes it far more likely to bite: every Run_Podman call opens a fresh SSH connection, and the wrapper polls ps plus stats per task every few seconds, so a long task accumulates thousands of chances to hit the drop.

Why the SSH connection drops at all is a podman/gvproxy-on-macOS question, outside BOINC's control. But the wrapper's reaction to it is what converts a transient blip into lost work.

Reported at https://github.com/lfield/lhcathome/issues/21

Where this host stands

With the /cvmfs, VM memory and output-permission workarounds in place: 16 Sep saw 122 uploads against 9 losses, about 93% success. The remaining failures are this connection bug and the missing aarch64 generator builds.

If you are on macOS and see tasks die after running normally for several minutes, with "absent" output and nothing useful in the task log, check for "ssh: handshake failed: EOF". Setting verbose = 2 in job.toml makes it visible — note that keeping the byte count identical (verbose = 1 to verbose = 2) avoids BOINC re-downloading the file over its size check.

In reply to Erich56's message of 4 Sep 2026:
on one of my machines, a Theory task with 100.000 events has now been running for 68 days und has so far fnished some 67.000 events.
Question: does it make sense to keep the task running? Or is it already invalid due to overly long runtime ?
OMG!
Check the WU status on the project website. If there is already a canonical result from another host, abort the task immediately.
If not, you can continue - Ok, I think so. :)

In reply to airgunnut's message of 29 Aug 2026:
at the time of posting i have 10 tasks, 2 at 23 days 1 at 5 days, the rest in between and all at 100%, im running AMD Ryzen 9 9950X3D 16-Core Processor (4.30 GHz) im not going to say how much ram there is as i do not want a target on my back
I would have to brag if I had 95.65 GB ram in one host
With your long running tasks just check to see if they are running in the *slots* for each one if not delete it and that info is in the thread on this subject https://lhcathome.cern.ch/lhcathome/forum_thread.php?id=6251
But yours says it has nothing running
https://lhcathome.cern.ch/lhcathome/results.php?hostid=11015514

# ==========================================
# Bunker Zombie Exterminator v1.1, English
# LHC@home Theory startup VM zombie killer
# Full plug principle, DST-safe, daily log cleanup edition
# ==========================================
#
# Rule of the shovel:
#
# 1. If the BOINC slot root contains Theory*.vdi,
# this slot is treated as an LHC Theory VM.
#
# 2. VM start time is taken from this slot stderr.txt line:
# Starting VM using VBoxManage interface
#
# 3. If shared\runRivet.log exists and is not empty,
# the VM is alive. It has coughed into the Rivet jar.
#
# 4. If GraceSeconds seconds have passed since VM start
# and shared\runRivet.log does not exist or is empty,
# this is a startup zombie.
#
# 5. Then the orderly prescribes a brick.
#
# Clock-twisting safety catch:
# - start time and current time are converted to DateTimeOffset
# using the local timezone UTC offset for that exact moment;
# - if the hour is ambiguous, the most conservative smallest
# non-negative difference is used;
# - if the time is invalid or looks like it came from the future, do not touch.
#
# Log cleanup:
# - when the calendar day changes, the log file is cleaned;
# - if the script is started on the next day, the old log is cleaned too;
# - if the log grows above MaxLogMB during a day, fallback rotation still works.
#
# Stderr is used only to find VM start time.
# Stderr content is not flea-combed for CERN tea leaves.
# BOINC XML is used only to map an active slot to a task/result name.
#
# Run without kill:
# powershell.exe -NoProfile -ExecutionPolicy Bypass -File "C:\BOINC_Tools\Bunker-Zombie-Killer.ps1" -Loop -GraceSeconds 900
#
# Run with kill:
# powershell.exe -NoProfile -ExecutionPolicy Bypass -File "C:\BOINC_Tools\Bunker-Zombie-Killer.ps1" -Kill -Loop -GraceSeconds 900
#

param(
[string]$BoincDataDir = "C:\ProgramData\BOINC",
[string]$BoincCmd = "C:\Program Files\BOINC\boinccmd.exe",

[int]$GraceSeconds = 900,
[int]$CheckIntervalSeconds = 60,

[switch]$Kill,
[switch]$Loop,

[string]$DefaultProjectUrl = "https://lhcathome.cern.ch/lhcathome/",
[string]$LogFile = "C:\BOINC_Tools\Bunker-Zombie-Killer.log",

[int]$MaxLogMB = 5
)

$slotsDir = Join-Path $BoincDataDir "slots"
$clientStatePath = Join-Path $BoincDataDir "client_state.xml"

$LastLogByKey = @{}
$KilledThisRun = @{}

$script:InvariantCulture = [System.Globalization.CultureInfo]::InvariantCulture
$script:LogTimestampFormat = "yyyy-MM-dd HH':'mm':'ss"
$script:StderrTimestampFormat = "yyyy-MM-dd HH':'mm':'ss"

# Initial day marker. Needed for automatic log floor-washing after midnight.
$script:LogDay = (Get-Date).Date
$script:LogStartupChecked = $false

function Clear-BunkerLogIfNewDay {
param(
[datetime]$Now
)

$today = $Now.Date
$cleared = $false

try {
# First log write after script start:
# if the log file was left from a previous day, clean it.
if (-not $script:LogStartupChecked) {
$script:LogStartupChecked = $true
$script:LogDay = $today

if (Test-Path $LogFile) {
$info = Get-Item $LogFile -ErrorAction Stop

if ($info.LastWriteTime.Date -lt $today) {
Clear-Content -Path $LogFile -Force -ErrorAction Stop
$cleared = $true
}
}

return $cleared
}

# Normal patrol:
# if the calendar day changed, clean the log.
# No need to hit exactly 00:00:00 like a drunk cuckoo clock.
if ($today -ne $script:LogDay) {
$script:LogDay = $today

if (Test-Path $LogFile) {
Clear-Content -Path $LogFile -Force -ErrorAction Stop
}

$cleared = $true
}
}
catch {
# If cleanup fails, the orderly does not explode.
$cleared = $false
}

return $cleared
}

function Rotate-BunkerLogIfNeeded {
try {
if (-not (Test-Path $LogFile)) {
return
}

$maxBytes = $MaxLogMB * 1024 * 1024
$fileInfo = Get-Item $LogFile -ErrorAction Stop

if ($fileInfo.Length -lt $maxBytes) {
return
}

$oldLog = "$LogFile.old"

if (Test-Path $oldLog) {
Remove-Item $oldLog -Force -ErrorAction SilentlyContinue
}

Rename-Item -Path $LogFile -NewName (Split-Path $oldLog -Leaf) -Force
}
catch {
# If log rotation fails, keep writing. The bunker survives.
}
}

function Write-BunkerLog {
param(
[string]$Message,
[string]$Level = "INFO",
[ConsoleColor]$Color = "Gray"
)

$now = Get-Date
$ts = $now.ToString($script:LogTimestampFormat, $script:InvariantCulture)
$line = "$ts [$Level] $Message"

Write-Host $line -ForegroundColor $Color

try {
$logDir = Split-Path $LogFile -Parent

if (-not (Test-Path $logDir)) {
New-Item -ItemType Directory -Path $logDir -Force | Out-Null
}

$dailyCleared = Clear-BunkerLogIfNewDay -Now $now

if ($dailyCleared) {
$clearLine = "$ts [INFO] New day. Yesterday's log droppings were swept away."
Write-Host $clearLine -ForegroundColor DarkGray
Add-Content -Path $LogFile -Value $clearLine
}
else {
Rotate-BunkerLogIfNeeded
}

Add-Content -Path $LogFile -Value $line
}
catch {
Write-Host "Could not write to log file: $($_.Exception.Message)" -ForegroundColor Yellow
}
}

function Write-BunkerLogThrottled {
param(
[string]$Key,
[string]$Message,
[string]$Level = "INFO",
[ConsoleColor]$Color = "Gray",
[int]$IntervalSeconds = 1800
)

$now = Get-Date

if (-not $LastLogByKey.ContainsKey($Key)) {
$LastLogByKey[$Key] = $now
Write-BunkerLog $Message $Level $Color
return
}

$age = ($now - $LastLogByKey[$Key]).TotalSeconds

if ($age -ge $IntervalSeconds) {
$LastLogByKey[$Key] = $now
Write-BunkerLog $Message $Level $Color
}
}

function Get-ClientStateXml {
if (-not (Test-Path $clientStatePath)) {
Write-BunkerLogThrottled `
-Key "NO_CLIENT_STATE" `
-Message "client_state.xml not found: $clientStatePath" `
-Level "ERROR" `
-Color Red `
-IntervalSeconds 300
return $null
}

for ($i = 1; $i -le 3; $i++) {
try {
return [xml](Get-Content $clientStatePath -Raw -ErrorAction Stop)
}
catch {
Start-Sleep -Milliseconds 250
}
}

Write-BunkerLogThrottled `
-Key "BAD_CLIENT_STATE" `
-Message "Could not read client_state.xml. BOINC may be rewriting it right now. Do not touch." `
-Level "WARN" `
-Color Yellow `
-IntervalSeconds 300

return $null
}

function Get-SlotPath {
param([int]$Slot)

return Join-Path $slotsDir ([string]$Slot)
}

function Get-SharedPath {
param([int]$Slot)

return Join-Path (Get-SlotPath $Slot) "shared"
}

function Get-StderrPath {
param([int]$Slot)

return Join-Path (Get-SlotPath $Slot) "stderr.txt"
}

function Test-SlotHasTheoryVdi {
param([int]$Slot)

$slotPath = Get-SlotPath $Slot

if (-not (Test-Path $slotPath)) {
return $false
}

try {
$vdi = Get-ChildItem -Path $slotPath -Filter "Theory*.vdi" -File -ErrorAction SilentlyContinue |
Select-Object -First 1

return ($null -ne $vdi)
}
catch {
return $false
}
}

function Get-StderrText {
param([int]$Slot)

$stderrPath = Get-StderrPath $Slot

if (-not (Test-Path $stderrPath)) {
return ""
}

try {
return Get-Content $stderrPath -Raw -ErrorAction Stop
}
catch {
return ""
}
}

function Get-VMStartTimeFromStderr {
param([int]$Slot)

$text = Get-StderrText -Slot $Slot

if ([string]::IsNullOrWhiteSpace($text)) {
return $null
}

$pattern = "(?m)^(?<ts>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2})\s+\(\d+\):\s+Starting VM using VBoxManage interface\."

$matches = [regex]::Matches($text, $pattern)

if ($matches.Count -eq 0) {
return $null
}

# Take the last VM start, because stderr may contain more than one segment.
$last = $matches[$matches.Count - 1]
$tsText = $last.Groups["ts"].Value

try {
return [datetime]::ParseExact(
$tsText,
$script:StderrTimestampFormat,
$script:InvariantCulture
)
}
catch {
return $null
}
}

function Get-NowLocalUnspecified {
$now = Get-Date

return [datetime]::new(
$now.Year,
$now.Month,
$now.Day,
$now.Hour,
$now.Minute,
$now.Second
)
}

function Get-OffsetsForLocalTime {
param(
[datetime]$LocalTime
)

$tz = [System.TimeZoneInfo]::Local

try {
if ($tz.IsInvalidTime($LocalTime)) {
return @()
}

if ($tz.IsAmbiguousTime($LocalTime)) {
return @($tz.GetAmbiguousTimeOffsets($LocalTime))
}

return @($tz.GetUtcOffset($LocalTime))
}
catch {
return @()
}
}

function Get-SafeElapsedSeconds {
param(
[datetime]$StartLocal
)

$nowLocal = Get-NowLocalUnspecified

$startOffsets = @(Get-OffsetsForLocalTime -LocalTime $StartLocal)
$nowOffsets = @(Get-OffsetsForLocalTime -LocalTime $nowLocal)

if ($startOffsets.Count -eq 0) {
return [pscustomobject]@{
IsUsable = $false
Elapsed = 0.0
NowLocal = $nowLocal
OffsetChanged = $false
Message = "VM start time does not exist in local timezone. Do not touch."
}
}

if ($nowOffsets.Count -eq 0) {
return [pscustomobject]@{
IsUsable = $false
Elapsed = 0.0
NowLocal = $nowLocal
OffsetChanged = $false
Message = "Current local time does not exist in local timezone. Do not touch."
}
}

$diffs = @()
$offsetChanged = $false

foreach ($so in $startOffsets) {
foreach ($no in $nowOffsets) {
if ($so.TotalSeconds -ne $no.TotalSeconds) {
$offsetChanged = $true
}

try {
$startDto = [System.DateTimeOffset]::new($StartLocal, $so)
$nowDto = [System.DateTimeOffset]::new($nowLocal, $no)
$diffs += (($nowDto - $startDto).TotalSeconds)
}
catch {
# Bad offset combination. Ignore it.
}
}
}

if ($diffs.Count -eq 0) {
return [pscustomobject]@{
IsUsable = $false
Elapsed = 0.0
NowLocal = $nowLocal
OffsetChanged = $offsetChanged
Message = "Could not calculate safe VM age. Do not touch."
}
}

# Conservative rule:
# if several differences are possible, use the smallest non-negative one.
# This prevents the autumn double-hour swamp from killing a VM too early.
$nonNegativeDiffs = @($diffs | Where-Object { $_ -ge 0 } | Sort-Object)

if ($nonNegativeDiffs.Count -gt 0) {
$elapsed = [double]$nonNegativeDiffs[0]
}
else {
# If all differences are negative, assume the clock is doing circus tricks.
$elapsed = [double](($diffs | Sort-Object -Descending)[0])
}

return [pscustomobject]@{
IsUsable = $true
Elapsed = $elapsed
NowLocal = $nowLocal
OffsetChanged = $offsetChanged
Message = "OK"
}
}

function Test-RunRivetLogNonEmpty {
param([int]$Slot)

$runRivetLog = Join-Path (Get-SharedPath $Slot) "runRivet.log"

if (-not (Test-Path $runRivetLog)) {
return $false
}

try {
$info = Get-Item $runRivetLog -ErrorAction Stop
return ($info.Length -gt 0)
}
catch {
return $false
}
}

function Get-ActiveTaskMapBySlot {
$state = Get-ClientStateXml
$map = @{}

if ($null -eq $state) {
return $map
}

$activeList = @()

if ($null -ne $state.client_state.active_task_set -and
$null -ne $state.client_state.active_task_set.active_task) {
$activeList = @($state.client_state.active_task_set.active_task)
}
elseif ($null -ne $state.client_state.active_task) {
$activeList = @($state.client_state.active_task)
}

foreach ($active in $activeList) {
$resultName = [string]$active.result_name
$slotText = [string]$active.slot

if ([string]::IsNullOrWhiteSpace($resultName) -or [string]::IsNullOrWhiteSpace($slotText)) {
continue
}

$slotNumber = 0

if (-not [int]::TryParse($slotText, [ref]$slotNumber)) {
continue
}

$projectUrl = ""

$result = @($state.client_state.result) | Where-Object {
[string]$_.name -eq $resultName
} | Select-Object -First 1

if ($null -ne $result) {
$projectUrl = [string]$result.project_url
}

$map[$slotNumber] = [pscustomobject]@{
ResultName = $resultName
ProjectUrl = $projectUrl
}
}

Write-BunkerLogThrottled `
-Key "ACTIVE_TASK_XML_COUNT" `
-Message "client_state.xml active BOINC tasks found: $($activeList.Count)." `
-Level "DEBUG" `
-Color DarkGray `
-IntervalSeconds 3600

return $map
}

function Get-ActiveTheoryTasksByPlough {
$activeMap = Get-ActiveTaskMapBySlot
$tasks = @()

if (-not (Test-Path $slotsDir)) {
Write-BunkerLogThrottled `
-Key "NO_SLOTS_DIR" `
-Message "BOINC slots directory not found: $slotsDir" `
-Level "ERROR" `
-Color Red `
-IntervalSeconds 300
return @()
}

$slotDirs = Get-ChildItem -Path $slotsDir -Directory -ErrorAction SilentlyContinue

foreach ($slotDir in $slotDirs) {
$slotNumber = 0

if (-not [int]::TryParse($slotDir.Name, [ref]$slotNumber)) {
continue
}

if (-not (Test-SlotHasTheoryVdi -Slot $slotNumber)) {
continue
}

if (-not $activeMap.ContainsKey($slotNumber)) {
Write-BunkerLogThrottled `
-Key "THEORY_VDI_NO_ACTIVE_TASK_SLOT_$slotNumber" `
-Message "Slot $slotNumber has Theory*.vdi, but no active BOINC task mapping. Do not touch." `
-Level "WARN" `
-Color Yellow `
-IntervalSeconds 300
continue
}

$vmStartTime = Get-VMStartTimeFromStderr -Slot $slotNumber

if ($null -eq $vmStartTime) {
Write-BunkerLogThrottled `
-Key "NO_VM_START_TIME_SLOT_$slotNumber" `
-Message "Slot $slotNumber has Theory*.vdi, but VM start time was not found in stderr.txt. Do not touch." `
-Level "WARN" `
-Color Yellow `
-IntervalSeconds 300
continue
}

$info = $activeMap[$slotNumber]

$tasks += [pscustomobject]@{
Slot = $slotNumber
ResultName = $info.ResultName
ProjectUrl = $info.ProjectUrl
VMStartTime = $vmStartTime
}
}

return $tasks
}

function Abort-BoincTask {
param(
[string]$ResultName,
[string]$ProjectUrlFromTask,
[string]$Reason
)

if ([string]::IsNullOrWhiteSpace($ResultName)) {
Write-BunkerLog "Abort impossible: empty task name. The orderly lost the toe tag." "ERROR" Red
return
}

if ($KilledThisRun.ContainsKey($ResultName)) {
Write-BunkerLogThrottled `
-Key "ALREADY_KILLED_$ResultName" `
-Message "Task $ResultName already got a brick in this run. No double-tapping the corpse." `
-Level "INFO" `
-Color DarkGray `
-IntervalSeconds 300
return
}

if (-not (Test-Path $BoincCmd)) {
Write-BunkerLog "boinccmd.exe not found: $BoincCmd. Brick is ready, but..." "ERROR" Red
return
}

$urlToUse = $DefaultProjectUrl

if (-not [string]::IsNullOrWhiteSpace($ProjectUrlFromTask)) {
$urlToUse = $ProjectUrlFromTask
}

if (-not $Kill) {
Write-BunkerLog "DRY RUN: $ResultName would be aborted here. Reason: $Reason" "DRYRUN" Red
return
}

Write-BunkerLog "Orderly prescribes a brick to zombie: $ResultName. Reason: $Reason" "KILL" Red

try {
& $BoincCmd --task $urlToUse $ResultName abort | Out-Null
$KilledThisRun[$ResultName] = Get-Date
Write-BunkerLog "Abort command sent: $ResultName" "KILL" Red
}
catch {
Write-BunkerLog "Abort failed: $($_.Exception.Message)" "ERROR" Yellow
}
}

function Check-Once {
$tasks = Get-ActiveTheoryTasksByPlough

if ($tasks.Count -eq 0) {
Write-BunkerLogThrottled `
-Key "NO_THEORY_VDI_SLOTS" `
-Message "No active slots with Theory*.vdi found. The bunker is quiet, or the plough found no furrow." `
-Level "INFO" `
-Color DarkGray `
-IntervalSeconds 1800
return
}

Write-BunkerLogThrottled `
-Key "ACTIVE_THEORY_COUNT" `
-Message "Active Theory slots by Theory*.vdi: $($tasks.Count). The orderly patrols silently." `
-Level "INFO" `
-Color Cyan `
-IntervalSeconds 1800

foreach ($task in $tasks) {
$slot = [int]$task.Slot

$elapsedInfo = Get-SafeElapsedSeconds -StartLocal $task.VMStartTime

if (-not $elapsedInfo.IsUsable) {
Write-BunkerLogThrottled `
-Key "BAD_TIME_SLOT_$slot" `
-Message "Slot $slot, task $($task.ResultName): $($elapsedInfo.Message)" `
-Level "WARN" `
-Color Yellow `
-IntervalSeconds 300
continue
}

$ageSeconds = [double]$elapsedInfo.Elapsed
$ageMin = [math]::Round($ageSeconds / 60, 1)

if ($ageSeconds -lt -60) {
Write-BunkerLogThrottled `
-Key "NEGATIVE_AGE_SLOT_$slot" `
-Message "Slot $slot, task $($task.ResultName): VM start time looks like future time. Computer clock is crazy. Do not touch." `
-Level "WARN" `
-Color Yellow `
-IntervalSeconds 300
continue
}

if ($elapsedInfo.OffsetChanged) {
Write-BunkerLogThrottled `
-Key "OFFSET_CHANGED_SLOT_$slot" `
-Message "Slot $slot, task $($task.ResultName): timezone offset changed. Safe VM age: $ageMin min." `
-Level "TIME" `
-Color DarkYellow `
-IntervalSeconds 300
}

$hasRunRivetLog = Test-RunRivetLogNonEmpty -Slot $slot

if ($hasRunRivetLog) {
Write-BunkerLogThrottled `
-Key "OK_SLOT_$slot" `
-Message "Slot $slot, task $($task.ResultName): shared\runRivet.log exists and is not empty. VM is alive." `
-Level "OK" `
-Color Green `
-IntervalSeconds 3600
continue
}

if ($ageSeconds -lt $GraceSeconds) {
$left = [math]::Round(($GraceSeconds - $ageSeconds) / 60, 1)

Write-BunkerLogThrottled `
-Key "WAIT_SLOT_$slot" `
-Message "Slot $slot, task $($task.ResultName): no runRivet.log yet. VM still has $left min to create it." `
-Level "INFO" `
-Color Gray `
-IntervalSeconds 300
continue
}

$reason = "No non-empty shared\runRivet.log appeared within $GraceSeconds s after DST-safe VM start time."

Write-BunkerLog "ZOMBIE caught! Slot $slot, task $($task.ResultName) is doing nothing for $ageMin min." "ZOMBIE" Red
Write-BunkerLog "Theory*.vdi is present, VM started according to stderr, but runRivet.log is empty or unborn. The orderly raises the brick." "ZOMBIE" Red

Abort-BoincTask `
-ResultName $task.ResultName `
-ProjectUrlFromTask $task.ProjectUrl `
-Reason $reason
}
}

Write-BunkerLog "Bunker radar is online. LHC Theory zombie hunt begins." "START" Cyan
Write-BunkerLog "Mode: Kill=$Kill, GraceSeconds=$GraceSeconds, Loop=$Loop, CheckIntervalSeconds=$CheckIntervalSeconds" "START" Cyan

do {
Check-Once

if ($Loop) {
Start-Sleep -Seconds $CheckIntervalSeconds
}
}
while ($Loop)

Write-BunkerLog "Check finished. The orderly puts the brick back on the shelf." "END" Cyan

I had to do quite a bit extra to get it working with systemd, it was not so hard to get it working with cgroupv2 but this is not supported I was told, I documented my results on the BOINC Q&A

https://github.com/BOINC/boinc/discussions/6833

I don't there is anything we can do when you see HTTP gateway timeout. its the backend infrastructure overloaded, it normally comes back eventually

Try Podman instead of Docker.

Further advice can't be given as long as your computers are hidden.
You may make them visible so other volunteers have access to the logs.
https://lhcathome.cern.ch/lhcathome/prefs.php?subset=project

Nothing wrong with your setup, there is an issue with the way the wrapper parses the name of the container when they have long names.

its in this thread. https://lhcathome.cern.ch/lhcathome/forum_thread.php?id=6501

I'm running the nightly one and its good, they didn't change the code the way I did but same out comes.

https://lhcathome.cern.ch/lhcathome/results.php?hostid=9930009&offset=0&show_names=0&state=4&appid=13

I guess there should be a request for a v23

In reply to Toby Broom's message of 4 Jun 2026:
I have an overide in appconfig for the VM that it has 1.5 GB to aviod this, I think its just a bad WU?
OK. That might be the case.