Skip to content

False disk space warning on immutable/composefs root filesystems (Directory.GetDirectoryRoot always returns / on Linux) #4380

@apvd

Description

@apvd

Description

Self-hosted runners on immutable Linux systems (bootc, Fedora CoreOS, ostree-based) report a false disk space warning every job:

Warning: You are running out of disk space. The runner will stop working when the machine runs out of disk space. Free space left: 0 MB

There is plenty of real disk space available. The warning fires because of a platform-specific bug in CheckDiskSpaceAsync.

Root Cause

In src/Runner.Worker/JobExtension.cs, the disk space check does:

var workDirRoot = Directory.GetDirectoryRoot(HostContext.GetDirectory(WellKnownDirectory.Work));
var driveInfo = new DriveInfo(workDirRoot);
var freeSpaceInMB = driveInfo.AvailableFreeSpace / 1024 / 1024;

On Linux, Directory.GetDirectoryRoot() always returns / regardless of which filesystem the work directory is actually mounted on. This means DriveInfo always inspects / — not the filesystem backing the work directory.

On immutable systems, / is a composefs mount (the read-only image layer). composefs correctly reports 0 bytes available because it is a read-only filesystem. The runner interprets this as the disk being full.

Environment

  • Runner version: 2.334.0
  • OS: Fedora CoreOS / MethidOS (bootc-managed image)
  • Root filesystem: composefs (immutable, read-only — this is by design)
  • Actual writable partitions: hundreds of GB free (verified with df -h)

Example df -h output from an affected machine:

Filesystem      Size  Used Avail Use% Mounted on
composefs        47M   47M     0 100% /              ← always 0, read-only by design
/dev/nvme0n1p5  141G   24G  110G  18% /etc
/dev/nvme0n1p3  619G  106G  514G  17% /var/lib/docker
/dev/nvme0n1p4  186G    2G  175G   2% /var/home

The runner work directory (/var/lib/github-runner/_work) is on /dev/nvme0n1p5 with 110 GB free. GetDirectoryRoot returns / for this path, causing DriveInfo to check composefs instead.

Expected Behavior

The disk space check should inspect the filesystem that actually backs the work directory, not always /.

Suggested Fix

Replace Directory.GetDirectoryRoot() + DriveInfo with a direct statvfs(2) call (via P/Invoke or DriveInfo on the full work path) on Linux, or use the df-equivalent .NET API that resolves the actual mount point for a given path:

// Instead of:
var workDirRoot = Directory.GetDirectoryRoot(HostContext.GetDirectory(WellKnownDirectory.Work));
var driveInfo = new DriveInfo(workDirRoot);

// Use the work path directly — DriveInfo accepts a directory path, not just a root:
var driveInfo = new DriveInfo(HostContext.GetDirectory(WellKnownDirectory.Work));

new DriveInfo(path) on .NET resolves the actual volume for the given path on Windows, but on Linux it still resolves to /. The correct Linux fix is to call statvfs directly on the work directory path so the kernel returns stats for the actual mount.

Additional Context

This affects any self-hosted runner on an ostree/bootc/composefs-based OS, including Fedora CoreOS (41+), Universal Blue images, and any custom bootc image. These systems are increasingly common for edge/embedded CI workloads. The warning is purely cosmetic today but causes user confusion and will become a hard failure if the runner ever starts refusing jobs based on this check.

There is no runner-side configuration knob to suppress or redirect this check — system.runner.lowdiskspacethreshold is pushed by GitHub's service in the job payload and cannot be overridden from the runner's .env file.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions