bug(vm): capability probe cannot read /proc/self/status on rootless RHEL HPC host

[Andrews2017]

User Story

As a user on a locked-down HPC cluster,
I want the MicroVM driver to start a sandbox using the available unprivileged KVM access,
so that I can use OpenShell where Docker and Podman are unavailable.

Problem Statement

With OpenShell 0.1.2 and the opt-in MicroVM driver, openshell sandbox create --name demo allocates the sandbox, pulls and boots the image, completes privileged guest initialization, and then enters the Error phase. Immediately after the guest reports starting OpenShell VM sandbox, the capability-free startup probe fails to read /proc/self/status with EACCES.

The gateway and VM compute driver remain healthy. The failure occurs after the guest reconciles the sandbox identity to UID/GID 1000 and before the workload starts.

Impact / Why This Matters

No sandbox can be created on this host despite /dev/kvm being available and the MicroVM booting successfully. Docker and Podman are not installed, and the site-provided Apptainer runtime is not supported by OpenShell, so there is currently no usable OpenShell compute driver on this cluster.

Increasing VM memory does not appear relevant: the host has ample available memory, no process or cgroup PID cap was reached, and the first actionable failure is the procfs permission denial. The later workqueue message appears while the failed guest is shutting down.

Acceptance Criteria

• A MicroVM sandbox reaches the Running phase on this host configuration.
• The capability-free runtime qualification can inspect its own capability state, or reports the exact unsupported guest procfs configuration and an actionable remediation.
• The documented VM prerequisites identify any required host or guest procfs, namespace, SELinux, or sysctl settings.

Reproduction Steps

1. Install the v0.1.2 Linux release archives without root privileges.
2. Configure the gateway with compute_driver = "vm", the released openshell-driver-vm, and nvcr.io/nvidia/base/ubuntu:24.04 as both the default and bootstrap image.
3. Start the user-level gateway service.
4. Confirm openshell status is connected and openshell gateway info reports the v0.1.2 VM driver healthy.
5. Run openshell sandbox create --name demo.
6. Observe that image preparation and VM boot succeed, followed by the procfs permission failure below.

Environment

• OpenShell CLI, gateway, VM driver, and guest sandbox: 0.1.2
• Host OS: RHEL 9.8
• Host kernel: 5.14.0-687.52.1.el9_8.x86_64
• Architecture: x86_64
• Deployment: rootless user-level gateway, libkrun MicroVM driver, KVM accessible
• Host security: SELinux enforcing
• Host storage: home and OpenShell state under NFS
• Host user.max_user_namespaces: 0
• Host kernel.yama.ptrace_scope: 0
• Host fs.suid_dumpable: 0
• VM allocation: 2 vCPUs, 2048 MiB RAM
• Sandbox/bootstrap image: nvcr.io/nvidia/base/ubuntu:24.04
• Docker: unavailable
• Podman: unavailable
• Apptainer/Singularity: 1.5.3 available

Logs

Created sandbox: demo

VM guest console:
[0.010s] setting up writable overlay root
[0.237s] reconciled sandbox account (1000:1000)
[0.272s] prepared /sandbox ownership (1000:1000)
[0.303s] hostname=demo
[0.370s] OPENSHELL_SANDBOX_ID=<redacted>
[0.375s] starting OpenShell VM sandbox
Error:   x read /proc/self/status
  `-> Permission denied (os error 13)

[    0.755621] workqueue: Failed to create a worker thread: -12

openshell gateway info reports the gateway and openshell-driver-vm as healthy. Host diagnostics showed ample available RAM, pids.max=max, and no session memory limit.

Initial Investigation

The v0.1.2 guest startup path changes to the configured non-root identity, clears capabilities, sets no_new_privs, and then the mandatory runtime qualification reads /proc/self/status to verify the capability sets. That read is the first failing operation. This localizes the failure but does not establish whether the incompatible procfs behavior originates in the guest mount options, the embedded libkrun runtime/kernel, or another host-specific interaction.

Related Apptainer Roadmap Question

This environment is representative of an HPC cluster where Apptainer is provided but Docker/Podman and unprivileged user namespaces are unavailable. The directly relevant Apptainer proposal #1393 was closed as not planned, while the open native OCI driver proposal #2255 lists rootless mode as a non-goal for its initial release.

Is there now a supported path or roadmap estimate for Apptainer-compatible HPC environments? If Apptainer support is still not planned, confirmation that there is no expected timeline would help users plan around this class of cluster.