Lightweight Linux containerization tool written from scratch for the ReCodEx assignment evaluation system.
This project uses advanced Linux kernel features (namespaces, cgroups, UID/GID mappings, etc.) to create secure sandboxes for running untrusted code. It provides fine-grained control over system resources and filesystem access. To get more insight into the details, you can take a look at my thesis in the docs/thesis.pdf file.
- Guardian — This tool as a whole, providing containerization capabilities.
- Instance — One run of the Guardian, from parsing the configuration file to executing tasks and generating metadata.
- Sandbox — The isolated environment created based on configuration, including namespaces, cgroups, UID/GID mappings, environment variables, and filesystem mounts.
- Task — A single unit of execution within the sandbox, running an executable with specified arguments and resource limits.
An instance runs three different processes with distinct responsibilities:
- Root process — Reserves necessary global resources (directories, cgroups) and prepares the environment.
- Proxy process — Creates and configures the sandbox environment (namespaces, filesystem mounts, cgroups).
- Task process — Executes the isolated program with the specified resource limits.
- Linux kernel with cgroupv2 enabled, with the
cpu,cpuset,memoryandpidscontrollers available (every--runenables all four). - CMake 3.20+ and a compiler supporting C++23 (on Rocky 9 that means
gcc-toolset-14; the system GCC 11.5 is not enough) libcapdevelopment headers (libcap-devel/libcap-dev)- Network access at configure time: yaml-cpp is fetched by CMake and statically
linked. For an offline build, point
FETCHCONTENT_SOURCE_DIR_YAML-CPPat a pre-fetched tree. rpm-buildonly if you want to produce the RPM- For disk quotas (
--quota, standalonedisk-usage), the box tree (/var/lib/recodex-guardian/boxes) must be on a filesystem that enforces user quotas (QUOTACTL(2), e.g. ext4 mountedusrquota)
One script drives the whole lifecycle — scripts/guardian.sh MODE:
| Mode | What it does |
|---|---|
build |
configure (only if needed) + build |
package |
build, then cpack -G RPM — produces the package, installs nothing |
install |
build, then cmake --install |
uninstall |
remove what install put there, alias included |
purge |
destroy every box in /var/lib/recodex-guardian/boxes (keeping the RPM-owned directory) and tear down the shared cgroup parent |
Defaults are the same for every mode: Release, prefix /usr, no isolate
alias, no test tiers. scripts/guardian.sh --help lists the flags.
On Rocky 9, enter the toolset first — for every mode that compiles:
scl enable gcc-toolset-14 -- bashThe system compiler is GCC 11.5, which has no
<format>. Configure refuses without a toolset and tells you which one to enter; changing compiler under an existing build tree makesguardian.shwipe it. Modern dev distros need nothing special.
No installation at all: build in the tree and invoke the binary through sudo.
This is what the workload and mock-evaluator test tiers use.
scripts/guardian.sh build # into ./build
sudo ./build/src/recodex-guardian --yaml=config.ymlAdd --dev (Debug + -DTESTING=ON) when you want the test tiers built too —
the workload tier needs it, since a default build compiles no test code.
Nothing is placed on the system; the only persistent state is the box tree
(/var/lib/recodex-guardian/boxes) and the shared cgroup parent, both created lazily on
first run and removable with scripts/guardian.sh purge.
Because the binary is not setuid here, every invocation needs root — the
Isolate-compatible drop-in path (a non-root caller) is not exercised by this
option.
Use this on hosts where you cannot or do not want to build an RPM. install
builds first, so this is the whole procedure from a clean tree:
scripts/guardian.sh install # add --alias on a ReCodEx Worker hostIt installs:
| Path | What |
|---|---|
/usr/bin/recodex-guardian |
the binary, mode 4755 (setuid root) |
/var/lib/recodex-guardian/boxes |
persistent box tree |
/usr/share/man/man1/recodex-guardian.1 |
man page |
Only the cmake --install step is elevated (via sudo); configure and build
never are, so the build tree stays yours. The script prints the mode that
actually landed, because cmake --install applies permissions directly and a
restrictive umask can file the setuid bit off — expect -rwsr-xr-x. (The RPM in
Option C forces 4755 regardless of the build user's umask.)
Configure-time choices are remembered in the build tree, so install after a
build --alias still installs the alias; naming a flag again reconfigures and
says so. --destdir DIR stages the install under DIR instead, which needs no
root at all — useful for inspecting exactly what would land.
scripts/guardian.sh uninstall # -n / --dry-run to previewRun it from the same build tree you installed from — it reads that tree's
install_manifest.txt, and errors out rather than guessing if the manifest is
gone. It removes one path the manifest does not contain: the isolate symlink,
which CMake never records because an install(CODE ...) step creates it. Left
behind, that symlink dangles at the front of PATH and anything invoking
isolate fails confusingly instead of falling through to another install.
The box tree is reclaimed only when empty, exactly as dnf remove treats the
package's directory. A non-empty tree means live or leftover boxes, so it is
reported and left alone; uninstall --purge (or purge on its own) removes it
and the shared cgroup parent.
On a host where the Guardian came from an RPM, uninstall refuses and points
you at dnf: deleting RPM-owned files behind rpm's back leaves the package
database convinced it is still installed. And when there's no manifest to work
from, it surveys the host instead of just complaining — listing every recodex-guardian
it can find with the right removal route for each, since an RPM install under
/usr and a source install under /usr/local can coexist (and the /usr/local
one wins on a default PATH).
The ReCodEx Worker cluster runs Rocky Linux 9 and its Worker RPM declares
Requires: isolate, which our package satisfies with Provides: isolate when
built with the alias — so a package destined for a Worker host wants --alias:
scl enable gcc-toolset-14 -- bash
scripts/guardian.sh package --alias # -> build/recodex-guardian-0.1.0-1.el9.x86_64.rpm
sudo dnf install ./build/recodex-guardian-0.1.0-1.el9.x86_64.rpmpackage produces the RPM and stops there; installing and removing it is dnf's
job, deliberately. The default Release build type matters here: CPack emits no
-debuginfo subpackage, so a RelWithDebInfo package carries debug symbols
inside the setuid binary — measured on el9, 17 MB of binary in a 4.4 MB
package, against 3.1 MB in 955 KB for Release.
The package owns the same file list as Option B, with /usr/bin/recodex-guardian forced
to %attr(4755,root,root) and /var/lib/recodex-guardian/boxes owned as a directory so
dnf remove cleans it up. The binary statically links libstdc++/libgcc and
yaml-cpp, so it has no dependency on the gcc-toolset SCL runtime at execution
time — build under the toolset, run against the plain system.
An alias-built package owns /usr/bin/isolate, so RPM will refuse to install it
alongside the upstream isolate package (that file conflict is the
coexistence guard); remove upstream Isolate first. A default
(alias-less) package owns no such path and installs beside it.
There is no boot-time service or --init-style system setup step to enable.
The alias is what makes the drop-in path work: the ReCodEx Worker execvps a
PATH-resolved binary literally named isolate, and its RPM declares
Requires: isolate. One CMake option covers all three of its parts:
ISOLATE_ALIAS=OFF (default) |
ISOLATE_ALIAS=ON |
|
|---|---|---|
<bindir>/isolate symlink |
not installed | installed |
man isolate redirect page |
not installed | installed |
RPM Provides: isolate |
not declared | declared |
It is opt-in because taking over /usr/bin/isolate displaces upstream
Isolate on the host. A default build is the neutral one: it installs only
recodex-guardian, coexists with upstream Isolate, and is all you need for standalone
--yaml mode.
Turn it on by adding --alias on any mode that configures:
scripts/guardian.sh install --alias
scripts/guardian.sh package --aliasTwo consequences of the default. A default-built package cannot satisfy the
Worker's Requires: isolate, so a Worker deployment must be handed an alias
build. And flipping to --no-alias and re-installing does not remove an alias
an earlier install left behind — uninstall (which knows about the symlink) is
what cleans it up.
ls -l /usr/bin/recodex-guardian # -rwsr-xr-x, owner root
man -w recodex-guardian # man page resolves
recodex-guardian --init --box-id=999 # as a non-root user: prints the box root
recodex-guardian --cleanup --box-id=999 # and tears it back downThe --init / --cleanup round-trip as an unprivileged user is the meaningful
check for Options B and C: it only succeeds if the setuid bit is in place. On an
alias build, repeat it as isolate — that command -v isolate resolves to our
symlink is the drop-in path's precondition.
No system initialization step is required: on its first --run the Guardian
lazily creates the shared cgroup parent (/sys/fs/cgroup/recodex-guardian) and
enables its controllers. To tear the shared cgroup tree and box
directory (/var/lib/recodex-guardian/boxes) back down:
scripts/guardian.sh purgescripts/guardian.sh build(That is installation Option A above — see the Installation section for the system-wide and RPM options.)
sudo ./build/src/recodex-guardian --yaml=/path/to/config.ymlTo reclaim the host-wide state Guardian instances leave behind (the box tree and the shared cgroup parent):
scripts/guardian.sh purgeThe repository has two tiers of test plus the ReCodEx integration suite:
Host-side GoogleTest tests of pure
logic — CLI/config parsing and the box lock — needing no root. Fetched via
CMake only when -DTESTING=ON, and run with ctest:
scripts/guardian.sh build --dev # Debug + -DTESTING=ON
cd build && ctest --output-on-failurepytest tests that drive the whole recodex-guardian binary
end-to-end, running workloads (small in-box payload programs under
tests/workload/workloads/) inside the sandbox to verify resource limits and
isolation boundaries actually bite. These need root (cgroups + namespaces):
scripts/guardian.sh build --dev # builds the binary and the workloads
scripts/workload_tests.sh # venv + sudo pytestA default build compiles no test code, so --dev (or --testing) is what
puts the workloads in tests/workload/build/; without them the tier skips and
tells you which command to run.
scripts/workload_tests.sh is idempotent: on first run it creates a local
.venv (gitignored) from tests/requirements.txt, then runs pytest under
sudo. Pass pytest args through (scripts/workload_tests.sh -k limits, or
--co to collect without root). The suite skips itself cleanly when the binary
isn't built or root isn't available, so collection off a root host is a safe
no-op. (System Python on Arch/PEP-668 distros is externally managed, hence the
venv rather than a global pip install.)
Replays real, production-harvested ReCodEx job configs (C, Python, C#, Maven) to
validate toolchains and limits. It drives the Guardian in standalone mode
(--yaml=), so it deliberately covers no part of the compatibility CLI as the
Worker actually emits it; that validation belongs to ReCodEx's own integration
pipeline, against an installed alias build.
Running these tests will:
- Clone the ReCodEx worker repository from GitHub
- Install Python dependencies (pandas)
- Install .NET runtime (requires sudo)
- Download approximately 1GB of test data from an external server
- Build additional components
These tests are primarily intended for integration with the ReCodEx evaluation system.
To run the ReCodEx tests:
cd tests/recodex
sudo ./recodex_init.py # Setup (downloads ~1GB data)
./recodex_mock.py [language_groups] # Run tests for specified language groupsWhere language_groups is a subset of [C#, Python, C++, AdvC++]
The test suites use variable substitution with the ${VARIABLE} syntax to run without additional setup. For example, the test scripts use this feature to replace ${FILE_DIR} with the absolute path to the test directory:
box-fs:
dir-rules:
- "build=${FILE_DIR}/build"
- "res=${FILE_DIR}/res:rw"When creating manual configuration files, you should either replace these variables with absolute paths or implement similar substitution logic.
The Guardian uses YAML configuration files to define sandbox environments and tasks.
# Global settings and credentials
root-dir: "/var/lib/recodex-guardian/boxes" # Root directory for all sandboxes
root-cgroup: "/sys/fs/cgroup/recodex-guardian" # Root cgroup path
share-net: false # Whether to share network namespace with parent
credentials:
id: "unique-instance-id" # Unique identifier for this instance
name: "example-instance" # Descriptive name for this instance
as-uid: 1000 # User ID for running processes in the sandbox
as-gid: 1000 # Group ID for running processes in the sandbox
# Environment variables configuration
env:
vars: # Environment variables visible in the sandbox
- "PATH=/usr/bin" # Set a specific variable
- "HOME" # Inherit HOME from parent environment
- "TEMP=/tmp" # Define a new variable
- "LANG=" # Remove variable from environment
inherit-all: false # Whether to inherit environment from the host
# Filesystem configuration
box-fs:
use-defaults: true # Mount standard directories (/bin, /lib, etc.)
dir-rules: # Directories visible in the sandbox
- "tests=/home/user/project/tests" # Mount external directory
- "tests=/home/user/project/tests:rw" # Mount with read-write permissions
- "tmp:tmp" # Create temporary directory
- "proc:fs" # Mount /proc
- "dev:dev" # Mount /dev with device access
- "lib64:maybe" # Mount only if exists
- "data:rw,noexec" # Multiple options (rw, noexec)
- "custom=/path/to/dir:norec" # Do not bind mount recursively
# Tasks to execute in the sandbox
tasks:
- task-id: "example1" # Name for this task (used in output files)
stats-yaml: "results1.yml" # Path to output results file
cmd:
bin: "tests/example" # Path to the executable inside the sandbox
args: # Optional arguments for the executable
- "Hello"
- "World"
stdin: "input.txt" # Redirect stdin from file (optional)
stdout: "output.txt" # Redirect stdout to file (optional)
stderr: "error.txt" # Redirect stderr to file (optional)
stderr-to-stdout: false # Redirect stderr to stdout (optional)
chdir: "tests" # Change directory before execution (optional)
limits: # Resource limits for this task
mem: 5000000 # Memory limit in bytes
as-size: 10000000 # Address space size limit in bytes
stack: 8192 # Stack size limit in KB
cpu-time: 3 # CPU time limit in seconds
wall-time: 5 # Wall clock time limit in seconds
extra-time: 0.5 # Grace period after CPU limit in seconds
disk-usage: 1000000 # Disk quota in bytes (box-wide, no inode cap)
processes: 10 # Maximum number of processes/threads
open-files: 64 # Maximum open file descriptors
fsize: 1024 # Maximum file size in KB
core: 0 # Maximum core dump size in KB
- task-id: "example2" # A second task in the same sandbox
cmd:
bin: "/bin/echo"
args:
- "Another task"
stdout: "output2.txt"
limits:
mem: 1000000
cpu-time: 1Every limit is optional; omit it to mean "no limit". A limit written as 0 is
never silently read as "unlimited" — with the one exception of the disk quota:
mem,as-size,fsize,open-filesandcoreenforce a literal0— ask for zero and you get zero.disk-usage: 0is unlimited: the kernel's quota interface stores a 0 limit as "no limit" and has no way to write a zero cap. Any non-zero value is rounded up to whole 1 KiB quota blocks, so a small cap never becomes 0.stack: 0andprocesses: 0are refused as a usage error (exit code 2). A zero stack leaves the task unable toexecveat all, and a process cap of0merely duplicates1, so both are mistakes rather than strict settings.- An omitted
stackis the one limit still applied, as unlimited, rather than inheriting the caller's (typically 8 MiB) stack — otherwise a deeply recursive task's verdict would depend on the shell that launched the Guardian.
The Guardian creates a secure sandbox environment with a strictly controlled filesystem. The box-fs section in the configuration defines how the filesystem should be structured within the sandbox.
Directory rules use the following syntax:
box-fs:
dir-rules:
- "target=/host/path[:options]"Where:
targetis the path inside the sandbox (relative to sandbox root)/host/pathis the absolute path on the host systemoptionsare optional access modifiers described in the above example, separated by commas
For example:
box-fs:
dir-rules:
- "bin=/bin" # Mount /bin as read-only
- "tmp=/tmp/mytmp:rw, dev" # Mount with read-write access and allow devicesAll paths specified within a task configuration are relative to the sandbox root, not the host filesystem. This is crucial to understand when configuring:
chdir- Working directory for the command (relative to sandbox root)cmd.bin- Path to the executable (relative to sandbox root)stats-yaml- Output file for statistics (relative to sandbox root)stdin/stdout/stderr- I/O file paths (relative to sandbox root)
For example, if you have the following directory rule:
box-fs:
dir-rules:
- "build=/home/user/myproject/build"
- "res=/home/user/myproject/results:rw"Then your task configuration would reference these paths as:
tasks:
- task-id: "example-task"
chdir: "/res" # Inside the sandbox at /res
stats-yaml: "/res/stats.yaml" # Save results to /res/stats.yaml
cmd:
bin: "/build/myprogram" # Run /build/myprogram
args: ["input.txt"]After execution, the Guardian generates metadata in YAML format with information about the run — this is the stats-yaml of standalone mode (--yaml=):
status: OK # Status: OK, killed, memory, wall-time, cpu-time
exitcode: 0 # Process exit code
exitsig: 0 # Signal that terminated the process (if any)
time: 0.125 # CPU time used (seconds)
memory: 8520 # Memory usage (KB)
wall-time: 0.135 # Wall clock time (seconds)Possible status values:
OK: Task completed successfullykilled: Task was terminated by a signalnon zero exit code: Task exited with non-zero codewall-time: Wall time limit exceededcpu-time: CPU time limit exceededmemory: Memory limit exceeded
Compatibility mode writes a different file. Driven as a drop-in replacement
for Isolate (--init / --run / --cleanup with --meta=FILE), the Guardian
emits Isolate's key:value meta-file instead, as a superset of what Isolate
writes so the ReCodEx Worker parses it unchanged:
| Key | When |
|---|---|
status |
only on failure — RE / SG / TO / XX; absent means success, as in Isolate |
exitcode |
the task exited normally (not on TO) |
exitsig |
the task died on a signal (not on TO: a time-out's SIGKILL is ours, as in Isolate) |
killed:1 |
we SIGKILLed it — the wall/CPU-timeout path |
time, time-wall, max-rss, csw-voluntary, csw-forced |
always |
cg-mem |
cgroup memory was measurable (omitted otherwise, leaving max-rss as the memory signal) |
cg-oom-killed:1 |
the kernel OOM-killed something in the box |
The YAML above is never written on that path, and conversely --meta has no
effect on the standalone path — the two output formats belong to the two modes.
- Running the Guardian itself requires root — unless it is installed setuid, which is the point of Options B and C. Of the helper modes,
buildandpackageneed no privilege at all;install,uninstallandpurgeelevate the single command that needs it (viasudo) rather than running wholesale as root, so your build tree never ends up root-owned. - If tests fail with filesystem errors, ensure that the directories specified in the configuration exist and have appropriate permissions.
- The Guardian self-arranges its cgroup parent on first
--run; if the shared cgroup tree or box directory gets into a bad state, reset it withscripts/guardian.sh purge. - Always use absolute paths in host filesystem references but remember that paths inside the task configuration are relative to the sandbox root.
- When testing, inspect the content of
/var/lib/recodex-guardian/boxes/to see the actual sandbox structure. - Run the Guardian binary with --debug to see detailed logs.