[heft-native] Opt-in Rust front end and plan-driven Node host

apps/heft-native/.gitignore

@@ -0,0 +1,2 @@
+target/
+!Cargo.lock

apps/heft-native/Cargo.toml

@@ -0,0 +1,18 @@
+[package]
+name = "heft-native"
+version = "1.3.1"
+edition = "2021"
+publish = false
+
+[[bin]]
+name = "heft"
+path = "src/main.rs"
+
+[dependencies]
+
+[profile.release]
+opt-level = "s"
+lto = true
+codegen-units = 1
+panic = "abort"
+strip = true

apps/heft-native/README.md

@@ -0,0 +1,108 @@
+# heft-native
+
+A self-contained, std-only Rust implementation of the `heft` command line. It is an experiment: it is
+**opt-in**, it is not a Rush project (there is no `package.json` here), and nothing else in the repository
+depends on it. `rush install` and `rush build` never look at this folder and work without Cargo.
+
+## Building
+
+```bash
+cd apps/heft-native
+cargo build --release
+```
+
+The binary is written to `apps/heft-native/target/release/heft` (`target/` is ignored by Git). The crate has
+no dependencies; `Cargo.lock` only lists `heft-native` itself.
+
+## Using it
+
+Run the binary instead of `heft` from a project folder:
+
+```bash
+cd build-tests/heft-node-everything-test
+../../apps/heft-native/target/release/heft build --help
+```
+
+The binary implements `@rushstack/heft` of the version in `Cargo.toml` (kept equal to `apps/heft/package.json` by
+a unit test). It first makes the same decision as Heft's JavaScript version selector
+(`lib-commonjs/startWithVersionSelector.js`): with `--unmanaged`, outside of any project, or when the project's
+`package.json` has no `@rushstack/heft` dependency, it runs itself; when the project's
+`node_modules/@rushstack/heft` is the binary's own JavaScript companion (the same folder, for example through a
+workspace or pnpm link) with the `lib-commonjs` layout, it runs itself too. In every other case (an older or newer
+local Heft, a copy of the same version in another folder such as a patched or linked package, the legacy `lib/`
+layout, a missing entry point, a `package.json` that `JSON.parse()` rejects, a companion of another version) it
+prints nothing and delegates the whole invocation to its JavaScript companion, whose version selector then behaves
+exactly like before.
+
+Whatever the binary does not handle natively is delegated the same way: `node <companion>/bin/heft <arguments>`.
+The companion is the `@rushstack/heft` package that belongs to the binary, found in this order:
+
+1. `HEFT_NATIVE_JS_BIN`: the path of a Heft `bin/heft` script
+2. next to the executable: `apps/heft-native/target/<profile>/heft` uses `apps/heft/bin/heft` of the same
+   checkout, and a future `@rushstack/<platform package>/bin/heft` uses `@rushstack/heft/bin/heft` next to it
+3. `<project>/node_modules/@rushstack/heft/bin/heft`
+
+## What runs without Node.js
+
+- Help, usage and command line errors (`--help`, `<action> --help`, `--version`, unknown actions or parameters,
+  unknown phases, "No phases were selected") are printed by the binary itself.
+- `heft clean` runs natively for any project: it deletes `temp/<phase>/<task>` and the `cleanFiles` of the
+  selected phases and prints Heft's summary.
+- A build runs natively when every task of the project is a `copy-files-plugin`, `delete-files-plugin` or
+  `set-environment-variables-plugin` task of `@rushstack/heft`, there are no lifecycle plugins, and the selected
+  phases and tasks form a single chain (so the output order does not depend on timing). The binary produces the
+  same output, exit code, copied files and `file-copy.json` incremental state as the JavaScript Heft.
+- It does so only when standard input is the null device (as for builds started by Rush) and without `--watch`,
+  `--verbose` or `--debug`, and only for glob patterns of the forms `**/*`, `*`, `**/*.ext`, `**/*.{a,b}`,
+  `name.*` and literal paths without `excludeGlobs`. Symbolic links, unusual file names, files outside those
+  patterns or a `file-copy.json` that is not plain JSON make it hand the invocation to the JavaScript Heft before
+  anything is printed.
+- If standard output is closed while it writes (`heft build | head`), it stops where Heft would stop and lets
+  `node` report the error exactly like Heft does.
+
+## Plugin host
+
+Invocations that need JavaScript plugins run in the plugin host of the selected Heft package
+(`node <heft>/lib-commonjs/host/HostEntry.js --heft-plan-fd=<n>`). The binary writes the plan to an unnamed
+temporary file (`O_TMPFILE` in `$XDG_RUNTIME_DIR`, `$TMPDIR`, `/dev/shm` or `/tmp`), which `node` inherits and
+the host reads and closes before any plugin is loaded; nothing is added to the environment or left on disk. On
+Unix the binary replaces itself with `node` (`exec`), so signals, exit codes and memory use are the ones of the
+JavaScript Heft; on other platforms it runs `node` and waits for it (only Linux is tested). `node` is looked up
+in `PATH`, like `#!/usr/bin/env node` does.
+
+## Warm plugin host (opt-in)
+
+With `HEFT_WARM_HOST=1`, plugin-host runs whose standard input is the null device and whose standard output and
+error are not terminals (for example builds started by Rush) are sent to a pre-warmed Node.js host over a Unix
+domain socket in `$XDG_RUNTIME_DIR/heft-host-<uid>` (or `/tmp/heft-host-<uid>`). The folder must be a real
+directory owned by the user with mode `0700`; otherwise the warm host is not used. The host's output is streamed
+back and the binary exits with the host's exit code; Ctrl+C and `SIGTERM` are forwarded.
+
+Each host serves one run and then starts its successor; a host exits after `HEFT_WARM_HOST_IDLE_MS` milliseconds
+without a run. Any mismatch (Heft or Node.js version, environment, umask, changed Heft files, Rush reporter or IPC
+file descriptors) makes the host refuse and the binary run cold. When no host accepts the run, the binary runs
+cold as usual and starts a host (`lib-commonjs/host/WarmHostEntry.js`) in the background for the next run, unless
+four hosts are already running for the user.
+
+## Static build (optional)
+
+`cargo build --release --target x86_64-unknown-linux-musl` (after `rustup target add x86_64-unknown-linux-musl`)
+produces a statically linked binary in `target/x86_64-unknown-linux-musl/release/heft`. It is about as fast as the
+default build and uses less memory (about 1 MB instead of 2.8 MB peak for a native build), but it is about 90 KB
+larger.
+
+## Layout
+
+Each folder under `src/` is a module with a single owner:
+
+| Module | Responsibility |
+| --- | --- |
+| `json`, `schema`, `regex` | JSONC parsing, JSON schema validation, the regular expression subset used by schemas |
+| `cli` | command line model, parsing, help and error rendering |
+| `config` | `heft.json`, rigs, `heft-plugin.json`, plugin options, package resolution |
+| `graph`, `run`, `builtin`, `terminal` | phase and task graph, execution, native built-in plugins, terminal output |
+| `sys` | the only place with `unsafe` code (minimal operating system calls) |
+| `process`, `version`, `host_link` | running Node.js, selecting the Heft version, the connection to the JavaScript plugin host |
+
+The code follows these rules: only the Rust standard library, `#![deny(unsafe_code)]` outside `src/sys`, no
+comments in the code, source files of at most 200 lines, and a stripped release binary of at most 1 MB.

apps/heft-native/src/builtin/base64.rs

@@ -0,0 +1,44 @@
+const STANDARD_ALPHABET: &[u8; 64] =
+    b"ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/";
+
+pub fn append_standard_base64(bytes: &[u8], output: &mut String) {
+    output.reserve(bytes.len().div_ceil(3) * 4);
+    let (chunks, remainder) = bytes.as_chunks::<3>();
+    for chunk in chunks {
+        let combined_bits =
+            ((chunk[0] as u32) << 16) | ((chunk[1] as u32) << 8) | (chunk[2] as u32);
+        output.push(STANDARD_ALPHABET[((combined_bits >> 18) & 0x3f) as usize] as char);
+        output.push(STANDARD_ALPHABET[((combined_bits >> 12) & 0x3f) as usize] as char);
+        output.push(STANDARD_ALPHABET[((combined_bits >> 6) & 0x3f) as usize] as char);
+        output.push(STANDARD_ALPHABET[(combined_bits & 0x3f) as usize] as char);
+    }
+    if remainder.len() == 1 {
+        let combined_bits = (remainder[0] as u32) << 16;
+        output.push(STANDARD_ALPHABET[((combined_bits >> 18) & 0x3f) as usize] as char);
+        output.push(STANDARD_ALPHABET[((combined_bits >> 12) & 0x3f) as usize] as char);
+        output.push('=');
+        output.push('=');
+    } else if remainder.len() == 2 {
+        let combined_bits = ((remainder[0] as u32) << 16) | ((remainder[1] as u32) << 8);
+        output.push(STANDARD_ALPHABET[((combined_bits >> 18) & 0x3f) as usize] as char);
+        output.push(STANDARD_ALPHABET[((combined_bits >> 12) & 0x3f) as usize] as char);
+        output.push(STANDARD_ALPHABET[((combined_bits >> 6) & 0x3f) as usize] as char);
+        output.push('=');
+    }
+}
+
+pub fn sha256_digest_as_base64(digest: &[u8; 32]) -> [u8; 44] {
+    let mut encoded = [b'='; 44];
+    let (chunks, remainder) = digest.as_chunks::<3>();
+    for (chunk_index, chunk) in chunks.iter().enumerate() {
+        let combined_bits = ((chunk[0] as u32) << 16) | ((chunk[1] as u32) << 8) | (chunk[2] as u32);
+        for (offset, shift) in [18, 12, 6, 0].into_iter().enumerate() {
+            encoded[chunk_index * 4 + offset] = STANDARD_ALPHABET[((combined_bits >> shift) & 0x3f) as usize];
+        }
+    }
+    let combined_bits = ((remainder[0] as u32) << 16) | ((remainder[1] as u32) << 8);
+    for (offset, shift) in [18, 12, 6].into_iter().enumerate() {
+        encoded[40 + offset] = STANDARD_ALPHABET[((combined_bits >> shift) & 0x3f) as usize];
+    }
+    encoded
+}

apps/heft-native/src/builtin/build_info.rs

@@ -0,0 +1,116 @@
+use std::collections::HashMap;
+use std::fs::{self, File};
+use std::io::{BufWriter, ErrorKind, Write};
+
+use super::build_info_json::{is_array_index_key, parse_build_info_json};
+use super::javascript_json::append_json_string;
+use super::node_file_system_error::{is_node_not_exist_error, NodeFileSystemError};
+use super::posix_path::{directory_name, relative_path, resolve_path};
+
+pub struct IncrementalBuildInfo {
+    pub configuration_hash: String,
+    pub input_file_versions: Vec<(String, String)>,
+}
+
+pub enum BuildInfoReadResult {
+    Missing,
+    Found(IncrementalBuildInfo),
+    NeedsJavaScript,
+}
+
+pub fn try_read_build_info(build_info_path: &str) -> BuildInfoReadResult {
+    let bytes = match fs::read(build_info_path) {
+        Ok(bytes) => bytes,
+        Err(error) if is_node_not_exist_error(&error) => return BuildInfoReadResult::Missing,
+        Err(_) => return BuildInfoReadResult::NeedsJavaScript,
+    };
+    let Some(parsed) = std::str::from_utf8(&bytes).ok().and_then(parse_build_info_json) else {
+        return BuildInfoReadResult::NeedsJavaScript;
+    };
+    drop(bytes);
+    let base_folder_path = directory_name(build_info_path);
+    let mut input_file_versions: Vec<(String, String)> = parsed
+        .input_file_versions
+        .into_iter()
+        .map(|(relative_file_path, version)| (resolve_path(base_folder_path, &relative_file_path), version))
+        .collect();
+    let mut duplicates: Vec<(usize, usize)> = Vec::new();
+    {
+        let mut first_index_by_path: HashMap<&str, usize> = HashMap::with_capacity(input_file_versions.len());
+        for (index, (absolute_file_path, _)) in input_file_versions.iter().enumerate() {
+            if let Some(&first_index) = first_index_by_path.get(absolute_file_path.as_str()) {
+                duplicates.push((first_index, index));
+            } else {
+                first_index_by_path.insert(absolute_file_path, index);
+            }
+        }
+    }
+    if !duplicates.is_empty() {
+        for &(first_index, duplicate_index) in &duplicates {
+            input_file_versions[first_index].1 = std::mem::take(&mut input_file_versions[duplicate_index].1);
+        }
+        let mut is_duplicate = vec![false; input_file_versions.len()];
+        for &(_, duplicate_index) in &duplicates {
+            is_duplicate[duplicate_index] = true;
+        }
+        let mut flags = is_duplicate.into_iter();
+        input_file_versions.retain(|_| !flags.next().unwrap_or(false));
+    }
+    BuildInfoReadResult::Found(IncrementalBuildInfo {
+        configuration_hash: parsed.configuration_hash,
+        input_file_versions,
+    })
+}
+
+pub fn write_build_info<'entries>(
+    configuration_hash: &str,
+    input_file_versions: impl Iterator<Item = (&'entries str, &'entries str)> + Clone,
+    build_info_path: &str,
+) -> Result<(), NodeFileSystemError> {
+    let base_folder_path = directory_name(build_info_path);
+    let relative_entries = input_file_versions.map(|(absolute_file_path, version)| (relative_path(base_folder_path, absolute_file_path), version));
+    let mut array_index_entries: Vec<(String, &str)> =
+        relative_entries.clone().filter(|(key, _)| is_array_index_key(key)).collect();
+    array_index_entries.sort_by_key(|(key, _)| key.parse::<u64>().unwrap_or(0));
+    let named_entries = relative_entries.filter(|(key, _)| !is_array_index_key(key));
+    let file = create_file_ensuring_folder_exists(build_info_path)?;
+    let mut writer = BufWriter::with_capacity(16384, file);
+    let mut chunk = String::with_capacity(256);
+    chunk.push_str("{\"configHash\":");
+    append_json_string(configuration_hash, &mut chunk);
+    chunk.push_str(",\"inputFileVersions\":{");
+    for (index, (key, version)) in array_index_entries.into_iter().chain(named_entries).enumerate() {
+        if index > 0 {
+            chunk.push(',');
+        }
+        append_json_string(&key, &mut chunk);
+        chunk.push(':');
+        append_json_string(version, &mut chunk);
+        write_chunk(&mut writer, &mut chunk, build_info_path)?;
+    }
+    chunk.push_str("}}");
+    write_chunk(&mut writer, &mut chunk, build_info_path)?;
+    writer
+        .flush()
+        .map_err(|error| NodeFileSystemError::new(error, "write", build_info_path, None))
+}
+
+fn write_chunk(writer: &mut BufWriter<File>, chunk: &mut String, file_path: &str) -> Result<(), NodeFileSystemError> {
+    writer
+        .write_all(chunk.as_bytes())
+        .map_err(|error| NodeFileSystemError::new(error, "write", file_path, None))?;
+    chunk.clear();
+    Ok(())
+}
+
+fn create_file_ensuring_folder_exists(file_path: &str) -> Result<File, NodeFileSystemError> {
+    match File::create(file_path) {
+        Ok(file) => Ok(file),
+        Err(error) if error.kind() == ErrorKind::NotFound => {
+            fs::create_dir_all(directory_name(file_path))
+                .map_err(|error| NodeFileSystemError::new(error, "mkdir", directory_name(file_path), None))?;
+            File::create(file_path).map_err(|error| NodeFileSystemError::new(error, "open", file_path, None))
+        }
+        Err(error) => Err(NodeFileSystemError::new(error, "open", file_path, None)),
+    }
+}