Report total Prestissimo worker memory usage (calculated by LinuxMemoryChecker) via Presto coordinator endpoint

[acarpente-denodo]

Problem / Context

Currently, there is no way to monitor the total memory usage of a Prestissimo worker process through the Presto Coordinator.

Existing endpoints, such as /v1/cluster/workerMemory, /v1/worker/<worker-id>/status, and /v1/info/metrics (e.g., reading presto_cpp_memory_manager_total_bytes), only report memory tracked by active/running queries. They miss memory consumed by:

• The presto_server executable itself
• Shared libraries
• Memory-mapped files
• Other process-level overheads

When no active queries are running, these endpoints report reservedBytes: 0 and full freeBytes, failing to reflect actual OS-level process footprint (e.g., VmRSS, RssAnon, etc.).

While users can query OS stats directly on the worker node (e.g., via /proc/<pid>/status or /proc/self/smaps_rollup), this requires connecting directly to each individual worker and knowing its environment, rather than querying the central Presto Coordinator.

Proposed Solution

1. Report LinuxMemoryChecker Metric in the worker metrics: Expose the total process memory usage calculated by LinuxMemoryChecker as a new metric in the worker's metrics endpoint (/v1/info/metrics or /v1/status).
2. Propagate to Coordinator: Update the Presto Coordinator behavior to collect, aggregate and expose this metric so users can monitor overall worker memory usage across the cluster via a single Coordinator endpoint call.

Key Considerations

• Why not a sidecar process? Relying on an external sidecar process requires additional open ports and distinct connections per worker node. The Presto Coordinator already acts as the central management plane that knows all worker nodes and routinely collects worker-related statistics.
• Accuracy vs. Complexity: While LinuxMemoryChecker may not be 100% exact (e.g., missing kernel dirty page writes when spilling or using SSD caches) and OS metrics can vary, returning the memory footprint calculated by LinuxMemoryChecker provides a much closer approximation of actual worker memory consumption without adding heavy OS-dependent overhead.

Suggested Integration

• Worker Metric: Expose a new metric (e.g., presto_cpp_used_memory_total_bytes or similar) on /v1/info/metrics or /v1/status.
• Coordinator: Aggregate the total memory usage by workers on a coordinator endpoint (e.g., /v1/cluster/workerMemory or /v1/worker/<worker-id>/status).