First settings poll after supervisor start always reports provider_env_changed:true and can drop an in-flight request

[fede-kamel]

User Story

As an operator, I want a freshly started sandbox to keep serving requests without a spurious provider environment reload 10 seconds after start, so that short-lived agents and early requests do not fail with closed connections.

Problem Statement

Every sandbox supervisor logs, exactly one settings poll (10 s) after Starting sandbox supervision:

OCSF CONFIG:DETECTED [INFO] Settings poll: config change detected [old_revision:X new_revision:X policy_changed:false provider_env_changed:true]

with an identical old and new config revision, and it does so for sandboxes with no provider attached too. The supervisor then rebuilds the provider environment, which advances the policy generation. A request in flight at that moment fails on the proxy side with

OCSF NET:OPEN [MED] DENIED <host>:443 [reason:L7 tunnel closed before inspection because policy changed: policy generation is stale]

and the client sees the connection closed without a response.

Likely cause, from the source: current_provider_env_revision is seeded from the local credential snapshot (ctx.provider_credentials.snapshot().revision, crates/openshell-supervisor/src/lib.rs around line 3878), which does not equal the server-computed provider_env_revision, so the comparison in the poll loop (around line 4064, result.provider_env_revision != current_provider_env_revision || ctx.provider_readiness.needs_environment(...)) is true on the first poll regardless of any change.

Impact / Why This Matters

• Sandboxes that start working immediately (agents, --no-keep jobs) can have a request dropped about 10 s in. Observed once with real traffic: the sandbox was created at 19:37:11, the detection fired at 19:37:21.963, the stale-generation denial at 19:37:21.970, and the client got "Remote end closed connection without response".
• Every sandbox emits a change detection when nothing changed, which is noise in OCSF pipelines and hides real changes.
• Workaround: clients retry once on a closed connection. Not acceptable for arbitrary workloads.

Acceptance Criteria

• A sandbox whose config and providers did not change does not report provider_env_changed:true on its first poll (for example, seed the initial revision from the server's value at startup, or treat the first observation as the baseline).
• No policy-generation bump and no tunnel closure when nothing changed.
• A test covering the first settings poll after start.

Reproduction Steps

1. openshell sandbox create --name sb --detach -- sleep infinity, with or without --provider.
2. Read the supervisor log. Docker driver: docker logs openshell-default--sb-<uuid>-supervisor.
3. At start plus 10 s, observe CONFIG:DETECTED ... policy_changed:false provider_env_changed:true with identical revisions. Reproduced 4 of 4 times on 0.1.2, one sandbox with no provider at all.
4. To see the drop: start a request loop right after create (one request per second to an allowed host). With a request in flight at the 10 s mark, the supervisor logs the stale-generation denial and the client sees a closed connection.

Environment

• OpenShell 0.1.2 CLI and gateway (Homebrew install), macOS 26.7 arm64.
• Docker compute driver through Rancher Desktop (Docker 29.5, Alpine VM).
• Sandbox image ghcr.io/astral-sh/uv:python3.12-bookworm-slim.

Logs

Four sandboxes, start time and first detection:

probe-a  20:04:12.034 Starting sandbox supervision
probe-a  20:04:22.131 CONFIG:DETECTED ... [old_revision:9754802694322051917 new_revision:9754802694322051917 policy_changed:false provider_env_changed:true]
probe-b  20:06:18.424 Starting sandbox supervision
probe-b  20:06:28.523 CONFIG:DETECTED ... [old_revision:9754802694322051917 new_revision:9754802694322051917 policy_changed:false provider_env_changed:true]
probe-c  20:06:33.839 Starting sandbox supervision            (no provider attached)
probe-c  20:06:43.950 CONFIG:DETECTED ... [old_revision:16709790733110020355 new_revision:16709790733110020355 policy_changed:false provider_env_changed:true]
probe-d  20:07:54.065 CONFIG:LOADED Acknowledged initial policy revision
probe-d  20:08:04.066 CONFIG:DETECTED ... [old_revision:9754802694322051917 new_revision:9754802694322051917 policy_changed:false provider_env_changed:true]

The observed drop (sandbox created 19:37:11, request in flight):

19:37:21.963 OCSF CONFIG:DETECTED [INFO] Settings poll: config change detected [old_revision:9754802694322051917 new_revision:9754802694322051917 policy_changed:false provider_env_changed:true]
19:37:21.968 OCSF CONFIG:CONFIGURED [INFO] OPA runtime binary identity mode configured [source:proto require_binary_identity:true]
19:37:21.970 OCSF NET:OPEN [MED] DENIED inference.generativeai.us-chicago-1.oci.oraclecloud.com:443 [reason:L7 tunnel closed before inspection because policy changed: policy generation is stale]

[fede-kamel]

Update after digging further: this is already fixed on main, and the fix is not in a release yet.

Root cause (0.1.2). At startup the supervisor captures the provider environment while loading the policy and installs the captured ProviderCredentialState directly. In 0.1.2 that path never called readiness.credentials_installed(...), so the ProviderReadinessTracker stayed at its initial WaitingForCredentials state with a default identity. On the first settings poll, needs_environment(&desired_identity) is therefore true even though the revision matches, the supervisor refetches and reinstalls the environment, the policy generation advances, and any tunnel opened under the previous generation is closed. That is the provider_env_changed:true at start + 10 s with identical revisions, and the "policy generation is stale" denial for a request in flight.

Fix. b8932d4 Fix/startup provider readiness (#3819), merged 2026-09-30 06:03 UTC (after v0.1.2). It introduces CapturedProviderEnvironment and has the startup path call readiness.credentials_installed with the environment's identity, plus a test that covers startup provider polling.

Verification.

• 0.1.2 gateway + 0.1.2 supervisor: 4 of 4 fresh sandboxes log the spurious detection at start + 10 s (this issue).
• Gateway and supervisor built from main (Docker driver, same host): a fresh sandbox logs no CONFIG:DETECTED for the first 18 s.
• A server-side check that GetSandboxConfig and GetSandboxProviderEnvironment return the same policy_hash, provider_env_revision, and provider_attachment_epoch passes on main, so the gateway side was never the cause.

Suggest closing this as fixed by #3819, or keeping it open only to track the release that ships it. Until then, 0.1.2 users should expect one environment reload about 10 s after a sandbox starts and retry a request that fails with a closed connection at that moment.