trace: git push hangs on long AI sessions because secret redaction is slow

[migmartri]

Problem

git push hangs for a long time when the branch contains commits from a long AI coding session. The pre-push hook runs chainloop trace push. The push does not finish, and the git connection to the remote closes:

$ git push
INF parsing AI-assisted commit records ai_commit_count=15
INF discovered AI agent config files agent=claude files=7
INF uploading ai-agent-config-claude.json - sha256:...
INF successfully collected AI agent configuration agent=claude
Connection to github.com closed by remote host.

In the reported case, the push ran for approximately 7 minutes before the remote closed the connection.

Cause

The time is spent in secret redaction of the CHAINLOOP_AI_CODING_SESSION evidence (internal/redaction). The upload is not the cause.

For a long session, the evidence document is very large. The document contains the full main transcript and all subagent transcripts. In the reported case, the document is 190 MB with 154 subagent transcripts. One betterleaks scan pass over this document takes approximately 10 minutes, and redaction does 2 or more passes. The pre-push hook keeps the git connection open during this time, so the remote closes it.

Expected behavior

Redaction of a large session completes in seconds, and git push does not block long enough for the remote to close the connection.

Research and measurements are in the first comment.

🤖 Posted by Maximus bot (Claude Code) on behalf of @migmartri

[migmartri]

Research

I reproduced the push pipeline offline with the reported session: ParseSession, evidence encoding, crafter validation, and redaction. The upload was not included. Machine: 8 cores.

Where the time goes

Step	Time
Parse the session, encode the evidence (190 MB), decode and validate it	approximately 8 s (heap peak 1.3 GB)
One betterleaks scan pass, stdlib regexp (current code)	10m23s
Full Redact (3 passes for this session), stdlib regexp	approximately 31 min (estimated: 3 × one pass)
Full Redact, RE2 engine	5m20s (measured)

In the CPU profile, 98% of the time is in regexp.(*Regexp).FindAllStringIndex called from detect.(*Detector).detectFragmentWithRule. The scan uses only one core.

Root causes

1. Slow regex engine. The betterleaks CLI selects the RE2 engine (github.com/betterleaks/go-re2, which runs on wazero) by default. When betterleaks is used as a library, the default engine is stdlib regexp. We use the library, so we get stdlib regexp.
2. The document is scanned as one fragment. betterleaksScanner.Scan sends the whole indented document as one sources.Fragment. The betterleaks keyword prefilter selects a rule when one of its keywords occurs anywhere in the fragment. A 190 MB fragment contains almost all keywords, so almost all rules run their regex over all 190 MB. The betterleaks file source reads input in 100 KB chunks for this reason.
3. Each pass rescans the full document. A clean result needs 2 passes, and this session needed 3. Each pass encodes the full document again and scans all of it. Encoding and rewriting add approximately 19 s per pass.
4. Much of the transcript is repeated content. attachment entries are 72 MB of the 160 MB raw transcripts (45%). The command field of hook_success attachments is 36 MB: one third-party hook command of 2,966 bytes occurs 12,778 times with the same text. Other examples are total_tokens_reminder, prompt_snapshot and skill_listing.

Per-rule timings (100 KB chunks): generic-api-key uses approximately 31% of the scan time. Approximately 10 rules with common keywords (key, token, secret, ...) run on almost every chunk (1,951 of 1,952).

Measured options (one scan pass, 190 MB)

Configuration	Time	Findings
stdlib, 1 fragment (current)	10m23s	319
stdlib, 100 KB chunks	1m41s	319
stdlib, 100 KB chunks, 8 workers	26.5 s	—
RE2, 1 fragment	1m27s	319
RE2, 100 KB chunks	13.7 s	319
RE2, 100 KB chunks, 8 workers	3.95 s	—

The findings column shows counts from our scanner, which includes composite components. The parallel runs used Detector.DetectString, which does not include components, so their counts are not comparable and are not shown.

RE2 adds approximately 4.5 MB to the binary. It works with CGO_ENABLED=0.

Proposed changes, by impact

• A. Use the RE2 engine. Call regexp.SetEngine(re2.RE2{}) before the detector is built. Result: 7x faster, with the same findings.
• B. Scan in chunks. Split the document at record or leaf boundaries, so that no secret crosses a chunk boundary. Keep each key next to its value, because the generic rules need that context. With A: 13.7 s per pass.
• C. Scan chunks in parallel. Use one detector for each worker, because the detector keeps state for each run. With A and B: approximately 4 s per pass.
• D. Rescan only what changed. After the first pass, scan only the chunks that had replacements, and rewrite only the leaves that had findings. This removes most of the 19 s encoding and rewrite cost for each pass.
• F. Cache scan results. Keep a cache keyed by the content hash of each record in the trace state, or scan records incrementally from the agent hooks. Each push scans the same growing session again, so with a cache only new records are scanned. This is necessary only if A–D are not sufficient.
• G. Do not block git push. Run the attestation in the background, so that a large session cannot cause the remote to close the connection.

Recommendation: do A, B and C first. They are a small change in internal/redaction and change the worst case from approximately 30 minutes to seconds. Then do D, and G as a safeguard.

Removing the repeated content (root cause 4) is not proposed. Each entry is a real transcript event with its own timestamp, and the evidence stores the transcript as it was captured. Item E, which proposed this, was withdrawn. F reduces the scan cost of repeated content without changing the transcript.

Compaction does not help. The Claude Code transcript only appends entries, so old entries stay in the file.

Related: #3481 / #3489 changes the rewrite step (redactLeaf). That change is about correctness, not speed, but it will conflict with D.

🤖 Posted by Maximus bot (Claude Code) on behalf of @migmartri