chore(search): add paced projection replacement and data retirement by icecrasher321 · Pull Request #8534 · simstudioai/sim

icecrasher321 · 2026-10-01T18:51:13Z

Summary

Follow chore(search): remove legacy indexed enterprise search #8528, now merged into staging. Add operator-run, checkpointed replacement of embedding_search that retains ordinary-KB vectors and retires indexed Search chunks in paced batches. Deploying this PR starts no maintenance.
Build shared HNSW indexes on the empty replacement and populate them incrementally. Validate retained data and concurrent changes before an explicit, fail-fast cutover; retire the backup and remove capture at separate operator gates. No final reindex or vacuum runs.
Require a verified direct PlanetScale primary connection, fresh primary/replica/application telemetry, bounded transactions and cooldowns. Preserve ordinary KBs, canonical document ACLs and live Search configuration. Stop after connection loss instead of reconnecting without the maintenance lock.
Require the code-removal release to be fully deployed and older workers drained. Start with single-page runs while watching telemetry; verify ordinary-KB retrieval, ACL denials, connector ingestion and live Search before begin-purge. The CLI help contains the command sequence and health-file contract.
Fence the entire schema-push command chain against retirement, refuse pushes once retirement starts, and disable the old bulk-cleanup CLI. Remaining legacy compatibility columns and tables need a separate writer/bootstrap cleanup and schema contraction.

Type of Change

Improvement

Testing

23 real PostgreSQL cases with both schema-push and versioned-migration provisioning, covering concurrent writes, all vector widths, deferred repair, snapshot/lock refusal, late writes, dependency drift, CLI health/connection gates, schema-push contention and process connection loss. JSON reports use INTEGRATION_REPORT_PATH.
61 focused health and schema-push unit tests. The corrected late-write precondition, both schema-push races, endpoint admission and lost-session checkpoint protection were verified failing before their fixes.
Database and explicit CLI type-checks, lint, all 54 audits, artifact generation, docs manifest, block registry and migration checks.

Checklist

Code follows project style guidelines
Self-reviewed my changes
Tests added/updated and passing (new tests pass the test-audit authoring gate)
No new warnings introduced
I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

vercel · 2026-10-01T18:51:17Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
docs	Ready	Preview	Oct 1, 2026 8:12pm UTC

icecrasher321 · 2026-10-01T18:51:27Z

@greptile

icecrasher321 · 2026-10-01T18:51:29Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-10-01T18:51:35Z

@cubic-dev-ai review this PR

@icecrasher321 I have started the AI code review. It will take a few minutes to complete.

greptile-apps · 2026-10-01T18:56:45Z

Confidence Score: 5/5

[Critical risk] Implements database schema migration and search index retirement.

The PR appears safe to merge based on this review; no new findings remain, and all previous threads are resolved.

Summary

This PR adds an operator-run, checkpointed replacement for embedding_search, with paced copying, validation, explicit cutover, and separate gates for backup removal and Search-data retirement. It also adds health and connection checks, fences schema pushes during retirement, and disables the legacy bulk-cleanup CLI.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  A[Prepare empty replacement and capture writes] --> B[Copy and reconcile in bounded pages]
  B --> C[Validate and wait for cutover approval]
  C --> D[Swap projection; retain backup]
  D --> E[Explicitly retire backup]
  E --> F[Purge Search chunks and retire documents]
  F --> G[Finalize and remove capture]

_{Reviews (6) · Last reviewed commit: "fix(db): preserve test connection URL pa..."}

cubic-dev-ai

All reported issues were addressed across 14 files

_{Tip: instead of fixing issues one by one fix them all with cubic

Re-trigger cubic}

icecrasher321 · 2026-10-01T19:09:01Z

@greptile

icecrasher321 · 2026-10-01T19:09:02Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-10-01T19:09:09Z

@cubic-dev-ai review this PR

@icecrasher321 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

All reported issues were addressed across 14 files

_{Requires human review: Auto-approval blocked because this review re-detected 1 unresolved issue already reported by Cubic.
Tip: cubic can generate docs of your entire codebase and keep them up to date. Try it here.

Re-trigger cubic}

gitguardian · 2026-10-01T19:37:37Z

️✅ There are no secrets present in this pull request anymore.

If these secrets were true positive and are still valid, we highly recommend you to revoke them.
While these secrets were previously flagged, we no longer have a reference to the
specific commits where they were detected. Once a secret has been leaked into a git
repository, you should consider it compromised, even if it was deleted immediately.
Find here more information about risks.

^{_{🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.}}

icecrasher321 · 2026-10-01T19:38:39Z

@greptile

icecrasher321 · 2026-10-01T19:38:40Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-10-01T19:38:47Z

@cubic-dev-ai review this PR

@icecrasher321 I have started the AI code review. It will take a few minutes to complete.

icecrasher321 · 2026-10-01T19:43:12Z

@greptile

icecrasher321 · 2026-10-01T19:43:13Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-10-01T19:43:20Z

@cubic-dev-ai review this PR

@icecrasher321 I have started the AI code review. It will take a few minutes to complete.

icecrasher321 · 2026-10-01T19:44:10Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-10-01T19:44:19Z

@cubic-dev-ai review this PR

@icecrasher321 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

No issues found across 14 files

Confidence score: 5/5

Automated review surfaced no issues in the provided summaries.
No files require special attention.

_{Tip: cubic can generate docs of your entire codebase and keep them up to date. Try it here.

Re-trigger cubic}

cubic-dev-ai

All reported issues were addressed across 14 files

_{You've manually re-run cubic several times on this PR. Each manual re-review checks the full PR again and counts toward your usage quota. To preserve your usage limits, we recommend letting cubic automatically review new commits.
Tip: cubic can generate docs of your entire codebase and keep them up to date. Try it here.

Fix all with cubic | Re-trigger cubic}

icecrasher321 · 2026-10-01T19:53:09Z

@greptile

icecrasher321 · 2026-10-01T19:53:16Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-10-01T19:53:22Z

@cubic-dev-ai review this PR

@icecrasher321 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

No issues found across 14 files

Confidence score: 5/5

Automated review surfaced no issues in the provided summaries.
No files require special attention.

_{You've manually re-run cubic several times on this PR. Each manual re-review checks the full PR again and counts toward your usage quota. To preserve your usage limits, we recommend letting cubic automatically review new commits.
Tip: cubic can generate docs of your entire codebase and keep them up to date. Try it here.

Re-trigger cubic}

icecrasher321 · 2026-10-01T20:07:46Z

@greptile

icecrasher321 · 2026-10-01T20:07:51Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-10-01T20:07:58Z

@cubic-dev-ai review this PR

@icecrasher321 I have started the AI code review. It will take a few minutes to complete.

icecrasher321 · 2026-10-01T20:08:42Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-10-01T20:08:48Z

@cubic-dev-ai review this PR

@icecrasher321 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

No issues found across 14 files

Confidence score: 5/5

Automated review surfaced no issues in the provided summaries.
No files require special attention.

_{You've manually re-run cubic several times on this PR. Each manual re-review checks the full PR again and counts toward your usage quota. To preserve your usage limits, we recommend letting cubic automatically review new commits.
Tip: cubic can generate docs of your entire codebase and keep them up to date. Try it here.

Re-trigger cubic}

icecrasher321 requested a review from a team as a code owner October 1, 2026 18:51

vercel Bot deployed to Preview October 1, 2026 18:54 View deployment

greptile-apps Bot reviewed Oct 1, 2026

View reviewed changes

Comment thread packages/db/maintenance/search-retirement.ts

Comment thread packages/db/maintenance/search-retirement.ts

cubic-dev-ai Bot reviewed Oct 1, 2026

View reviewed changes

vercel Bot deployed to Preview October 1, 2026 19:11 View deployment

cubic-dev-ai Bot reviewed Oct 1, 2026

View reviewed changes

greptile-apps Bot reviewed Oct 1, 2026

View reviewed changes

Comment thread packages/db/scripts/retire-indexed-search.ts

icecrasher321 mentioned this pull request Oct 1, 2026

chore(search): remove legacy indexed enterprise search #8528

Merged

6 tasks done

vercel Bot deployed to Preview October 1, 2026 19:40 View deployment

icecrasher321 force-pushed the codex/safe-search-data-retirement branch from fa63c49 to 7932613 Compare October 1, 2026 19:42

vercel Bot deployed to Preview October 1, 2026 19:45 View deployment

cubic-dev-ai Bot reviewed Oct 1, 2026

View reviewed changes

Comment thread packages/db/maintenance/search-retirement.integration.ts Outdated

vercel Bot deployed to Preview October 1, 2026 19:55 View deployment

cubic-dev-ai Bot reviewed Oct 1, 2026

View reviewed changes

Base automatically changed from codex/remove-indexed-enterprise-search to staging October 1, 2026 20:03

icecrasher321 added 4 commits October 1, 2026 13:04

chore(search): add paced projection replacement and data retirement

46e9c73

chore(search): keep retirement usage in the CLI

ce207dd

fix(db): fence schema push and require direct retirement sessions

6ec913c

fix(db): preserve test connection URL parameters

5593bea

icecrasher321 force-pushed the codex/safe-search-data-retirement branch from 5f428d2 to 5593bea Compare October 1, 2026 20:07

cubic-dev-ai Bot reviewed Oct 1, 2026

View reviewed changes

vercel Bot deployed to Preview October 1, 2026 20:12 View deployment

icecrasher321 merged commit 491ab88 into staging Oct 1, 2026
32 of 33 checks passed

icecrasher321 deleted the codex/safe-search-data-retirement branch October 1, 2026 20:25

waleedlatif1 mentioned this pull request Oct 1, 2026

v0.9.10: geo, chat fixes, sandbox fixes #8539

Open

Conversation

icecrasher321 commented Oct 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Type of Change

Testing

Checklist

Uh oh!

vercel Bot commented Oct 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

icecrasher321 commented Oct 1, 2026

Uh oh!

icecrasher321 commented Oct 1, 2026

Uh oh!

cubic-dev-ai Bot commented Oct 1, 2026

Uh oh!

greptile-apps Bot commented Oct 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Confidence Score: 5/5

Summary

Uh oh!

Uh oh!

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

icecrasher321 commented Oct 1, 2026

Uh oh!

icecrasher321 commented Oct 1, 2026

Uh oh!

cubic-dev-ai Bot commented Oct 1, 2026

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

gitguardian Bot commented Oct 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

️✅ There are no secrets present in this pull request anymore.

Uh oh!

icecrasher321 commented Oct 1, 2026

Uh oh!

icecrasher321 commented Oct 1, 2026

Uh oh!

cubic-dev-ai Bot commented Oct 1, 2026

Uh oh!

icecrasher321 commented Oct 1, 2026

Uh oh!

icecrasher321 commented Oct 1, 2026

Uh oh!

cubic-dev-ai Bot commented Oct 1, 2026

Uh oh!

icecrasher321 commented Oct 1, 2026

Uh oh!

cubic-dev-ai Bot commented Oct 1, 2026

Uh oh!

cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

icecrasher321 commented Oct 1, 2026

Uh oh!

icecrasher321 commented Oct 1, 2026

Uh oh!

cubic-dev-ai Bot commented Oct 1, 2026

Uh oh!

cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

icecrasher321 commented Oct 1, 2026 •

edited

Loading

vercel Bot commented Oct 1, 2026 •

edited

Loading

greptile-apps Bot commented Oct 1, 2026 •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading

gitguardian Bot commented Oct 1, 2026 •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading