Repository navigation
fix(service-analytics): native SQL judges a comparand against a declared number column by the spec's verdict, as the comparand walk's second arm - #21446
Conversation
…erdict as the walk's second arm The native strategy compiles its own SQL past the engine's field-aware walk, so a comparand against a declared number column reached the driver as written: "abc" counted no row on SQLite and was a 500 on PostgreSQL, true bound 1, and a bare-day $lte met the window rule and counted every row, where the engine door answers INVALID_FILTER / 400. The boolean walk becomes one walk with two arms (number first, the classes are disjoint), each reading its own spec verdict and operator lists, at the same three positions: where (runtimeFilter merged), each measure filter, the dataset scope. A numeric string narrows to its number, copy-on-write. Claude-Session: https://claude.ai/code/session_01DiCSbmJrkzNhuEAier4VoJ Co-authored-by: Claude <noreply@anthropic.com>
…he engine door's answers Each cell is asked at the cube read and the dataset door on both faces (native statement, engine aggregate), on SQLite and on live PostgreSQL where OS_TEST_POSTGRES_URL is set: the card's four cells refuse INVALID_FILTER / 400 before any statement runs, a number is the control, and a numeric string binds the number the engine handed its driver. A registered dataset's own scope and measure filters are judged too, frozen so narrowing must be copy-on-write. Claude-Session: https://claude.ai/code/session_01DiCSbmJrkzNhuEAier4VoJ Co-authored-by: Claude <noreply@anthropic.com>
…rand arm Claude-Session: https://claude.ai/code/session_01DiCSbmJrkzNhuEAier4VoJ Co-authored-by: Claude <noreply@anthropic.com>
📓 Docs Drift Check16 anchor(s) derived from 1 changed package(s); no hand-written page names any of them, so this run has nothing to list — not a clean bill of health. This check sees only pages that NAME a derived anchor: one that documents this change in prose, or enumerates it in an authoring dialect, names none and stays invisible to it on every run. What this run could not see
Coarse fallback — 10 page(s) merely mention a changed package (the pre-#9192 predicate, kept for the deliberately-wide backstop): Which tree this was computed onThis run read A worktree cut from an older # while this PR is open — GitHub drops the merge commit once it closes
git fetch origin 711b79109677defb3dd2ccdd6ca4967e6d3b2051 && git checkout 711b79109677defb3dd2ccdd6ca4967e6d3b2051
# afterwards, rebuild it from the two parents, which stay fetchable
git fetch origin b79301000c85fd5986c0656bde27bb7a70eadf60 7a626eb094e77281f8e7bf4c4869f042ea9de843 && git checkout -B drift-repro b79301000c85fd5986c0656bde27bb7a70eadf60 && git merge --no-ff 7a626eb094e77281f8e7bf4c4869f042ea9de843
node scripts/docs-audit/affected-docs.mjs --json b79301000c85fd5986c0656bde27bb7a70eadf60 |
…t-set narrowing The native face now refuses INVALID_FILTER / 400 where it answered 200 (a 500 on PostgreSQL), the same class of change the boolean arm declared: minor, Clause-② no (narrowing), a BREAKING banner and an ADR-0087 not-required disposition stating this change's facts. Claude-Session: https://claude.ai/code/session_01DiCSbmJrkzNhuEAier4VoJ Co-authored-by: Claude <noreply@anthropic.com>
…arand-shape face, whatever the column type (objectstack-ai#21448) (objectstack-ai#21484) Fixes objectstack-ai#21448 Clause-②: no (narrowing) ## What this changes The shared comparand-shape face (`assertListComparandShapes`, `@objectstack/spec/data`) now refuses a LIST at every scalar operator, whatever the column type. That covers `$gt`, `$gte`, `$lt`, `$lte`, the text operators (`$contains`, `$notContains`, `$startsWith`, `$endsWith`, `$icontains`, `$like`, `$ilike`) and the flags (`$null`, `$exists`, `$empty`). `$eq` and `$ne` keep their own ruled arms. - **Envelope:** `INVALID_FILTER` / 400, before any read. - **Sentence:** one sentence naming the operator, the field, the list and the path. Its leading clause is `driver-memory`'s `arrayComparandError` for the same condition, word for word. - **Remedy:** one value; `$in` (authoring `in`) for "one of these values"; `$between` (authoring `between`) for a range. This implements triage's ruling (5958292323) as written. There is one verdict, at the shared face. It is not in the number or boolean declared-type verdicts. The lowering gains no second rule and no `values[0]` read of a list. ## Measured on `origin/main` `b94a2a727`, SQLite and PostgreSQL 16.14 alike Through `AnalyticsService.query` / `.queryDataset` (what `POST /api/v1/analytics/query` and `/api/v1/analytics/dataset/query` relay), on both faces, and through `engine.find` on the real `ObjectQL`: | filter | engine-aggregate face | native face | `engine.find` | |---|---|---|---| | `{ amount: { $gt: [10, 99] } }` (number) | **200, 2** (the driver got `$gt: 10`) | 400, the number verdict | 400, the number verdict | | `{ amount: { $gt: [10] } }` | **200, 2** | 400, the number verdict | 400, the number verdict | | `{ amount: { $lte: [12, 1] } }` | **200, 2** (`$lte: 12`) | 400, the number verdict | 400, the number verdict | | `{ note: { $gt: ['a', 'z'] } }` (text) | **200, 3** (`$gt: 'a'`) | **200, 3** (bound `'a'`) | 400, driver-sql's bind refusal | | `[['note', '>', ['a', 'z']]]` | **200, 3** | **200, 3** | 400, driver-sql's | | `{ note: { $eq: ['b'] } }`, `{ note: { $ne: ['b'] } }` | 400, the face | 400, the face | 400, the face | | `{ note: { $contains: ['b', 'm'] } }` | 400, this package's LIKE gate | 400, the same | 400, driver-sql's | | controls: `$in: ['b']`, `$nin: ['b']`, `$gt: 10` | 1 / 2 / 2 | the same | the same | **Triage's "measure first": the engine door's own answer for the text cell.** The engine door does NOT bind the first member. On `driver-sql` it refused 400 in the driver's own words ("…cannot be bound as a SQL parameter…"). So its answer was right and only its wording was per driver. The same verdict now answers it first, in the face's words (pinned: the `engine.find` text cell). One position over, `driver-memory` ANSWERS a list at a text operator (`memory-matcher-array-and-date-comparand.test.ts`). The face now refuses that before any driver runs. **Dispatch assumptions this measurement corrected:** - `$eq: [x]` / `$ne: [x]` already answered one 400 on both faces (objectstack-ai#19757 / objectstack-ai#19886's arms). The live defect was the ordering operators, plus a text column's native face. - The lowering reaches the face through `filter-normalizer.ts`'s `assertWhereComparandShapes` (objectstack-ai#20010), not through `comparand-shape.ts`. ## Design - **The operator set is the spec's own:** `SCALAR_COMPARAND_OPERATORS`, the comparand-TYPE face's split, which `filter-comparand-type.test.ts` reconciles against `FieldOperatorsSchema`'s keys. - It moved verbatim from `filter-comparand-type.ts` into a new module outside the `data` barrel (`filter-comparand-operators.ts`). Both faces and the save door read ONE split, and nothing is published: `check:api-surface` is unchanged. - The face test also derives the arm's operators from the schema: every declared operator whose enforced slot refuses an array, which is all but `$in` / `$nin` / `$between`. - **No rule in the lowering.** `lowerAnalyticsWhere` already hands every field entry to the face before any leaf exists. So the arm reaches both analytics faces at every position (`where`, `runtimeFilter`, a dataset's scope, a measure's `filter`) with no code change there. `filter-normalizer.ts` and `comparand-shape.ts` change docblocks only; they state the invariant that only a list operator's array is spread into a leaf's `values`. - **The save door asks the same face** (`filter-save-door-refusals.ts`). - A stored dataset, measure, widget or report filter carrying the shape is refused on save, in the face's sentence less its location. The parity test's §2 requires a save-door sentence for every face arm. - The HTTP routes that Zod-parse a filter in their body therefore answer `VALIDATION_FAILED` / 400, located on the member, as for every other face arm. In-process callers get `INVALID_FILTER` / 400. Both layers are pinned. - **The native number arm (PR objectstack-ai#21446).** `judgedComparands` lowers through `lowerAnalyticsWhere` first, so that arm's `array` refusal is no longer reached at a scalar operator from any native position. `native-sql-strategy.ts` is untouched; the now-unreachable branch is a note, not an edit. - **Flags.** A list at `$null` / `$exists` / `$empty` now reads in the shape sentence, because how many values comes before which value. A non-boolean scalar flag keeps the boolean rule's sentence. ## Pins - **New, `service-analytics`:** `list-at-scalar-operator-both-faces.test.ts`, run on SQLite and PostgreSQL. Each measured cell, plus `$gte` / `$lt: []` / `$contains` / `$startsWith`, `$or` / `$not`, and the FilterArray spelling, is checked at both doors. - Each cell answers one 400 on both faces, with the same message on each face and no raw statement, engine aggregate or driver read. - A registered dataset's scope and measure filter are refused the same way. `DatasetSchema` refuses the stored filter on save, in the sentence less its location. - `engine.find` refuses the text cell in the face's words, not the driver's. - Controls (`$in`, `$nin`, scalar `$gt`, `$between`) count alike on both faces. - **Both-faces pin:** PR objectstack-ai#21446's native-only `$gt: [10]` cell is now a both-faces cell in `native-sql-number-comparand-door.test.ts`. - **New, `@objectstack/spec`:** a `filter-comparand-shape.test.ts` block covering the derived operator set; every list shape (pair, one member, strings, empty); nested paths; every AST spelling that carries a value; the message and remedy (`$in`, `$between`, both declared); flags; controls; the 500-char bound. - **Moved because the face now answers first** (each row left its old table and is pinned as the shape face's): - the declared-type corpora (`filter-number-` / `filter-boolean-comparand-declared-type.ts`) and their tests: list rows only at list members now; - `filter-save-door-face-parity.test.ts` (§1: every declared operator is face-judged; §2: new rows); - objectql's number, boolean and aggregate-flag doors; - REST's number and boolean data doors; - `analytics-filter-refusal-envelope.test.ts` (a new HTTP cell); - analytics' flag, `$empty` and type-face tests. ## Ablations Each leg was committed first, mutated through `scripts/ablation-replace.mjs` (WRAP, with the restore trapped), and its restore proven by blob == HEAD and an empty `git diff HEAD`. - **A: the spec arm deleted.** Rebuilt; `ablation-dist-preflight.mjs @objectstack/spec 'throw arrayScalarComparandError(' --absent` exit 0. - **Predicted:** each list-at-scalar cell goes back to a 200 (or to the native number verdict on a number column); `$eq` / `$ne` and the controls stay green. - **Observed:** analytics went 60 failed / 142 passed (30 cells × SQLite and PG): - the text and number cells were answered 200 on the engine-aggregate face; - the LIKE cells fell to the analytics LIKE gate's words; - `engine.find` answered in driver-sql's words; - the save door accepted the stored filter; - every `$eq` / `$ne` and control cell stayed green. - The face test's new block went 12 red. Restore leg: rebuilt, marker present in `dist/`, tree clean, 202 / 202 green. - **B: the lowering's consumption deleted** (`assertWhereComparandShapes`' face hand-over). The subject resolves from `src`, so no rebuild is owed. - **Predicted:** the object-spelling analytics cells go red; the FilterArray spelling, `$eq`, the save door and `engine.find` stay green. - **Observed:** 48 failed (24 cells × 2 drivers: the object-spelling cells, `$ne` included, and the registered scope and measure). 0 failures among the FilterArray, `$eq`, save-door and `engine.find` cells. ## Verification, at the merged head `2b9fd4f5e` (`origin/main` merged in) - **Full suites:** - `@objectstack/spec` test: 602 files / 17754 tests green. - `@objectstack/service-analytics` test, with PostgreSQL 16.14: 172 files green. One file's 4 live-PG cells need a UTC server; see the acceptance notes. - `@objectstack/objectql` test: 366 files green. One barrel-import test timed out at 5 s at load ~7, then 34 / 34 when run alone. - REST door pins: 4 files, 67 tests (MySQL cells are named skips). - **Typecheck:** spec, service-analytics, objectql and rest all green. - **Face importers, at the pre-merge head:** driver-memory, driver-mongodb, driver-turso, driver-sql (with a non-UTC PostgreSQL server), lint, metadata-core, metadata-protocol, plugin-security and plugin-sharing are all green. - **Gates:** `dispatch-gates.mjs --commands` at `2b9fd4f5e` derives 90 families. All 90 ran with recorded exit codes, all 0, including `check:dual-build-cjs-loads` (105 require entries across 66 packages load). `--ran` reconciles 90 run / 0 NOT MEASURED. - **Lint (narrowed, measured):** `eslint --no-inline-config --format json` over the 24 changed `.ts` files gives 24 files, 0 errors, 0 warnings, none ignored. - The population is read from eslint's own output. - Invariance: `eslint.config.mjs` has no type-aware linting (no `parserOptions.project` / `projectService`), so this diff cannot move an untouched file's verdict. - **Docs:** grepping `content/docs/**` (outside `releases/`) and `skills/**` for the comparand-shape rules and the filter operators found no sentence made false. ## Blast radius - **Shipped producers** writing a list at a scalar operator (object form, `[field, op, value]` and `{ field, operator, value }`, across `examples/`, `skills/`, `content/docs/`, `apps/` and `packages/**` non-test sources): none. The CEL lowering already refuses one (`cel-to-filter.ts`). - **NOT MEASURED:** objectui's console filter builder. `../objectui` is not checked out here, and `packages/console/dist` is not built. ## File surface against the claim The claim named `filter-normalizer.ts`, `comparand-shape.ts`, spec `filter-comparand-shape.ts` and its test, the pins and the changeset. Added, each a consequence of the narrowing inside the rule's consumer radius: - `filter-comparand-operators.ts` (new, internal); - `filter-comparand-type.ts` (the split's import, and one now-false sentence); - `filter-comparand-refusal-text.ts` (the shared sentence); - `filter-save-door-refusals.ts` (the save door's sentence); - the two declared-type corpora and the parity test; - the objectql, REST and analytics pins listed above. None of `native-sql-strategy.ts`, `objectql-strategy.ts`, `analytics-service.ts` or `preview-evaluator.ts` is touched. ## Acceptance notes - **Out of scope; reported to the seat, not filed here.** On PostgreSQL with the server TimeZone set to `Asia/Shanghai`, `objectql-face-order-limit.test.ts`'s live cells answer the newest month bucket of a `date` column holding `2026-06-01` as `2026-05` (2 rows). At UTC the answer is `2026-06` (1 row). The cell is the engine-aggregate face, since the native face declines granularity. Not touched by this diff. - The compilers' `values[0]` reads stay as they are. With the face's arm, no list reaches a scalar leaf through any analytics door. - PR objectstack-ai#21452 (which held the analytics strategies) landed while this was open. It was merged in at `2b9fd4f5e` cleanly, with no overlap. --- _Generated by [Claude Code](https://claude.ai/code/session_01DiCSbmJrkzNhuEAier4VoJ)_ --------- Co-authored-by: Claude <noreply@anthropic.com>
Fixes #21426
Clause-②: no (narrowing)
What this changes
NativeSQLStrategycompiles its own SQL past the engine's field-aware filter walk, so it skipped the spec's number-comparand verdict (numberComparandDoorVerdict,@objectstack/spec/data). It now runs that verdict as the second arm of the one walk that the boolean arm added (#21376, PR #21424). The arm runs at the same three positions:where, with the dataset door'sruntimeFiltermerged into it;filter;Details:
One walk, two arms.
judgedBooleanComparands/narrowBooleanComparandsare renamedjudgedComparands/narrowComparands. A walk named for booleans would lie once it also judges numbers. Each arm is aComparandArmrow with three parts, and nothing in it is copied from the spec (no table, regex or refusal words):numberComparandFieldVerdict/booleanComparandFieldVerdict,judgedalone);NUMBER_COMPARAND_DOOR_*_OPERATORS/BOOLEAN_COMPARAND_DOOR_*_OPERATORS);The two classes are disjoint, so at most one arm judges a member. The number arm is asked first, which is the engine walk's order.
Refusal envelope. The arm throws
invalidFilterError(from the strategy'sfilter-normalizer.ts):INVALID_FILTER/ 400, the same constructor the boolean arm throws through. The message isnumberComparandRefusalMessage's sentence behind the[analytics]prefix. Every native position is bound by the driver (a measure filter compiles into its conditional aggregate's bind), so the sentence uses the spec's default, driver-bound reading.Narrowing. A numeric string narrows to the number the verdict names, so the native statement binds what the engine hands its driver:
12, never"12". It is copy-on-write: a subtree that nothing narrowed is returned by reference. The pins deep-freeze every filter handed in and every registered dataset.Member reader. Unchanged. The declared type comes from the host's
declaredFieldTypehook, for the (object, column) pair thatresolveStorageTargetreturns. Socurrencyandpercentare judged, like everyNUMERIC_VALUE_TYPESmember. A relationship-path member is judged at the related object's declared column. Aformulareaches the walk with noreturnType(the plugin relays none), so both verdicts answerdeferredfor it.{ amount: true }. The boolean arm never touched it: its field verdict isnot-judgedfor a number column. The number arm refuses it as thebooleanform.No
packages/specedit and noobjectql-strategy.tsedit.windowClauseSqlis untouched.Measured
Setup:
amount5, 12 and 30.AnalyticsServicePluginover a realObjectQLengine andSqlDriver.AnalyticsService.query(whatPOST /api/v1/analytics/queryrelays) andAnalyticsService.queryDatasetwith aruntimeFilter(whatPOST /api/v1/analytics/dataset/queryrelays).where3a6d92f78{ amount: "abc" }DATABASE_ERRORINVALID_FILTERINVALID_FILTER{ amount: { $lte: "9999-12-31" } }{ amount: { $ne: "abc" } }{ amount: true }{ amount: 12 }(the control){ amount: "12" }"12""12"1212runtimeFilteranswered identically on every cell.{ amount: "abc" }and a measure filter{ amount: { $ne: "abc" } }answered 200 on SQLite and 500 on PG on the native face, and 400 on the engine face. Both answer 400 now.$lteand$neat count 2 over different rows. The shape is the same: 200 where the engine refuses.Pins:
native-sql-number-comparand-door.test.tsThe file mirrors
native-sql-boolean-comparand-door.test.ts. Each parity cell asks both faces, at the cube read and at the dataset door. It asserts:where.What it covers:
number,currencyandpercent;$or, as a list member, a blank string and"+5";[12, 30]for$in ["12", 30].The PostgreSQL cell runs where
OS_TEST_POSTGRES_URLis set, and is a named skip otherwise, as in the boolean twin. It ran here against the live server: SQLite plus PG is 114 tests.Two cells are pinned on the native face alone, because the engine face does not reach this verdict there:
account_credit, the cube dimension overaccount.credit). The engine face refuses every cross-object filter (INVALID_FIELD/ 400). The native face joins and judges the related object'snumbercolumn:"abc"is refused, and{ $gt: "100" }binds100.{ amount: { $gt: [10] } }. The shared analytics lowering hands the engine only the list's first member, so the engine face answers 200, 2. The spec's verdict refuses a list where one number belongs, and the native face now does too. See the acceptance notes.Ablations (one-shot and restored; nothing left in the tree)
How each leg ran:
native-sql-strategy.tswas mutated throughscripts/ablation-replace.mjs. Its anchor must hit exactly once.git diff HEADis empty. Each leg also ran inside a script with an EXIT/INT/TERM restore trap.../plugin.jsby relative path, so it reads the source, notdist/.The predicted direction was named before each run. The observation matched the prediction on every leg, on SQLite and PG:
comparandArmFornever returnsNUMBER_ARM)narrowsverdict read aspasses)whereposition's call removedThe narrowing leg's first attempt did not run.
ablation-replacerefused it because the replacement text contained the anchor (count 1 → 1), restored the file, and ran no test. The row above is the re-run with a non-overlapping replacement.Gates
Patch round (declaration only), at HEAD
7a626eb09. The only change is the changeset; no code moved.dispatch-gates --commandsderived the same 63 commands as round 1. All 63 ran in a freshly recreated worktree (fullturbo run build, 72 of 72 tasks from cache, first) and exited 0.--ranreconciled 63 derived, 63 run, 0 NOT-MEASURED, 0 UNRUN.check-adr-0087-registration --base origin/main: reads the changeset as[BREAKING+clause-②-narrowing]and accepts the dispositionnot-required (no-migration-prescription).check-changeset-no-major: nomajorbump. Its level axis needs apull_requestpayload, so it was also driven offline with--eventcarrying this body (see the report on the card).check:changeset-gate-self-tests: exit 0.Round 1, at HEAD
a17f21b80:node scripts/pm/dispatch-gates.mjs --repo objectstack-ai/objectstack --commandsderived 63 commands. All 63 ran and exited 0.--ranreconciled 63 derived, 63 run, 0 NOT-MEASURED, 0 UNRUN.pnpm check:dual-build-cjs-loads: the first attempt answeredPREREQUISITE NOT MET(exit 3, nodist/for unbuilt packages). It was re-run after a fullturbo run build(72 of 72 tasks, 71 cached) and exited 0.For
@objectstack/service-analytics:typecheck: exit 0, andtsc --listFilesincludes both edited files.test: 170 files and 4065 tests passed, withOS_TEST_POSTGRES_URLpointing at the live server. This ran atd77d545a1; the later commit adds only the changeset.Docs
I grepped
content/docs/**(outsidereleases/) andskills/**for the analytics filter's comparand handling. No sentence describes the native face's number comparands, so none became false.Acceptance notes
@objectstack/service-analyticsminor,Clause-②: no (narrowing), a BREAKING banner and an ADR-0087not-required (no-migration-prescription)disposition. This matches the boolean twin's declaration for the same class of change; the seat's review on the card corrected the claim's line tono (narrowing).where(for examplewhere.amount.$ne), as the boolean arm already did. The engine roots it ataggregations[i].filter. This is wording only, and no one is set to carry it.{ note: { $gt: ["a", "z"] } }binds"a". On the engine face,{ amount: { $gt: [10, 99] } }answers 200, 2 (bound10). Filed by the seat as analytics: a list comparand at a scalar operator ({ amount: { $gt: [10, 99] } }) answers 200 bound to its first member on the engine-aggregate face (and on both faces for a text column), where the spec's verdict refuses the list form #21448.$lterule ignores the column type.{ note: { $lte: "9999-12-31" } }on atextcolumn binds nothing and counts every row (3), where the engine face counts 0. The seat's carrier is #5930 step 4 (domain:services): the analytics faces delete their hand-copied filter meaning — the read scope (F9), thewheretree and its compilers (F10), the draft preview (F11) — each naming its typed column reader #21417, which deletes that copy.Generated by Claude Code