Skip to content

Arrays grow by doubling (#453 step 3) - #482

Merged
ASDAlexander77 merged 21 commits into
mainfrom
array-reference-growth
Oct 4, 2026
Merged

ASDAlexander77 merged 21 commits into
mainfrom
array-reference-growth

Conversation

@ASDAlexander77

Copy link
Copy Markdown
Owner

Step 3 of #453 (spec section 6, docs/superpowers/specs/2026-10-03-array-reference-design.md).

#480 is merged; main is merged into this branch, so the diff is this PR's own.

What changes

  • ArrayLayout::ensureCapacity(arrayType, header, needed): when needed is past the header's capacity the data block grows to max(4, 2 * capacity, needed) elements and the slots past the old capacity are zeroed; it returns the data pointer, already stored in the header. It still goes through MemoryRealloc, so an rc data block keeps its count across a move and a block grown from null is born with the header's reference.
  • push, unshift and splice call it instead of reallocating to the exact new length on every call; with room left they write into the block they have.
  • pop, shift, a shrinking splice and a smaller length = keep the block and zero each slot they vacate, so gc does not keep a removed element alive and a later growth reads zero. pop/shift of an empty array leave it empty and give a zeroed element (before, they read the slot before the first, an rc block's count word, and set the length to -1).
  • length = n zeroes the slots between the old and the new length in every model, gc included (the needsGCRuntime() exception is gone): after a pop the slots past length are no longer fresh memory from the allocator. The comment in SetLengthOfOpLowering says why; the default library's Set, Map and Array.map still rely on new slots reading zero.
  • A smaller length = now releases the elements it drops under rc and own before zeroing their slots (it leaked on main too). The loop of 10 pushed strings then length = 0, 200k rounds, peaks at 4.4 MB under rc and own (was 65.9 MB), the same as the splice loop.
  • view() clamps its range at run time (offset to [0, length], count to what is left), so an out-of-range end no longer copies, and under rc retains, memory past the array.
  • main refuses an array-typed parameter in any position in non-DLL builds (JIT, exe, obj). A one-parameter main(args: string[]) compiled ahead of time used to segfault. argv is Ref<string> only.
  • LengthOf no longer has a struct fallback, so no ARRAY_*_INDEX remains outside ArrayLayout.h/Defines.h.
  • Docs: the spec and plan are brought up to date. This covers ABI (old and new objects most likely link, then fail at run time, so rebuild everything including the default library), rc strings over array data (A string cast over an array's data block dangles after the array grows; under rc it counts a block it does not own #481), and the empty pop/shift and shrink release behaviour.
  • New test 00array_growth.ts (1000 pushes through an alias, a popped slot reading zero when length grows again, shift/unshift/push around a growth, class elements across growth and pops), registered as compile, JIT and corpus tests. It also pins this PR's edges: empty pop/shift, 20 unshifts across growth, a shrinking splice then length = reading zero, a growing 9-item splice insert, and the shrink release. 00array_view.ts gains the clamped cases, 00array_rest_copy.ts a push onto an empty rest and [...all] = src, and a new lowering-error test covers the one-parameter main.

Timing

1,000,000 pushes onto a number[], AOT (--emit=obj --opt --opt_level=3, the test runner's link line), median of 3 runs:

model before after
gc 515 ms 18 ms
rc 2282 ms 15 ms

Gates

  • Windows Release suite (ctest -C Release -j 12 --timeout 300, staged default library): 3743/3743 passed, 0 failed (1 disabled test not run).
  • WSL Linux suite: 3728/3728 passed, 0 failed (1 disabled test not run).
  • -mm=own corpus check (every test compiled with --emit=llvm -mm=own, plain and --opt): no test changes between ok and not ok against An array is a reference to a header that never moves (#453, #477) #480; 455/594 and 453/594 ok (the +1 is the new test).
  • 00array_growth.ts and an edge-case program (empty pop/shift, splice delete/insert past capacity/replace, 100 unshifts then 99 shifts, string and nested arrays across shrink and regrow) pass under gc, rc, none and own, with and without --opt.

Not changed

🤖 Generated with Claude Code

ASDAlexander77 and others added 21 commits October 3, 2026 22:48
An array value becomes one pointer to a heap header { data, length,
capacity } that never moves, so every alias sees push/pop/splice/length=.
Rollout in three PRs: a behaviour-free layout helper, the switch, growth.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Eight tasks in three PRs. Rulings folded into the spec: a null header
reads as an empty array (R1), main's string[] argv is made by a new op
(R2), the rest element is built in MLIRGen (R3).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
….cpp)

ArrayLayout reads and writes an array's data and length, and makes one.
No change to the generated IR: --emit=llvm identical for every suite test
under gc, rc and own.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… ArrayLayout

The last places outside the helper that knew the { data, length } layout.
Also drops the llvmArrayType/loc locals Task 1 left unused.
No change to the generated IR.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Closes #477

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An array value is one pointer to { data, length, capacity }. Assigning,
passing and storing an array copy the pointer, so a push, pop, splice or
length= through any name is seen through every other. Under rc the header
is the counted block; a null header reads as an empty array and gets a
header before a change through its slot.

Also: realloc and free of a null payload (an empty array's data) pass
null on instead of null minus the block word; `null` as an array is a
null header; .view() copies its slice; a constant array inside global
data gets a static, immortal header.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`let g: C[];` left its slot unwritten. Reading an element of it now
compares the header pointer with null and branches on the result, and a
branch on an unwritten slot is undefined behaviour the optimizer took:
newWithSpread.ts faulted in every model. The slot starts null instead,
which reads as an empty array (ruling R1).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`.view()` copies its slice into a new array, and the new array releases
its elements like any other; under rc it now takes a reference to each
one it copied, so releasing the view no longer frees elements the source
still holds. push, pop, shift, unshift, splice and length= write the
capacity with the length, keeping capacity = length until growth uses it.

The spec records the accepted limit of ruling R1 for a null slot passed
by value, and stale { data, length } comments describe the header.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
view(from, to) includes `to`, so view(0, 1) is two elements: the class
case asserts it, and the string case asserts its one. Also reflows a
comment and drops two doubled blank lines.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts:
#	tslang/docs/superpowers/specs/2026-10-03-array-reference-design.md
#	tslang/include/TypeScript/LowerToLLVM/ArrayLayout.h
#	tslang/include/TypeScript/LowerToLLVM/CastLogicHelper.h
#	tslang/include/TypeScript/LowerToLLVM/LLVMCodeHelper.h
#	tslang/include/TypeScript/LowerToLLVM/OwnershipRoutineLogic.h
#	tslang/lib/TypeScript/LowerToLLVM.cpp
A string[] argv would read C's char ** as an array header. The main shape
check now accepts only Ref<string> and, for an array argv, says to use
Ref<string> and Deref(argv[i]). Spec section 5 and the plan are updated
(owner's ruling 2026-10-04; the ArrayFromCStrings op is dropped).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On Linux (and wherever GC_LIB_PATH/TSLANG_LIB_PATH are unset) the DLL link
could not find gc or the runtime. -mm=none needs only the runtime, which
--tslang-lib-path points at in the build tree; the check is model-independent.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A string made over an array's data block (`<string><Opaque>Ref(buffer[0])`,
the default library's convertNumber, convertInteger and date formatters)
takes a reference to the block. The release routine freed the block when
the header died, without reading its count, and a string over it then read
freed memory: five default-library tests failed ahead of time under rc.

The data block now gets a count of 1 when a header is made over it (or when
an empty array's first change allocates it), and the header's release drops
that reference; the elements are released and the block freed on its last
reference, as on main. Spec section 4 amended. 00array_data_string_alias
fails under rc (JIT, -O0 and -O3) without the change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Only rc counts the data block (the header's reference plus any string made
over it). Under own the header is the block's only owner; the count test
there kept LLVM from removing a dead block, and own_try_local's known
throw-path leak grew from 6.7 to 8.3 MB.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…h loop

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
push, unshift and splice grow the data block through
ArrayLayout::ensureCapacity to max(4, 2 * capacity, needed) elements and
otherwise write into the block they have, instead of reallocating to the
exact new length on every call. pop, shift, a shrinking splice and a
smaller `length =` keep the block and zero each slot they vacate, so gc
does not keep a removed element alive and a later growth reads zero.
`length = n` zeroes the slots between the old and the new length in
every model, gc included: after a pop the slots past length are no
longer fresh memory. pop and shift of an empty array leave it empty and
give a zeroed element.

ensureCapacity still goes through MemoryRealloc, which keeps an rc data
block's count across a move and births a block grown from null with the
header's reference.

1,000,000 pushes onto a number[] (AOT, --opt): gc 515 -> 18 ms,
rc 2280 -> 15 ms.

Spec section 6 (docs/superpowers/specs/2026-10-03-array-reference-design.md).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts:
#	tslang/include/TypeScript/LowerToLLVM/ArrayLayout.h
#	tslang/lib/TypeScript/LowerToLLVM.cpp
… array parameter

Final whole-branch review fixes for the arrays-as-references work (#453).

- SetLengthOf: a smaller `length =` releases the elements it drops,
  [new length, old length), before zeroing their slots, as splice does
  (emitReleaseArrayElements: nothing under gc and none). 200k rounds of ten
  pushed strings and `length = 0` peaked at 65.9 MB under rc and own; now
  4.4 MB, the same as the loop with `splice(0, 10)`.
- `main` with any array-typed parameter is refused in non-DLL builds, not
  only a second one: `main(args: string[])` compiled and faulted ahead of
  time. New error test main_args_string_array.ts for jit, exe and obj.
- ArrayView clamps the offset to [0, length] and the count to
  [0, length - offset] at run time: `view(1, 40)` of three elements copied
  40 and, under rc, retained the garbage past the end.
- LengthOf's fallback no longer extracts ARRAY_SIZE_INDEX from a struct no
  type lowers to; anything but an array is an error naming the type.
- ArrayLayout: ensureCapacity's comment says ops on a static header are
  broken (#479); make() states its contract (literal null or fresh
  allocation) and asserts against static data.
- Comments: splice's release ordering is described against ensureCapacity.
- Unused locals removed (push, SetLengthOf, the ConstArray cast).
- Tests: 00array_growth pins PR 3 (empty pop/shift, unshift across growth,
  splice shrink/grow, shrink then regrow reads zero, length = 0 drops
  strings and class elements); 00array_rest_copy adds the index-0 rest and
  a push onto an empty rest; 00array_view adds clamped views;
  00array_reference checks an empty array is not null.
- Spec and plan: status, the byValue=false null path (T4-A), the ABI note
  (mixed objects likely link and fail at run time), #481, the argv error
  tests, empty pop/shift and the shrink release; plan R3 is an indexed copy.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ASDAlexander77
ASDAlexander77 merged commit 77d9a6a into main Oct 4, 2026
2 checks passed
@ASDAlexander77
ASDAlexander77 deleted the array-reference-growth branch October 4, 2026 10:45
ASDAlexander77 added a commit that referenced this pull request Oct 4, 2026
…484)

A tuple or record literal whose values are all constants is folded to a
ts.Constant, and each array in it was lowered to a header in constant
global data over the literal's constant data. push reallocated that
global; since #482 pop, shift, splice and `length =` wrote into it, and
the optimiser folded the reads back to the initializer, so they gave
silently wrong results.

Such a literal is now rebuilt with heap copies of its arrays wherever it
becomes a value that can be changed:
- a `const` of it gets storage and is widened to a tuple, as a `const`
  of a constant array is (processConstRef, adjustLocalVariableType,
  adjustGlobalVariableType);
- its cast to a tuple or a class rebuilds it from the literal's
  attributes (copyArraysOfConstTuple), nested tuple literals included;
- a constant array of such tuples is built element by element
  (castConstArrayToArray);
- inside a tuple literal built at run time, it is cast to a tuple;
- a record literal built in a slot (a field known only at run time, or
  boxed) sets its constant array fields as heap copies.

Under rc a module-level array or tuple literal is born at count 0 and
nothing took a count for the global, so the first local that read it
and let go freed it: `const ga = [6, 7]` read through a `const` twice
crashed (since #480 the length lives in the freed header). The global
now retains what its initializer builds (OwnedReturnConsumptionPass,
claimGlobalInitializers), as it already did for a call's result.

Closes #479

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
ASDAlexander77 added a commit that referenced this pull request Oct 4, 2026
… (#489)

Array sizes were never checked. `a.length = -1` made the length a huge
unsigned size, `a.length = 2^42` asked the allocator for a block it
could not give, and growth zeroed the new slots through the null it got
back: an access violation in every model since #482, and before it, a
length ignored or stored with no block behind it.

TypeScript throws a RangeError for a length that is not an integer in
[0, 2^32 - 1]. tslang now stops the program instead, as a failing
assert does (the message, after the output printed before it, with the
file and line ahead of time):

- `length =` with a number checks it is >= 0, <= 2^32 - 1 and survives
  the round trip through an index (no fraction, not NaN), in MLIRGen,
  before the conversion drops what would show it;
- growth (ArrayLayout::ensureCapacity: push, unshift, splice, a longer
  `length =`) checks the length it needs is at most 2^32 - 1 - which a
  negative integer, sign-extended, is not - that the byte count does
  not overflow, and that the allocator gave a block ("Out of memory");
- a new array of a length (`new A(n)` for `type A = number[]`) checks
  the same.

"Invalid array length" is the message for a length, as TypeScript's
RangeError says; "Out of memory" for a failed allocation. Neither is an
exception: try/catch does not see them.

AssertLogic's failure path is shared by `assert` and the new
mid-lowering `check`, and it depends on LLVMCodeHelperBase only, so
ArrayLayout can use it.

Closes #483

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant