Skip to content

Keep UniqueList, the casted dicts and SliceableDeque consistent - #52

Merged
wolph merged 13 commits into
developfrom
fix/unique-list-membership
Oct 2, 2026
Merged

wolph merged 13 commits into
developfrom
fix/unique-list-membership

Conversation

@wolph

@wolph wolph commented Oct 2, 2026 •

Copy link
Copy Markdown
Owner

Builds on #51 by @shkyyy18, whose commit is the first one on this branch. That PR fixes indexed assignment in UniqueList. This one applies the same rule to every other way the containers can change: a class that subclasses a builtin has to cover every route into it.

The branch also contains #53, so the test hooks pass on any machine. Merge #53 first, or merge this and #53 is included.

UniqueList

UniqueList tracks membership in a set next to the list. Only append, insert, __setitem__ and __delitem__ updated that set, so the two drifted apart:

values = UniqueList(1, 2, 3, on_duplicate='raise')
values[0:2] = [8, 9]
1 in values        # True, although the list is [8, 9, 3]

values = UniqueList(1, 2)
values.pop()
values.append(2)   # ignored, although the list is [1]

values = UniqueList(1, 2)
values.extend([2, 2])
values             # [1, 2, 2, 2]
  • Slice assignment stores the values first and updates the membership afterwards. It accepts one-shot iterables, lets a slice reuse the values it replaces, and rejects a slice that repeats a value.
  • extend, += and *= follow on_duplicate. In raise mode extend checks every value before it changes the list.
  • pop, remove and clear release the values they remove. A failed insert no longer reserves its value.
  • in answers for an unhashable value the way a list does.
  • copy.copy and copy.deepcopy returned an empty list in ignore mode and raised in raise mode. Both now return a working copy.

CastedDict and LazyCastedDict

values = CastedDict(int, int, {'1': '2'})
pickle.loads(pickle.dumps(values))   # AttributeError: no attribute '_value_cast'
values.setdefault('3', '4')          # stored as '3': '4', not cast
values |= {'5': '6'}                 # stored as '5': '6', not cast
CastedDict(None, None, {'a': 1}, a=2)  # {'a': 1}, dict gives {'a': 2}
  • pickle restored the items through __setitem__ before the casts were back, so protocol 2 and up could not be loaded. copy did it after and cast every value twice. The raw items now travel in the state.
  • setdefault and |= go through the casts.
  • update and the constructor apply keyword arguments last, like dict.

SliceableDeque

  • d == [1, 2] and d != [1, 2] were both True. __ne__ now follows __eq__.
  • == against a set is False when an item cannot be hashed. It raised TypeError.

Compatibility

Checked against the released 4.0.1 in five adversarial rounds. They found regressions in earlier versions of this branch, all fixed here and pinned by tests:

  • A subclass that sets on_duplicate on the class keeps its policy.
  • A subclass with __slots__ keeps its slot values through pickle and copy.
  • pop, remove, del and index assignment do not raise for a value whose hash changed after it was added.
  • A subclass with its own __setstate__ or __reduce_ex__, or with a mixin that brings one, pickles and copies as it did.
  • An empty casted dict is pickled in the form that 4.0.1 can load.
  • UniqueList.remove is one list operation, and insert, index assignment and slice assignment reserve the membership before they write. Under threads the list is never worse off than in 4.0.1. A stress run with six threads showed about 36000 duplicates for the first version of insert and none now.

The last round re-ran every reproduction and harness from all rounds and found no line that differs from the expected outcome.

Intended behaviour changes:

  • UniqueList.extend and += no longer add duplicates and *= no longer repeats items. The documentation already promised this for extend.
  • Slice assignment is more permissive for values the slice itself replaces, and stricter for a slice that repeats a value.
  • setdefault and |= on the casted dicts cast what they store. A None default is stored as it is.
  • A LazyCastedDict casts a key once when it stores it. It cast the key twice, which only shows for a cast that is not idempotent.
  • A copy of a LazyCastedDict keeps the raw values, where it stored the cast ones.
  • When the same key is passed to a casted dict both in the mapping and as a keyword, the keyword wins.
  • SliceableDeque != equal_list is False.
  • The DictUpdateArgs alias no longer lists an iterable of mappings. That shape raised or stored a key as the value. Nothing changes at runtime.

Pickles written by 4.0.1 still load, including casted dict pickles on protocol 2 and up that 4.0.1 itself could not read back. The released 4.0.1 test_containers.py passes unchanged against this branch.

Left alone on purpose, because the change would be visible to working code:

  • LazyCastedDict.get, pop and popitem return the stored value without casting it.
  • A slice of a bounded SliceableDeque has no maxlen.
  • UniqueList.remove with an argument that is equal to more than one item can release the wrong member. Releasing the stored item needs two list operations, which is not safe between threads.
  • UniqueList.extend appends value by value, so a batch is not contiguous when other threads extend at the same time.

shkyyy18 and others added 2 commits October 1, 2026 09:23
Indexed assignment was the only mutator that released the value it
replaced. Slice assignment, extend, pop, remove, clear, += and *= all
left the membership set out of step with the items, so a removed value
could not be added again and extend could add duplicates.

- Slice assignment stores the values first and updates the membership
  afterwards, accepts one-shot iterables, lets a slice reuse the values it
  replaces and rejects a slice that repeats a value.
- extend, += and *= follow the on_duplicate mode.
- pop, remove and clear release the values they remove.
- __new__ creates the membership set and __setstate__ rebuilds it from the
  items, so pickle keeps working now that extend is overridden, pickles
  written by 4.0.1 still load, and copy.copy and copy.deepcopy return a
  working list instead of an empty one.

Builds on the indexed assignment fix from PR #51.
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

Comment thread _python_utils_tests/test_containers.py Fixed
Comment thread _python_utils_tests/test_containers.py Fixed
Comment thread python_utils/containers.py Fixed
`values *= 2` repeats every item once, which is what `values.extend(values)`
does. Routing the repeat through extend keeps the duplicate policy in one
place and removes the ValueError that CodeQL flagged inside __imul__.

The pop test asserted on the calls themselves. It stores the popped values
first so the asserts have no side effects.
wolph added 3 commits October 2, 2026 13:48
The timeout tests counted items against real sleeps and left 10 to 40 ms
of slack. A sleep only promises to take at least as long as requested, so
the counts changed on a busy machine and on a coarse clock:

- Blocking sleeps that overshoot by 40 ms or more made timeout_generator
  yield one item fewer, in five test cases and in its doctest.
- A 15.6 ms event loop clock resolution, the Windows default, let the
  0.05 s timeout fire together with a 0.04 s sleep, so the detector tests
  stopped at 3 instead of 4.

The sync tests and the doctest now run on a fake clock that only moves
when it is slept on, and they check the requested sleeps as well. The
total timeout tests advance the same clock. The per-item timeout tests
yield without waiting and then stall for 10 s against a 0.05 s timeout.
One test stays on the real clock and only checks what holds for any sleep
accuracy.

The fixtures are loaded from a conftest.py in the repository root so the
doctests can use them, and the sdist ships that file.
An adversarial pass over the containers found three regressions in the
UniqueList change on this branch, and the same kind of bug in the other
classes that subclass a builtin.

UniqueList, restoring 4.0.1 behaviour:
- The default on_duplicate lives on the class again. __new__ wrote it on
  the instance, which shadowed a policy that a subclass set on its class.
- __setstate__ accepts the state of a subclass with slots. It treated the
  dict and slots pair as a plain dict and lost the slot values.
- pop, remove and index assignment no longer raise after the list changed
  when the value is missing from the membership set. That happens for a
  value whose hash changed and for an equal but unhashable argument. The
  set is rebuilt from the list in that case.

UniqueList, older bugs:
- insert stores the value before it reserves it, so a failed insert leaves
  no phantom member.
- in falls back to the list for an unhashable value.

CastedDict and LazyCastedDict:
- pickle restored the items through __setitem__ before the casts were
  back, so protocol 2 and up could not be loaded. copy did it after, and
  cast every value twice. The raw items now travel in the state. The casts
  default to None on the class, which also makes pickles written by 4.0.1
  loadable.
- setdefault and |= go through the casts.
- update and the constructor apply keyword arguments last, like dict.

SliceableDeque:
- __ne__ follows __eq__. Both were True against an equal list.
- == against a set is False when an item cannot be hashed.

DictUpdateArgs no longer lists an iterable of mappings, which raised or
stored a key as the value.
@wolph wolph changed the title Keep UniqueList membership in sync across all mutators Keep UniqueList, the casted dicts and SliceableDeque consistent Oct 2, 2026
Comment thread conftest.py
root because the doctests in ``python_utils`` need them as well.
"""

pytest_plugins: tuple[str, ...] = ('_python_utils_tests.clock',)
wolph added 2 commits October 2, 2026 14:45
PyPy 3.10 words the dict.update error for a second positional argument
differently. The test only needs the TypeError.
A second adversarial pass compared the container fixes on this branch
with 4.0.1 and found behaviour that had changed without being a fix.

Casted dicts:
- A subclass with its own __setstate__ gets the default state again. It
  received the new state that carries the raw items, and copy failed.
- __reduce_ex__ returns five items like the default does. It returned
  three, which broke a subclass that unpacks the result.
- An empty dict is described the default way, so 4.0.1 can still load a
  pickle of it.
- setdefault stores a None default without casting it. Casting it raised
  for int and stored the string 'None' for str.
- setdefault casts its key once. LazyCastedDict.__setitem__ cast the key
  twice, an old bug that made the new setdefault raise for a key cast
  that is not idempotent.

UniqueList:
- pop, remove and index assignment never raise after the list changed,
  whatever the hash of the value does. The membership set is rebuilt, and
  left as it is when that fails as well.
- remove releases the item that was stored. It looked the argument up in
  the set, which could drop another member with the same hash.
return function, arguments, state, list_items, dict_items


class BrokenHash:
wolph added 5 commits October 2, 2026 15:11
test_aio_timeout_generator still counted items against real sleeps. The
case with five sleeps of 0.06 s against a 0.3 s timeout ends one item
short as soon as the sleeps run 15 ms late in total. It failed 3 of 25
runs on a busy machine, and fails every time when asyncio.sleep is made
20 ms late.

The test now lets asyncio.sleep advance the fake clock. The default
iterable test in test_lazy_imports uses the fake clock too, so its 0.05 s
timeout cannot end the loop before the second item.
A third adversarial pass over the container fixes found four more
differences with 4.0.1.

- A __setstate__ from a mixin that follows UniqueList or a casted dict in
  the method resolution order runs again. The new __setstate__ methods
  shadowed it. They now hand the state to the next one in line.
- A casted dict subclass with its own __reduce_ex__ gets the default
  state and items to build on. It received the state that carries the raw
  items, so editing the state as a dict failed.
- UniqueList.remove is one list operation again. Looking the value up and
  popping it by index let another thread get in between, which removed
  the wrong item or raised IndexError. A stress run showed about 2800
  wrong removals in 3 million, and none with list.remove.
- UniqueList.__delitem__ deletes from the list first and releases the
  membership afterwards, like every other mutator. It raised halfway for
  an item whose hash had changed and left the set short.

The base methods for the subclass checks come from the class dictionary.
CastedDictBase[...].__reduce_ex__ is the method of the generic alias.
insert, index assignment and slice assignment wrote the list first and
the membership set afterwards, so that a failed write left no phantom
member. That opened a window between the two steps: another thread
inserting the same value still saw it missing from the set, and the list
got duplicates. With six threads inserting the same values a stress run
showed about 36000 duplicates, where 4.0.1 has none on CPython 3.11 and
up.

The membership is reserved right after the duplicate check again, as in
4.0.1, and taken back when the list refuses the write.
Slice assignment dropped every old value from the membership set and
added the new ones back. A value that the slice reuses, as in a swap,
was out of the set in between, and another thread appending it at that
moment got a duplicate in. Only the values that really leave the list
are dropped now.

Index assignment of a value over itself no longer releases and re-adds
it either.
@wolph
wolph merged commit e13d62f into develop Oct 2, 2026
17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants