Keep UniqueList, the casted dicts and SliceableDeque consistent - #52
Merged
Merged
Conversation
Indexed assignment was the only mutator that released the value it replaced. Slice assignment, extend, pop, remove, clear, += and *= all left the membership set out of step with the items, so a removed value could not be added again and extend could add duplicates. - Slice assignment stores the values first and updates the membership afterwards, accepts one-shot iterables, lets a slice reuse the values it replaces and rejects a slice that repeats a value. - extend, += and *= follow the on_duplicate mode. - pop, remove and clear release the values they remove. - __new__ creates the membership set and __setstate__ rebuilds it from the items, so pickle keeps working now that extend is overridden, pickles written by 4.0.1 still load, and copy.copy and copy.deepcopy return a working list instead of an empty one. Builds on the indexed assignment fix from PR #51.
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
`values *= 2` repeats every item once, which is what `values.extend(values)` does. Routing the repeat through extend keeps the duplicate policy in one place and removes the ValueError that CodeQL flagged inside __imul__. The pop test asserted on the calls themselves. It stores the popped values first so the asserts have no side effects.
The timeout tests counted items against real sleeps and left 10 to 40 ms of slack. A sleep only promises to take at least as long as requested, so the counts changed on a busy machine and on a coarse clock: - Blocking sleeps that overshoot by 40 ms or more made timeout_generator yield one item fewer, in five test cases and in its doctest. - A 15.6 ms event loop clock resolution, the Windows default, let the 0.05 s timeout fire together with a 0.04 s sleep, so the detector tests stopped at 3 instead of 4. The sync tests and the doctest now run on a fake clock that only moves when it is slept on, and they check the requested sleeps as well. The total timeout tests advance the same clock. The per-item timeout tests yield without waiting and then stall for 10 s against a 0.05 s timeout. One test stays on the real clock and only checks what holds for any sleep accuracy. The fixtures are loaded from a conftest.py in the repository root so the doctests can use them, and the sdist ships that file.
… fix/unique-list-membership
An adversarial pass over the containers found three regressions in the UniqueList change on this branch, and the same kind of bug in the other classes that subclass a builtin. UniqueList, restoring 4.0.1 behaviour: - The default on_duplicate lives on the class again. __new__ wrote it on the instance, which shadowed a policy that a subclass set on its class. - __setstate__ accepts the state of a subclass with slots. It treated the dict and slots pair as a plain dict and lost the slot values. - pop, remove and index assignment no longer raise after the list changed when the value is missing from the membership set. That happens for a value whose hash changed and for an equal but unhashable argument. The set is rebuilt from the list in that case. UniqueList, older bugs: - insert stores the value before it reserves it, so a failed insert leaves no phantom member. - in falls back to the list for an unhashable value. CastedDict and LazyCastedDict: - pickle restored the items through __setitem__ before the casts were back, so protocol 2 and up could not be loaded. copy did it after, and cast every value twice. The raw items now travel in the state. The casts default to None on the class, which also makes pickles written by 4.0.1 loadable. - setdefault and |= go through the casts. - update and the constructor apply keyword arguments last, like dict. SliceableDeque: - __ne__ follows __eq__. Both were True against an equal list. - == against a set is False when an item cannot be hashed. DictUpdateArgs no longer lists an iterable of mappings, which raised or stored a key as the value.
| root because the doctests in ``python_utils`` need them as well. | ||
| """ | ||
|
|
||
| pytest_plugins: tuple[str, ...] = ('_python_utils_tests.clock',) |
PyPy 3.10 words the dict.update error for a second positional argument differently. The test only needs the TypeError.
A second adversarial pass compared the container fixes on this branch with 4.0.1 and found behaviour that had changed without being a fix. Casted dicts: - A subclass with its own __setstate__ gets the default state again. It received the new state that carries the raw items, and copy failed. - __reduce_ex__ returns five items like the default does. It returned three, which broke a subclass that unpacks the result. - An empty dict is described the default way, so 4.0.1 can still load a pickle of it. - setdefault stores a None default without casting it. Casting it raised for int and stored the string 'None' for str. - setdefault casts its key once. LazyCastedDict.__setitem__ cast the key twice, an old bug that made the new setdefault raise for a key cast that is not idempotent. UniqueList: - pop, remove and index assignment never raise after the list changed, whatever the hash of the value does. The membership set is rebuilt, and left as it is when that fails as well. - remove releases the item that was stored. It looked the argument up in the set, which could drop another member with the same hash.
| return function, arguments, state, list_items, dict_items | ||
|
|
||
|
|
||
| class BrokenHash: |
test_aio_timeout_generator still counted items against real sleeps. The case with five sleeps of 0.06 s against a 0.3 s timeout ends one item short as soon as the sleeps run 15 ms late in total. It failed 3 of 25 runs on a busy machine, and fails every time when asyncio.sleep is made 20 ms late. The test now lets asyncio.sleep advance the fake clock. The default iterable test in test_lazy_imports uses the fake clock too, so its 0.05 s timeout cannot end the loop before the second item.
A third adversarial pass over the container fixes found four more differences with 4.0.1. - A __setstate__ from a mixin that follows UniqueList or a casted dict in the method resolution order runs again. The new __setstate__ methods shadowed it. They now hand the state to the next one in line. - A casted dict subclass with its own __reduce_ex__ gets the default state and items to build on. It received the state that carries the raw items, so editing the state as a dict failed. - UniqueList.remove is one list operation again. Looking the value up and popping it by index let another thread get in between, which removed the wrong item or raised IndexError. A stress run showed about 2800 wrong removals in 3 million, and none with list.remove. - UniqueList.__delitem__ deletes from the list first and releases the membership afterwards, like every other mutator. It raised halfway for an item whose hash had changed and left the set short. The base methods for the subclass checks come from the class dictionary. CastedDictBase[...].__reduce_ex__ is the method of the generic alias.
… fix/unique-list-membership
insert, index assignment and slice assignment wrote the list first and the membership set afterwards, so that a failed write left no phantom member. That opened a window between the two steps: another thread inserting the same value still saw it missing from the set, and the list got duplicates. With six threads inserting the same values a stress run showed about 36000 duplicates, where 4.0.1 has none on CPython 3.11 and up. The membership is reserved right after the duplicate check again, as in 4.0.1, and taken back when the list refuses the write.
Slice assignment dropped every old value from the membership set and added the new ones back. A value that the slice reuses, as in a swap, was out of the set in between, and another thread appending it at that moment got a duplicate in. Only the values that really leave the list are dropped now. Index assignment of a value over itself no longer releases and re-adds it either.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Builds on #51 by @shkyyy18, whose commit is the first one on this branch. That PR fixes indexed assignment in
UniqueList. This one applies the same rule to every other way the containers can change: a class that subclasses a builtin has to cover every route into it.The branch also contains #53, so the test hooks pass on any machine. Merge #53 first, or merge this and #53 is included.
UniqueList
UniqueListtracks membership in a set next to the list. Onlyappend,insert,__setitem__and__delitem__updated that set, so the two drifted apart:extend,+=and*=followon_duplicate. Inraisemodeextendchecks every value before it changes the list.pop,removeandclearrelease the values they remove. A failedinsertno longer reserves its value.inanswers for an unhashable value the way a list does.copy.copyandcopy.deepcopyreturned an empty list inignoremode and raised inraisemode. Both now return a working copy.CastedDict and LazyCastedDict
picklerestored the items through__setitem__before the casts were back, so protocol 2 and up could not be loaded.copydid it after and cast every value twice. The raw items now travel in the state.setdefaultand|=go through the casts.updateand the constructor apply keyword arguments last, likedict.SliceableDeque
d == [1, 2]andd != [1, 2]were bothTrue.__ne__now follows__eq__.==against a set isFalsewhen an item cannot be hashed. It raisedTypeError.Compatibility
Checked against the released 4.0.1 in five adversarial rounds. They found regressions in earlier versions of this branch, all fixed here and pinned by tests:
on_duplicateon the class keeps its policy.__slots__keeps its slot values through pickle and copy.pop,remove,deland index assignment do not raise for a value whose hash changed after it was added.__setstate__or__reduce_ex__, or with a mixin that brings one, pickles and copies as it did.UniqueList.removeis one list operation, andinsert, index assignment and slice assignment reserve the membership before they write. Under threads the list is never worse off than in 4.0.1. A stress run with six threads showed about 36000 duplicates for the first version ofinsertand none now.The last round re-ran every reproduction and harness from all rounds and found no line that differs from the expected outcome.
Intended behaviour changes:
UniqueList.extendand+=no longer add duplicates and*=no longer repeats items. The documentation already promised this forextend.setdefaultand|=on the casted dicts cast what they store. ANonedefault is stored as it is.LazyCastedDictcasts a key once when it stores it. It cast the key twice, which only shows for a cast that is not idempotent.LazyCastedDictkeeps the raw values, where it stored the cast ones.SliceableDeque != equal_listisFalse.DictUpdateArgsalias no longer lists an iterable of mappings. That shape raised or stored a key as the value. Nothing changes at runtime.Pickles written by 4.0.1 still load, including casted dict pickles on protocol 2 and up that 4.0.1 itself could not read back. The released 4.0.1
test_containers.pypasses unchanged against this branch.Left alone on purpose, because the change would be visible to working code:
LazyCastedDict.get,popandpopitemreturn the stored value without casting it.SliceableDequehas nomaxlen.UniqueList.removewith an argument that is equal to more than one item can release the wrong member. Releasing the stored item needs two list operations, which is not safe between threads.UniqueList.extendappends value by value, so a batch is not contiguous when other threads extend at the same time.