Repository navigation
gh-47369: Make the pure-Python pickler avoid recursion limit on deeply nested objects - #160157
Open
coreythompson643-lang wants to merge 1 commit into
Open
coreythompson643-lang wants to merge 1 commit into
coreythompson643-lang wants to merge 1 commit into
Conversation
… deeply nested objects Save containers using an explicit stack of generators instead of recursive calls in pickle._Pickler.
|
All commit authors signed the Contributor License Agreement. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes gh-47369.
Problem
The pure-Python pickler (
pickle._Pickler) saves nested containers through recursivesave()calls, so pickling deeply nested lists, tuples, dicts, sets, frozensets or object states raisesRecursionErrorwell before memory is a concern.Approach
save()now drives the save methods as generators on an explicit stack. A save method that needs a child object saved yields a request instead of callingsave()recursively; exceptions are thrown back into the requesting generator so they propagate (and collect notes) as they did with recursion. Subclass overrides ofsave(),save_reduce()and dispatch table entries are still honoured. The C pickler (_pickle) is not changed.Tests
Added tests in
Lib/test/pickletester.pyfor deeply nested structures with the pure-Python pickler../python -m test test_pickle test_pickletools test_copyreg test_shelvepasses on a local build of main (1,538 tests run, 96 skipped).Performance
The pure-Python pickler is about 1.5–1.9x slower in my measurements because of the generator machinery. The C pickler, which is what
pickle.dumpsuses by default, is unaffected. Feedback on whether this trade-off is acceptable is welcome.