Repository navigation
Conversation
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Snapshotting a partially read text file can skip unread characters or raise
OverflowError. The reducer copies decoded text intoStringIO, then seeks using the originalTextIOWrapper.tell()cookie. That cookie is not a character offset.For a UTF-8 file containing
"éabc":Universal-newline translation also breaks ordinary CRLF files. With a lone CR, the decoder can return a very large cookie that makes the snapshot's
StringIO.seek()fail.Count the unread decoded characters and derive the snapshot's character position from the complete decoded contents. Use the original cookie to restore the source stream in
finally, including after read failures. Public underlying-stream positions distinguish an actual seek past EOF so its existing behavior is preserved; no decoder internals are inspected.Validation on Windows / CPython 3.12.14:
4af5936production code gives 15 failed, 17 passed. Fixed: 32 passed.git diff --checkpass for the changed files.The regressions use real files and cover UTF-8, UTF-8 BOM, UTF-16/32, ASCII/Latin-1, translated/preserved CRLF and CR, a position between CR and LF, empty files, EOF, positions beyond EOF, and restoration when either the suffix or full-file read raises a decode error.
python -m pytest tests -q --maxfail=5 -rsLocal
PYTHONPATHincluded the checkout,tests, and an isolated directory containing the test dependencies.The 10 skips comprise seven missing historical CPython 3.12 pickle fixtures and three existing runtime/protocol guards. Other Python/PyPy/OS matrices were not run locally. Computing the character position adds one read of the unread suffix; its temporary text is released before reading the complete snapshot. No performance improvement is claimed.
Implementation and independent review used Codex agents under the user's authorization; the reported tests were executed locally. No human review is being claimed.