Skip to content

Fix text-file snapshot positions for Unicode and translated newlines - #619

Open
chivector wants to merge 1 commit into
cloudpipe:masterfrom
chivector:fix/text-file-snapshot-position
Open

chivector wants to merge 1 commit into
cloudpipe:masterfrom
chivector:fix/text-file-snapshot-position

Conversation

@chivector

@chivector chivector commented Oct 26, 2026 •

Copy link
Copy Markdown

Snapshotting a partially read text file can skip unread characters or raise OverflowError. The reducer copies decoded text into StringIO, then seeks using the original TextIOWrapper.tell() cookie. That cookie is not a character offset.

For a UTF-8 file containing "éabc":

with open(path, encoding="utf-8") as source:
    source.read(1)
    restored = pickle.loads(cloudpickle.dumps(source))
    assert restored.read() == "abc"  # currently returns "bc"

Universal-newline translation also breaks ordinary CRLF files. With a lone CR, the decoder can return a very large cookie that makes the snapshot's StringIO.seek() fail.

Count the unread decoded characters and derive the snapshot's character position from the complete decoded contents. Use the original cookie to restore the source stream in finally, including after read failures. Public underlying-stream positions distinguish an actual seek past EOF so its existing behavior is preserved; no decoder internals are inspected.

Validation on Windows / CPython 3.12.14:

  • Added 25 focused regression cases; together with the existing file suite, unchanged 4af5936 production code gives 15 failed, 17 passed. Fixed: 32 passed.
  • Full repository suite with Tornado, ipykernel and the repository test package available: 291 passed, 10 skipped, 2 xfailed.
  • Independent agent review and file-suite rerun passed.
  • Pinned Black 25.9.0, Ruff 0.14.3 and git diff --check pass for the changed files.

The regressions use real files and cover UTF-8, UTF-8 BOM, UTF-16/32, ASCII/Latin-1, translated/preserved CRLF and CR, a position between CR and LF, empty files, EOF, positions beyond EOF, and restoration when either the suffix or full-file read raises a decode error.

python -m pytest tests -q --maxfail=5 -rs

Local PYTHONPATH included the checkout, tests, and an isolated directory containing the test dependencies.

The 10 skips comprise seven missing historical CPython 3.12 pickle fixtures and three existing runtime/protocol guards. Other Python/PyPy/OS matrices were not run locally. Computing the character position adds one read of the unread suffix; its temporary text is released before reading the complete snapshot. No performance improvement is claimed.

Implementation and independent review used Codex agents under the user's authorization; the reported tests were executed locally. No human review is being claimed.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant