Repository navigation
Conversation
…sis stubs Hypothesis lets a test pin a known-failing input with `@example(...).xfail(reason=..., raises=...)`, which is the natural way to record a bug found by a property test without taking the input out of the strategy or writing a separate expectedFailure test. The stub `example` is a plain decorator function, so any test using `.xfail()` or `.via()` failed to import when Hypothesis is not installed. This turns the stub into a class with those two methods. An expected failure is strict, as in Hypothesis: when the examples are replayed the test must raise one of the `raises` exceptions for that input, and an unexpected pass fails the test. `.via()` is a no-op, as it is upstream.
bedevere-app
Bot
added
awaiting core review
type-feature
pganssle
added a commit
to pganssle/cpython
that referenced
this pull request
Oct 22, 2026
…mples Previously each known bug had its own expectedFailure test generating only the failing inputs. Hypothesis already has a way to say that: an `@example(...).xfail(reason=..., raises=...)` on the test the input belongs to, which is strict in both Hypothesis and (after pythongh-160356) the stubs. Each bug is now a few such examples on test_decode_in_chunks, test_encode_in_chunks, test_encode_decode or test_reader_matches_decode, and the carve-outs in the strategies point at them. Also from review: - MULTIBYTE_ENCODINGS is an issubclass check against _multibytecodec.MultibyteIncrementalDecoder rather than a look at the repr of the MRO. - Only raw_unicode_escape is listed out of the surrogateescape codecs by hand, and for the right reason (a truncated \u escape raises); unicode_escape and utf_7 already fail _lossless_bytes. - xmlcharrefreplace text leaves out the control characters and noncharacters whose references html.unescape drops, not only "&" and the C1 range. - The charmap encoding tables leave out U+FFFE, which charmap_build treats as undefined, and the two encode tests share the strategy. - test_unicode_escape_with_surrogates compared an expression with itself; it now checks the unicode_escape round trip and the raw_unicode_escape round trip on text without backslashes. - StreamReader.read(size) takes a byte count, so the reads are no longer expected to return at most that many characters (big5hkscs decodes one sequence to two). test_reader_read_chars covers the chars argument of the pure Python reader, which the multibyte readers do not have, and test_reader_matches_decode compares the reader with the stateless decoder under every error handler.
This was referenced Oct 22, 2026
pganssle
added a commit
to pganssle/cpython
that referenced
this pull request
Oct 23, 2026
Previously each known bug replaced the affected test for its backend with an expectedFailure version, so the test as a whole was expected to fail and nothing checked that the failure was the known one. With example.xfail() in the stubs (pythongh-160356), the known inputs are now expected-failure examples on the affected test: the generated input leaves the failing region out, as the comments say, and each example must fail, so the test fails again once the bug is fixed. For the needs_input bug (pythongh-159999) the generated calls for bz2 and zlib are unlimited, which is the case they get right; the examples depend on where the compressor puts block boundaries, so each has been checked to fail with both zlib 1.3.2 and zlib-ng 1.3.1, since the expected failures are strict. The FDICT examples (pythongh-160001) go on the zlib wrapper's test_invalid_input, which now clears the flag in generated garbage rather than skipping the input, and they are only registered in release builds, where the missing exception is a SystemError rather than a fatal assertion. The max_length=0 tests for lzma and zstd stop short of the failing call (pythongh-160000, pythongh-160121). Hypothesis also found that the broad zstd tests could reach pythongh-160121 with sixteen zero-length calls in a row, so zstd now shares the lzma carve-out of max_length=0, in the companions and the state machine. The subTest calls inside the invalid-input and zdict checks are gone: Hypothesis disables subTest under @given anyway, and inside the stub replay they swallowed the exception an expected-failure example has to raise.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #107862. Base for the codecs property tests in #160155, where the known-failing inputs should be
@example(...).xfail(...)on the broad tests rather than separateexpectedFailuretest cases.The stub
examplewas a plain decorator function, so a test using.xfail()or.via()failed to import without Hypothesis. This makes it a class with those two methods. In stub mode an expected failure is strict, as it is in Hypothesis: when the examples are replayed, that input must raise one of theraisesexceptions (any exception by default), and an unexpected pass fails the test with the same message Hypothesis gives..via()is a no-op, as upstream.Checked by replaying a scratch test in both modes: a passing
.xfail()example and one raising the wrong type fail, the rest pass, and the existing property tests (test_math_property,test_zoneinfo,test_base64,test_binascii,test_makeunicodedata) are unchanged in stub mode.