Repository navigation
docs(crawl): say that a crawl whose patterns match nothing completes with 0 results - #57
Merged
Merged
Conversation
… a page as a file A crawl reports `discovery`, the strategies it finds URLs with on each page it walks, and a create can ask for them. `links` follows the links in the page's HTML, and is what a crawl runs with when the create leaves it out. The field reads `["links"]` when the server does not send it, so the model parses an older answer too. `content(..., download=True)` asks for the page as a file to save rather than one to show. The text is the same; the server sends a file name built from the content id. `CrawlErrorCode` gains `api_key_cap_reached`, which a crawl ends with when the key that created it reaches one of its credit caps. It parsed as an unknown value before.
… stopped one has none The check that reads a page as a file borrowed the crawl another check stops mid-flight. That stop can land before any page is fetched, and the check then skipped instead of covering anything. It now falls back to a one-item crawl of its own and waits for it, as the check beside it already did.
The lock names the API description this surface was synced to, by its blob id, so a later drift check compares against it instead of reporting the surface as never synced. Two operations in that description are not served to callers and no surface exposes them; the lock lists them as pending, so a change to either does not read as drift here.
… set The Crawl API now ends a crawl that runs out of credits, or whose API key hits a credit cap, as `stopped` with `error.code` `insufficient_credits` or `api_key_cap_reached` and no `stop_reason`, instead of `failed`. The pages it fetched before that stay readable. The docstrings, README and example said `error` appears only on a `failed` crawl and that `stopped` always means the caller stopped it. They now describe both cases. A unit test pins that `wait` returns such a crawl with its error and no stop reason.
The fixtures now carry the error.detail the server returns when credits end a crawl, one per error code.
…it opens The Crawl API now treats include and exclude patterns as a filter on what a crawl returns. The crawl still opens on-site pages that fail them, within `depth` and `max_pages`, to follow their links to pages that match, and never returns those pages. Every page it opens counts toward `max_pages` and is billed. The `create` docstring and the README now say so, and that a crawl whose patterns match nothing ends `completed` with no results.
…eted Acceptance check 19 asserted that an include pattern matching nothing ends the crawl `failed` with `no_items_found`. The Crawl API now ends such a crawl `completed` with 0 results and no `error`, so the check asserts that, and reads the crawl's results to confirm none came back. Check 21 still reads the defaults from the same crawl. The unit test fixture for a failed crawl carries the server's current `no_items_found` detail.
…esults is not a failure A URL that fails include_patterns or exclude_patterns is neither returned nor opened, so the docs no longer say the crawl opens such pages or bills them. A crawl whose pages held nothing new to follow ends completed with no results and no error. seed_unreachable covers an empty start page, and no_items_found means Extract found no data on the start page.
zenrows-dev-pool
Bot
changed the title
…duct names no_items_found now reads as the crawl finding no data it could use on the start page, with the advice to start from a listing or product page.
…hat match nothing The check expects a completed crawl with 0 results, so the report no longer calls it a failed crawl.
StefanoGuerrini
approved these changes
Oct 16, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Before: The SDK said that an include pattern matching nothing makes a crawl end
failedwithno_items_found. Acceptance check 19 expected that too. The SDK did not say whatseed_unreachableandno_items_foundmean.After: A crawl whose pages held nothing new to follow ends
completedwith no results and noerror. That covers a page with no link, only links to other sites, or only links the patterns reject. The README says to check the patterns against the URLs the start page links to.CrawlErrorCodesays thatseed_unreachablecovers a start page that could not be reached, was blocked, or came back empty. It says thatno_items_foundmeans the crawl read the start page but found no data it could use, so it had nothing to follow. The patterns still limit what a crawl opens as well as what it returns, somax_pageskeeps its wording frommain.How:
README.md: one paragraph on the 0-result outcome, next to howwaitreturns afailedcrawl.src/zenrows/crawl/client.py: thecreatedocstring forinclude_patterns/exclude_patternssays a crawl whose patterns reject every link endscompleted.src/zenrows/crawl/models.py: theCrawlErrorCodedocstring gives the narrowed meaning ofseed_unreachableandno_items_found.tests/test_crawl_client.py: theno_items_foundfixture carries the same wording as its doc.acceptance/crawl: row 19, its check and its report title expectcompleted,items_found0, no results and noerror.acceptance/crawl/contract.lockrecords the contract this surface follows.maininto the branch.Tests:
ty check src: all checks passed.ruff check src/ tests/ acceptance/: all checks passed.ruff format --check src/ tests/ acceptance/: 34 files already formatted.pytest: 251 passed, 1 deselected (e2e).bash acceptance/crawl/run.sh --no-key: PASS 7, SKIP 21 (the live checks need a key), 0 findings.