Skip to content

Check each line a listing page adds against the data - #521

Merged
JesperDramsch merged 3 commits into
mainfrom
desk/listing-lines
Oct 28, 2026
Merged

JesperDramsch merged 3 commits into
mainfrom
desk/listing-lines

Conversation

@JesperDramsch

@JesperDramsch JesperDramsch commented Oct 28, 2026 •

Copy link
Copy Markdown
Owner

What this does

Some changedetection watches look at a page that lists many events: PyData upcoming events, SciPy upcoming conferences, the PSF trademark list. When the watch filter keeps one event per line, the diff that changedetection sends is already a list of finds.

Before this change, such a listing-change dispatch went to the desk queue as one signal for the whole page. The desk then had to read the diff and decide line by line.

Now check-conference-update.yml sends the added lines to utils/desk_leads.py listing. For each line:

  • The line names an edition the data tracks (for example "PyCon US 2027"): it is dropped.
  • The line has no year and names a series the data tracks: it is dropped.
  • Else: it goes to the queue as a candidate with the flag listing. The worklist routes it like other candidates: a missing edition of a tracked series goes to the new-year-checker, a new series goes to the scouter.

A line with quoted names, as on the PSF trademark list, is matched by each quoted name.

The PSF trademark list moves to this path

The watch on the PSF PyCon trademark page now keeps one list item per line, such as "PyCon NP", "PyCon Nepal" in Nepal. Its diff goes through the same listing check, so the psf-trademarks source in desk_leads.py is no longer needed. This PR removes:

  • the call to the python.org page API, trademark_items and psf_records;
  • the registry source, which read the trademark list and pycon.org together. pycon-org is now the only registry, and it runs as its own step in desk-leads.yml.

A registry-change dispatch that names psf-trademarks is now refused. On the instance, the trademark watch now sends listing-change.

Files

  • utils/desk_leads.py: listing_names, listing_records, and the listing source with --lines and --url.
  • .github/workflows/check-conference-update.yml: the listing path writes the lines and the resolved page URL to $RUNNER_TEMP. Two new steps, which run only for a listing, set up Pixi and run the data check.
  • .claude/skills/scouter/SKILL.md: the listing flag. A listing page is a lead, never the evidence.
  • utils/desk_queue.py: the queue README row for the flag, and the registry values.
  • .github/workflows/desk-leads.yml: pycon-org as its own step; psf-trademarks and registry removed.
  • Tests in tests/test_desk_leads.py and tests/test_check_conference_update.py.

Watch set-up

A listing watch must keep one event per line. A watch that gives two lines per event (name, then place and dates) makes the second line a false new series. The filter on each listing watch is set on the instance, not in this repository.

Checks

  • pre-commit run on the changed files: clean.
  • Full test suite with the pinned command from REVIEW.md.

https://claude.ai/code/session_016a2XGjzRsX9nEb4swsKYms


Generated by Claude Code

…s the desk

A watch on a page that lists many events keeps one event per line, so its diff is already the list of finds. check-conference-update now hands those lines to desk_leads.py listing, which drops a line whose edition the data tracks (or, for a line without a year, whose series it tracks) and queues the rest as candidates with the flag listing. The worklist routes them like any other candidate: a missing edition of a tracked series to the new-year-checker, a new series to the scouter. Quoted names on a line, as on the PSF trademark list, are each matched.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016a2XGjzRsX9nEb4swsKYms

Copy link
Copy Markdown
Owner Author

Routine review

1. Request

No issue is linked, so the PR description is the request. A listing-change dispatch in .github/workflows/check-conference-update.yml should send each added line to a new listing source in utils/desk_leads.py (listing_names, listing_records, --lines, --url). That source drops a line naming an edition the data tracks, and drops a year-less line naming a series the data tracks. Every other line is queued as a candidate with the flag listing, which the worklist routes to the new-year-checker or the scouter. A line with quoted names is matched by each quoted name. The PR also documents the flag in .claude/skills/scouter/SKILL.md and in the utils/desk_queue.py README table, and adds tests in tests/test_desk_leads.py and tests/test_check_conference_update.py.

2. Checked against

3. Needs your decision

None.

4. Findings

Important: None.

Nits:

  1. utils/desk_leads.py:1143-1144: the comment says markers are stripped "in case a body sends diff_added", but the regex ^(?:\(added\)|\+)\s* strips only (added) and +. According to the changedetection wiki cited above, diff_added can also emit (changed) lines (the old text) and (into) lines (the new text).
    • Run here: name_variants("(into) PyData Foo 2027") returns ['(into) PyData Foo', 'PyData Foo'], and clean_name returns (into) PyData Foo. So with such a body, fields.conference carries the marker.
    • A (changed) line for a series the data does not track is queued as a NEW_SERIES candidate, even though the event left the page.
    • This only happens if a watch sends diff_added and not diff_added_clean.
    • Smallest fix: strip (into) as well, and skip lines that start with (changed) or (removed). Or drop the comment's promise.
  2. .claude/skills/new-year-checker/SKILL.md:17-20: the candidate flags list there has no listing entry, but the PR routes MISSING_EDITION listing candidates to the new-year-checker (test_records_reach_their_skills). The "a listing page is a lead, never the evidence" rule is only in the scouter skill. Suggest the same bullet there.
  3. .github/workflows/check-conference-update.yml, step "Write to the desk queue": for a listing, the commit message is still "Queue a changedetection signal for …", even though it now queues candidates. Cosmetic.

5. Tests

  • Invariant coverage: TestListing.test_drops_what_the_data_tracks_and_routes_the_rest runs one set of lines through listing_records and covers every rule in the request:
    • a tracked edition with an (added) marker is dropped;
    • a missing edition of a tracked series becomes MISSING_EDITION;
    • an untracked series with a year becomes NEW_SERIES;
    • a year-less tracked series with a + marker is dropped;
    • a duplicate line is dropped;
    • quoted names, both tracked (dropped) and untracked (kept) with curly quotes;
    • a plain new name becomes a candidate.
  • test_records_reach_their_skills takes those records through add_items and build_worklist to the new-year-checker and scouter skills.
  • test_the_listing_steps_run_only_for_a_listing pins the workflow guards.
  • This is a feature, not a fix, so the fail-on-base run does not apply. The new functions do not exist on main.

Change (5832146), with the pinned commands from REVIEW.md:

uv run --python 3.11 --with "pandas<2.3",pytest,... pytest tests/ -q -x -p no:cacheprovider
2563 passed, 5 skipped, 3 xfailed, 2 xpassed, 251 warnings in 190.50s (0:03:10)

uvx ruff@0.4.4 check utils tests
All checks passed!

PR status checks on the head commit:

  • pre-commit.ci - pr: success ("checks completed successfully")
  • CI Status, Python Tests, Data Validation, Pre-commit Checks, CodeQL: success

6. Bounds

  • Route: the PR head was fetched by ref and git rev-parse HEAD matched 5832146. The diff and PR text came from the REST API.
  • Read in full:
    • the diff;
    • check-conference-update.yml at the head;
    • .github/actions/desk-queue-write/action.yml. A missing or empty records file writes nothing, so a listing with no new lines is a clean no-op.
  • Read in part, the parts the diff relies on:
    • utils/desk_leads.py: TrackingIndex.classify, is_tracked_name, name_series, clean_name, write_records;
    • utils/desk_queue.py: item_id, which leaves matched out of a candidate's id, so a re-sent line dedupes on source_id;
    • the scouter and new-year-checker skills.
  • The desk-queue branch holds no queued registry: listing item from the old path, so dropping that row from the README and the scouter skill orphans nothing.
  • Not run: the workflow itself (no Actions runner here), and pixi. The pixi default environment carries pyyaml and requests, and desk-leads.yml already runs desk_leads.py under the same setup-pixi@v0.11.2.
  • Pre-existing: none noticed.

Generated by Claude Code

…f its own

The changedetection watch on the trademark page now keeps one list item per line, so its diff goes through the listing check like the other listing pages. That retires the psf-trademarks source: the page API call, trademark_items and psf_records. With pycon.org the only registry left, the registry source that read both goes too, and desk-leads.yml runs pycon-org as its own step.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016a2XGjzRsX9nEb4swsKYms
Comment thread tests/test_desk_leads.py Fixed
…ut a hostname

CodeQL read the test's substring check on pycon.org as URL sanitisation. The tests now check the HTTP error and the summary line instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016a2XGjzRsX9nEb4swsKYms
@JesperDramsch
JesperDramsch merged commit b34718c into main Oct 28, 2026
11 checks passed
@JesperDramsch
JesperDramsch deleted the desk/listing-lines branch October 28, 2026 16:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants