Repository navigation
feat: emit credly expiries from the badge state table with a daily sync - #5141
Conversation
A daily expirySync reads the accepted badges past their date that were not emitted yet, builds expired-badge from the stored row, emits it on the segment of the first sight and marks the row. No Credly call. A shadow run emits shadow records and marks nothing. Refs linuxfoundation/lfx-cdp-insights-ops#873 Signed-off-by: Umberto Sgueglia <usgueglia@contractor.linuxfoundation.org>
PR SummaryMedium Risk Overview The sync loads due rows in 500-row batches (accepted, past Data layer: Reviewed by Cursor Bugbot for commit d31d762. Bugbot is set up for automated code reviews on this repo. Configure here. |
There was a problem hiding this comment.
🟡 Changes recommended
Expiry processing can indefinitely block valid records behind 500 unmappable rows and publish and permanently mark expiries for soft-deleted integrations or integrations moved to another segment.
2 open issues
What changed in this PR
Adds a daily Credly expiry sync that generates expired-badge activities from stored badge state without calling Credly.
Changes:
- Processes expiries in batches, emits to stored targets, and marks each group after emission.
- Shares earner mapping between live and stored badges.
- Adds coverage for batching, run-budget deferral, skipped rows, shadow mode, and dependency wiring.
| File | Description |
|---|---|
services/libs/connectors/src/connectors/credly/syncs/expirySync.ts |
Implements batched expiry emission and completion tracking. |
services/libs/connectors/src/connectors/credly/syncs/expirySync.test.ts |
Tests expiry mapping and sync behavior. |
services/libs/connectors/src/connectors/credly/syncs/badgePoll.test.ts |
Updates dependency mocks for the expanded interface. |
services/libs/connectors/src/connectors/credly/mapper.ts |
Maps stored expiries and extracts shared earner mapping. |
services/libs/connectors/src/connectors/credly/index.ts |
Registers the daily expiry sync. |
services/libs/connectors/src/connectors/credly/discover.test.ts |
Verifies the new sync registration and cadence. |
services/libs/connectors/src/connectors/credly/deps.ts |
Exposes due-expiry reads and emission marking. |
services/libs/connectors/src/connectors/credly/deps.test.ts |
Tests expiry dependency delegation. |
services/apps/connectors_worker/src/bin/dry-run-credly-badge-poll.ts |
Adds expiry dependency stubs to the dry-run script. |
Review effort: Balanced
| await ctx.emit(activities, target) | ||
| counters.emitted += activities.length | ||
| if (ctx.emitEnabled) counters.marked += await ctx.deps.markExpiriesEmitted(badgeIds) |
…tions Due expiries are read only for badges whose integration is still live on their segment, and the sync advances on a (expiresAt, badgeId) cursor so a row it could not mark never blocks the rest. Refs linuxfoundation/lfx-cdp-insights-ops#873 Signed-off-by: Umberto Sgueglia <usgueglia@contractor.linuxfoundation.org>
Closes linuxfoundation/lfx-cdp-insights-ops#873
What
Last piece of the badge state table:
expirySync, a second sync on the Credly manifest, every 24 hours.integration."credlyBadges". No Credly call.expired-badgefrom the stored row (expiredBadgeActivityFromRow): timestampexpiresAt, template name and url from the row, badge url from its id, earner email and name of the first sight. Same member shape as the live badge, through one shared helper.expiredEmittedAt). A badge is emitted as expired once; an expiry extended afterwards stays silent, as agreed.complete: falsewhen the run budget ends with rows still due, so the worker resumes in a minute rather than a day. Run logcredly_expiry_sync_runwithdue,emitted,marked,skipped,drained.Operations
The manifest change does not create the sync unit:
pnpm script:seed-credly-sync-unit --segment-id <anchor segment>must be re-run once in staging and later in prod. It leaves thebadgePollunit untouched and createsexpirySyncwithemitEnabled = false; switch it on like the poll's. There is one Credly organization, so oneexpirySyncunit drains the whole table by design.Code review triage
Applied: short defer instead of the daily cadence when the budget runs out; short-page check on the raw batch; mark per target group after its emit; groups carry only the target; comments trimmed.
Not applied: scoping the due list by integration (one organization, one unit); dropping the pre-emit schema parse (same pattern as the poll, it turns a bad row into a warn and skip).
Verification
169 Credly tests. Staging: re-run the seed, force the
expirySyncunit; the table already holds a CKAD badge expired on 2025-02-28, which must come out as oneexpired-badgeon umbertoTest5.1 withexpiredEmittedAtset; a second run must emit nothing.No reviewer requested.