feat(newsroom): implement summary backfill job and error classification (task 3.4) #249
No reviewers
Labels
No milestone
No project
No assignees
2 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
prskr/tools!249
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "paperclip/ICB-134-summary-backfill"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Implements task 3.4 Summary backfill job of the OpenSpec change
newsroom(newsroom/tasks/23-summary-backfill.md).Changes
ErrLLMUnavailableininternal/enrichwrapping transport errors, timeouts (ctx.Err() == nil), and HTTP status codes 404, 408, 429, 502, 503, 504. Other non-2xx responses and malformed payloads are treated as entry-specific errors.recordFailureleaves entries inpendingand does not incrementenrichment_attemptswhen encounteringErrLLMUnavailable.ListPendingSummariesquery across all feeds ordered byfirst_seen_at ASCand bounded by limit. NarrowedListPendingEnrichmentto skip entries whereextracted_contentis already present.SummaryBackfillLockID(hash ofsummary-backfill) andAcquireSummaryBackfillLockonStore.Backfillerininternal/enrichwith in-process skip-if-running guard, session-level advisory lock, early exit on first unavailable failure, structured logging (selected,summarized,failed,stopped_unavailable), andRegister(s *schedule.Scheduler).llm.backfill.schedule(default*/15 * * * *) andllm.backfill.limit(default50) with startup validation. Updatedconfig.yamlwith commented-out section.Co-Authored-By: Paperclip noreply@paperclip.ing
Verdict: FAIL: two major defects (one config bug, one spec drift). Everything else is close.
Success condition tested:
newsroom/tasks/23-summary-backfill.mdplus thenewsroom-enrichment,newsroom-ingestionandnewsroom-configspec deltas:pendingand does not count an attemptllm.backfill.schedulethat selects entries missing only their summary (body stored, *or providerextract: false), oldest first, bounded byllm.backfill.limit, stops at the first unavailable error, and is advisory-lockedbazel test //newsroom/...passesReviewed head
26f7bbc, one commit on top ofmain(0947f6e).Steps run
Findings (most severe first)
1. MAJOR:
llm.backfill.limitfrom config is ignored (internal/enrich/backfill.go,NewBackfiller/resolveLimit)NewBackfillersetsb.limit = DefaultBackfillLimit(50). Because of that,resolveLimit()always returns at its first branch and never readscfg.LLM.Backfill.Limit. Only theWithBackfillLimitoption changes the bound, and the existing limit test only uses that option.limit: 2and 5 candidates, 2 entries are selected.b.limitzero by default so the config fallback is reached, or resolve the limit from config in the constructor. Add a test that sets the limit only throughconfig.Config.2. MAJOR (spec drift): entries from
extract: falseproviders are never backfilled, and the ingest sweep still sends them to the LLMextract: false)".newsroom-ingestionsays the ingest sweep "skips it and leaves it to the summary backfill job, so the two never call the LLM for the same entry".extract: false, summarize: trueprovider,extracted_contentstays NULL (Enricher.extractBodyreturnsentry.Contentwithout storing anything). Then:ListPendingSummariesrequiresextracted_content IS NOT NULL AND != '', so the job never selects these entries.ListPendingEnrichmentselectsextracted_content IS NULL, so the ingest sweep keeps picking them up and calls the LLM once per entry on every ingest. It never stops at the first unavailable error.ResolveSettings(keep them only when!Extract && Summarize). Mirror that in the sweep (skip when!Extract). Add a backfill test for anextract: falseprovider.3. MINOR: summarization logic is copied instead of reused (
processEntry/recordFailureinbackfill.go)The task says "call only the summarization step of the enricher". The backfiller re-implements it with the package defaults
DefaultMaxAttemptsandDefaultMaxContentLength. TheEnricherhonoursWithMaxAttemptsandWithMaxContentLength, so the two paths will diverge as soon as either option is used. Prefer exposing the enricher's summarize step, for example aSummarizeOnly(ctx, entry, settings), and calling that.4. MINOR: a
summarize: falseentry stayspendingforever and takes up a limit slot on every runThis only happens after a provider is switched to
summarize: falsewhile it still has pending entries. Once there are at leastlimitsuch entries at the head of the oldest-first order, the job starves.Suggest marking such entries
done, which matchesEnricher.ensureExistingDone.5. NIT
fakeBackfillStore.ListPendingSummariesiterates a map, so the unit test can't check "oldest first". The Postgres integration testTestListPendingSummariesdoes cover it, so this is fine.DefaultBackfillScheduleis defined in bothconfigandenrich. Keep one.panic("Unreachable: ...")guards in the integration test are noise next toassert.Equalon the length.What's good
ctx.Err()==nil, plus 404/408/429/502/503/504, are wrapped inErrLLMUnavailable. Other statuses and malformed responses stay entry failures.recordFailuredoes not count unavailable errors, and there is a 3×unavailable test.summary-backfill, documented).llm.backfill.schedule/llm.backfill.limit; the defaults are tested.lock.gois in scope and keeps behaviour the same.Not covered
cmd/newsroom/main.godoesn't construct the scheduler (3.3) or the backfiller, so nothing runsRegisterin the service yet. The 3.3 scheduler isn't wired inmain.goeither, so I'm not blocking 3.4 on this. The spec scenario "service starts and backfills every 15 min" is not true end-to-end until it is wired.Pushed fixes addressing review findings to
paperclip/ICB-134-summary-backfill(head63152a2):llm.backfill.limitfrom config honoured:NewBackfillerdefaults limit to 0 soresolveLimit()readscfg.LLM.Backfill.Limit. AddedTestBackfiller_ConfigLimitHonouredsetting limit viaconfig.Configonly.extract: falseproviders supported & skipped in sweep:ListPendingSummariesselects pending unsummarized entries without requiringextracted_content. Ingest sweep skips feeds with!ExtractandsweepOnePendingskips entries when!Extract && Summarize. AddedTestBackfiller_ExtractFalseProviderandTestIngest_ExtractFalseProvider_SummaryOnlyEntrySkippedInSweep.SummarizeOnly(ctx, entry, settings)toEnricher(reusingWithMaxAttempts,WithMaxContentLength, error classification, and failure recording), and Backfiller invokes it viaSummarizerStepinterface.summarize: falseentries todone:SummarizeOnlymarks themdoneviaensureExistingDoneto prevent queue starvation. Verified inTestBackfiller_SkipProviderWithSummarizeFalse_MarksDone.DefaultBackfillScheduleintoconfigpackage; removedpanicguards in integration test;bazel test //newsroom/...(12/12) andbazel run //:formatall green.QA re-review (round 2): FAIL, head
63152a2Verdict: FAIL. Round-1 findings 1 and 3 are fixed. The fix for finding 2 introduced a data-correctness regression, and one of the PR's tests is flaky, which makes
bazel test //newsroom/...red on my run.Correction to my round-1 finding 2 (my error)
extract: false+summarize: trueis rejected at startup (internal/config/provider.go:157,ErrExtractDisabled; covered by the config test "summarize without extract"). So the "summary-only entry from anextract: falseprovider" that task 23 §3 mentions can't exist in a validated config. Round-1 finding 2 was unreachable, and asking for it caused the widened query below. Sorry for the churn.Success condition tested: task 23 §3/§4. Select pending, unfiltered, unsummarized entries whose extraction step is complete. Skip
summarize: false. Never call the extractor. Stop at the first unavailability. The sweep leaves summary-only entries alone.bazel test //newsroom/...is green.Steps run
Findings (by severity)
1. MAJOR: the backfill finalizes entries whose extraction is still pending (regression).
ListPendingSummariesdroppedextracted_content IS NOT NULL AND != '', so it now also returns entries that are waiting for an extraction retry. For anextract: true(default),summarize: falseprovider,processEntry→SummarizeOnly→ensureExistingDonemarks such an entrydonewith no body. The ingest sweep then never retries extraction./g/nosum(summarize: false), entry{state: pending, attempts: 1, extracted_content: nil}, thenBackfiller.Run.pendingfor the ingest sweep.stats={Selected:1 Summarized:0 Failed:0} state=done extracted_nil=true.2. MAJOR: entries awaiting extraction take limit slots and starve the backfill. This has the same root cause as finding 1.
processEntrysilently skipsExtract && no bodyentries, but they still count againstllm.backfill.limit, and they sort oldest-first. During an extractor outage, or with any backlog of older extraction retries, the job summarizes nothing.summarize: true, entries 1 and 2 pending without a body, entry 3 pending with a body (ordered by ID, mirroringORDER BY first_seen_at).stats={Selected:2 Summarized:0 Failed:0} gatewayCalls=0.Fix for 1+2: restore the body predicate in
ListPendingSummaries. Theextract: falsebranch is unreachable, see the correction above. Keepsummarize: false→donefor entries with a body (round-1 #4 still holds), and add tests for both repros. The!settings.Extractearly-return inrunPendingSweepand the extra sweep condition can go back to the round-1 shape too. They're harmless, but they handle an impossible config.3. MAJOR (CI honesty):
TestBackfiller_EntryFailureIncrementsAttemptsAndContinuesis flaky.fakeBackfillStore.ListPendingSummariesiterates a Go map, so which entry receives the 400 is random. The test asserts that entry 1 got it.bazel test //newsroom/...→Executed 11 out of 12 tests: 11 tests pass and 1 fails locally.,backfill_test.go:536: Expected values to be equal: -1 +0.-count=200→ 41 failures.Fix: sort the fake's result (by
IDor aFirstSeenAt) so it mirrors the SQLORDER BY. This also makes the limit tests deterministic.Verified fixed from round 1
NewBackfillerno longer seedslimit, andTestBackfiller_ConfigLimitHonouredcovers the config-only path ✅Enricher.SummarizeOnly, which usesmaxAttempts/maxContentLength/recordFailure✅DefaultBackfillScheduleis deduplicated intoconfig✅Not covered
main.go: still not constructed, same as 3.3.QA re-review, round 3: PASS (head
a870fd2)Success condition: task 23 / 3.4 as specified, and the three round-2 findings fixed: (1) the backfill must not mark no-body
summarize: falseentriesdone, (2) no-body entries must not use up limit slots, (3) no flaky test / green CI.Steps run (clean clone, PR branch at
a870fd2)bazel test //newsroom/...Executed 11 out of 12 tests: 12 tests pass.EXIT=0go test ./newsroom/internal/enrich/ -run TestBackfiller -count=200ok(round 2 had 41/200 failures)go test ./newsroom/internal/{enrich,ingest,config}/ -covergo vet ./newsroom/...TestQA_R3_SecondRunAndMixedNoSum(deleted, not committed)Round-2 findings
extracted_content IS NOT NULL AND != ''. The regenerated sqlc matches (write_dbgen_testpasses).processEntryalso guards against an empty body before thesummarize: falsebranch.TestBackfiller_EntriesWaitingForExtractionAreNotSelectedOrStarved: with limit 2 and two older no-body entries, the newest bodied entry is still summarized.Throwaway repro (failure path and second run): one
summarize: falseprovider with a no-body entry and a bodied entry, and a gateway that returns 500. Ran twice:run1={Selected:1 Summarized:0} run2={Selected:0} calls=0 e1=pending e2=done. The no-body entry stayspendingfor extraction retry, the bodied one goesdonewith no LLM call, and the second run is a no-op.Earlier-confirmed scope (round 1, unchanged since): unavailable vs entry-specific error classification, stop at first unavailability, advisory lock, schedule/limit config with validation, ingest sweep leaves bodied summary-only entries to the job.
Nit (non-blocking)
ListPendingSummariesorders only byfirst_seen_at, while the fake also tie-breaks onid. This is harmless, butORDER BY first_seen_at, idwould make the real query deterministic too.Not covered
storage_integration_testistags=manualand skips without Docker (--- SKIP: TestListPendingSummaries). The SQL is a 2-line predicate andwrite_dbgen_testconfirms the generated code is in sync, but I did not run it against a real DB. I also did not test multi-replica locking against real Postgres.main.gowiring: the scheduler and backfiller are not constructed yet, same as 3.3, so the job does not run in the service.Not merged; the final review and merge are the owner's.