feat(newsroom): implement summary backfill job and error classification (task 3.4) #249

Merged
prskr merged 5 commits from paperclip/ICB-134-summary-backfill into main 2026-10-13 10:38:52 +00:00
Contributor

Implements task 3.4 Summary backfill job of the OpenSpec change newsroom (newsroom/tasks/23-summary-backfill.md).

Changes

  • Error classification: Added exported ErrLLMUnavailable in internal/enrich wrapping transport errors, timeouts (ctx.Err() == nil), and HTTP status codes 404, 408, 429, 502, 503, 504. Other non-2xx responses and malformed payloads are treated as entry-specific errors.
  • Attempt rule: recordFailure leaves entries in pending and does not increment enrichment_attempts when encountering ErrLLMUnavailable.
  • Storage: Added ListPendingSummaries query across all feeds ordered by first_seen_at ASC and bounded by limit. Narrowed ListPendingEnrichment to skip entries where extracted_content is already present.
  • Advisory lock: Added SummaryBackfillLockID (hash of summary-backfill) and AcquireSummaryBackfillLock on Store.
  • Backfiller job: Implemented Backfiller in internal/enrich with in-process skip-if-running guard, session-level advisory lock, early exit on first unavailable failure, structured logging (selected, summarized, failed, stopped_unavailable), and Register(s *schedule.Scheduler).
  • Configuration: Added llm.backfill.schedule (default */15 * * * *) and llm.backfill.limit (default 50) with startup validation. Updated config.yaml with commented-out section.
  • Tests: Comprehensive unit and integration tests covering error classification, attempt preservation, backfill single-request on 404, advisory locking, ordering/limit, and config defaults/validation.

Co-Authored-By: Paperclip noreply@paperclip.ing

Implements task **3.4 Summary backfill job** of the OpenSpec change `newsroom` (`newsroom/tasks/23-summary-backfill.md`). ### Changes - **Error classification**: Added exported `ErrLLMUnavailable` in `internal/enrich` wrapping transport errors, timeouts (`ctx.Err() == nil`), and HTTP status codes 404, 408, 429, 502, 503, 504. Other non-2xx responses and malformed payloads are treated as entry-specific errors. - **Attempt rule**: `recordFailure` leaves entries in `pending` and does not increment `enrichment_attempts` when encountering `ErrLLMUnavailable`. - **Storage**: Added `ListPendingSummaries` query across all feeds ordered by `first_seen_at ASC` and bounded by limit. Narrowed `ListPendingEnrichment` to skip entries where `extracted_content` is already present. - **Advisory lock**: Added `SummaryBackfillLockID` (hash of `summary-backfill`) and `AcquireSummaryBackfillLock` on `Store`. - **Backfiller job**: Implemented `Backfiller` in `internal/enrich` with in-process skip-if-running guard, session-level advisory lock, early exit on first unavailable failure, structured logging (`selected`, `summarized`, `failed`, `stopped_unavailable`), and `Register(s *schedule.Scheduler)`. - **Configuration**: Added `llm.backfill.schedule` (default `*/15 * * * *`) and `llm.backfill.limit` (default `50`) with startup validation. Updated `config.yaml` with commented-out section. - **Tests**: Comprehensive unit and integration tests covering error classification, attempt preservation, backfill single-request on 404, advisory locking, ordering/limit, and config defaults/validation. Co-Authored-By: Paperclip <noreply@paperclip.ing>
feat(newsroom): implement summary backfill job and error classification (task 3.4)
All checks were successful
Validate / validate (pull_request) Successful in 2m1s
26f7bbc8d0
- Classify OpenAI-compatible gateway errors into unavailable (ErrLLMUnavailable wrapping transport errors, timeouts, HTTP 404, 408, 429, 502, 503, 504) vs entry-specific errors
- Ensure ErrLLMUnavailable preserves pending state without incrementing enrichment_attempts
- Add ListPendingSummaries storage query across feeds (oldest first, limit-bounded) and narrow ListPendingEnrichment to skip summary-only entries
- Implement Backfiller job with bounded passes, early stop on first unavailable error, advisory locking, in-process skip, and scheduler registration
- Add llm.backfill.schedule and llm.backfill.limit config schema with validation and defaults
- Tick task 3.4 in OpenSpec change newsroom

Co-Authored-By: Paperclip <noreply@paperclip.ing>
paperclip left a comment

Verdict: FAIL: two major defects (one config bug, one spec drift). Everything else is close.

Success condition tested: newsroom/tasks/23-summary-backfill.md plus the newsroom-enrichment, newsroom-ingestion and newsroom-config spec deltas:

  • errors classified per the table
  • an unavailable error keeps the entry pending and does not count an attempt
  • a backfill job on llm.backfill.schedule that selects entries missing only their summary (body stored, *or provider extract: false), oldest first, bounded by llm.backfill.limit, stops at the first unavailable error, and is advisory-locked
  • the ingest sweep leaves summary-only entries alone
  • bazel test //newsroom/... passes

Reviewed head 26f7bbc, one commit on top of main (0947f6e).

Steps run

git worktree add --detach /tmp/rv249 origin/paperclip/ICB-134-summary-backfill
cd /tmp/rv249 && bazel test //newsroom/...
  -> Executed 12 out of 12 tests: 12 tests pass.   (exit 0)
# throwaway repro tests, not committed: newsroom/internal/enrich/qa_repro_test.go
go test ./newsroom/internal/enrich/ -run 'TestQA_' -v

Findings (most severe first)

1. MAJOR: llm.backfill.limit from config is ignored (internal/enrich/backfill.go, NewBackfiller / resolveLimit)
NewBackfiller sets b.limit = DefaultBackfillLimit (50). Because of that, resolveLimit() always returns at its first branch and never reads cfg.LLM.Backfill.Limit. Only the WithBackfillLimit option changes the bound, and the existing limit test only uses that option.

  • Expected: with config limit: 2 and 5 candidates, 2 entries are selected.
  • Actual:
=== RUN   TestQA_ConfigLimitHonoured
INFO summary backfill run completed selected=5 summarized=5 failed=0 stopped_unavailable=false
    qa_repro_test.go:43: config limit=2 -> selected=5 summarized=5 gatewayCalls=5
--- FAIL: TestQA_ConfigLimitHonoured
  • Fix: leave b.limit zero by default so the config fallback is reached, or resolve the limit from config in the constructor. Add a test that sets the limit only through config.Config.

2. MAJOR (spec drift): entries from extract: false providers are never backfilled, and the ingest sweep still sends them to the LLM

  • Task 23 §3 selects entries where "the extraction step is complete (body stored, or the provider has extract: false)".
  • newsroom-ingestion says the ingest sweep "skips it and leaves it to the summary backfill job, so the two never call the LLM for the same entry".
  • For an extract: false, summarize: true provider, extracted_content stays NULL (Enricher.extractBody returns entry.Content without storing anything). Then:
    • ListPendingSummaries requires extracted_content IS NOT NULL AND != '', so the job never selects these entries.
    • The narrowed ListPendingEnrichment selects extracted_content IS NULL, so the ingest sweep keeps picking them up and calls the LLM once per entry on every ingest. It never stops at the first unavailable error.
  • Expected: after an outage these entries get their summary from the backfill job, and the ingest sweep skips them.
  • Actual: the backfill job never picks them up. They are only retried by the ingest sweep, when the feed is next ingested.
  • Fix (one option): let the selection query also return pending, summary-NULL entries with an empty body, then decide per feed with ResolveSettings (keep them only when !Extract && Summarize). Mirror that in the sweep (skip when !Extract). Add a backfill test for an extract: false provider.

3. MINOR: summarization logic is copied instead of reused (processEntry / recordFailure in backfill.go)
The task says "call only the summarization step of the enricher". The backfiller re-implements it with the package defaults DefaultMaxAttempts and DefaultMaxContentLength. The Enricher honours WithMaxAttempts and WithMaxContentLength, so the two paths will diverge as soon as either option is used. Prefer exposing the enricher's summarize step, for example a SummarizeOnly(ctx, entry, settings), and calling that.

4. MINOR: a summarize: false entry stays pending forever and takes up a limit slot on every run
This only happens after a provider is switched to summarize: false while it still has pending entries. Once there are at least limit such entries at the head of the oldest-first order, the job starves.

run 1: selected=1 state=pending
run 2: selected=1 state=pending
run 3: selected=1 state=pending
--- FAIL: TestQA_SummarizeFalseEntryStaysPending

Suggest marking such entries done, which matches Enricher.ensureExistingDone.

5. NIT

  • fakeBackfillStore.ListPendingSummaries iterates a map, so the unit test can't check "oldest first". The Postgres integration test TestListPendingSummaries does cover it, so this is fine.
  • DefaultBackfillSchedule is defined in both config and enrich. Keep one.
  • The panic("Unreachable: ...") guards in the integration test are noise next to assert.Equal on the length.

What's good

  • The error table is implemented in one place, the client: transport errors with ctx.Err()==nil, plus 404/408/429/502/503/504, are wrapped in ErrLLMUnavailable. Other statuses and malformed responses stay entry failures.
  • recordFailure does not count unavailable errors, and there is a 3×unavailable test.
  • The job stops at the first unavailable error: one gateway request with 5 candidates.
  • In-process skip-if-running, plus a fixed advisory lock ID (summary-backfill, documented).
  • One structured log line per run.
  • Config validation names llm.backfill.schedule / llm.backfill.limit; the defaults are tested.
  • The advisory-lock refactor in lock.go is in scope and keeps behaviour the same.
  • No gateway product names; no secrets in the diff.

Not covered

  • Wiring: cmd/newsroom/main.go doesn't construct the scheduler (3.3) or the backfiller, so nothing runs Register in the service yet. The 3.3 scheduler isn't wired in main.go either, so I'm not blocking 3.4 on this. The spec scenario "service starts and backfills every 15 min" is not true end-to-end until it is wired.
  • I didn't run the service against a real gateway or a real model outage.
  • I didn't measure coverage percentage (acceptance asks >70%).
  • I didn't test two replicas against one Postgres; the advisory lock is covered only by a fake.
**Verdict: FAIL**: two major defects (one config bug, one spec drift). Everything else is close. **Success condition tested:** `newsroom/tasks/23-summary-backfill.md` plus the `newsroom-enrichment`, `newsroom-ingestion` and `newsroom-config` spec deltas: - errors classified per the table - an unavailable error keeps the entry `pending` and does not count an attempt - a backfill job on `llm.backfill.schedule` that selects entries missing only their summary (body stored, *or provider `extract: false`), oldest first, bounded by `llm.backfill.limit`, stops at the first unavailable error, and is advisory-locked - the ingest sweep leaves summary-only entries alone - `bazel test //newsroom/...` passes Reviewed head `26f7bbc`, one commit on top of `main` (`0947f6e`). ### Steps run ``` git worktree add --detach /tmp/rv249 origin/paperclip/ICB-134-summary-backfill cd /tmp/rv249 && bazel test //newsroom/... -> Executed 12 out of 12 tests: 12 tests pass. (exit 0) # throwaway repro tests, not committed: newsroom/internal/enrich/qa_repro_test.go go test ./newsroom/internal/enrich/ -run 'TestQA_' -v ``` ### Findings (most severe first) **1. MAJOR: `llm.backfill.limit` from config is ignored** (`internal/enrich/backfill.go`, `NewBackfiller` / `resolveLimit`) `NewBackfiller` sets `b.limit = DefaultBackfillLimit` (50). Because of that, `resolveLimit()` always returns at its first branch and never reads `cfg.LLM.Backfill.Limit`. Only the `WithBackfillLimit` option changes the bound, and the existing limit test only uses that option. - Expected: with config `limit: 2` and 5 candidates, 2 entries are selected. - Actual: ``` === RUN TestQA_ConfigLimitHonoured INFO summary backfill run completed selected=5 summarized=5 failed=0 stopped_unavailable=false qa_repro_test.go:43: config limit=2 -> selected=5 summarized=5 gatewayCalls=5 --- FAIL: TestQA_ConfigLimitHonoured ``` - Fix: leave `b.limit` zero by default so the config fallback is reached, or resolve the limit from config in the constructor. Add a test that sets the limit only through `config.Config`. **2. MAJOR (spec drift): entries from `extract: false` providers are never backfilled, and the ingest sweep still sends them to the LLM** - Task 23 §3 selects entries where "the extraction step is complete (body stored, **or the provider has `extract: false`**)". - `newsroom-ingestion` says the ingest sweep "skips it and leaves it to the summary backfill job, so the two never call the LLM for the same entry". - For an `extract: false, summarize: true` provider, `extracted_content` stays NULL (`Enricher.extractBody` returns `entry.Content` without storing anything). Then: - `ListPendingSummaries` requires `extracted_content IS NOT NULL AND != ''`, so the job never selects these entries. - The narrowed `ListPendingEnrichment` selects `extracted_content IS NULL`, so the ingest sweep keeps picking them up and calls the LLM once per entry on every ingest. It never stops at the first unavailable error. - Expected: after an outage these entries get their summary from the backfill job, and the ingest sweep skips them. - Actual: the backfill job never picks them up. They are only retried by the ingest sweep, when the feed is next ingested. - Fix (one option): let the selection query also return pending, summary-NULL entries with an empty body, then decide per feed with `ResolveSettings` (keep them only when `!Extract && Summarize`). Mirror that in the sweep (skip when `!Extract`). Add a backfill test for an `extract: false` provider. **3. MINOR: summarization logic is copied instead of reused** (`processEntry` / `recordFailure` in `backfill.go`) The task says "call only the summarization step of the enricher". The backfiller re-implements it with the package defaults `DefaultMaxAttempts` and `DefaultMaxContentLength`. The `Enricher` honours `WithMaxAttempts` and `WithMaxContentLength`, so the two paths will diverge as soon as either option is used. Prefer exposing the enricher's summarize step, for example a `SummarizeOnly(ctx, entry, settings)`, and calling that. **4. MINOR: a `summarize: false` entry stays `pending` forever and takes up a limit slot on every run** This only happens after a provider is switched to `summarize: false` while it still has pending entries. Once there are at least `limit` such entries at the head of the oldest-first order, the job starves. ``` run 1: selected=1 state=pending run 2: selected=1 state=pending run 3: selected=1 state=pending --- FAIL: TestQA_SummarizeFalseEntryStaysPending ``` Suggest marking such entries `done`, which matches `Enricher.ensureExistingDone`. **5. NIT** - `fakeBackfillStore.ListPendingSummaries` iterates a map, so the unit test can't check "oldest first". The Postgres integration test `TestListPendingSummaries` does cover it, so this is fine. - `DefaultBackfillSchedule` is defined in both `config` and `enrich`. Keep one. - The `panic("Unreachable: ...")` guards in the integration test are noise next to `assert.Equal` on the length. ### What's good - The error table is implemented in one place, the client: transport errors with `ctx.Err()==nil`, plus 404/408/429/502/503/504, are wrapped in `ErrLLMUnavailable`. Other statuses and malformed responses stay entry failures. - `recordFailure` does not count unavailable errors, and there is a 3×unavailable test. - The job stops at the first unavailable error: one gateway request with 5 candidates. - In-process skip-if-running, plus a fixed advisory lock ID (`summary-backfill`, documented). - One structured log line per run. - Config validation names `llm.backfill.schedule` / `llm.backfill.limit`; the defaults are tested. - The advisory-lock refactor in `lock.go` is in scope and keeps behaviour the same. - No gateway product names; no secrets in the diff. ### Not covered - **Wiring**: `cmd/newsroom/main.go` doesn't construct the scheduler (3.3) or the backfiller, so nothing runs `Register` in the service yet. The 3.3 scheduler isn't wired in `main.go` either, so I'm not blocking 3.4 on this. The spec scenario "service starts and backfills every 15 min" is not true end-to-end until it is wired. - I didn't run the service against a real gateway or a real model outage. - I didn't measure coverage percentage (acceptance asks >70%). - I didn't test two replicas against one Postgres; the advisory lock is covered only by a fake.
fix(newsroom): address review feedback for summary backfill job
All checks were successful
Validate / validate (pull_request) Successful in 2m1s
63152a2d28
- Honour llm.backfill.limit from config when option is not set
- Support extract: false providers in summary backfill and skip them in ingest sweep
- Reuse Enricher.SummarizeOnly in Backfiller instead of duplicate summarization logic
- Transition summarize: false entries to done to avoid starvation
- Deduplicate DefaultBackfillSchedule and remove panic guards in tests

Co-Authored-By: Paperclip <noreply@paperclip.ing>
Author
Contributor

Pushed fixes addressing review findings to paperclip/ICB-134-summary-backfill (head 63152a2):

  1. llm.backfill.limit from config honoured: NewBackfiller defaults limit to 0 so resolveLimit() reads cfg.LLM.Backfill.Limit. Added TestBackfiller_ConfigLimitHonoured setting limit via config.Config only.
  2. extract: false providers supported & skipped in sweep: ListPendingSummaries selects pending unsummarized entries without requiring extracted_content. Ingest sweep skips feeds with !Extract and sweepOnePending skips entries when !Extract && Summarize. Added TestBackfiller_ExtractFalseProvider and TestIngest_ExtractFalseProvider_SummaryOnlyEntrySkippedInSweep.
  3. Reused Enricher summarization: Added SummarizeOnly(ctx, entry, settings) to Enricher (reusing WithMaxAttempts, WithMaxContentLength, error classification, and failure recording), and Backfiller invokes it via SummarizerStep interface.
  4. Transition summarize: false entries to done: SummarizeOnly marks them done via ensureExistingDone to prevent queue starvation. Verified in TestBackfiller_SkipProviderWithSummarizeFalse_MarksDone.
  5. Nits: Deduplicated DefaultBackfillSchedule into config package; removed panic guards in integration test; bazel test //newsroom/... (12/12) and bazel run //:format all green.
Pushed fixes addressing review findings to `paperclip/ICB-134-summary-backfill` (head `63152a2`): 1. **`llm.backfill.limit` from config honoured**: `NewBackfiller` defaults limit to 0 so `resolveLimit()` reads `cfg.LLM.Backfill.Limit`. Added `TestBackfiller_ConfigLimitHonoured` setting limit via `config.Config` only. 2. **`extract: false` providers supported & skipped in sweep**: `ListPendingSummaries` selects pending unsummarized entries without requiring `extracted_content`. Ingest sweep skips feeds with `!Extract` and `sweepOnePending` skips entries when `!Extract && Summarize`. Added `TestBackfiller_ExtractFalseProvider` and `TestIngest_ExtractFalseProvider_SummaryOnlyEntrySkippedInSweep`. 3. **Reused Enricher summarization**: Added `SummarizeOnly(ctx, entry, settings)` to `Enricher` (reusing `WithMaxAttempts`, `WithMaxContentLength`, error classification, and failure recording), and Backfiller invokes it via `SummarizerStep` interface. 4. **Transition `summarize: false` entries to `done`**: `SummarizeOnly` marks them `done` via `ensureExistingDone` to prevent queue starvation. Verified in `TestBackfiller_SkipProviderWithSummarizeFalse_MarksDone`. 5. **Nits**: Deduplicated `DefaultBackfillSchedule` into `config` package; removed `panic` guards in integration test; `bazel test //newsroom/...` (12/12) and `bazel run //:format` all green.
Author
Contributor

QA re-review (round 2): FAIL, head 63152a2

Verdict: FAIL. Round-1 findings 1 and 3 are fixed. The fix for finding 2 introduced a data-correctness regression, and one of the PR's tests is flaky, which makes bazel test //newsroom/... red on my run.

Correction to my round-1 finding 2 (my error)

extract: false + summarize: true is rejected at startup (internal/config/provider.go:157, ErrExtractDisabled; covered by the config test "summarize without extract"). So the "summary-only entry from an extract: false provider" that task 23 §3 mentions can't exist in a validated config. Round-1 finding 2 was unreachable, and asking for it caused the widened query below. Sorry for the churn.

Success condition tested: task 23 §3/§4. Select pending, unfiltered, unsummarized entries whose extraction step is complete. Skip summarize: false. Never call the extractor. Stop at the first unavailability. The sweep leaves summary-only entries alone. bazel test //newsroom/... is green.

Steps run

git diff 26f7bbc8 63152a2d                                          # round-2 delta
go test ./newsroom/internal/enrich/ -run TestQA_R2 -v               # 2 throwaway repro tests (not committed, deleted)
bazel test //newsroom/...                                           # 11/12 pass, enrich_test FAILED
go test ./newsroom/internal/enrich/ -run TestBackfiller_EntryFailureIncrementsAttemptsAndContinues -count=200   # 41/200 fail
go test ./newsroom/... ; go vet ./newsroom/internal/{enrich,ingest}/   # pass (lucky ordering) / clean

Findings (by severity)

1. MAJOR: the backfill finalizes entries whose extraction is still pending (regression).
ListPendingSummaries dropped extracted_content IS NOT NULL AND != '', so it now also returns entries that are waiting for an extraction retry. For an extract: true (default), summarize: false provider, processEntry → SummarizeOnly → ensureExistingDone marks such an entry done with no body. The ingest sweep then never retries extraction.

  • Repro: feed /g/nosum (summarize: false), entry {state: pending, attempts: 1, extracted_content: nil}, then Backfiller.Run.
  • Expected: entry stays pending for the ingest sweep.
  • Actual: stats={Selected:1 Summarized:0 Failed:0} state=done extracted_nil=true.

2. MAJOR: entries awaiting extraction take limit slots and starve the backfill. This has the same root cause as finding 1. processEntry silently skips Extract && no body entries, but they still count against llm.backfill.limit, and they sort oldest-first. During an extractor outage, or with any backlog of older extraction retries, the job summarizes nothing.

  • Repro: limit 2, summarize: true, entries 1 and 2 pending without a body, entry 3 pending with a body (ordered by ID, mirroring ORDER BY first_seen_at).
  • Expected: entry 3 summarized.
  • Actual: stats={Selected:2 Summarized:0 Failed:0} gatewayCalls=0.

Fix for 1+2: restore the body predicate in ListPendingSummaries. The extract: false branch is unreachable, see the correction above. Keep summarize: false → done for entries with a body (round-1 #4 still holds), and add tests for both repros. The !settings.Extract early-return in runPendingSweep and the extra sweep condition can go back to the round-1 shape too. They're harmless, but they handle an impossible config.

3. MAJOR (CI honesty): TestBackfiller_EntryFailureIncrementsAttemptsAndContinues is flaky. fakeBackfillStore.ListPendingSummaries iterates a Go map, so which entry receives the 400 is random. The test asserts that entry 1 got it.

  • bazel test //newsroom/... → Executed 11 out of 12 tests: 11 tests pass and 1 fails locally., backfill_test.go:536: Expected values to be equal: -1 +0.
  • -count=200 → 41 failures.

Fix: sort the fake's result (by ID or a FirstSeenAt) so it mirrors the SQL ORDER BY. This also makes the limit tests deterministic.

Verified fixed from round 1

  • #1 config limit: NewBackfiller no longer seeds limit, and TestBackfiller_ConfigLimitHonoured covers the config-only path ✅
  • #3 reuse: the backfiller now calls Enricher.SummarizeOnly, which uses maxAttempts/maxContentLength/recordFailure ✅
  • DefaultBackfillSchedule is deduplicated into config ✅

Not covered

  • Service wiring in main.go: still not constructed, same as 3.3.
  • Real gateway or outage, and real-Postgres advisory lock across replicas.
  • Coverage %.
## QA re-review (round 2): **FAIL**, head `63152a2` **Verdict: FAIL.** Round-1 findings 1 and 3 are fixed. The fix for finding 2 introduced a data-correctness regression, and one of the PR's tests is flaky, which makes `bazel test //newsroom/...` red on my run. ### Correction to my round-1 finding 2 (my error) `extract: false` + `summarize: true` is **rejected at startup** (`internal/config/provider.go:157`, `ErrExtractDisabled`; covered by the config test "summarize without extract"). So the "summary-only entry from an `extract: false` provider" that task 23 §3 mentions can't exist in a validated config. Round-1 finding 2 was unreachable, and asking for it caused the widened query below. Sorry for the churn. **Success condition tested:** task 23 §3/§4. Select pending, unfiltered, unsummarized entries whose extraction step is complete. Skip `summarize: false`. Never call the extractor. Stop at the first unavailability. The sweep leaves summary-only entries alone. `bazel test //newsroom/...` is green. ### Steps run ``` git diff 26f7bbc8 63152a2d # round-2 delta go test ./newsroom/internal/enrich/ -run TestQA_R2 -v # 2 throwaway repro tests (not committed, deleted) bazel test //newsroom/... # 11/12 pass, enrich_test FAILED go test ./newsroom/internal/enrich/ -run TestBackfiller_EntryFailureIncrementsAttemptsAndContinues -count=200 # 41/200 fail go test ./newsroom/... ; go vet ./newsroom/internal/{enrich,ingest}/ # pass (lucky ordering) / clean ``` ### Findings (by severity) **1. MAJOR: the backfill finalizes entries whose extraction is still pending (regression).** `ListPendingSummaries` dropped `extracted_content IS NOT NULL AND != ''`, so it now also returns entries that are waiting for an extraction retry. For an `extract: true` (default), `summarize: false` provider, `processEntry` → `SummarizeOnly` → `ensureExistingDone` marks such an entry `done` with **no body**. The ingest sweep then never retries extraction. - Repro: feed `/g/nosum` (`summarize: false`), entry `{state: pending, attempts: 1, extracted_content: nil}`, then `Backfiller.Run`. - Expected: entry stays `pending` for the ingest sweep. - Actual: `stats={Selected:1 Summarized:0 Failed:0} state=done extracted_nil=true`. **2. MAJOR: entries awaiting extraction take limit slots and starve the backfill.** This has the same root cause as finding 1. `processEntry` silently skips `Extract && no body` entries, but they still count against `llm.backfill.limit`, and they sort oldest-first. During an extractor outage, or with any backlog of older extraction retries, the job summarizes nothing. - Repro: limit 2, `summarize: true`, entries 1 and 2 pending without a body, entry 3 pending with a body (ordered by ID, mirroring `ORDER BY first_seen_at`). - Expected: entry 3 summarized. - Actual: `stats={Selected:2 Summarized:0 Failed:0} gatewayCalls=0`. **Fix for 1+2:** restore the body predicate in `ListPendingSummaries`. The `extract: false` branch is unreachable, see the correction above. Keep `summarize: false` → `done` for entries **with** a body (round-1 #4 still holds), and add tests for both repros. The `!settings.Extract` early-return in `runPendingSweep` and the extra sweep condition can go back to the round-1 shape too. They're harmless, but they handle an impossible config. **3. MAJOR (CI honesty): `TestBackfiller_EntryFailureIncrementsAttemptsAndContinues` is flaky.** `fakeBackfillStore.ListPendingSummaries` iterates a Go map, so which entry receives the 400 is random. The test asserts that entry 1 got it. - `bazel test //newsroom/...` → `Executed 11 out of 12 tests: 11 tests pass and 1 fails locally.`, `backfill_test.go:536: Expected values to be equal: -1 +0`. - `-count=200` → 41 failures. Fix: sort the fake's result (by `ID` or a `FirstSeenAt`) so it mirrors the SQL `ORDER BY`. This also makes the limit tests deterministic. ### Verified fixed from round 1 - #1 config limit: `NewBackfiller` no longer seeds `limit`, and `TestBackfiller_ConfigLimitHonoured` covers the config-only path ✅ - #3 reuse: the backfiller now calls `Enricher.SummarizeOnly`, which uses `maxAttempts`/`maxContentLength`/`recordFailure` ✅ - `DefaultBackfillSchedule` is deduplicated into `config` ✅ ### Not covered - Service wiring in `main.go`: still not constructed, same as 3.3. - Real gateway or outage, and real-Postgres advisory lock across replicas. - Coverage %.
fix(newsroom): require extracted body in summary backfill and sort fake store
All checks were successful
Validate / validate (pull_request) Successful in 2m6s
a870fd21e7
- Restore extracted_content IS NOT NULL AND != '' predicate in ListPendingSummaries
- Defensively guard processEntry against entries lacking extracted body
- Add TestBackfiller_EntriesWaitingForExtractionAreNotSelectedOrStarved verifying unextracted entries are not prematurely completed or starve the backfill queue
- Sort fakeBackfillStore candidates by FirstSeenAt and ID to mirror SQL ORDER BY and eliminate flakiness in TestBackfiller_EntryFailureIncrementsAttemptsAndContinues
- Update storage integration test assertions for ListPendingSummaries
- Clean up unreachable extract: false provider test in ingest_test

Co-Authored-By: Paperclip <noreply@paperclip.ing>
Author
Contributor

QA re-review, round 3: PASS (head a870fd2)

Success condition: task 23 / 3.4 as specified, and the three round-2 findings fixed: (1) the backfill must not mark no-body summarize: false entries done, (2) no-body entries must not use up limit slots, (3) no flaky test / green CI.

Steps run (clean clone, PR branch at a870fd2)

Command Result
bazel test //newsroom/... Executed 11 out of 12 tests: 12 tests pass. EXIT=0
go test ./newsroom/internal/enrich/ -run TestBackfiller -count=200 ok (round 2 had 41/200 failures)
go test ./newsroom/internal/{enrich,ingest,config}/ -cover enrich 88.5%, ingest 90.5%, config 89.6%
go vet ./newsroom/... clean
Throwaway test TestQA_R3_SecondRunAndMixedNoSum (deleted, not committed) PASS, see below

Round-2 findings

  1. Fixed. The query again requires extracted_content IS NOT NULL AND != ''. The regenerated sqlc matches (write_dbgen_test passes). processEntry also guards against an empty body before the summarize: false branch.
  2. Fixed. TestBackfiller_EntriesWaitingForExtractionAreNotSelectedOrStarved: with limit 2 and two older no-body entries, the newest bodied entry is still summarized.
  3. Fixed. The fake store now filters and sorts the same way the SQL does, and 200/200 runs are stable.

Throwaway repro (failure path and second run): one summarize: false provider with a no-body entry and a bodied entry, and a gateway that returns 500. Ran twice:
run1={Selected:1 Summarized:0} run2={Selected:0} calls=0 e1=pending e2=done. The no-body entry stays pending for extraction retry, the bodied one goes done with no LLM call, and the second run is a no-op.

Earlier-confirmed scope (round 1, unchanged since): unavailable vs entry-specific error classification, stop at first unavailability, advisory lock, schedule/limit config with validation, ingest sweep leaves bodied summary-only entries to the job.

Nit (non-blocking)

  • ListPendingSummaries orders only by first_seen_at, while the fake also tie-breaks on id. This is harmless, but ORDER BY first_seen_at, id would make the real query deterministic too.

Not covered

  • Real-Postgres check of the query change: storage_integration_test is tags=manual and skips without Docker (--- SKIP: TestListPendingSummaries). The SQL is a 2-line predicate and write_dbgen_test confirms the generated code is in sync, but I did not run it against a real DB. I also did not test multi-replica locking against real Postgres.
  • main.go wiring: the scheduler and backfiller are not constructed yet, same as 3.3, so the job does not run in the service.
  • Real gateway or real outage.

Not merged; the final review and merge are the owner's.

## QA re-review, round 3: **PASS** (head `a870fd2`) **Success condition:** task 23 / 3.4 as specified, and the three round-2 findings fixed: (1) the backfill must not mark no-body `summarize: false` entries `done`, (2) no-body entries must not use up limit slots, (3) no flaky test / green CI. ### Steps run (clean clone, PR branch at `a870fd2`) | Command | Result | |---|---| | `bazel test //newsroom/...` | `Executed 11 out of 12 tests: 12 tests pass.` EXIT=0 | | `go test ./newsroom/internal/enrich/ -run TestBackfiller -count=200` | `ok` (round 2 had 41/200 failures) | | `go test ./newsroom/internal/{enrich,ingest,config}/ -cover` | enrich 88.5%, ingest 90.5%, config 89.6% | | `go vet ./newsroom/...` | clean | | Throwaway test `TestQA_R3_SecondRunAndMixedNoSum` (deleted, not committed) | PASS, see below | ### Round-2 findings 1. **Fixed.** The query again requires `extracted_content IS NOT NULL AND != ''`. The regenerated sqlc matches (`write_dbgen_test` passes). `processEntry` also guards against an empty body before the `summarize: false` branch. 2. **Fixed.** `TestBackfiller_EntriesWaitingForExtractionAreNotSelectedOrStarved`: with limit 2 and two older no-body entries, the newest bodied entry is still summarized. 3. **Fixed.** The fake store now filters and sorts the same way the SQL does, and 200/200 runs are stable. **Throwaway repro (failure path and second run):** one `summarize: false` provider with a no-body entry and a bodied entry, and a gateway that returns 500. Ran twice: `run1={Selected:1 Summarized:0} run2={Selected:0} calls=0 e1=pending e2=done`. The no-body entry stays `pending` for extraction retry, the bodied one goes `done` with no LLM call, and the second run is a no-op. **Earlier-confirmed scope** (round 1, unchanged since): unavailable vs entry-specific error classification, stop at first unavailability, advisory lock, schedule/limit config with validation, ingest sweep leaves bodied summary-only entries to the job. ### Nit (non-blocking) - `ListPendingSummaries` orders only by `first_seen_at`, while the fake also tie-breaks on `id`. This is harmless, but `ORDER BY first_seen_at, id` would make the real query deterministic too. ### Not covered - **Real-Postgres check of the query change:** `storage_integration_test` is `tags=manual` and skips without Docker (`--- SKIP: TestListPendingSummaries`). The SQL is a 2-line predicate and `write_dbgen_test` confirms the generated code is in sync, but I did not run it against a real DB. I also did not test multi-replica locking against real Postgres. - **`main.go` wiring:** the scheduler and backfiller are not constructed yet, same as 3.3, so the job does not run in the service. - **Real gateway or real outage.** Not merged; the final review and merge are the owner's.
test(newsroom): add nilaway guards to storage integration test
All checks were successful
Validate / validate (pull_request) Successful in 2m2s
77bf4a1ca4
Co-Authored-By: Paperclip <noreply@paperclip.ing>
test: run newsroom integration tests by default, prefer asserts over nilaway guards
All checks were successful
Validate / validate (pull_request) Successful in 5m0s
f8ec602382
- Drop the `manual` tag from the newsroom ingest and storage integration
  tests so `//...` runs them; the remote executor already provides
  DOCKER_HOST via `test:remote`.
- Bump go.uber.org/nilaway and exclude `_test.go` files from it in
  .nogo.json. The analyzer's own exclude_files replaces `_base`, so the
  external/generated-code exclusions are repeated there.
- Remove the "Unreachable" panic guards that only existed for nilaway and
  replace guards that were the sole check with assert.Error,
  assert.NotZero or assert.True. Guards on values that flow into
  production receivers stay: nilaway reports those at the production
  dereference regardless of the test-file exclusion.
- Retry builds when the remote executor's CAS evicted an input
  (--experimental_remote_cache_eviction_retries=5).
- Document the assertions-over-guards rule in AGENTS.md and fix the stale
  "integration tests are manual" notes in AGENTS.md and the justfile.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
prskr approved these changes 2026-10-13 09:51:40 +00:00
prskr merged commit 7c8cd93d48 into main 2026-10-13 10:38:51 +00:00
prskr deleted branch paperclip/ICB-134-summary-backfill 2026-10-13 10:38:52 +00:00
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
prskr/tools!249
No description provided.