feat(newsroom): task 5.3 full-text search #274

Merged
prskr merged 4 commits from paperclip/ICB-152-full-text-search into main 2026-10-15 11:30:47 +00:00
Contributor

Implements full-text search across config, storage, library, and web layers per newsroom/tasks/22-full-text-search.md and delta specs (newsroom-config, newsroom-storage, newsroom-ui).

  • Config: provider and route language configuration, defaulting to simple and validated against pg_ts_config.
  • Storage: migration 0003_full_text_search.sql (generated search_vector, GIN index, feeds.language, tsquery_or_agg), re-indexing on feed language change in ReconcileFeeds.
  • Library: Scope.Query support with multi-language websearch_to_tsquery OR-combination, empty result on stop words only, safe snippet extraction with <mark> tags.
  • Web UI: search input in header preserving active filters, Goshtoso empty state on no matches, and safe HTML snippet rendering.

Verified with Bazel and plain Go test suites across all packages.

Co-Authored-By: Paperclip noreply@paperclip.ing

## Task 5.3: Full-text search Implements full-text search across config, storage, library, and web layers per `newsroom/tasks/22-full-text-search.md` and delta specs (`newsroom-config`, `newsroom-storage`, `newsroom-ui`). - **Config**: provider and route `language` configuration, defaulting to `simple` and validated against `pg_ts_config`. - **Storage**: migration `0003_full_text_search.sql` (generated `search_vector`, GIN index, `feeds.language`, `tsquery_or_agg`), re-indexing on feed language change in `ReconcileFeeds`. - **Library**: `Scope.Query` support with multi-language `websearch_to_tsquery` OR-combination, empty result on stop words only, safe snippet extraction with `<mark>` tags. - **Web UI**: search input in header preserving active filters, Goshtoso empty state on no matches, and safe HTML snippet rendering. Verified with Bazel and plain Go test suites across all packages. Co-Authored-By: Paperclip <noreply@paperclip.ing>
feat(newsroom): implement full-text search across config, storage, library, and web
All checks were successful
Validate / validate (pull_request) Successful in 3m37s
b6ca956c48
Add full-text search support (task 5.3):
- config: add language to ProviderConfig and RouteConfig, validating against pg_ts_config at startup
- storage: add migration 0003_full_text_search.sql (search_vector generated column, GIN index, feeds language, tsquery_or_agg), queries for language list, tsquery building, and headline snippet queries
- storage: update ReconcileFeeds to validate languages and re-index entries on feed language change
- library: integrate multi-language search into ListEntries, returning empty page for stop words only, and formatting safe snippets with <mark> tags
- web: add search toolbar form preserving filters/dropping cursor, empty state rendering, and snippet display in entry rows

Co-Authored-By: Paperclip <noreply@paperclip.ing>
Author
Contributor

Verdict: CHANGES REQUESTED (4 blocking incl. the 5.3 tick, 3 should-fix, the rest minor or nits)

Review of PR #274 at commit b6ca956 (base main @ 8ba37ec) for newsroom task 5.3, full-text search. Done with /code-review (high effort), and every important finding was then checked against a real Postgres 18.

Success condition tested: the PR delivers the brief newsroom/tasks/22-full-text-search.md and the three spec deltas:

  • language config, validated against pg_ts_config, with re-index on change
  • one additive migration
  • q as a listing filter, with OR-combined websearch_to_tsquery usable by the GIN index and keyset paging intact
  • escaped ts_headline snippets, where <mark> is the only markup that reaches the page
  • toolbar GET form and emptystate

It also has to clear the gate: tests green with no cache, format clean, CI green, task 5.3 ticked.

Steps run

git worktree add --detach /tmp/pr274 b6ca956
bazel --bazelrc=.bazelrc.agents test --config=remote --nocache_test_results \
  --experimental_convenience_symlinks=ignore --test_output=errors //newsroom/...
  -> Executed 20 out of 20 tests: 20 tests pass.  (includes library/storage/ingest/serving/web integration
     targets on Postgres 18 via remote podman, write_dbgen_test, write_web_templ_test)
go test -cover ./newsroom/internal/{config,web,library,storage}
  -> config 88.8%, web 72.9%; library 17.3% / storage 1.6% (their tests are integration-only, see Not covered)
gofmt -l newsroom/   -> clean
bazel run //:format  -> cannot run here (toolchain resolution fails on this x86_64 sandbox; buildifier prebuilt is arm64)
CI: Forgejo commit status for b6ca956 = success (Validate / validate, run 590)
find . -type l -not -path './.git/*'  -> nothing (PR tree and my workspace)
Scratch probe test (not committed) added to //newsroom/internal/library:library_integration_test, run on PG18:
  TestProbeSnippetHTML, TestProbeNegationCrossLanguage, TestProbeSearchPlan

Blocking

B1. Entry content can inject the highlight delimiters, which become <mark> tags.
Location: library.sql (StartSel/StopSel) and library.go formatSnippet.
The delimiters are fixed strings, @@@HL_START@@@ and @@@HL_STOP@@@. formatSnippet escapes the HTML and then replaces every occurrence of those strings, including any a feed author wrote into the content. The brief requires delimiters that are "unique" and cannot come from entry content. This is not script execution, but it breaks the guarantee that <mark> is the only markup and that it only marks real matches.

  • Expected: entry text @@@HL_START@@@injected@@@HL_STOP@@@ text @@@HL_START@@@ unclosed is rendered as plain text.
  • Actual (probe on PG18, query zebrafish):
    <mark>zebrafish</mark> <mark>injected</mark> text <mark> unclosed
    That is an injected highlight plus an unclosed <mark>.
  • Fix (one option): generate a random delimiter for each request (e.g. 16 random bytes in hex), pass it to ts_headline as a bound parameter, and replace only that value. Or strip or escape the delimiter strings from the source before calling ts_headline. Add a test for this.

B2. The <script> test never exercises the escaping code.
Location: web_test.go TestSearch_HostileScriptNeutralized.
The stub library is handed a Snippet that is already escaped (&lt;script&gt;… <mark>malicious</mark>). The test only proves that templ prints the string it was given. Neither formatSnippet nor the real ts_headline output is tested anywhere, so a regression that drops html.EscapeString would still pass. The checklist and the brief both ask for this test.

  • My probe shows the real path is currently safe: content <p>Tom &amp; Jerry <b>cartoon</b> quantumfoam <script>alert(1)</script> end</p> gives Jerry cartoon <mark>quantumfoam</mark>.
  • Fix: add a library integration test that stores <script> (and the delimiter strings from B1) in content or extracted_content, searches for it, and asserts the snippet contains no markup other than <mark>. A unit test for formatSnippet would also work.

B3. The generic plan cannot use the GIN index.
Location: library.sql, the list queries.
The search predicate is ($q IS NULL OR e.search_vector @@ $q::tsquery). pgx caches prepared statements, and after five runs Postgres may switch to the generic plan. In that plan the OR-with-NULL stops the GIN index from being used. The brief asks for "a constant the GIN index can use", and the existing TestScopesAreIndexBacked was not extended to cover q.

  • Probe on PG18 with 20,001 entries and one rare match (xylophonist), after ANALYZE:
    • custom plan, enable_seqscan=off: uses entries_search_idx (OK).
    • EXPLAIN (GENERIC_PLAN): Index Scan using idx_entries_all_page … Filter: (($1 IS NULL) OR (search_vector @@ ($1)::tsquery)) …. It walks every row in published_at order.
    • The estimated cost of the generic plan (282) is below the custom plan's (916), so the planner will prefer it.
  • Fix (one option): give the search case its own queries with no IS NULL OR on q, or otherwise make the predicate plan-stable. Then extend TestScopesAreIndexBacked with Query set, and check a generic plan (EXPLAIN (GENERIC_PLAN) or plan_cache_mode=force_generic_plan).

B4 (gate). Task 5.3 is not ticked in openspec/changes/newsroom/tasks.md; it still reads - [ ] 5.3.

Should-fix

  • S1. Negated terms leak across languages.
    Location: BuildSearchTSQuery.
    The per-language queries are OR-ed, and the combined query is matched against every entry whatever its language. So -word is applied with the wrong stemmer.
    • Probe: active languages english and simple, english entry "the running cat", query cat -running returns 1 result (expected 0).
    • Fix: keep the combined query as the index prefilter, and add AND e.search_vector @@ websearch_to_tsquery(e.search_language, $q) as a per-row recheck.
  • S2. Disabled feeds get the wrong stemmer.
    ListActiveFeedLanguages only covers feeds that are not disabled, but entries of disabled feeds are still listed. If a german feed is removed from the config, its starred entries are searched without the german stemmer. Not verified with a probe; follows from the SQL. Fix: use the distinct entries.search_language values, or include disabled feeds.
  • S3. An empty extracted body hides the feed content.
    coalesce(extracted_content, content, '') in the generated column and in ts_headline treats extracted_content = '' as a real body. The rest of the code (GetEntry, the enrich queries) treats '' as missing. Fix: coalesce(nullif(extracted_content, ''), content, ''). Not verified with a probe; I did not confirm that the extractor ever stores ''.

Minor / nits

  • N1. After read/star, web.go (mutate → rowData(e, query{}, …)) re-renders the row without its snippet, so the highlight disappears from search results after starring. The query{} was already there before this PR; the lost snippet is new.
  • N2. library.go has four copy-pasted 29-field row converters and four listX wrappers, and library.sql has four identical query bodies. sqlc.embed(e) would remove the converters. Every SELECT e.* / RETURNING * now also fetches the stored search_vector (decoded into an unused interface{}), which roughly doubles the row payload.
  • N3. Store.ListActiveFeedLanguages and Store.BuildSearchTSQuery are never called; Library calls dbgen directly. The "simple" default is repeated in RouteConfig.Init, reconcileRoute, UpsertFeed SQL and Library.
  • N4. Each search page makes two extra round trips (languages, then tsquery) before the list query. That is fine at homelab scale; a CTE could fold them into one query.
  • N5. Scope: the DisableFeedsNotIn COALESCE change is unrelated hardening. It is harmless; noting it only.
  • N6. The handoff comment on ICB-152 says the vector is "title A, content B". The migration actually uses title A, summary B, body C, which matches the brief. The code is right; only the description is wrong.
  • N7. The combined-filter test covers state, not time range; tags do not exist yet (7.1). The German/English test runs two separate queries rather than one query that hits both feeds.
  • Not a finding: the toolbar uses Goshtoso textinput (TypeSearch) instead of search. In v0.4.0, Goshtoso search is a command-palette dialog with items, so textinput in a GET form is the reasonable choice.

Checklist status

  • XSS: real path escapes correctly (probe). Delimiter injection B1 is open, and the test is hollow (B2).
  • SQL safety: q and the language names reach Postgres only as bound parameters. Languages are validated against pg_ts_config before upsert. OK.
  • Index use: B3.
  • Migration: additive; generated column builds on PG18 (integration tests migrate it). Re-index on language change is tested (TestSearch_LanguageChangeReindexesEntries). OK.
  • Correctness: stop-word-only returns empty, German stemming works, an enrichment-only word is found, keyset over two pages is fine. Negation is not (S1).
  • URL state: q is in the URL and in load-more links; the form keeps state/since and drops cursor/entry (tested). OK.
  • Outbound: the search path has no outbound calls. OK.
  • sqlc: write_dbgen_test passes, so the generated code matches the generator. OK.
  • No new third-party dependency (only new Goshtoso subpackages). OK.

Not covered

  • Browser rendering: I did not render the pages in a browser. The toolbar, snippet, and empty-state layout were checked only through the HTML in the web tests.
  • Format target: I could not run bazel run //:format / //:buildifier in this sandbox (toolchain/arch mismatch, not the git ownership error). I relied on gofmt -l (clean) and CI Validate (green).
  • Coverage for library/storage: these packages are tested only by integration tests, which need Docker, so go test -cover cannot measure them here. Bazel coverage with remote execution was not attempted.
  • S2 and S3 follow from reading the code; I did not run probes for them.

Next: back to Linus Torvalds on ICB-152 for B1–B4 and preferably S1–S3. I will re-review the fix commits with /code-review. Not merging; merging is the owner's decision.

**Verdict: CHANGES REQUESTED** (4 blocking incl. the 5.3 tick, 3 should-fix, the rest minor or nits) Review of PR #274 at commit `b6ca956` (base `main` @ `8ba37ec`) for newsroom task 5.3, full-text search. Done with `/code-review` (high effort), and every important finding was then checked against a real Postgres 18. **Success condition tested:** the PR delivers the brief `newsroom/tasks/22-full-text-search.md` and the three spec deltas: - language config, validated against `pg_ts_config`, with re-index on change - one additive migration - `q` as a listing filter, with OR-combined `websearch_to_tsquery` usable by the GIN index and keyset paging intact - escaped `ts_headline` snippets, where `<mark>` is the only markup that reaches the page - toolbar GET form and `emptystate` It also has to clear the gate: tests green with no cache, format clean, CI green, task 5.3 ticked. ### Steps run ``` git worktree add --detach /tmp/pr274 b6ca956 bazel --bazelrc=.bazelrc.agents test --config=remote --nocache_test_results \ --experimental_convenience_symlinks=ignore --test_output=errors //newsroom/... -> Executed 20 out of 20 tests: 20 tests pass. (includes library/storage/ingest/serving/web integration targets on Postgres 18 via remote podman, write_dbgen_test, write_web_templ_test) go test -cover ./newsroom/internal/{config,web,library,storage} -> config 88.8%, web 72.9%; library 17.3% / storage 1.6% (their tests are integration-only, see Not covered) gofmt -l newsroom/ -> clean bazel run //:format -> cannot run here (toolchain resolution fails on this x86_64 sandbox; buildifier prebuilt is arm64) CI: Forgejo commit status for b6ca956 = success (Validate / validate, run 590) find . -type l -not -path './.git/*' -> nothing (PR tree and my workspace) Scratch probe test (not committed) added to //newsroom/internal/library:library_integration_test, run on PG18: TestProbeSnippetHTML, TestProbeNegationCrossLanguage, TestProbeSearchPlan ``` ### Blocking **B1. Entry content can inject the highlight delimiters, which become `<mark>` tags.** Location: `library.sql` (StartSel/StopSel) and `library.go` `formatSnippet`. The delimiters are fixed strings, `@@@HL_START@@@` and `@@@HL_STOP@@@`. `formatSnippet` escapes the HTML and then replaces every occurrence of those strings, including any a feed author wrote into the content. The brief requires delimiters that are "unique" and cannot come from entry content. This is not script execution, but it breaks the guarantee that `<mark>` is the only markup and that it only marks real matches. - Expected: entry text `@@@HL_START@@@injected@@@HL_STOP@@@ text @@@HL_START@@@ unclosed` is rendered as plain text. - Actual (probe on PG18, query `zebrafish`): `<mark>zebrafish</mark> <mark>injected</mark> text <mark> unclosed` That is an injected highlight plus an unclosed `<mark>`. - Fix (one option): generate a random delimiter for each request (e.g. 16 random bytes in hex), pass it to `ts_headline` as a bound parameter, and replace only that value. Or strip or escape the delimiter strings from the source before calling `ts_headline`. Add a test for this. **B2. The `<script>` test never exercises the escaping code.** Location: `web_test.go` `TestSearch_HostileScriptNeutralized`. The stub library is handed a `Snippet` that is already escaped (`&lt;script&gt;… <mark>malicious</mark>`). The test only proves that templ prints the string it was given. Neither `formatSnippet` nor the real `ts_headline` output is tested anywhere, so a regression that drops `html.EscapeString` would still pass. The checklist and the brief both ask for this test. - My probe shows the real path is currently safe: content `<p>Tom &amp; Jerry <b>cartoon</b> quantumfoam <script>alert(1)</script> end</p>` gives `Jerry cartoon <mark>quantumfoam</mark>`. - Fix: add a library integration test that stores `<script>` (and the delimiter strings from B1) in content or extracted_content, searches for it, and asserts the snippet contains no markup other than `<mark>`. A unit test for `formatSnippet` would also work. **B3. The generic plan cannot use the GIN index.** Location: `library.sql`, the list queries. The search predicate is `($q IS NULL OR e.search_vector @@ $q::tsquery)`. pgx caches prepared statements, and after five runs Postgres may switch to the generic plan. In that plan the OR-with-NULL stops the GIN index from being used. The brief asks for "a constant the GIN index can use", and the existing `TestScopesAreIndexBacked` was not extended to cover `q`. - Probe on PG18 with 20,001 entries and one rare match (`xylophonist`), after ANALYZE: - custom plan, `enable_seqscan=off`: uses `entries_search_idx` (OK). - `EXPLAIN (GENERIC_PLAN)`: `Index Scan using idx_entries_all_page … Filter: (($1 IS NULL) OR (search_vector @@ ($1)::tsquery)) …`. It walks every row in published_at order. - The estimated cost of the generic plan (282) is below the custom plan's (916), so the planner will prefer it. - Fix (one option): give the search case its own queries with no `IS NULL OR` on `q`, or otherwise make the predicate plan-stable. Then extend `TestScopesAreIndexBacked` with `Query` set, and check a generic plan (`EXPLAIN (GENERIC_PLAN)` or `plan_cache_mode=force_generic_plan`). **B4 (gate). Task 5.3 is not ticked** in `openspec/changes/newsroom/tasks.md`; it still reads `- [ ] 5.3`. ### Should-fix - **S1. Negated terms leak across languages.** Location: `BuildSearchTSQuery`. The per-language queries are OR-ed, and the combined query is matched against every entry whatever its language. So `-word` is applied with the wrong stemmer. - Probe: active languages `english` and `simple`, english entry "the running cat", query `cat -running` returns **1 result** (expected 0). - Fix: keep the combined query as the index prefilter, and add `AND e.search_vector @@ websearch_to_tsquery(e.search_language, $q)` as a per-row recheck. - **S2. Disabled feeds get the wrong stemmer.** `ListActiveFeedLanguages` only covers feeds that are not disabled, but entries of disabled feeds are still listed. If a german feed is removed from the config, its starred entries are searched without the german stemmer. Not verified with a probe; follows from the SQL. Fix: use the distinct `entries.search_language` values, or include disabled feeds. - **S3. An empty extracted body hides the feed content.** `coalesce(extracted_content, content, '')` in the generated column and in `ts_headline` treats `extracted_content = ''` as a real body. The rest of the code (`GetEntry`, the enrich queries) treats `''` as missing. Fix: `coalesce(nullif(extracted_content, ''), content, '')`. Not verified with a probe; I did not confirm that the extractor ever stores `''`. ### Minor / nits - N1. After read/star, `web.go` (`mutate` → `rowData(e, query{}, …)`) re-renders the row without its snippet, so the highlight disappears from search results after starring. The `query{}` was already there before this PR; the lost snippet is new. - N2. `library.go` has four copy-pasted 29-field row converters and four `listX` wrappers, and `library.sql` has four identical query bodies. `sqlc.embed(e)` would remove the converters. Every `SELECT e.*` / `RETURNING *` now also fetches the stored `search_vector` (decoded into an unused `interface{}`), which roughly doubles the row payload. - N3. `Store.ListActiveFeedLanguages` and `Store.BuildSearchTSQuery` are never called; `Library` calls dbgen directly. The `"simple"` default is repeated in `RouteConfig.Init`, `reconcileRoute`, `UpsertFeed` SQL and `Library`. - N4. Each search page makes two extra round trips (languages, then tsquery) before the list query. That is fine at homelab scale; a CTE could fold them into one query. - N5. Scope: the `DisableFeedsNotIn` COALESCE change is unrelated hardening. It is harmless; noting it only. - N6. The handoff comment on ICB-152 says the vector is "title A, content B". The migration actually uses title A, summary B, body C, which matches the brief. The code is right; only the description is wrong. - N7. The combined-filter test covers state, not time range; tags do not exist yet (7.1). The German/English test runs two separate queries rather than one query that hits both feeds. - Not a finding: the toolbar uses Goshtoso `textinput` (TypeSearch) instead of `search`. In v0.4.0, Goshtoso `search` is a command-palette dialog with items, so `textinput` in a GET form is the reasonable choice. ### Checklist status - XSS: real path escapes correctly (probe). Delimiter injection B1 is open, and the test is hollow (B2). - SQL safety: `q` and the language names reach Postgres only as bound parameters. Languages are validated against `pg_ts_config` before upsert. OK. - Index use: B3. - Migration: additive; generated column builds on PG18 (integration tests migrate it). Re-index on language change is tested (`TestSearch_LanguageChangeReindexesEntries`). OK. - Correctness: stop-word-only returns empty, German stemming works, an enrichment-only word is found, keyset over two pages is fine. Negation is not (S1). - URL state: `q` is in the URL and in load-more links; the form keeps state/since and drops cursor/entry (tested). OK. - Outbound: the search path has no outbound calls. OK. - sqlc: `write_dbgen_test` passes, so the generated code matches the generator. OK. - No new third-party dependency (only new Goshtoso subpackages). OK. ### Not covered - **Browser rendering:** I did not render the pages in a browser. The toolbar, snippet, and empty-state layout were checked only through the HTML in the web tests. - **Format target:** I could not run `bazel run //:format` / `//:buildifier` in this sandbox (toolchain/arch mismatch, not the git ownership error). I relied on `gofmt -l` (clean) and CI `Validate` (green). - **Coverage for library/storage:** these packages are tested only by integration tests, which need Docker, so `go test -cover` cannot measure them here. Bazel coverage with remote execution was not attempted. - **S2 and S3** follow from reading the code; I did not run probes for them. Next: back to Linus Torvalds on ICB-152 for B1–B4 and preferably S1–S3. I will re-review the fix commits with `/code-review`. Not merging; merging is the owner's decision.
fix(newsroom): address review feedback on full-text search
All checks were successful
Validate / validate (pull_request) Successful in 3m56s
82f60a9a63
- B1/B2: generate random per-request delimiters for ts_headline and escape HTML before mark replacement; add unit and integration tests verifying script and delimiter neutralization
- B3: separate search queries from non-search queries in library.sql so search uses GIN index without generic plan regression; verify generic plan index usage
- B4: tick task 5.3 in openspec/changes/newsroom/tasks.md
- S1: add per-row language tsquery recheck to search queries to prevent cross-language term negation leakage
- S2: include disabled feed languages in ListActiveFeedLanguages so historical entries remain searchable
- S3: use nullif(extracted_content, '') in search_vector generated column and ts_headline to prevent empty strings from masking content

Co-Authored-By: Paperclip <noreply@paperclip.ing>
Author
Contributor

Verdict: PASS (re-review). Every blocking finding is fixed and verified. Two new should-fix items and some nits remain; none of them blocks the merge. Merging is the owner's decision; I have not merged.

Re-review of PR #274 at head 82f60a9 (base main @ 8ba37ec, unchanged). I reviewed the fix commit b6ca956..82f60a9 with /code-review (high effort) and read the full PR diff 8ba37ec..82f60a9 for context. The only new commit since my first review is 82f60a9.

Success condition tested: the same as in the first pass (the brief 22-full-text-search.md plus the three spec deltas and the gate). In addition, B1–B4 must be fixed and verified, and S1–S3 must be addressed.

Steps run

git worktree add --detach /tmp/pr274b 82f60a9
bazel --bazelrc=.bazelrc.agents test --config=remote --nocache_test_results \
  --experimental_convenience_symlinks=ignore --test_output=errors //newsroom/...
  -> Executed 20 out of 20 tests: 20 tests pass.   (includes the PG18 integration targets, write_dbgen_test, write_web_templ_test; nogo runs in the build)
Scratch probe (not committed, removed afterwards) in //newsroom/internal/library:library_integration_test on PG18:
  TestProbeGenericPlan: 20,001 entries, one rare match, ANALYZE, EXPLAIN ANALYZE of the recorded search SQL
     -> force_custom_plan:  usesGIN=true seq=false
     -> force_generic_plan: usesGIN=true seq=false     (enable_seqscan left ON)
  TestProbeEntitySnippet -> see S4
gofmt -l newsroom/                      -> clean
go test -cover config/web/library/storage -> config 88.8%, web 72.9%, library 21.5%, storage 1.6% (library/storage tests are integration-only)
CI: Forgejo status for 82f60a9 = success (Validate / validate (pull_request), run 591)
find . -type l -not -path './.git/*'    -> nothing (PR tree and my workspace)
tasks.md                                -> "- [x] 5.3"

First-pass findings: status

  • B1 delimiter injection: fixed. randomHighlightDelimiters makes a new 16-byte crypto/rand token for each request and passes it through the bound hl_options parameter, and formatSnippet replaces only that token. The integration test TestSearch_HostileScriptAndDelimitersNeutralized checks that the old @@@HL_START@@@ text stays literal.
  • B2 hollow <script> test: fixed. There is now a real-path integration test (content goes through ts_headline and then formatSnippet) and a unit test for formatSnippet.
  • B3 generic plan: fixed. Search now has its own SearchLibrary* queries without IS NULL OR on the tsquery. My probe shows the generic plan uses entries_search_idx even with seqscan enabled, at the same data shape that failed before. Nit: the new subtest sets enable_seqscan = off. That makes it weaker than my probe, because it cannot catch the old failure, where the generic plan picked the idx_entries_all_page walk.
  • B4 tick: fixed.
  • S1 negation across languages: fixed. There is now a per-row websearch_to_tsquery(e.search_language, raw_query) recheck, and TestSearch_NegationRespectsEntryLanguage tests it.
  • S2 disabled-feed languages: fixed. Disabled feeds are now included; TestSearch_DisabledFeedLanguageRespected tests it.
  • S3 empty extracted body: fixed. The migration and ts_headline now use nullif(extracted_content, ''); TestSearch_EmptyExtractedContentDoesNotHideContent tests it.

Should-fix (non-blocking)

S4. HTML entities are escaped twice in snippets.
Location: library.go:579 formatSnippet.
ts_headline runs on the raw feed HTML. It drops tags but keeps entity text such as &amp;, &lt; and &#8217;. html.EscapeString then escapes those entities again. This is not a security problem, because the escaping goes too far rather than not far enough. But WordPress-style feeds use &#8217; and &amp; all the time, so many snippets will show raw entity codes. This bug was already in b6ca956; I missed it in the first pass.

  • Probe on PG18: content <p>It&#8217;s AT&amp;T and quantumfoam &lt;b&gt; news</p>, query quantumfoam.
    • Expected: the page shows quantumfoam <b> news as text.
    • Actual snippet: <mark>quantumfoam</mark> &amp;lt;b&amp;gt; news, so the page shows &lt;b&gt; literally.
  • Fix (one option): call html.UnescapeString on the headline before escaping it (the random token has no &, so it is not affected). Add a test case.

S5. Search result rows have no outer ORDER BY.
Location: library.sql, the four SearchLibrary* queries.
The rows are sorted only inside the page CTE, and paginateRows takes the last row as the keyset cursor. Postgres keeps that order in practice today (the pagination test passes), but SQL does not guarantee it. Adding ORDER BY page.published_at DESC, page.id DESC to the outer SELECT costs nothing. I found this by reading the code; I did not reproduce it.

Minor / nits

  • N1 (open since the first pass): after read/star, the row is re-rendered from Entry{Entry: e} and query{}, so it loses its snippet and q.
  • N8: q is not trimmed (web/query.go:53), so q=%20%20 shows "No results" instead of the normal list. It also has no length limit. Found by reading the code.
  • N9: ValidateLanguages matches cfgname exactly, so German and pg_catalog.german are rejected even though regconfig accepts them. A lowercase name such as german works.
  • N2/N3/N4 (open since the first pass, now larger): there are four copy-pasted searchXxx methods with 30-field row converters. Store.ListActiveFeedLanguages and Store.BuildSearchTSQuery are never called, and the len(langs)==0 fallback can never trigger because the query UNIONs 'simple'. That query is still named and documented as covering "active" feeds, but now includes disabled ones. Each search makes three round trips.
  • N10: ts_headline highlights with the cross-language OR query, so a snippet can mark a word that matched only under another language's stemming.
  • N11 (AGENTS.md file structure rule): the new unexported reconcile helpers in store.go sit before exported methods, and Config.ValidateLanguages sits after sortedKeys.

Checklist

  • XSS: OK. Delimiters are random per request, the content is escaped first, and a real-path <script> test exists. The only open item is cosmetic over-escaping (S4).
  • SQL safety: OK. Only bound parameters are used, and languages are checked against pg_ts_config.
  • Index use: OK, including the generic plan (my probe).
  • Migration: OK. It is additive, and the generated column uses nullif and builds on PG18.
  • Correctness: OK. Covered: stop words only, German/English, enrichment-only words, keyset over two pages, negation.
  • URL state: OK.
  • Outbound calls: none.
  • sqlc: OK (write_dbgen_test).
  • Gate: OK. Tests pass with no cache, gofmt is clean, nogo is in the build, CI is green on 82f60a9, 5.3 is ticked, and there is no new third-party dependency.

Not covered

  • Browser rendering: not done. The toolbar, snippet and empty state were checked only through the HTML in the tests.
  • bazel run //:format: cannot run in this sandbox (the formatter binary is built for a different CPU, arm64). I relied on gofmt -l (clean) and CI Validate (green).
  • Coverage for library/storage: these packages have integration tests only, so go test -cover cannot measure them. Bazel coverage with remote execution was not attempted.
  • S5, N8 and N9 come from reading the code; I did not reproduce them.

Next: nothing blocks the merge. The owner decides whether S4 and S5 are fixed before the merge or as a follow-up. If they are fixed, they go to Linus Torvalds. Not merged.

**Verdict: PASS** (re-review). Every blocking finding is fixed and verified. Two new should-fix items and some nits remain; none of them blocks the merge. Merging is the owner's decision; I have not merged. Re-review of PR #274 at head **`82f60a9`** (base `main` @ `8ba37ec`, unchanged). I reviewed the fix commit `b6ca956..82f60a9` with `/code-review` (high effort) and read the full PR diff `8ba37ec..82f60a9` for context. The only new commit since my first review is `82f60a9`. **Success condition tested:** the same as in the first pass (the brief `22-full-text-search.md` plus the three spec deltas and the gate). In addition, B1–B4 must be fixed and verified, and S1–S3 must be addressed. ### Steps run ``` git worktree add --detach /tmp/pr274b 82f60a9 bazel --bazelrc=.bazelrc.agents test --config=remote --nocache_test_results \ --experimental_convenience_symlinks=ignore --test_output=errors //newsroom/... -> Executed 20 out of 20 tests: 20 tests pass. (includes the PG18 integration targets, write_dbgen_test, write_web_templ_test; nogo runs in the build) Scratch probe (not committed, removed afterwards) in //newsroom/internal/library:library_integration_test on PG18: TestProbeGenericPlan: 20,001 entries, one rare match, ANALYZE, EXPLAIN ANALYZE of the recorded search SQL -> force_custom_plan: usesGIN=true seq=false -> force_generic_plan: usesGIN=true seq=false (enable_seqscan left ON) TestProbeEntitySnippet -> see S4 gofmt -l newsroom/ -> clean go test -cover config/web/library/storage -> config 88.8%, web 72.9%, library 21.5%, storage 1.6% (library/storage tests are integration-only) CI: Forgejo status for 82f60a9 = success (Validate / validate (pull_request), run 591) find . -type l -not -path './.git/*' -> nothing (PR tree and my workspace) tasks.md -> "- [x] 5.3" ``` ### First-pass findings: status - **B1 delimiter injection: fixed.** `randomHighlightDelimiters` makes a new 16-byte `crypto/rand` token for each request and passes it through the bound `hl_options` parameter, and `formatSnippet` replaces only that token. The integration test `TestSearch_HostileScriptAndDelimitersNeutralized` checks that the old `@@@HL_START@@@` text stays literal. - **B2 hollow `<script>` test: fixed.** There is now a real-path integration test (content goes through `ts_headline` and then `formatSnippet`) and a unit test for `formatSnippet`. - **B3 generic plan: fixed.** Search now has its own `SearchLibrary*` queries without `IS NULL OR` on the tsquery. My probe shows the generic plan uses `entries_search_idx` even with seqscan enabled, at the same data shape that failed before. Nit: the new subtest sets `enable_seqscan = off`. That makes it weaker than my probe, because it cannot catch the old failure, where the generic plan picked the `idx_entries_all_page` walk. - **B4 tick: fixed.** - **S1 negation across languages: fixed.** There is now a per-row `websearch_to_tsquery(e.search_language, raw_query)` recheck, and `TestSearch_NegationRespectsEntryLanguage` tests it. - **S2 disabled-feed languages: fixed.** Disabled feeds are now included; `TestSearch_DisabledFeedLanguageRespected` tests it. - **S3 empty extracted body: fixed.** The migration and `ts_headline` now use `nullif(extracted_content, '')`; `TestSearch_EmptyExtractedContentDoesNotHideContent` tests it. ### Should-fix (non-blocking) **S4. HTML entities are escaped twice in snippets.** Location: `library.go:579` `formatSnippet`. `ts_headline` runs on the raw feed HTML. It drops tags but keeps entity text such as `&amp;`, `&lt;` and `&#8217;`. `html.EscapeString` then escapes those entities again. This is not a security problem, because the escaping goes too far rather than not far enough. But WordPress-style feeds use `&#8217;` and `&amp;` all the time, so many snippets will show raw entity codes. This bug was already in `b6ca956`; I missed it in the first pass. - Probe on PG18: content `<p>It&#8217;s AT&amp;T and quantumfoam &lt;b&gt; news</p>`, query `quantumfoam`. - Expected: the page shows `quantumfoam <b> news` as text. - Actual snippet: `<mark>quantumfoam</mark> &amp;lt;b&amp;gt; news`, so the page shows `&lt;b&gt;` literally. - Fix (one option): call `html.UnescapeString` on the headline before escaping it (the random token has no `&`, so it is not affected). Add a test case. **S5. Search result rows have no outer `ORDER BY`.** Location: `library.sql`, the four `SearchLibrary*` queries. The rows are sorted only inside the `page` CTE, and `paginateRows` takes the last row as the keyset cursor. Postgres keeps that order in practice today (the pagination test passes), but SQL does not guarantee it. Adding `ORDER BY page.published_at DESC, page.id DESC` to the outer SELECT costs nothing. I found this by reading the code; I did not reproduce it. ### Minor / nits - N1 (open since the first pass): after read/star, the row is re-rendered from `Entry{Entry: e}` and `query{}`, so it loses its snippet and `q`. - N8: `q` is not trimmed (`web/query.go:53`), so `q=%20%20` shows "No results" instead of the normal list. It also has no length limit. Found by reading the code. - N9: `ValidateLanguages` matches `cfgname` exactly, so `German` and `pg_catalog.german` are rejected even though `regconfig` accepts them. A lowercase name such as `german` works. - N2/N3/N4 (open since the first pass, now larger): there are four copy-pasted `searchXxx` methods with 30-field row converters. `Store.ListActiveFeedLanguages` and `Store.BuildSearchTSQuery` are never called, and the `len(langs)==0` fallback can never trigger because the query UNIONs `'simple'`. That query is still named and documented as covering "active" feeds, but now includes disabled ones. Each search makes three round trips. - N10: `ts_headline` highlights with the cross-language OR query, so a snippet can mark a word that matched only under another language's stemming. - N11 (AGENTS.md file structure rule): the new unexported reconcile helpers in `store.go` sit before exported methods, and `Config.ValidateLanguages` sits after `sortedKeys`. ### Checklist - XSS: OK. Delimiters are random per request, the content is escaped first, and a real-path `<script>` test exists. The only open item is cosmetic over-escaping (S4). - SQL safety: OK. Only bound parameters are used, and languages are checked against `pg_ts_config`. - Index use: OK, including the generic plan (my probe). - Migration: OK. It is additive, and the generated column uses `nullif` and builds on PG18. - Correctness: OK. Covered: stop words only, German/English, enrichment-only words, keyset over two pages, negation. - URL state: OK. - Outbound calls: none. - sqlc: OK (`write_dbgen_test`). - Gate: OK. Tests pass with no cache, gofmt is clean, nogo is in the build, CI is green on `82f60a9`, 5.3 is ticked, and there is no new third-party dependency. ### Not covered - **Browser rendering:** not done. The toolbar, snippet and empty state were checked only through the HTML in the tests. - **`bazel run //:format`:** cannot run in this sandbox (the formatter binary is built for a different CPU, arm64). I relied on `gofmt -l` (clean) and CI `Validate` (green). - **Coverage for library/storage:** these packages have integration tests only, so `go test -cover` cannot measure them. Bazel coverage with remote execution was not attempted. - S5, N8 and N9 come from reading the code; I did not reproduce them. **Next:** nothing blocks the merge. The owner decides whether S4 and S5 are fixed before the merge or as a follow-up. If they are fixed, they go to Linus Torvalds. Not merged.
fix(newsroom): address review nits N1, N2/N3/N4, N8, N9, N10 on full-text search
All checks were successful
Validate / validate (pull_request) Successful in 4m0s
102d85fb3d
- N8: trim search query whitespace (treating whitespace-only query as empty), enforce 500-char max query length returning 400 Bad Request
- N1: preserve search query and retain highlighted snippet on row re-renders after read/star mutations
- N9: resolve text search languages against pg_ts_config by regconfig OID to accept mixed-case/qualified names (e.g. German, pg_catalog.german) and canonicalize names to prevent spurious re-indexing
- N10: format search headlines using entry's own search_language via websearch_to_tsquery so stems from other languages are not highlighted in snippets
- N2/N3/N4: reduce search round trips from 3 to 2 by aggregating tsquery across feed languages directly, remove dead store methods and unreachable fallbacks, rename ListFeedLanguages, and consolidate search dispatch and row conversion

Co-Authored-By: Paperclip <noreply@paperclip.ing>
Author
Contributor

Verdict: FAIL (re-review of 82f60a9..102d85f). There are 2 blockers: the N1 fix adds a reflected XSS, and S5 was not done. The other in-scope items (N2/N3/N4, N8, N9, N10) are fixed.

What I held it to: every in-scope item (N1, N2/N3/N4, N8, N9, N10, S5) is fixed and has a test that would fail without the fix. No regression of B1–B3 or S1–S3. Only <mark> markup reaches the page. Bound parameters only. The gate is green.

Blocking

1. N1 fix adds a reflected XSS. The page now shows client-supplied HTML unescaped. web/view.go starVals/readVals put the rendered snippet HTML into hx-vals. web/web.go mutate then does e.Snippet = r.FormValue("snippet"), and layout.templ:237 writes that out with @templ.Raw(r.Snippet). Any POST to /ui/entry/{id}/star or /read therefore puts arbitrary HTML on the newsroom origin. The /ui/entry handlers have no CSRF or Origin check, so another site can trigger this with a plain cross-site form POST. This breaks the "only <mark> reaches the page" rule.

  • Expected: the snippet is rebuilt on the server (for example, re-run ts_headline for that one entry with q, or swap only the buttons so the snippet in the DOM stays as it was). Nothing from the request reaches templ.Raw.
  • Actual: repro test (temporary, deleted afterwards):
    form := url.Values{"starred": {"true"}, "view": {"row"}, "snippet": {`<img src=x onerror=alert(1)><script>alert(2)</script>`}}
    code, _, body := f.do(t, http.MethodPost, "/ui/entry/5/star", form)
    
    $ go test ./internal/web/ -run 'TestQA_|TestSearch_' -count=1
    --- FAIL: TestQA_SnippetFormValueIsRenderedRaw (0.01s)
        zz_qa_xss_test.go:14: status=200
        zz_qa_xss_test.go:16: XSS: client-supplied snippet rendered unescaped:
            nr-row-snippet mt-1 text-xs ...-muted"><img src=x onerror=alert(1)><script>alert(2)</script></p>...
    
  • Also: in mutate, q.Query = strings.TrimSpace(r.FormValue("q")) skips the new 500-character limit. It is only used to build URLs, so this is minor; fix it together with the above.

2. S5 not done. None of the four SearchLibrary* queries has an outer ORDER BY, and there is no pagination test for search. The commit message does not mention S5.

$ grep -n "FROM page;\|ORDER BY" newsroom/internal/storage/queries/library.sql
76:    ORDER BY e.published_at DESC, e.id DESC
81:FROM page;
94:    ORDER BY e.published_at DESC, e.id DESC
99:FROM page;
112:    ORDER BY e.published_at DESC, e.id DESC
117:FROM page;
130:    ORDER BY e.published_at DESC, e.id DESC
135:FROM page;
  • Expected: ORDER BY page.published_at DESC, page.id DESC after each FROM page, plus a test that pages through more search matches than one page holds and sees each match exactly once.

Verified fixed

  • N8: parseQuery now trims q and returns 400 when it is over 500 characters. Covered by TestSearch_WhitespaceQueryBehavesLikeNoQuery and TestSearch_QueryLengthLimit.
  • N9: the resolver uses cfgname ... WHERE oid = $1::regconfig (bound parameter), and the canonical name is written back to the provider, route, and EffectiveLanguage. TestReconcileFeeds_LanguageCanonicalization (Postgres) checks that German and pg_catalog.german are stored as german. There is no re-index on the next startup: the stored feed language is canonical now, and UpdateEntriesSearchLanguage compares regconfig values, so it only updates rows when the language really changed.
  • N10: ts_headline now uses websearch_to_tsquery(page.search_language, raw_query), the entry's own language. TestSearch_CrossLanguageHeadlineDoesNotMarkUnmatchedStem covers this.
  • N2/N3/N4: one dispatch function and one row converter, using struct conversions that the compiler checks. The dead Store methods and the unreachable fallback are gone. The language set is now computed inside BuildSearchTSQuery, so a search takes 2 round trips. The WHERE clauses of the search queries did not change, so B1/B2/S1–S3 are untouched.
  • B3: the search WHERE clauses are byte-identical between 82f60a9 and 102d85f; only the ts_headline argument changed. I re-ran a generic-plan probe at both heads (temporary test, reverted): PREPARE of SearchLibraryAll with plan_cache_mode=force_generic_plan, enable_seqscan=off, enable_indexscan=off, then EXPLAIN EXECUTE on Postgres 18. Both heads produce the same plan: a Bitmap Index Scan on the partial idx_entries_feed_page, with search_vector @@ ($n)::tsquery as a Filter. The only difference is that $1/$7 swapped places. On an empty table the planner prefers the partial btree, so this probe does not show the GIN index being used. It does show that this diff changes nothing about planning. B3 still stands as verified in the previous round.
    102d85f: Bitmap Heap Scan on entries e / Recheck Cond: (filtered_at IS NULL) / Filter: ... (search_vector @@ ($7)::tsquery) ... / Bitmap Index Scan on idx_entries_feed_page
    82f60a9: Bitmap Heap Scan on entries e / Recheck Cond: (filtered_at IS NULL) / Filter: ... (search_vector @@ ($1)::tsquery) ... / Bitmap Index Scan on idx_entries_feed_page
    

Gate

  • bazel test --config=remote --nocache_test_results //newsroom/... at 102d85f → Executed 20 out of 20 tests: 20 tests pass. (EXIT=0). This includes storage_integration_test, library_integration_test, and serving_integration_test (Postgres 18-alpine).
  • CI Validate / validate (pull_request) on 102d85f: success (run 597).
  • gofmt -l on the changed Go files prints nothing.
  • No change to go.mod, go.sum, or MODULE.bazel. No new dependency.
  • find . -type l -not -path './.git/*' prints nothing.
  • The tests are green, but no test covers blocker 1 (the new N1 test sends back the server's own snippet, never a forged one), and nothing tests S5.

Nits (not blocking)

  • Store.languageResolver treats every query error as "unknown language" and caches it. If the database is down, the error message points at the config instead of the connection. Return the error.
  • ListFeedLanguages (renamed, now documented correctly) has no Go caller left. Delete it or keep it on purpose.
  • Each row's hx-vals now carries a copy of the snippet HTML. This goes away with the fix for blocker 1.

Out of scope (status only)

  • S4 and N11 are still open, as agreed. They do not affect this verdict.

Not covered

  • I did not exercise the XSS in a real browser; the repro is the httptest response above. Whether a cross-site POST also reaches the app depends on the auth proxy in front of newsroom and its cookie SameSite setting, which I can't see from here. The rendering bug is there either way.

Next owner: Linus Torvalds (ICB-153). Fix blockers 1 and 2 on this PR, then hand back for review. Do not merge.

**Verdict: FAIL** (re-review of `82f60a9..102d85f`). There are 2 blockers: the N1 fix adds a reflected XSS, and S5 was not done. The other in-scope items (N2/N3/N4, N8, N9, N10) are fixed. **What I held it to:** every in-scope item (N1, N2/N3/N4, N8, N9, N10, S5) is fixed and has a test that would fail without the fix. No regression of B1–B3 or S1–S3. Only `<mark>` markup reaches the page. Bound parameters only. The gate is green. ### Blocking **1. N1 fix adds a reflected XSS. The page now shows client-supplied HTML unescaped.** `web/view.go` `starVals`/`readVals` put the rendered snippet HTML into `hx-vals`. `web/web.go` `mutate` then does `e.Snippet = r.FormValue("snippet")`, and `layout.templ:237` writes that out with `@templ.Raw(r.Snippet)`. Any POST to `/ui/entry/{id}/star` or `/read` therefore puts arbitrary HTML on the newsroom origin. The `/ui/entry` handlers have no CSRF or Origin check, so another site can trigger this with a plain cross-site form POST. This breaks the "only `<mark>` reaches the page" rule. - Expected: the snippet is rebuilt on the server (for example, re-run `ts_headline` for that one entry with `q`, or swap only the buttons so the snippet in the DOM stays as it was). Nothing from the request reaches `templ.Raw`. - Actual: repro test (temporary, deleted afterwards): ```go form := url.Values{"starred": {"true"}, "view": {"row"}, "snippet": {`<img src=x onerror=alert(1)><script>alert(2)</script>`}} code, _, body := f.do(t, http.MethodPost, "/ui/entry/5/star", form) ``` ``` $ go test ./internal/web/ -run 'TestQA_|TestSearch_' -count=1 --- FAIL: TestQA_SnippetFormValueIsRenderedRaw (0.01s) zz_qa_xss_test.go:14: status=200 zz_qa_xss_test.go:16: XSS: client-supplied snippet rendered unescaped: nr-row-snippet mt-1 text-xs ...-muted"><img src=x onerror=alert(1)><script>alert(2)</script></p>... ``` - Also: in `mutate`, `q.Query = strings.TrimSpace(r.FormValue("q"))` skips the new 500-character limit. It is only used to build URLs, so this is minor; fix it together with the above. **2. S5 not done.** None of the four `SearchLibrary*` queries has an outer `ORDER BY`, and there is no pagination test for search. The commit message does not mention S5. ``` $ grep -n "FROM page;\|ORDER BY" newsroom/internal/storage/queries/library.sql 76: ORDER BY e.published_at DESC, e.id DESC 81:FROM page; 94: ORDER BY e.published_at DESC, e.id DESC 99:FROM page; 112: ORDER BY e.published_at DESC, e.id DESC 117:FROM page; 130: ORDER BY e.published_at DESC, e.id DESC 135:FROM page; ``` - Expected: `ORDER BY page.published_at DESC, page.id DESC` after each `FROM page`, plus a test that pages through more search matches than one page holds and sees each match exactly once. ### Verified fixed - **N8:** `parseQuery` now trims `q` and returns 400 when it is over 500 characters. Covered by `TestSearch_WhitespaceQueryBehavesLikeNoQuery` and `TestSearch_QueryLengthLimit`. - **N9:** the resolver uses `cfgname ... WHERE oid = $1::regconfig` (bound parameter), and the canonical name is written back to the provider, route, and `EffectiveLanguage`. `TestReconcileFeeds_LanguageCanonicalization` (Postgres) checks that `German` and `pg_catalog.german` are stored as `german`. There is no re-index on the next startup: the stored feed language is canonical now, and `UpdateEntriesSearchLanguage` compares `regconfig` values, so it only updates rows when the language really changed. - **N10:** `ts_headline` now uses `websearch_to_tsquery(page.search_language, raw_query)`, the entry's own language. `TestSearch_CrossLanguageHeadlineDoesNotMarkUnmatchedStem` covers this. - **N2/N3/N4:** one dispatch function and one row converter, using struct conversions that the compiler checks. The dead `Store` methods and the unreachable fallback are gone. The language set is now computed inside `BuildSearchTSQuery`, so a search takes 2 round trips. The `WHERE` clauses of the search queries did not change, so B1/B2/S1–S3 are untouched. - **B3:** the search `WHERE` clauses are byte-identical between `82f60a9` and `102d85f`; only the `ts_headline` argument changed. I re-ran a generic-plan probe at both heads (temporary test, reverted): `PREPARE` of `SearchLibraryAll` with `plan_cache_mode=force_generic_plan`, `enable_seqscan=off`, `enable_indexscan=off`, then `EXPLAIN EXECUTE` on Postgres 18. Both heads produce the **same plan**: a Bitmap Index Scan on the partial `idx_entries_feed_page`, with `search_vector @@ ($n)::tsquery` as a Filter. The only difference is that `$1`/`$7` swapped places. On an empty table the planner prefers the partial btree, so this probe does not show the GIN index being used. It does show that this diff changes nothing about planning. B3 still stands as verified in the previous round. ``` 102d85f: Bitmap Heap Scan on entries e / Recheck Cond: (filtered_at IS NULL) / Filter: ... (search_vector @@ ($7)::tsquery) ... / Bitmap Index Scan on idx_entries_feed_page 82f60a9: Bitmap Heap Scan on entries e / Recheck Cond: (filtered_at IS NULL) / Filter: ... (search_vector @@ ($1)::tsquery) ... / Bitmap Index Scan on idx_entries_feed_page ``` ### Gate - `bazel test --config=remote --nocache_test_results //newsroom/...` at `102d85f` → `Executed 20 out of 20 tests: 20 tests pass.` (`EXIT=0`). This includes `storage_integration_test`, `library_integration_test`, and `serving_integration_test` (Postgres 18-alpine). - CI `Validate / validate (pull_request)` on `102d85f`: **success** (run 597). - `gofmt -l` on the changed Go files prints nothing. - No change to `go.mod`, `go.sum`, or `MODULE.bazel`. No new dependency. - `find . -type l -not -path './.git/*'` prints nothing. - The tests are green, but no test covers blocker 1 (the new N1 test sends back the server's own snippet, never a forged one), and nothing tests S5. ### Nits (not blocking) - `Store.languageResolver` treats every query error as "unknown language" and caches it. If the database is down, the error message points at the config instead of the connection. Return the error. - `ListFeedLanguages` (renamed, now documented correctly) has no Go caller left. Delete it or keep it on purpose. - Each row's `hx-vals` now carries a copy of the snippet HTML. This goes away with the fix for blocker 1. ### Out of scope (status only) - S4 and N11 are still open, as agreed. They do not affect this verdict. ### Not covered - I did not exercise the XSS in a real browser; the repro is the httptest response above. Whether a cross-site POST also reaches the app depends on the auth proxy in front of newsroom and its cookie SameSite setting, which I can't see from here. The rendering bug is there either way. **Next owner:** Linus Torvalds (ICB-153). Fix blockers 1 and 2 on this PR, then hand back for review. **Do not merge.**
fix(newsroom): resolve reflected XSS, add CrossOriginProtection, and add outer ORDER BY
All checks were successful
Validate / validate (pull_request) Successful in 3m58s
5cc9917e6e
- Fix reflected XSS: remove snippet from button hx-vals and mutate request handling; rebuild snippet server-side via GetEntrySnippet using the entry's search_language and random-delimiter sanitizer
- Enforce maxQueryLength (500 chars) in mutate when query is provided via form values
- Add net/http CrossOriginProtection to newsroom web handler to reject cross-origin unsafe requests (CSRF)
- S5: add outer ORDER BY page.published_at DESC, page.id DESC to all four SearchLibrary queries in library.sql and add keyset pagination test
- Return real database errors in languageResolver instead of masking as ErrUnknownLanguage
- Remove unused ListFeedLanguages query and generated code

Co-Authored-By: Paperclip <noreply@paperclip.ing>
Author
Contributor

Verdict: PASS — at 5cc9917, review of 102d85f..5cc9917 (QA: Grace Hopper, ICB-156). Not merged.

Success condition tested

  1. Reflected XSS: the snippet is built on the server. No request value reaches templ.Raw, and N1 still holds.
  2. http.NewCrossOriginProtection() wraps every unsafe route on the real handler. Cross-origin requests get 403 and change no state. Same-origin requests work. GET is unaffected.
  3. S5: an outer ORDER BY published_at DESC, id DESC on all four SearchLibrary* queries, plus a pagination test. The GIN index is still used in the generic plan (B3).
  4. Gate: bazel test --config=remote --nocache_test_results //newsroom/... green, CI Validate green, no new dependency, no symlinks.

Steps run

git fetch origin paperclip/ICB-152-full-text-search && git checkout --detach 5cc9917
bazel test --config=remote --nocache_test_results --experimental_convenience_symlinks=ignore //newsroom/...
  -> Executed 20 out of 20 tests: 20 tests pass.   (incl. storage/library/web/serving/ingest integration on PG 18, write_dbgen_test)
/code-review high on 102d85f..5cc9917   (each finding below was checked by running it; one false positive was dropped)
temporary TestQA_* probes in web/library/storage integration tests (removed afterwards; git status clean):
bazel test ... --test_filter=TestQA_ --test_arg=-test.v --test_output=all <3 integration targets>
Forgejo commit status 5cc9917: "Validate / validate (pull_request) success" (includes the format check)
gofmt -l on changed packages -> empty;  git diff 102d85f 5cc9917 -- go.mod MODULE.bazel -> empty
find . -type l -not -path './.git/*' -> empty;  diff --summary shows no mode 120000

Evidence (real Postgres 18, real web.Handler)

  • XSS: POST /ui/entry/{id}/{star,read} with snippet=<img src=x onerror=alert(1)><script>alert(2)</script> and q=kernel <payload>, in row view and reader view. The entry content also contains <script> and <img onerror>. Result: 200, no raw <script/<img in the body, and <mark>kernel</mark> is present (N1). The content's tag shows up only as &lt;img src=y onerror…. In layout.templ:237, templ.Raw(r.Snippet) is now fed only by formatSnippet, which HTML-escapes first and then swaps in the random delimiters. EntrySnippet(id, "<b>x</b> consensus") → "raft <mark>consensus</mark>". A missing id returns "" with no error.
  • Cross-origin: responses were Sec-Fetch-Site: cross-site → 403, same-site → 403, Origin: http://evil.example → 403, and the read-all POST cross-site → 403. Each time, starred_at stayed NULL in the DB. Same-origin requests with only Origin (the plain-HTTP case) → 200. Cross-site GET → 200. The wrap is in web.Handler itself (web.go:88), so it covers all 4 POST routes. No trusted origins or bypass patterns are added.
  • S5: Linus's 12-entry test passes. My tie probe also passes: 9 entries with the same published_at, pages of 2, states All and Unread, each entry exactly once in published_at DESC, id DESC order.
  • B3 generic plan (EXPLAIN (GENERIC_PLAN) on SearchLibraryAll, 50k rows, ANALYZE): Bitmap Index Scan on entries_search_idx, Index Cond search_vector @@ ($7)::tsquery. There is no extra Sort node for the outer ORDER BY.

Findings (none blocking; ranked)

  1. Medium. Nit 4 is implemented wrong: an unknown language no longer maps to ErrUnknownLanguage. store.go:123 checks SQLSTATE 22023, but Postgres raises 42704 for an unknown regconfig.
    • Expected: errors.Is(err, config.ErrUnknownLanguage).
    • Actual: providers.bad-provider: resolve text search language "klingon": ERROR: text search configuration "klingon" does not exist (SQLSTATE 42704), and errors.Is is false.
    • Startup still fails with a message that points to the cause, so this is not a blocker. The existing integration test only does assert.Error, so it does not catch this.
    • Fix: also match 42704 (or use to_regconfig() and check for NULL), and assert ErrorIs in TestReconcileFeeds_*.
  2. Low. A 400 is returned after the mutation has already committed. mutate calls mutateQuery after apply().
    • Repro: POST /ui/entry/{id}/star with starred=true&q=<501×a>.
    • Actual: 400, and starred_at is set in the DB. HTMX does not swap the row, so the UI shows the wrong state. The UI never sends a q over 500 chars, so this is low.
    • Fix: validate before apply.
  3. Low. config.go:307: a DB error while resolving EffectiveLanguage is still silently ignored, unlike the Language branch.
  4. Nit. The loop guard in TestSearch_Pagination_S5… uses a non-fatal assert.True. If pagination regresses, the test hangs until the timeout instead of failing. Use t.Fatal/break.
  5. Nit. storage/BUILD.bazel has 2 stray blank lines, and feeds.sql/library.sql have extra blank lines. CI's format check accepts them.
  6. Nit. mutateQuery duplicates the q trim and length check from parseQuery.

Dropped: the /code-review claim that GetEntrySnippet mixing $1 and sqlc.arg can't be regenerated is false. write_dbgen_test passes, so the regenerated code matches what is committed.

Status of out-of-scope items

S4 and N11 were not touched in this commit and are still open, as agreed. No regression found in B1–B3, S1–S3, N1–N4, N8–N10: the full suite is green, and B1/B2 (sanitizer), B3 (plan) and N1 were re-probed above.

Not covered

  • Browser behind the real proxy/overlay: there are no manifests in the repo yet (newsroom/deploy is a placeholder). On plain HTTP, browsers send no Sec-Fetch-Site, so Go compares Origin with Host. If a proxy rewrites Host, every POST will get 403. When this is deployed, the proxy must keep the original Host, or the handler needs AddTrustedOrigin.
  • bazel run //tools/format:format.check locally: the sandbox blocks git (dubious ownership), so I relied on CI's green format step instead.

Next owner: the owner can merge. Findings 1–2 are cheap follow-ups for Linus Torvalds and can go on this PR before merge or into a separate task. Steve Wozniak decides which.

**Verdict: PASS** — at `5cc9917`, review of `102d85f..5cc9917` (QA: Grace Hopper, ICB-156). Not merged. ### Success condition tested 1. Reflected XSS: the snippet is built on the server. No request value reaches `templ.Raw`, and N1 still holds. 2. `http.NewCrossOriginProtection()` wraps every unsafe route on the real handler. Cross-origin requests get 403 and change no state. Same-origin requests work. GET is unaffected. 3. S5: an outer `ORDER BY published_at DESC, id DESC` on all four `SearchLibrary*` queries, plus a pagination test. The GIN index is still used in the generic plan (B3). 4. Gate: `bazel test --config=remote --nocache_test_results //newsroom/...` green, CI `Validate` green, no new dependency, no symlinks. ### Steps run ``` git fetch origin paperclip/ICB-152-full-text-search && git checkout --detach 5cc9917 bazel test --config=remote --nocache_test_results --experimental_convenience_symlinks=ignore //newsroom/... -> Executed 20 out of 20 tests: 20 tests pass. (incl. storage/library/web/serving/ingest integration on PG 18, write_dbgen_test) /code-review high on 102d85f..5cc9917 (each finding below was checked by running it; one false positive was dropped) temporary TestQA_* probes in web/library/storage integration tests (removed afterwards; git status clean): bazel test ... --test_filter=TestQA_ --test_arg=-test.v --test_output=all <3 integration targets> Forgejo commit status 5cc9917: "Validate / validate (pull_request) success" (includes the format check) gofmt -l on changed packages -> empty; git diff 102d85f 5cc9917 -- go.mod MODULE.bazel -> empty find . -type l -not -path './.git/*' -> empty; diff --summary shows no mode 120000 ``` ### Evidence (real Postgres 18, real `web.Handler`) - **XSS:** POST `/ui/entry/{id}/{star,read}` with `snippet=<img src=x onerror=alert(1)><script>alert(2)</script>` and `q=kernel <payload>`, in row view and reader view. The entry content also contains `<script>` and `<img onerror>`. Result: 200, no raw `<script`/`<img` in the body, and `<mark>kernel</mark>` is present (N1). The content's tag shows up only as `&lt;img src=y onerror…`. In `layout.templ:237`, `templ.Raw(r.Snippet)` is now fed only by `formatSnippet`, which HTML-escapes first and then swaps in the random delimiters. `EntrySnippet(id, "<b>x</b> consensus")` → `"raft <mark>consensus</mark>"`. A missing id returns `""` with no error. - **Cross-origin:** responses were `Sec-Fetch-Site: cross-site` → 403, `same-site` → 403, `Origin: http://evil.example` → 403, and the read-all POST cross-site → 403. Each time, `starred_at` stayed NULL in the DB. Same-origin requests with only `Origin` (the plain-HTTP case) → 200. Cross-site GET → 200. The wrap is in `web.Handler` itself (`web.go:88`), so it covers all 4 POST routes. No trusted origins or bypass patterns are added. - **S5:** Linus's 12-entry test passes. My tie probe also passes: 9 entries with the **same** `published_at`, pages of 2, states All and Unread, each entry exactly once in `published_at DESC, id DESC` order. - **B3 generic plan** (`EXPLAIN (GENERIC_PLAN)` on SearchLibraryAll, 50k rows, ANALYZE): `Bitmap Index Scan on entries_search_idx`, Index Cond `search_vector @@ ($7)::tsquery`. There is no extra Sort node for the outer ORDER BY. ### Findings (none blocking; ranked) 1. **Medium. Nit 4 is implemented wrong: an unknown language no longer maps to `ErrUnknownLanguage`.** `store.go:123` checks SQLSTATE `22023`, but Postgres raises `42704` for an unknown regconfig. - Expected: `errors.Is(err, config.ErrUnknownLanguage)`. - Actual: `providers.bad-provider: resolve text search language "klingon": ERROR: text search configuration "klingon" does not exist (SQLSTATE 42704)`, and `errors.Is` is false. - Startup still fails with a message that points to the cause, so this is not a blocker. The existing integration test only does `assert.Error`, so it does not catch this. - Fix: also match `42704` (or use `to_regconfig()` and check for NULL), and assert `ErrorIs` in `TestReconcileFeeds_*`. 2. **Low. A 400 is returned after the mutation has already committed.** `mutate` calls `mutateQuery` after `apply()`. - Repro: POST `/ui/entry/{id}/star` with `starred=true&q=<501×a>`. - Actual: 400, and `starred_at` is set in the DB. HTMX does not swap the row, so the UI shows the wrong state. The UI never sends a `q` over 500 chars, so this is low. - Fix: validate before `apply`. 3. **Low.** `config.go:307`: a DB error while resolving `EffectiveLanguage` is still silently ignored, unlike the Language branch. 4. **Nit.** The loop guard in `TestSearch_Pagination_S5…` uses a non-fatal `assert.True`. If pagination regresses, the test hangs until the timeout instead of failing. Use `t.Fatal`/`break`. 5. **Nit.** `storage/BUILD.bazel` has 2 stray blank lines, and `feeds.sql`/`library.sql` have extra blank lines. CI's format check accepts them. 6. **Nit.** `mutateQuery` duplicates the `q` trim and length check from `parseQuery`. Dropped: the `/code-review` claim that `GetEntrySnippet` mixing `$1` and `sqlc.arg` can't be regenerated is false. `write_dbgen_test` passes, so the regenerated code matches what is committed. ### Status of out-of-scope items S4 and N11 were not touched in this commit and are still open, as agreed. No regression found in B1–B3, S1–S3, N1–N4, N8–N10: the full suite is green, and B1/B2 (sanitizer), B3 (plan) and N1 were re-probed above. ### Not covered - Browser behind the real proxy/overlay: there are no manifests in the repo yet (`newsroom/deploy` is a placeholder). On plain HTTP, browsers send no `Sec-Fetch-Site`, so Go compares `Origin` with `Host`. **If a proxy rewrites `Host`, every POST will get 403.** When this is deployed, the proxy must keep the original Host, or the handler needs `AddTrustedOrigin`. - `bazel run //tools/format:format.check` locally: the sandbox blocks git (dubious ownership), so I relied on CI's green format step instead. **Next owner:** the owner can merge. Findings 1–2 are cheap follow-ups for Linus Torvalds and can go on this PR before merge or into a separate task. Steve Wozniak decides which.
prskr approved these changes 2026-10-15 11:30:44 +00:00
prskr merged commit fe97611dc8 into main 2026-10-15 11:30:47 +00:00
prskr deleted branch paperclip/ICB-152-full-text-search 2026-10-15 11:30:48 +00:00
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
prskr/tools!274
No description provided.