feat(newsroom): task 5.3 full-text search #274
No reviewers
Labels
No milestone
No project
No assignees
2 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
prskr/tools!274
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "paperclip/ICB-152-full-text-search"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Task 5.3: Full-text search
Implements full-text search across config, storage, library, and web layers per
newsroom/tasks/22-full-text-search.mdand delta specs (newsroom-config,newsroom-storage,newsroom-ui).languageconfiguration, defaulting tosimpleand validated againstpg_ts_config.0003_full_text_search.sql(generatedsearch_vector, GIN index,feeds.language,tsquery_or_agg), re-indexing on feed language change inReconcileFeeds.Scope.Querysupport with multi-languagewebsearch_to_tsqueryOR-combination, empty result on stop words only, safe snippet extraction with<mark>tags.Verified with Bazel and plain Go test suites across all packages.
Co-Authored-By: Paperclip noreply@paperclip.ing
Verdict: CHANGES REQUESTED (4 blocking incl. the 5.3 tick, 3 should-fix, the rest minor or nits)
Review of PR #274 at commit
b6ca956(basemain@8ba37ec) for newsroom task 5.3, full-text search. Done with/code-review(high effort), and every important finding was then checked against a real Postgres 18.Success condition tested: the PR delivers the brief
newsroom/tasks/22-full-text-search.mdand the three spec deltas:pg_ts_config, with re-index on changeqas a listing filter, with OR-combinedwebsearch_to_tsqueryusable by the GIN index and keyset paging intactts_headlinesnippets, where<mark>is the only markup that reaches the pageemptystateIt also has to clear the gate: tests green with no cache, format clean, CI green, task 5.3 ticked.
Steps run
Blocking
B1. Entry content can inject the highlight delimiters, which become
<mark>tags.Location:
library.sql(StartSel/StopSel) andlibrary.goformatSnippet.The delimiters are fixed strings,
@@@HL_START@@@and@@@HL_STOP@@@.formatSnippetescapes the HTML and then replaces every occurrence of those strings, including any a feed author wrote into the content. The brief requires delimiters that are "unique" and cannot come from entry content. This is not script execution, but it breaks the guarantee that<mark>is the only markup and that it only marks real matches.@@@HL_START@@@injected@@@HL_STOP@@@ text @@@HL_START@@@ unclosedis rendered as plain text.zebrafish):<mark>zebrafish</mark> <mark>injected</mark> text <mark> unclosedThat is an injected highlight plus an unclosed
<mark>.ts_headlineas a bound parameter, and replace only that value. Or strip or escape the delimiter strings from the source before callingts_headline. Add a test for this.B2. The
<script>test never exercises the escaping code.Location:
web_test.goTestSearch_HostileScriptNeutralized.The stub library is handed a
Snippetthat is already escaped (<script>… <mark>malicious</mark>). The test only proves that templ prints the string it was given. NeitherformatSnippetnor the realts_headlineoutput is tested anywhere, so a regression that dropshtml.EscapeStringwould still pass. The checklist and the brief both ask for this test.<p>Tom & Jerry <b>cartoon</b> quantumfoam <script>alert(1)</script> end</p>givesJerry cartoon <mark>quantumfoam</mark>.<script>(and the delimiter strings from B1) in content or extracted_content, searches for it, and asserts the snippet contains no markup other than<mark>. A unit test forformatSnippetwould also work.B3. The generic plan cannot use the GIN index.
Location:
library.sql, the list queries.The search predicate is
($q IS NULL OR e.search_vector @@ $q::tsquery). pgx caches prepared statements, and after five runs Postgres may switch to the generic plan. In that plan the OR-with-NULL stops the GIN index from being used. The brief asks for "a constant the GIN index can use", and the existingTestScopesAreIndexBackedwas not extended to coverq.xylophonist), after ANALYZE:enable_seqscan=off: usesentries_search_idx(OK).EXPLAIN (GENERIC_PLAN):Index Scan using idx_entries_all_page … Filter: (($1 IS NULL) OR (search_vector @@ ($1)::tsquery)) …. It walks every row in published_at order.IS NULL ORonq, or otherwise make the predicate plan-stable. Then extendTestScopesAreIndexBackedwithQueryset, and check a generic plan (EXPLAIN (GENERIC_PLAN)orplan_cache_mode=force_generic_plan).B4 (gate). Task 5.3 is not ticked in
openspec/changes/newsroom/tasks.md; it still reads- [ ] 5.3.Should-fix
Location:
BuildSearchTSQuery.The per-language queries are OR-ed, and the combined query is matched against every entry whatever its language. So
-wordis applied with the wrong stemmer.englishandsimple, english entry "the running cat", querycat -runningreturns 1 result (expected 0).AND e.search_vector @@ websearch_to_tsquery(e.search_language, $q)as a per-row recheck.ListActiveFeedLanguagesonly covers feeds that are not disabled, but entries of disabled feeds are still listed. If a german feed is removed from the config, its starred entries are searched without the german stemmer. Not verified with a probe; follows from the SQL. Fix: use the distinctentries.search_languagevalues, or include disabled feeds.coalesce(extracted_content, content, '')in the generated column and ints_headlinetreatsextracted_content = ''as a real body. The rest of the code (GetEntry, the enrich queries) treats''as missing. Fix:coalesce(nullif(extracted_content, ''), content, ''). Not verified with a probe; I did not confirm that the extractor ever stores''.Minor / nits
web.go(mutate→rowData(e, query{}, …)) re-renders the row without its snippet, so the highlight disappears from search results after starring. Thequery{}was already there before this PR; the lost snippet is new.library.gohas four copy-pasted 29-field row converters and fourlistXwrappers, andlibrary.sqlhas four identical query bodies.sqlc.embed(e)would remove the converters. EverySELECT e.*/RETURNING *now also fetches the storedsearch_vector(decoded into an unusedinterface{}), which roughly doubles the row payload.Store.ListActiveFeedLanguagesandStore.BuildSearchTSQueryare never called;Librarycalls dbgen directly. The"simple"default is repeated inRouteConfig.Init,reconcileRoute,UpsertFeedSQL andLibrary.DisableFeedsNotInCOALESCE change is unrelated hardening. It is harmless; noting it only.textinput(TypeSearch) instead ofsearch. In v0.4.0, Goshtososearchis a command-palette dialog with items, sotextinputin a GET form is the reasonable choice.Checklist status
qand the language names reach Postgres only as bound parameters. Languages are validated againstpg_ts_configbefore upsert. OK.TestSearch_LanguageChangeReindexesEntries). OK.qis in the URL and in load-more links; the form keeps state/since and drops cursor/entry (tested). OK.write_dbgen_testpasses, so the generated code matches the generator. OK.Not covered
bazel run //:format///:buildifierin this sandbox (toolchain/arch mismatch, not the git ownership error). I relied ongofmt -l(clean) and CIValidate(green).go test -covercannot measure them here. Bazel coverage with remote execution was not attempted.Next: back to Linus Torvalds on ICB-152 for B1–B4 and preferably S1–S3. I will re-review the fix commits with
/code-review. Not merging; merging is the owner's decision.Verdict: PASS (re-review). Every blocking finding is fixed and verified. Two new should-fix items and some nits remain; none of them blocks the merge. Merging is the owner's decision; I have not merged.
Re-review of PR #274 at head
82f60a9(basemain@8ba37ec, unchanged). I reviewed the fix commitb6ca956..82f60a9with/code-review(high effort) and read the full PR diff8ba37ec..82f60a9for context. The only new commit since my first review is82f60a9.Success condition tested: the same as in the first pass (the brief
22-full-text-search.mdplus the three spec deltas and the gate). In addition, B1–B4 must be fixed and verified, and S1–S3 must be addressed.Steps run
First-pass findings: status
randomHighlightDelimitersmakes a new 16-bytecrypto/randtoken for each request and passes it through the boundhl_optionsparameter, andformatSnippetreplaces only that token. The integration testTestSearch_HostileScriptAndDelimitersNeutralizedchecks that the old@@@HL_START@@@text stays literal.<script>test: fixed. There is now a real-path integration test (content goes throughts_headlineand thenformatSnippet) and a unit test forformatSnippet.SearchLibrary*queries withoutIS NULL ORon the tsquery. My probe shows the generic plan usesentries_search_idxeven with seqscan enabled, at the same data shape that failed before. Nit: the new subtest setsenable_seqscan = off. That makes it weaker than my probe, because it cannot catch the old failure, where the generic plan picked theidx_entries_all_pagewalk.websearch_to_tsquery(e.search_language, raw_query)recheck, andTestSearch_NegationRespectsEntryLanguagetests it.TestSearch_DisabledFeedLanguageRespectedtests it.ts_headlinenow usenullif(extracted_content, '');TestSearch_EmptyExtractedContentDoesNotHideContenttests it.Should-fix (non-blocking)
S4. HTML entities are escaped twice in snippets.
Location:
library.go:579formatSnippet.ts_headlineruns on the raw feed HTML. It drops tags but keeps entity text such as&,<and’.html.EscapeStringthen escapes those entities again. This is not a security problem, because the escaping goes too far rather than not far enough. But WordPress-style feeds use’and&all the time, so many snippets will show raw entity codes. This bug was already inb6ca956; I missed it in the first pass.<p>It’s AT&T and quantumfoam <b> news</p>, queryquantumfoam.quantumfoam <b> newsas text.<mark>quantumfoam</mark> &lt;b&gt; news, so the page shows<b>literally.html.UnescapeStringon the headline before escaping it (the random token has no&, so it is not affected). Add a test case.S5. Search result rows have no outer
ORDER BY.Location:
library.sql, the fourSearchLibrary*queries.The rows are sorted only inside the
pageCTE, andpaginateRowstakes the last row as the keyset cursor. Postgres keeps that order in practice today (the pagination test passes), but SQL does not guarantee it. AddingORDER BY page.published_at DESC, page.id DESCto the outer SELECT costs nothing. I found this by reading the code; I did not reproduce it.Minor / nits
Entry{Entry: e}andquery{}, so it loses its snippet andq.qis not trimmed (web/query.go:53), soq=%20%20shows "No results" instead of the normal list. It also has no length limit. Found by reading the code.ValidateLanguagesmatchescfgnameexactly, soGermanandpg_catalog.germanare rejected even thoughregconfigaccepts them. A lowercase name such asgermanworks.searchXxxmethods with 30-field row converters.Store.ListActiveFeedLanguagesandStore.BuildSearchTSQueryare never called, and thelen(langs)==0fallback can never trigger because the query UNIONs'simple'. That query is still named and documented as covering "active" feeds, but now includes disabled ones. Each search makes three round trips.ts_headlinehighlights with the cross-language OR query, so a snippet can mark a word that matched only under another language's stemming.store.gosit before exported methods, andConfig.ValidateLanguagessits aftersortedKeys.Checklist
<script>test exists. The only open item is cosmetic over-escaping (S4).pg_ts_config.nullifand builds on PG18.write_dbgen_test).82f60a9, 5.3 is ticked, and there is no new third-party dependency.Not covered
bazel run //:format: cannot run in this sandbox (the formatter binary is built for a different CPU, arm64). I relied ongofmt -l(clean) and CIValidate(green).go test -covercannot measure them. Bazel coverage with remote execution was not attempted.Next: nothing blocks the merge. The owner decides whether S4 and S5 are fixed before the merge or as a follow-up. If they are fixed, they go to Linus Torvalds. Not merged.
Verdict: FAIL (re-review of
82f60a9..102d85f). There are 2 blockers: the N1 fix adds a reflected XSS, and S5 was not done. The other in-scope items (N2/N3/N4, N8, N9, N10) are fixed.What I held it to: every in-scope item (N1, N2/N3/N4, N8, N9, N10, S5) is fixed and has a test that would fail without the fix. No regression of B1–B3 or S1–S3. Only
<mark>markup reaches the page. Bound parameters only. The gate is green.Blocking
1. N1 fix adds a reflected XSS. The page now shows client-supplied HTML unescaped.
web/view.gostarVals/readValsput the rendered snippet HTML intohx-vals.web/web.gomutatethen doese.Snippet = r.FormValue("snippet"), andlayout.templ:237writes that out with@templ.Raw(r.Snippet). Any POST to/ui/entry/{id}/staror/readtherefore puts arbitrary HTML on the newsroom origin. The/ui/entryhandlers have no CSRF or Origin check, so another site can trigger this with a plain cross-site form POST. This breaks the "only<mark>reaches the page" rule.ts_headlinefor that one entry withq, or swap only the buttons so the snippet in the DOM stays as it was). Nothing from the request reachestempl.Raw.mutate,q.Query = strings.TrimSpace(r.FormValue("q"))skips the new 500-character limit. It is only used to build URLs, so this is minor; fix it together with the above.2. S5 not done. None of the four
SearchLibrary*queries has an outerORDER BY, and there is no pagination test for search. The commit message does not mention S5.ORDER BY page.published_at DESC, page.id DESCafter eachFROM page, plus a test that pages through more search matches than one page holds and sees each match exactly once.Verified fixed
parseQuerynow trimsqand returns 400 when it is over 500 characters. Covered byTestSearch_WhitespaceQueryBehavesLikeNoQueryandTestSearch_QueryLengthLimit.cfgname ... WHERE oid = $1::regconfig(bound parameter), and the canonical name is written back to the provider, route, andEffectiveLanguage.TestReconcileFeeds_LanguageCanonicalization(Postgres) checks thatGermanandpg_catalog.germanare stored asgerman. There is no re-index on the next startup: the stored feed language is canonical now, andUpdateEntriesSearchLanguagecomparesregconfigvalues, so it only updates rows when the language really changed.ts_headlinenow useswebsearch_to_tsquery(page.search_language, raw_query), the entry's own language.TestSearch_CrossLanguageHeadlineDoesNotMarkUnmatchedStemcovers this.Storemethods and the unreachable fallback are gone. The language set is now computed insideBuildSearchTSQuery, so a search takes 2 round trips. TheWHEREclauses of the search queries did not change, so B1/B2/S1–S3 are untouched.WHEREclauses are byte-identical between82f60a9and102d85f; only thets_headlineargument changed. I re-ran a generic-plan probe at both heads (temporary test, reverted):PREPAREofSearchLibraryAllwithplan_cache_mode=force_generic_plan,enable_seqscan=off,enable_indexscan=off, thenEXPLAIN EXECUTEon Postgres 18. Both heads produce the same plan: a Bitmap Index Scan on the partialidx_entries_feed_page, withsearch_vector @@ ($n)::tsqueryas a Filter. The only difference is that$1/$7swapped places. On an empty table the planner prefers the partial btree, so this probe does not show the GIN index being used. It does show that this diff changes nothing about planning. B3 still stands as verified in the previous round.Gate
bazel test --config=remote --nocache_test_results //newsroom/...at102d85f→Executed 20 out of 20 tests: 20 tests pass.(EXIT=0). This includesstorage_integration_test,library_integration_test, andserving_integration_test(Postgres 18-alpine).Validate / validate (pull_request)on102d85f: success (run 597).gofmt -lon the changed Go files prints nothing.go.mod,go.sum, orMODULE.bazel. No new dependency.find . -type l -not -path './.git/*'prints nothing.Nits (not blocking)
Store.languageResolvertreats every query error as "unknown language" and caches it. If the database is down, the error message points at the config instead of the connection. Return the error.ListFeedLanguages(renamed, now documented correctly) has no Go caller left. Delete it or keep it on purpose.hx-valsnow carries a copy of the snippet HTML. This goes away with the fix for blocker 1.Out of scope (status only)
Not covered
Next owner: Linus Torvalds (ICB-153). Fix blockers 1 and 2 on this PR, then hand back for review. Do not merge.
Verdict: PASS — at
5cc9917, review of102d85f..5cc9917(QA: Grace Hopper, ICB-156). Not merged.Success condition tested
templ.Raw, and N1 still holds.http.NewCrossOriginProtection()wraps every unsafe route on the real handler. Cross-origin requests get 403 and change no state. Same-origin requests work. GET is unaffected.ORDER BY published_at DESC, id DESCon all fourSearchLibrary*queries, plus a pagination test. The GIN index is still used in the generic plan (B3).bazel test --config=remote --nocache_test_results //newsroom/...green, CIValidategreen, no new dependency, no symlinks.Steps run
Evidence (real Postgres 18, real
web.Handler)/ui/entry/{id}/{star,read}withsnippet=<img src=x onerror=alert(1)><script>alert(2)</script>andq=kernel <payload>, in row view and reader view. The entry content also contains<script>and<img onerror>. Result: 200, no raw<script/<imgin the body, and<mark>kernel</mark>is present (N1). The content's tag shows up only as<img src=y onerror…. Inlayout.templ:237,templ.Raw(r.Snippet)is now fed only byformatSnippet, which HTML-escapes first and then swaps in the random delimiters.EntrySnippet(id, "<b>x</b> consensus")→"raft <mark>consensus</mark>". A missing id returns""with no error.Sec-Fetch-Site: cross-site→ 403,same-site→ 403,Origin: http://evil.example→ 403, and the read-all POST cross-site → 403. Each time,starred_atstayed NULL in the DB. Same-origin requests with onlyOrigin(the plain-HTTP case) → 200. Cross-site GET → 200. The wrap is inweb.Handleritself (web.go:88), so it covers all 4 POST routes. No trusted origins or bypass patterns are added.published_at, pages of 2, states All and Unread, each entry exactly once inpublished_at DESC, id DESCorder.EXPLAIN (GENERIC_PLAN)on SearchLibraryAll, 50k rows, ANALYZE):Bitmap Index Scan on entries_search_idx, Index Condsearch_vector @@ ($7)::tsquery. There is no extra Sort node for the outer ORDER BY.Findings (none blocking; ranked)
ErrUnknownLanguage.store.go:123checks SQLSTATE22023, but Postgres raises42704for an unknown regconfig.errors.Is(err, config.ErrUnknownLanguage).providers.bad-provider: resolve text search language "klingon": ERROR: text search configuration "klingon" does not exist (SQLSTATE 42704), anderrors.Isis false.assert.Error, so it does not catch this.42704(or useto_regconfig()and check for NULL), and assertErrorIsinTestReconcileFeeds_*.mutatecallsmutateQueryafterapply()./ui/entry/{id}/starwithstarred=true&q=<501×a>.starred_atis set in the DB. HTMX does not swap the row, so the UI shows the wrong state. The UI never sends aqover 500 chars, so this is low.apply.config.go:307: a DB error while resolvingEffectiveLanguageis still silently ignored, unlike the Language branch.TestSearch_Pagination_S5…uses a non-fatalassert.True. If pagination regresses, the test hangs until the timeout instead of failing. Uset.Fatal/break.storage/BUILD.bazelhas 2 stray blank lines, andfeeds.sql/library.sqlhave extra blank lines. CI's format check accepts them.mutateQueryduplicates theqtrim and length check fromparseQuery.Dropped: the
/code-reviewclaim thatGetEntrySnippetmixing$1andsqlc.argcan't be regenerated is false.write_dbgen_testpasses, so the regenerated code matches what is committed.Status of out-of-scope items
S4 and N11 were not touched in this commit and are still open, as agreed. No regression found in B1–B3, S1–S3, N1–N4, N8–N10: the full suite is green, and B1/B2 (sanitizer), B3 (plan) and N1 were re-probed above.
Not covered
newsroom/deployis a placeholder). On plain HTTP, browsers send noSec-Fetch-Site, so Go comparesOriginwithHost. If a proxy rewritesHost, every POST will get 403. When this is deployed, the proxy must keep the original Host, or the handler needsAddTrustedOrigin.bazel run //tools/format:format.checklocally: the sandbox blocks git (dubious ownership), so I relied on CI's green format step instead.Next owner: the owner can merge. Findings 1–2 are cheap follow-ups for Linus Torvalds and can go on this PR before merge or into a separate task. Steve Wozniak decides which.