feat(newsroom): wire kong CLI app, health endpoints, and graceful shutdown (task 6.1) #286
No reviewers
Labels
No milestone
No project
No assignees
2 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
prskr/tools!286
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "paperclip/ICB-159-app-wiring"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Implements task 6.1 of OpenSpec change
newsroom(newsroom/tasks/16-app-wiring.md).Changes
cmd/newsroom): Rootappstruct with env-only secret flags (DB_PASSWORD,LLM_API_KEY,READECK_TOKEN) and default address:8081./health/*,/assets/*, UI pages, Atom routes, and/admin/*. Preserves PR #274 cross-origin protection across all unsafe UI requests.internal/health):/health/readiness(pings database) and/health/liveness(lightweight, no db ping). Removed old/healthz./ui,/assets,/admin,/health,/feed,/provider,/entry,/briefings.internal/serverskeleton and/v1/enrichproxy handler.Verification
bazel test //newsroom/...passed (22/22 tests pass).go run ./newsroom/cmd/newsroom --helpverified flags and secret env-vars.go test ./newsroom/...andgo vet ./newsroom/...passed cleanly.Co-Authored-By: Paperclip noreply@paperclip.ing
QA review: newsroom 6.1 App wiring (kong), PR #286 @
0c948f2Verdict: FAIL: 3 blocking findings. Most of the wiring is right: kong app, single mux, health split, reserved prefixes, cross-origin protection through the full app. It falls short on one acceptance criterion (
os.Getenv). It also adds a process crash on a config that passes validation, and its shutdown does not respect its own time limit.Success condition tested:
newsroom/tasks/16-app-wiring.mdacceptance criteria and the ICB-160 plan checks. Specs: newsroom-operations ("Readiness and liveness…", "Shutdown drains in-flight work") and newsroom-config ("Route namespace is collision-free", "Secrets come from the environment only").Steps run
/code-review(high) onorigin/main...origin/paperclip/ICB-159-app-wiring(merge-basefe97611, 1 commit, 14 files). Every finding below was checked against the code or reproduced; unconfirmed ones were dropped.bazel test --config=remote --nocache_test_results --experimental_convenience_symlinks=ignore //newsroom/...→ Executed 22 out of 22 tests: 22 tests pass.--test_output=all --test_arg=-test.von//newsroom/cmd/newsroom:newsroom_integration_test, plus a temporary QA probe file (not committed; worktree deleted). The container tests really ran and were not skipped:TestIntegration_WiringRoutes,…CrossOriginProtectionThroughFullAppand…GracefulShutdownall PASS.bazel run --experimental_convenience_symlinks=ignore //newsroom/cmd/newsroom -- --help, plus the real binary run against configs with reserved routes.os.Getenv,ListenAndServe,CrossOriginProtection,AddTrustedOrigin,v1/enrichandcurator; a diff ofgo.mod/MODULE.bazel;git diff --summarychecked for mode120000; Forgejo commit status.Blocking (Linus)
B1.
os.Getenvis still innewsroom/:cmd/newsroom/app.go:102(resolveDSN).os.Getenvinnewsroom/(task acceptance criterion), and all secrets come through kong flags so that--helplists every secret.grep -rn os.Getenv newsroom --include=*.go(non-test) →newsroom/cmd/newsroom/app.go:102: return os.Getenv(key). When--db.passwordis empty, the password comes from whatever variabledatabase.passwordEnvnames, and that variable never appears in--help. Also,injectPasswordsilently drops the password when the DSN has no user.c.Database.Password(DB_PASSWORD) and remove theos.Getenvfallback. IfpasswordEnvis set to anything other thanDB_PASSWORD, fail at startup with a clear error, or ignore it; howpasswordEnvchanges long-term belongs to newsroom-cleanup. Return an error instead of silently dropping the password when the DSN has no user.B2.
summarize: truewithoutllm.baseURLpasses validation, then crashes the whole process (nil-interface panic).app.go:252-255only builds a summarizer whencfg.LLM.BaseURL != "". Otherwiseenrich.New(store, extractor, nil), andvalidateSummarizedoes not require a base URL. ProbeTestQAProbe_SummarizeWithoutBaseURL:buildHandler err=<nil>, thenPOST /admin/ingest/g/test→ The panic happens in an ingest worker goroutine, so net/http does not recover it and the process exits. The same path runs from cron and from the always-registered backfill job.summarize: true(or summary backfill is active) andllm.baseURLis empty, or never pass a nil summarizer. Add a test for it.B3. Shutdown is not bounded when an ingest is in flight, and a normal SIGINT exits non-zero.
RunAllruns usecontext.Background()/WithoutCancelwith a 2-minute route timeout (schedule.go:220,249,368).Rungivess.Stopandsrv.Shutdownone shared 10s budget, one after the other, and nothing cancels the runs. ProbeTestQAProbe_ShutdownOverrunsBudget(feed upstream hangs for 20s,POST /admin/ingest→ 202, then cancel):Run returned after 20s: err=context deadline exceeded. That is twice the 10s limit, because the deferredstore.Close()waits for the run that is still holding a connection.srv.Shutdownalso got an already-expired context, andkctx.FatalIfErrorfturns this into exit 1.Runcancels when the drain budget runs out, so in-flight runs stop. Givesrv.Shutdownits own budget, or shut HTTP down first so new admin triggers stop. Treat "drained within budget" as a clean exit. Add a test where the ingest outlives the budget.TestQAProbe_ShutdownDrainsInFlightIngestNoLeaks→Run returned: <nil>,ingest status=200 feedDone=true. A strictgoleak.VerifyNonewith anIgnoreCurrent()baseline and noruntime_pollWaitignore passed.Non-blocking
main.go:13usessignal.NotifyContext(…, os.Interrupt, os.Kill).os.Kill(SIGKILL) cannot be caught, and SIGTERM, which Kubernetes sends, keeps Go's default behaviour of exiting immediately, so the drain never runs in a pod. The brief told Linus to use exactlyos.Kill(mirroring content-extractor), so the code matches the brief, but the spec's "termination signal" requirement is not met in a pod. It's a one-line fix (syscall.SIGTERM); I recommend making it in this round, before 6.3 deploys it.readeck:section is present butREADECK_TOKENis unset (app.go:260), export is silently disabled with no log line and no error. At least log a warning, or fail at startup.TestIntegration_GracefulShutdownignoresinternal/poll.runtime_pollWait, which hides any goroutine blocked on a socket (a leaked serve loop or pgx connection). My strict version shows there is no leak today; the guard is just weak. Use anIgnoreCurrent()baseline instead./admin/sits outsideCrossOriginProtection, as the plan requires for header-less clients. Wrapping it would still let curl through and would stop drive-by browser POSTs to/admin/ingest. Authentication and admin hardening are out of scope for 6.1; noting it for newsroom-cleanup.Nits
app.go:303:srv.Shutdownnever returnshttp.ErrServerClosed, so that check never runs.app.go:221-224: theservingLimitdefault duplicatesServingConfig.init(50) andserving.Register's own default.internal/healthis close to a copy ofcontent-extractor/internal/health, andCheckerFuncis unused. The brief asked for "mirroring", so this is fine for now; a sharedpkg/healthwould be a follow-up.--help:--http.read-header-timeout,--http.idle-timeoutand--http.max-header-bytesshow no env names. They come frompkg/cliand were like that before this PR.Acceptance checklist (checked by running)
bazel test //newsroom/...uncached, container tests included--helplists all flags with env names; secrets are env-only with no default--db.password=STRING ($DB_PASSWORD),--llm.api-key=STRING ($LLM_API_KEY),--readeck.token=STRING ($READECK_TOKEN),--http.address=":8081" ($HTTP_ADDRESS),--config="config.yaml" ($CONFIG),LOG_*. ❌ the DB secret can still arrive another way (B1)/healthzis gonereadiness=200 liveness=200 healthz=404, downreadiness=503 liveness=200/ui/x,/health,/assets→route collides with a reserved prefix. A single segment under/briefings/xis allowed on purpose (config/briefing.go:175, briefing feeds per tasks 03/21);/briefingsitself and deeper paths are rejectednewsroom: error: --config: stat /nope.yaml: no such file or directoryos.Getenv/http.ListenAndServeos.Getenvatapp.go:102(B1); ✅ onlysrv.ListenAndServe()on the configuredhttp.Serverinternal/curatorand/v1/enrichdeletedcurator,internal/serverorv1/enrichinnewsroom/Sec-Fetch-Site→ 403 withstarred_atunchanged; foreignOrigin→ 403; same-origin → 200 with DB updated;/admin/ingest/g/testwith no Origin and no Sec-Fetch-Site → 200. Exactly oneNewCrossOriginProtection(web.go:88); noAddTrustedOriginor bypassgo.mod/go.sum/MODULE.bazelunchanged (goleakwas already a dependency)ValidateValidate / validate (pull_request) successon0c948f2tasks.md120000in the diff;find . -type l -not -path './.git/*'prints nothingNot covered
Runcontext, which is whatNotifyContextdoes. SIGTERM behaviour (N1) comes from reading the code and Go's documented semantics, not from running it.storage.Openbefore the listener starts, so this holds by construction. I did not test it separately.Next owner: Linus Torvalds for B1–B3, routed by Steve Wozniak. Steve Wozniak also needs to decide on N1. I'll re-review with
/code-reviewonce there's a new SHA. Not merged.Verdict: PASS. PR #286 at
5979c3dis OK to merge. Nothing is blocking. The non-blocking items below are for Steve Wozniak to place: fold them into this PR or record them innewsroom-cleanup.Re-review of the fix round (ICB-162, commit
5979c3d) against my previous FAIL at0c948f2(ICB-160). I did not merge.Success condition tested
B1, B2, B3 and N1 are fixed, shown by running them. N2 and N3 are done. Nothing that passed last time has regressed. All
//newsroom/...tests are green without cached results, including the container integration tests. No symlinks, no new dependencies, CIValidateis green, and 6.1 is ticked at the final commit.Steps run
Results per item
Executed 22 out of 22 tests: 22 tests pass.EXIT=0os.Getenvnewsroom/config/database.go:52passwordEnvdatabase.passwordEnv "MY_PW" is not supported: database password must be set via DB_PASSWORD / --db.passworddatabase dsn has no user to attach password to--help--db.password ($DB_PASSWORD),--llm.api-key ($LLM_API_KEY),--readeck.token ($READECK_TOKEN), no defaults.DrainTimeoutis hiddensummarize: truewithoutllm.baseURLinvalid config: providers.p: summarize is true but llm.baseURL is empty. Backfill without baseURL also gives exit 1Runreturns within budget + margin, run cancelled, pool closedRun returned after 1.001985108s err=<nil>(budget 1s). Upstream request gotcontext canceled.backends after Run: 0. Strictgoleak.VerifyNone(baseline)passesTestQAProbe_ShutdownOverrunsBudgetPASSRunreturns nil, no leaksRun returned after 1.516754017s err=<nil> served=true,entries persisted: 1, 0 backends, strict goleak passesTestQAProbe_ShutdownDrainsInFlightIngestNoLeaksPASSNotifyContext(..., os.Interrupt, syscall.SIGTERM), andos.Killis gone. A realSIGTERMsent to the process running the realmain()(kong parse, real Postgres): readiness was 200 before the signal, andmain() returned normally 391µs after SIGTERM(the exit-0 path;FatalIfErrorfwas not hit, the process was not killed), 0 backends leftTestQAProbe_RealSIGTERMThroughMainPASSWARN readeck config present but READECK_TOKEN / --readeck.token is unset; export disabled, covered byTestIntegration_ReadeckWithoutTokenLogsWarningIgnoreCurrent()baseline, noruntime_pollWaitignorehttp.NewCrossOriginProtection()(web/web.go:88) and noAddTrustedOrigin. The health, reserved-prefix and routing code is not touched by the fix diff.TestIntegration_WiringRoutesandTestIntegration_CrossOriginProtectionThroughFullApppass120000entries,tasks.md6.1 is[x]at5979c3d, CIValidate / validate (pull_request)= successFindings, most severe first
Blocking: none.
Non-blocking (real, but none crashes, leaks a secret or loses data):
AfterApplybindsLogging.PrepareLogger()into kong, but nothing callsslog.SetDefault; I grepped the whole repo.ingest,enrichandopenaicompatlog through package-levelslog.WarnContext/ErrorContext/InfoContext(10+ call sites, e.g.ingest.go:462,:490,openaicompat.go:215). Their lines bypassLOG_FORMAT/LOG_LEVELand come out as Go's default text log next to the app's JSON. In the probe run,main()logged JSON and the ingest error logged plain text. This was already present at0c948f2and I missed it last round. The fix is one line inAfterApply.Stopalso times out, its error is discarded,Runreturns nil (exit 0), and the deferredstore.Close()(pgxpool.Close) waits for any connection still held.NotifyContext'scancelonly runs on exit, so a second Ctrl-C or SIGTERM is swallowed. In practice pgx honours ctx and my probe shows a clean 1.0s exit, so this is theoretical.POST /admin/ingest/{route}sent in that window gets 500 (ErrSchedulerStopped), not 503.passwordEnvis rejected in the CLI, not in config validation.resolveDSNrejects a non-defaultdatabase.passwordEnvat runtime, butconfig.Validateand thedatabase.gocomment still accept and describe any env var name. A DSN that sets the user with?user=(and has no userinfo) is also rejected when a password is set. Both match the agreed "error" behaviour; the check just sits in the wrong layer.ErrNoLLMBaseURLis reported once for each provider withsummarize: trueinstead of once overall (provider.go:165).llm.baseURLis set, even if no provider summarizes, becauseinit()always fills in a defaultBackfill.Nits:
executeRouteandrunGenericJob;context.AfterFuncwould avoid the extra goroutine per run.loadConfigis now just a pass-through toconfig.LoadFile, and theresolveDSNlookup closure is a constant stub.DisabledSummarizer.Summarizehas no doc comment and its params are named but unused (CI lint passed anyway).persistConnignores were widened fromIgnoreTopFunctiontoIgnoreAnyFunction. That works, but it is looser than needed.explicitBackfillis only set by YAML decoding, so aConfigbuilt in code skips the check.ingest.Stats→schedule.Statsfield-by-field copy could be a type conversion.healthis now the third copy of the probe handler (content-extractor, chargelog).store.Pool()instead ofstore.LISTEN_ADDRtoHTTP_ADDRESS(no deployment exists yet; 6.3 should useHTTP_ADDRESS).Not covered
SIGTERMto the test process running the realmain()against a real Postgres container on the remote executor (see the table). An actual process exit code was not observed. The not-killed / no-FatalIfErrorfpath is equivalent to exit 0.main(): not sent separately. It uses the sameNotifyContextand the same ctx-cancel path, which the probes cover.Probe files and the worktree are removed.
find . -type l -not -path './.git/*'prints nothing.Next owner: Steve Wozniak decides whether non-blocking items 1–6 go into this PR (Linus Torvalds) or into
newsroom-cleanup. Item 1 is a one-line fix and worth doing first. With that decision made, the owner can merge.Verdict: PASS — re-check of
5979c3d..4125c9f(one commit,fix(newsroom): configure slog default logger and address review nits). Not merged.Success condition tested
ingest/enrich/openaicompatlines come out as JSON underLOG_FORMAT=jsonand followLOG_LEVEL, when parsed through kong the waymain()does it.NewCrossOriginProtectionand noAddTrustedOrigin. No new dependency. No mode120000. 6.1 ticked. CI green.Steps run
Results
//newsroom/...; no target is taggedmanual)Executed 22 out of 22 tests: 22 tests pass.EXIT=0LOG_LEVEL=info(TestQAProbe_SlogDefaultThroughKong/info, kongParsefrom env likemain()){"level":"INFO","msg":"openai-compatible completion","model":"m",...}and{"level":"WARN","msg":"enrichment failed for entry","entry_id":2,...}LOG_LEVEL=warn{"level":"WARN","msg":"enrichment failed for entry",...}TestQAProbe_ShutdownOverrunsBudget, budget 1s)Run returned after 1.000957453s err=<nil>, upstreamcontext canceled,backends after Run: 0, strictgoleak.VerifyNone(baseline)passesTestQAProbe_ShutdownDrainsInFlightIngestNoLeaks)Run returned after 1.515296778s err=<nil>,entries persisted: 1, 0 backends, strict goleak passesAfterFuncrefactorGracefulShutdown*tests in the suiterunContextusescontext.AfterFuncand is used in both placesloadConfig/resolveDSNstubconfig.LoadFileis called directly. The lookup isfunc(_ string) string { return password }, which is safe because thePasswordEnvguard directly above only allows""orDB_PASSWORDDisabledSummarizer_persistConnignorespersistConn.readLoopgoroutine withinternal/poll.runtime_pollWait on top of the stack, soIgnoreTopFunctioncannot match it. A comment now explains thisexplicitBackfillfor aConfigbuilt in codeisDefaultBackfillflag is set byinit(); new testTestLLMConfig_Validate_CodeBuiltBackfillWithoutBaseURLErrorsStatsconversionschedule.Stats(stats)storehealth.PingChecker{Pinger: store}HTTP_ADDRESSnote on 6.3tasks.md6.3 andtasks/18-deployment-and-release.mdhealthcopyNewCrossOriginProtectiononce (web/web.go:88),AddTrustedOrigin0. No change togo.mod/go.sum/MODULE.bazel. No mode120000. 6.1[x]. CIValidate / validate (pull_request)= success (run 609) on4125c9fopening database poolline logs a redacted DSN (password replaced withxxxxx)Findings
Blocking: none.
Non-blocking: none.
Nits (optional; none of them is worth another round on their own):
schedule.go:164:abortChis now dead state. Nothing reads it any more, butCancelInFlightstill closes it next toabortCancel(). It can go.schedule.go:167:abortCtxis afunc() context.Contextthat always returns the same context. A plaincontext.Contextfield would be simpler.NewsetsabortCtxandabortCancel, so a zero-valueSchedulernow panics inrunContext. Nothing builds one that way today (onlyNewdoes, per grep), so this is theoretical.isDefaultBackfillis only set insideinit(). Code that replacesLLM.BackfillafterInit()would skip the base-URL check. This is an edge case and nothing does it today.TestAfterApply_ConfiguresSlogDefaultswaps the globalos.Stderr. That is fine today because the package has not.Parallel./code-review also raised these, which I dismissed:
slog.SetDefaultas global state (it was the requested fix), coupling through theStatsconversion (requested), andassert.NoErrornot stopping the test (alecthomasassertis fatal).Not covered
Probe files are removed, and
find . -type l -not -path './.git/*'prints nothing.Next owner: the owner. Nothing is outstanding for this PR, and it can be merged. Nits 1–4 can go into
newsroom-cleanupif wanted (Steve Wozniak's call).Addressed Grace Hopper's two optional nits from the re-review in commit
e5969e46d0e72e6d085cc066c96839e53c1cb2d4:schedule.go:164: removed deadabortChfield,close(s.abortCh)inCancelInFlight, and channel allocation inNew.schedule.go:167: convertedabortCtxfromfunc() context.Contextto a plaincontext.Contextstored directly inNewand passed tocontext.AfterFunc.Verified with
go test -race ./newsroom/internal/schedule/...,go vet ./newsroom/internal/schedule/..., and fullgo test ./newsroom/.... All passed.PR remains open and not merged.
@ -167,2 +176,4 @@// New creates a new Scheduler with the provided options.func New(opts ...Option) *Scheduler {abortCtx, abortCancel := context.WithCancel(context.Background())should the parent context not be the one of the app basically, so jobs are also canceled, on
[Ctrl]-[C]?Fixed the
containedctxnogo build error in commit44a661f1ae25a98dcbdeb6f55e6e7efaf1a79401and reviewed Peter's question regarding parent context:1. App Context vs. In-Flight Job Drain on
[Ctrl]-[C]Binding job runs to the app context (which cancels immediately on
[Ctrl]-[C]/SIGTERM) does not make sense with the graceful shutdown architecture:c.DrainTimeout, default 10s) vias.Stop(drainCtx)so active ingestion runs can finish cleanly without leaving partial records or dirty state.[Ctrl]-[C], defeating the drain period and failingTestIntegration_GracefulShutdown_DrainTimeout(which explicitly verifies that runs are given the drain window and only aborted if they exceed it).s.CancelInFlight()is specifically called when the drain budget is exceeded or ifsrv.ListenAndServe()fails.2. Fix for
containedctxnogo ErrorabortCtxfrom a directcontext.Contextstruct field back tofunc() context.ContextonScheduler.func() context.Contextis intentional forcontainedctxanalyzer compliance while supportingcontext.AfterFuncwithout background watcher goroutines.abortChremains removed (nit 1).Verification
bazel --bazelrc=.bazelrc.agents test //newsroom/internal/schedule/...: PASS (schedule_testpassed, nogo passed with zero errors).bazel --bazelrc=.bazelrc.agents test //newsroom/cmd/newsroom:newsroom_test: PASS (nogo onnewsroom_libandnewsroom_testpassed).bazel --bazelrc=.bazelrc.agents test //newsroom/internal/enrich/... //newsroom/internal/ingest/...: PASS (all 4 test targets passed).go vet ./newsroom/internal/schedule/...: PASS.go test -race ./newsroom/internal/schedule/...: PASS (1.147s).go build -buildvcs=false ./newsroom/...: PASS.