prskr/tools

Fork 0

tool: thumb #24

New issue

Open

opened 2026-08-05 16:49:23 +00:00 by prskr · 0 comments

prskr commented

2026-08-05 16:49:23 +00:00

Owner

thumb — Implementation Plan

Status: planning complete, implementation not started
Location: tools/thumb/ in the tools monorepo
Language: Go, built with Bazel (rules_go + gazelle)
Binaries: thumb (hub, local machine), thumb-agent (VM)

1. Problem

A cloud dev VM is used for regular software development. Commands like npm run dev,
dotnet run, or git fetch with an OAuth credential helper open TCP listeners on the
VM — sometimes on well-known ports, sometimes on random high ports. Those services need
to be reachable from the local machine.

Requirements:

Zero maintenance. No manual action when a port opens or closes.
Detection happens on the VM. Only the VM can see what's listening.
Single TCP connection. One outbound SSH connection from the local machine, nothing else.
No inbound reachability to the local machine. This is what rules out the current setup.

The current bore-based solution fails the last two: it needs the bore server reachable
from the VM on a set of random high ports as part of its handshake, which means the local
machine has to be directly connectable.

Performance is explicitly not a concern.

2. Design decisions and why

Recorded because the reasoning matters more than the conclusions if any of this gets
revisited later.

2.1 SSH as the transport, not QUIC or a custom protocol

QUIC is UDP-based and awkward to forward through an SSH tunnel. SSH already multiplexes
arbitrary channels over one TCP connection and has battle-tested forwarding primitives
(direct-tcpip, tcpip-forward). There is no reason to reimplement any of that.

2.2 Not ControlMaster + `ssh -O forward`

A viable approach exists using OpenSSH alone: remote-forward the ControlMaster socket
itself to the VM, then have a VM-side daemon issue ssh -S <forwarded-socket> -O forward
commands against it. This works and requires no custom binary.

Rejected because it means shelling out to the ssh CLI and parsing exit codes for every
port event, and because the ControlMaster mux protocol is an undocumented OpenSSH-internal
IPC format (mux.c), versioned per release, with no third-party client library in any
language. Speaking it natively is not a realistic target.

Once you're willing to run a custom binary on the VM, defining your own control protocol
is strictly simpler than reverse-engineering OpenSSH's.

2.3 Control plane and data plane run in opposite directions

This is the single most important thing to keep straight:

Data plane is -L-shaped. The hub listens on localhost:N locally and opens a
direct-tcpip channel to the VM's localhost:N. Driven entirely by the hub. The agent
is not involved and no VM-side cooperation is needed beyond the port existing.
Control plane is hub↔agent messaging. Its only purpose is letting the agent tell the
hub that a port appeared or disappeared, plus carrying agent logs and hub config.

All bulk traffic uses ordinary SSH forwarding. The control protocol carries only small
JSON messages.

2.4 Go over Rust

Considered Rust (existing Rust + Bazel + LLVM cross-compilation setup in the monorepo,
and bore was Rust). Chose Go because:

golang.org/x/crypto/ssh is extremely battle-tested — Tailscale SSH, Teleport, and most
Go infra tooling build on it. Client.Dial (direct-tcpip) is first-class.
GOOS=linux GOARCH=arm64 go build with no toolchain setup, no cgo, produces a static
binary. This matters concretely for the upload-and-exec deployment flow, where the agent
must just work on whatever arch the VM is without fussing over glibc/musl or target triples.
/proc/net/tcp parsing is trivial in either language.
The only Rust-specific asset was the bore crate, and bore is gone from the design.

2.5 JSON-RPC over gRPC

The deployment model kills gRPC's main advantage. The hub embeds a content-hashed agent
binary and uploads it, so hub and agent are always the same build — version skew is
structurally impossible, and protobuf's field-number discipline is insurance against a
scenario that cannot occur.

Beyond that:

The traffic is notification-heavy peer-to-peer messaging, which is JSON-RPC's native
model. Only one exchange (agent.hello → config) is a real request/response. gRPC would
mean tunnelling all of this through a single long-lived bidi stream — a streaming RPC
used as a message bus.
Dropping proto_library / go_proto_library removes the protoc toolchain from the
Bazel graph, which is one of the fiddlier parts of a rules_go setup — especially
combined with the platform transitions already needed for the embedded agent.
JSON overhead is irrelevant: a few dozen listeners at 1 Hz.
Debuggable by eye.

2.6 Stdio, not a forwarded Unix socket

Earlier drafts of this plan remote-forwarded a Unix domain socket
(streamlocal-forward@openssh.com) to carry the control protocol. That is unnecessary.

The agent is exec'd over an SSH session channel, and that channel already provides stdin,
stdout, and stderr. Stdout and stdin together form a full-duplex byte stream — exactly what
JSON-RPC needs. This is how every LSP server works.

Note the distinction: it is not that hub→agent signalling is unneeded (hub.setConfig
for dynamic log level is a real feature). It's that stdin already provides it, making
the forwarded socket pure redundancy.

Deleting the socket removes:

The streamlocal-forward@openssh.com global request and forwarded-streamlocal
channel-open handling — the one genuinely hand-rolled, no-library-support part of the
entire SSH layer.
The channel→net.Listener adapter and the net.Conn wrapper with its fake addresses
and no-op deadline methods.
Socket path generation, the stale-socket janitor, and StreamLocalBindUnlink concerns.
A dependency on sshd's AllowStreamLocalForwarding being enabled — worth calling out
separately, since a hardened VM image could have it off and the forward would silently
not work.
The pre-connect log ring buffer. With stdio the stream is connected before the process
starts executing; there is no dial that can fail and no window where logs have nowhere
to go.
Two of the three agent-lifetime mechanisms. Reading stdin returns EOF when the channel
closes, so the agent exits naturally.

2.7 Logs travel in the protocol, not just on stderr

Stderr already flows hub-ward for free, but gives unstructured text with no levels, no
filtering, and no correlation with hub-side events. The protocol path gives structured
records, hub-controlled verbosity, and a single merged timeline.

Both are retained — they cover different failure modes. Stderr is the fallback that
still works when the RPC layer itself is broken.

3. Architecture

local machine (hub)                            VM (agent)
┌──────────────────────────┐                  ┌────────────────────────┐
│ JSON-RPC peer            │                  │ JSON-RPC peer          │
│   reads session stdout ◄─┼── ndjson ────────┤   writes stdout        │
│   writes session stdin ──┼── ndjson ───────►│   reads stdin          │
│   drains session stderr ◄┼── fallback logs ─┤   stderr               │
│                          │                  │                        │
│                          │                  │ polls /proc/net/tcp*   │
│                          │                  │ slog → agent.log       │
│                          │                  │                        │
│ localhost:5173 ──────────┼── direct-tcpip ──┼─► localhost:5173       │
│ localhost:5000 ──────────┼── direct-tcpip ──┼─► localhost:5000       │
└──────────────────────────┘                  └────────────────────────┘
        one TCP connection, one SSH session, one exec channel + N forward channels

Nothing binds a Unix socket. Nothing touches the local filesystem for IPC. No other local
process can reach the control channel.

4. Repository layout

tools/thumb/
├── README.md
├── PLAN.md                  # this document
├── BUILD.bazel
├── cmd/
│   ├── thumb/               # hub binary
│   └── thumb-agent/         # agent binary
└── internal/
    ├── wire/                # shared message types, method names, ndjson framing
    ├── config/              # hub config file parsing
    ├── sshx/                # connect, auth, knownhosts, exec, sftp, keepalive
    ├── hub/                 # session lifecycle, reconnect loop, reconciliation
    ├── forward/             # local listeners + direct-tcpip data plane
    ├── procnet/             # /proc/net/tcp{,6} parsing
    ├── agent/               # poll loop, RPC peer, lifetime management
    ├── logstream/           # slog handler → RPC notifications + stderr fanout
    └── embedbin/            # go:embed of cross-built agent binaries

Filesystem paths:

Purpose	Path
Agent binary cache (VM)	`~/.cache/thumb/agent-<contenthash>`
Hub forward state	`~/.local/state/thumb/forwards.json`
Hub config	`~/.config/thumb/config.toml`

5. Protocol

Peer-to-peer JSON-RPC 2.0, newline-delimited JSON framing, over the exec channel's
stdout (agent→hub) and stdin (hub→agent). Message types live in internal/wire/ and are
imported by both binaries — single source of truth, no codegen step.

5.1 Methods

Agent → Hub

Method	Kind	Payload
`agent.hello`	request	version, hostname, arch, kernel, baseline listener set
`agent.snapshot`	notification	complete current listener set + monotonic seq
`agent.log`	notification	batch of log records

Hub → Agent

Method	Kind	Payload
`hub.setConfig`	notification	poll interval, port filters, log level
`hub.ping`	request	liveness check (belt-and-braces; stdin EOF is load-bearing)
`hub.shutdown`	notification	graceful exit request

agent.hello returns the initial config in its response, so there is no window where
the agent runs unconfigured. hub.setConfig handles runtime changes afterwards.

5.2 Snapshots are full state, not deltas

Every poll sends the complete listener set. Idempotent, and reconciliation-by-diff on the
hub eliminates an entire category of state-drift bugs. At this scale the redundancy costs
nothing.

5.3 Types

package wire

type Listener struct {
    Port     uint16 `json:"port"`
    BindAddr string `json:"bind_addr"`          // "0.0.0.0", "127.0.0.1", "::", or specific IP
    IPv6     bool   `json:"ipv6"`
    PID      int    `json:"pid,omitempty"`      // phase 5
    Process  string `json:"process,omitempty"`  // phase 5
}

type Snapshot struct {
    Listeners []Listener `json:"listeners"`
    Seq       uint64     `json:"seq"`
}

type Hello struct {
    Version  string     `json:"version"`
    Hostname string     `json:"hostname"`
    Arch     string     `json:"arch"`
    Kernel   string     `json:"kernel"`
    Baseline []Listener `json:"baseline"`
}

type Config struct {
    PollInterval  Duration `json:"poll_interval"`
    LogLevel      string   `json:"log_level"`
    IgnorePorts   []uint16 `json:"ignore_ports,omitempty"`
    IgnoreRanges  []Range  `json:"ignore_ranges,omitempty"`
}

type LogRecord struct {
    Time  time.Time      `json:"t"`
    Level string         `json:"lvl"`
    Msg   string         `json:"msg"`
    Attrs map[string]any `json:"attrs,omitempty"`
}

type LogBatch struct {
    Records []LogRecord `json:"records"`
    Dropped uint64      `json:"dropped,omitempty"`
}

5.4 Library choice — still open

github.com/creachadair/jrpc2 vs github.com/sourcegraph/jsonrpc2. Both handle symmetric
peers and peer-initiated notifications.

jrpc2 is currently favoured because its channel constructors take a separate reader and
writer, which maps directly onto session.StdoutPipe() / session.StdinPipe().
sourcegraph/jsonrpc2 wants a single io.ReadWriteCloser, requiring a trivial adapter
joining the two halves — five lines, not a real obstacle, but one fewer thing.

Worth a 20-minute spike against the actual message set before committing, specifically
checking how each handles high-frequency notifications with backpressure, since that is
what the log path needs.

6. Stdio transport — required care

Three non-obvious failure modes:

6.1 Stdout must be sacred

One stray fmt.Println, one dependency that logs to stdout, and the JSON-RPC framing is
corrupt. Apply the standard LSP mitigation at agent startup:

Dup the real stdout to a private fd, use that fd for the RPC stream.
Reassign os.Stdout = os.Stderr.

After this, accidental stdout writes are harmless and visible rather than stream-corrupting.
Go panics already go to stderr and are fine.

6.2 The hub must actively drain stderr

SSH channels are windowed, and stderr is extended-data on the same channel as stdout.
If the hub does not continuously read the stderr pipe, the window fills and stalls the
entire channel — including the RPC stream. The failure looks like a total hang with nothing
obviously wrong on the stdout side.

6.3 No buffering

Write and flush per message on both sides.

7. Log streaming

Implemented in internal/logstream as a custom slog.Handler that fans out to both sinks.
The rest of the agent code just uses slog normally and never knows where records go.

Phase 1 (bootstrap and fallback) — stderr. Everything before agent.hello completes,
plus anything emitted after the RPC path fails. Reaches the hub via the exec channel's
stderr, so a total RPC failure is still debuggable. The hub prefixes these distinctly
(source=agent-stderr) so "the agent's logging subsystem is broken" is distinguishable
from normal operation.

Phase 2 (steady state) — agent.log notifications. Structured, level-filtered, merged
into the hub's own slog output with source=agent.

Details that matter:

Never block the poll loop. Log emission goes through a buffered channel with
drop-on-full semantics. Emit a dropped=N count on the next batch rather than blocking
or silently losing records.
Recursion guard. A failure inside the log transport must never be logged through
the log transport. That path goes to stderr only, rate-limited to ~once per 10s.
Batch on a ~100 ms flush interval, so a chatty burst is one notification, not fifty.
Clock skew. Keep the agent's timestamp as t and attach hub receipt time as a
separate attr. VM clocks drift; both are needed when ordering looks wrong.
Dynamic level via hub.setConfig — raise the agent to debug at runtime, no reconnect,
no restart. This is the feature that will get the most use; wire it to a hub CLI
subcommand or signal handler early.

No pre-connect ring buffer is needed (see §2.6) — stderr is connected before the agent's
first instruction executes.

8. Port detection (`internal/procnet`)

Parse /proc/net/tcp and /proc/net/tcp6 on a poll loop. The details that will bite:

Filter on st == 0A (TCP_LISTEN).
IPv4 local_address is byte-reversed (little-endian): 0100007F is 127.0.0.1,
not 1.0.0.127. IPv6 is reversed per 32-bit word.
Dedup across both files. A dual-stack listener bound to :: appears only in tcp6
but accepts IPv4 too; one bound to 0.0.0.0 appears only in tcp. Key the set on port
and keep the most permissive bind address seen.
Bind address determines the dial target. 0.0.0.0 and 127.0.0.1 → dial
127.0.0.1:N over direct-tcpip. A listener bound to a specific non-loopback interface
IP must be dialled at that IP; localhost will not reach it.
Baseline snapshot at agent startup. Everything already listening when the agent starts
(sshd, systemd-resolved, chrony, docker-proxy, …) goes on an implicit ignore list. This is
what makes the tool zero-config in practice — only ports opened by actual dev work ever
surface.
Debounce removals. A port must be absent for 2–3 consecutive polls before teardown.
Dev servers flap ports constantly on restart (every Vite full reload, every dotnet watch
rebuild) and listener churn on each one is not wanted.
Poll interval 500 ms – 1 s. There is no kernel notification for "a new listener
appeared" short of eBPF. Netlink sock_diag queries faster but still requires polling.
Reading a ~50-line text file at 1 Hz is free. eBPF is explicitly out of scope.

9. SSH layer (`internal/sshx`)

Built on golang.org/x/crypto/ssh plus github.com/pkg/sftp.

9.1 Connection

Auth via SSH_AUTH_SOCK (golang.org/x/crypto/ssh/agent), falling back to a configured
key file.
Host key verification via golang.org/x/crypto/ssh/knownhosts. Four lines — do not skip.
x/crypto/ssh does not parse ~/.ssh/config. Options were pulling in
github.com/kevinburke/ssh_config or defining a own config format. Chose own config:
thumb-specific settings (filters, collision policy, log level) are needed anyway, and two
sources of truth is worse than one.
Keepalives: send keepalive@openssh.com global requests every 20 s; a failed reply means
the connection is dead.

9.2 Agent deployment

Run uname -m over a session channel → x86_64 / aarch64.
SFTP-stat ~/.cache/thumb/agent-<contenthash>; upload and chmod 0755 if absent.
Content-hash naming makes upgrades automatic and old versions trivially GC-able.
Exec ~/.cache/thumb/agent-<contenthash> --stdio.
Attach stdin/stdout to the JSON-RPC peer; continuously drain stderr into hub logs.

9.3 Agent lifetime

Not to survive disconnect, by design. Three independent mechanisms, all cheap:

stdin EOF when the channel closes — load-bearing, the agent's read loop ends and it
exits.
sshd SIGHUPs exec'd processes on disconnect.
hub.ping watchdog — exit if no ping within 3× the ping interval. Belt-and-braces.

10. Data plane (`internal/forward`)

On port added (e.g. 5173, bind 0.0.0.0):

net.Listen("tcp", "127.0.0.1:5173").
On EADDRINUSE, apply the collision policy: default is a fixed offset (+10000 →
15173), falling back to an ephemeral port. Log the mapping loudly.
Per accepted connection: client.Dial("tcp", "127.0.0.1:5173") (opens a direct-tcpip
channel), then bidirectional copy.
Handle half-close properly — io.Copy in both directions with CloseWrite() on the
SSH channel when the local side EOFs. Without this, long-lived connections (SSE,
websockets, dotnet watch reload channels) hang on teardown.

On port removed: close the listener immediately; let in-flight connections drain with a
~5 s grace period before force-closing.

Mapping discoverability: because collisions remap ports, the actual mapping must be
inspectable. Write ~/.local/state/thumb/forwards.json on every change; thumb status
reads it. No IPC needed.

11. Bazel

Standard gazelle + rules_go for all Go targets.
No proto_library / go_proto_library — removing protoc from the build graph is one
of the concrete wins of the JSON-RPC decision.
The one non-obvious part: embedding cross-built agent binaries into the hub. Use
go_cross_binary to build //tools/thumb/cmd/thumb-agent for linux/amd64 and
linux/arm64, then feed both into the hub's embedsrcs. The agent must be fully built
before the hub compiles, and this needs platform transitions, not just GOOS/GOARCH
environment variables.
Keep the agent CGO-free (pure-Go everything, stdlib only where possible) so both
cross-builds are static and run on any VM regardless of libc.

12. Phasing

Phase	Deliverable	Done when
0	`wire` types + `procnet` parser	Golden-file tests pass against captured `/proc/net/tcp{,6}` samples including dual-stack, IPv6, and odd bind addresses
1	SSH connect + auth + knownhosts, exec agent, JSON-RPC over stdio, log streaming on both paths, stderr drain	Agent manually `scp`'d and exec'd; `agent.hello` lands and agent logs appear interleaved in hub output
2	Snapshot → reconciliation + data plane	`npm run dev` on the VM, browser on the laptop reaches `localhost:5173`
3	Auto-deploy: `uname`, SFTP, `go:embed`, content-hash versioning	Agent binary never manually touched again
4	Reconnect loop, keepalives, removal debounce, collision mapping, state file	Survives laptop suspend/resume and VM reboot with no intervention
5	Polish: pid→process name via `/proc/*/fd`, `thumb status`, runtime log-level control, systemd user unit for the hub	You forget it exists

Logging is deliberately in Phase 1, not Phase 5. It is the debugging substrate for
phases 2–4; building it afterwards means doing those phases blind.

13. Testing

procnet and wire: pure unit tests with golden files. Capture real
/proc/net/tcp{,6} content from the dev VM in interesting states (dual-stack listener,
IPv6-only, bound to a specific interface, thousands of entries) and commit them.
SSH layer: needs a real sshd. x/crypto/ssh's in-process server does not exercise
sshd's actual exec-channel and windowing behaviour, which is precisely what is under test.
Use a containerized OpenSSH for integration tests.
Keep integration tests out of the default test target so bazel test //... stays fast.
One integration test worth writing early: fill the stderr window without draining it and
assert the RPC stream stalls — locks in the §6.2 requirement so it cannot silently
regress.

14. Open items

JSON-RPC library: jrpc2 vs sourcegraph/jsonrpc2 (§5.4). Spike before Phase 1.
Collision policy default: fixed +10000 offset vs ephemeral-with-lookup. Offset is
more predictable and more memorable; ephemeral never fails. Currently leaning offset with
ephemeral fallback.
Port filters: whether the ignore-list needs anything beyond the startup baseline in
practice. Defer until it's actually annoying.
Multi-VM: the design assumes one VM. Nothing forbids running several hub instances,
but shared local port-space collision handling across them is unsolved. Out of scope
unless it comes up.

# thumb — Implementation Plan **Status:** planning complete, implementation not started **Location:** `tools/thumb/` in the tools monorepo **Language:** Go, built with Bazel (`rules_go` + `gazelle`) **Binaries:** `thumb` (hub, local machine), `thumb-agent` (VM) --- ## 1. Problem A cloud dev VM is used for regular software development. Commands like `npm run dev`, `dotnet run`, or `git fetch` with an OAuth credential helper open TCP listeners on the VM — sometimes on well-known ports, sometimes on random high ports. Those services need to be reachable from the local machine. Requirements: - **Zero maintenance.** No manual action when a port opens or closes. - **Detection happens on the VM.** Only the VM can see what's listening. - **Single TCP connection.** One outbound SSH connection from the local machine, nothing else. - **No inbound reachability to the local machine.** This is what rules out the current setup. The current `bore`-based solution fails the last two: it needs the `bore` server reachable from the VM on a set of random high ports as part of its handshake, which means the local machine has to be directly connectable. Performance is explicitly not a concern. --- ## 2. Design decisions and why Recorded because the reasoning matters more than the conclusions if any of this gets revisited later. ### 2.1 SSH as the transport, not QUIC or a custom protocol QUIC is UDP-based and awkward to forward through an SSH tunnel. SSH already multiplexes arbitrary channels over one TCP connection and has battle-tested forwarding primitives (`direct-tcpip`, `tcpip-forward`). There is no reason to reimplement any of that. ### 2.2 Not ControlMaster + `ssh -O forward` A viable approach exists using OpenSSH alone: remote-forward the ControlMaster socket itself to the VM, then have a VM-side daemon issue `ssh -S <forwarded-socket> -O forward` commands against it. This works and requires no custom binary. Rejected because it means shelling out to the `ssh` CLI and parsing exit codes for every port event, and because the ControlMaster mux protocol is an undocumented OpenSSH-internal IPC format (`mux.c`), versioned per release, with no third-party client library in any language. Speaking it natively is not a realistic target. Once you're willing to run a custom binary on the VM, defining your own control protocol is strictly simpler than reverse-engineering OpenSSH's. ### 2.3 Control plane and data plane run in opposite directions This is the single most important thing to keep straight: - **Data plane is `-L`-shaped.** The hub listens on `localhost:N` locally and opens a `direct-tcpip` channel to the VM's `localhost:N`. Driven entirely by the hub. The agent is not involved and no VM-side cooperation is needed beyond the port existing. - **Control plane is hub↔agent messaging.** Its only purpose is letting the agent tell the hub that a port appeared or disappeared, plus carrying agent logs and hub config. All bulk traffic uses ordinary SSH forwarding. The control protocol carries only small JSON messages. ### 2.4 Go over Rust Considered Rust (existing Rust + Bazel + LLVM cross-compilation setup in the monorepo, and `bore` was Rust). Chose Go because: - `golang.org/x/crypto/ssh` is extremely battle-tested — Tailscale SSH, Teleport, and most Go infra tooling build on it. `Client.Dial` (direct-tcpip) is first-class. - `GOOS=linux GOARCH=arm64 go build` with no toolchain setup, no cgo, produces a static binary. This matters concretely for the upload-and-exec deployment flow, where the agent must just work on whatever arch the VM is without fussing over glibc/musl or target triples. - `/proc/net/tcp` parsing is trivial in either language. - The only Rust-specific asset was the `bore` crate, and `bore` is gone from the design. ### 2.5 JSON-RPC over gRPC The deployment model kills gRPC's main advantage. The hub embeds a content-hashed agent binary and uploads it, so hub and agent are always the same build — **version skew is structurally impossible**, and protobuf's field-number discipline is insurance against a scenario that cannot occur. Beyond that: - The traffic is notification-heavy peer-to-peer messaging, which is JSON-RPC's native model. Only one exchange (`agent.hello` → config) is a real request/response. gRPC would mean tunnelling all of this through a single long-lived bidi stream — a streaming RPC used as a message bus. - Dropping `proto_library` / `go_proto_library` removes the protoc toolchain from the Bazel graph, which is one of the fiddlier parts of a `rules_go` setup — especially combined with the platform transitions already needed for the embedded agent. - JSON overhead is irrelevant: a few dozen listeners at 1 Hz. - Debuggable by eye. ### 2.6 Stdio, not a forwarded Unix socket Earlier drafts of this plan remote-forwarded a Unix domain socket (`streamlocal-forward@openssh.com`) to carry the control protocol. **That is unnecessary.** The agent is exec'd over an SSH session channel, and that channel already provides stdin, stdout, and stderr. Stdout and stdin together form a full-duplex byte stream — exactly what JSON-RPC needs. This is how every LSP server works. Note the distinction: it is *not* that hub→agent signalling is unneeded (`hub.setConfig` for dynamic log level is a real feature). It's that **stdin already provides it**, making the forwarded socket pure redundancy. Deleting the socket removes: - The `streamlocal-forward@openssh.com` global request and `forwarded-streamlocal` channel-open handling — the one genuinely hand-rolled, no-library-support part of the entire SSH layer. - The channel→`net.Listener` adapter and the `net.Conn` wrapper with its fake addresses and no-op deadline methods. - Socket path generation, the stale-socket janitor, and `StreamLocalBindUnlink` concerns. - A dependency on sshd's `AllowStreamLocalForwarding` being enabled — worth calling out separately, since a hardened VM image could have it off and the forward would silently not work. - The pre-connect log ring buffer. With stdio the stream is connected *before the process starts executing*; there is no dial that can fail and no window where logs have nowhere to go. - Two of the three agent-lifetime mechanisms. Reading stdin returns EOF when the channel closes, so the agent exits naturally. ### 2.7 Logs travel in the protocol, not just on stderr Stderr already flows hub-ward for free, but gives unstructured text with no levels, no filtering, and no correlation with hub-side events. The protocol path gives structured records, hub-controlled verbosity, and a single merged timeline. **Both are retained** — they cover different failure modes. Stderr is the fallback that still works when the RPC layer itself is broken. --- ## 3. Architecture ``` local machine (hub) VM (agent) ┌──────────────────────────┐ ┌────────────────────────┐ │ JSON-RPC peer │ │ JSON-RPC peer │ │ reads session stdout ◄─┼── ndjson ────────┤ writes stdout │ │ writes session stdin ──┼── ndjson ───────►│ reads stdin │ │ drains session stderr ◄┼── fallback logs ─┤ stderr │ │ │ │ │ │ │ │ polls /proc/net/tcp* │ │ │ │ slog → agent.log │ │ │ │ │ │ localhost:5173 ──────────┼── direct-tcpip ──┼─► localhost:5173 │ │ localhost:5000 ──────────┼── direct-tcpip ──┼─► localhost:5000 │ └──────────────────────────┘ └────────────────────────┘ one TCP connection, one SSH session, one exec channel + N forward channels ``` Nothing binds a Unix socket. Nothing touches the local filesystem for IPC. No other local process can reach the control channel. --- ## 4. Repository layout ``` tools/thumb/ ├── README.md ├── PLAN.md # this document ├── BUILD.bazel ├── cmd/ │ ├── thumb/ # hub binary │ └── thumb-agent/ # agent binary └── internal/ ├── wire/ # shared message types, method names, ndjson framing ├── config/ # hub config file parsing ├── sshx/ # connect, auth, knownhosts, exec, sftp, keepalive ├── hub/ # session lifecycle, reconnect loop, reconciliation ├── forward/ # local listeners + direct-tcpip data plane ├── procnet/ # /proc/net/tcp{,6} parsing ├── agent/ # poll loop, RPC peer, lifetime management ├── logstream/ # slog handler → RPC notifications + stderr fanout └── embedbin/ # go:embed of cross-built agent binaries ``` **Filesystem paths:** | Purpose | Path | |---|---| | Agent binary cache (VM) | `~/.cache/thumb/agent-<contenthash>` | | Hub forward state | `~/.local/state/thumb/forwards.json` | | Hub config | `~/.config/thumb/config.toml` | --- ## 5. Protocol Peer-to-peer JSON-RPC 2.0, newline-delimited JSON framing, over the exec channel's stdout (agent→hub) and stdin (hub→agent). Message types live in `internal/wire/` and are imported by both binaries — single source of truth, no codegen step. ### 5.1 Methods **Agent → Hub** | Method | Kind | Payload | |---|---|---| | `agent.hello` | request | version, hostname, arch, kernel, baseline listener set | | `agent.snapshot` | notification | complete current listener set + monotonic seq | | `agent.log` | notification | batch of log records | **Hub → Agent** | Method | Kind | Payload | |---|---|---| | `hub.setConfig` | notification | poll interval, port filters, log level | | `hub.ping` | request | liveness check (belt-and-braces; stdin EOF is load-bearing) | | `hub.shutdown` | notification | graceful exit request | `agent.hello` returns the initial config in its **response**, so there is no window where the agent runs unconfigured. `hub.setConfig` handles runtime changes afterwards. ### 5.2 Snapshots are full state, not deltas Every poll sends the complete listener set. Idempotent, and reconciliation-by-diff on the hub eliminates an entire category of state-drift bugs. At this scale the redundancy costs nothing. ### 5.3 Types ```go package wire type Listener struct { Port uint16 `json:"port"` BindAddr string `json:"bind_addr"` // "0.0.0.0", "127.0.0.1", "::", or specific IP IPv6 bool `json:"ipv6"` PID int `json:"pid,omitempty"` // phase 5 Process string `json:"process,omitempty"` // phase 5 } type Snapshot struct { Listeners []Listener `json:"listeners"` Seq uint64 `json:"seq"` } type Hello struct { Version string `json:"version"` Hostname string `json:"hostname"` Arch string `json:"arch"` Kernel string `json:"kernel"` Baseline []Listener `json:"baseline"` } type Config struct { PollInterval Duration `json:"poll_interval"` LogLevel string `json:"log_level"` IgnorePorts []uint16 `json:"ignore_ports,omitempty"` IgnoreRanges []Range `json:"ignore_ranges,omitempty"` } type LogRecord struct { Time time.Time `json:"t"` Level string `json:"lvl"` Msg string `json:"msg"` Attrs map[string]any `json:"attrs,omitempty"` } type LogBatch struct { Records []LogRecord `json:"records"` Dropped uint64 `json:"dropped,omitempty"` } ``` ### 5.4 Library choice — still open `github.com/creachadair/jrpc2` vs `github.com/sourcegraph/jsonrpc2`. Both handle symmetric peers and peer-initiated notifications. `jrpc2` is currently favoured because its channel constructors take a **separate reader and writer**, which maps directly onto `session.StdoutPipe()` / `session.StdinPipe()`. `sourcegraph/jsonrpc2` wants a single `io.ReadWriteCloser`, requiring a trivial adapter joining the two halves — five lines, not a real obstacle, but one fewer thing. Worth a 20-minute spike against the actual message set before committing, specifically checking how each handles high-frequency notifications with backpressure, since that is what the log path needs. --- ## 6. Stdio transport — required care Three non-obvious failure modes: ### 6.1 Stdout must be sacred One stray `fmt.Println`, one dependency that logs to stdout, and the JSON-RPC framing is corrupt. Apply the standard LSP mitigation at agent startup: 1. Dup the real stdout to a private fd, use that fd for the RPC stream. 2. Reassign `os.Stdout = os.Stderr`. After this, accidental stdout writes are harmless and visible rather than stream-corrupting. Go panics already go to stderr and are fine. ### 6.2 The hub must actively drain stderr SSH channels are windowed, and stderr is extended-data on the **same channel** as stdout. If the hub does not continuously read the stderr pipe, the window fills and stalls the entire channel — including the RPC stream. The failure looks like a total hang with nothing obviously wrong on the stdout side. ### 6.3 No buffering Write and flush per message on both sides. --- ## 7. Log streaming Implemented in `internal/logstream` as a custom `slog.Handler` that fans out to both sinks. The rest of the agent code just uses `slog` normally and never knows where records go. **Phase 1 (bootstrap and fallback) — stderr.** Everything before `agent.hello` completes, plus anything emitted after the RPC path fails. Reaches the hub via the exec channel's stderr, so a total RPC failure is still debuggable. The hub prefixes these distinctly (`source=agent-stderr`) so "the agent's logging subsystem is broken" is distinguishable from normal operation. **Phase 2 (steady state) — `agent.log` notifications.** Structured, level-filtered, merged into the hub's own `slog` output with `source=agent`. Details that matter: - **Never block the poll loop.** Log emission goes through a buffered channel with drop-on-full semantics. Emit a `dropped=N` count on the next batch rather than blocking or silently losing records. - **Recursion guard.** A failure inside the log transport must never be logged *through* the log transport. That path goes to stderr only, rate-limited to ~once per 10s. - **Batch on a ~100 ms flush interval**, so a chatty burst is one notification, not fifty. - **Clock skew.** Keep the agent's timestamp as `t` and attach hub receipt time as a separate attr. VM clocks drift; both are needed when ordering looks wrong. - **Dynamic level via `hub.setConfig`** — raise the agent to debug at runtime, no reconnect, no restart. This is the feature that will get the most use; wire it to a hub CLI subcommand or signal handler early. No pre-connect ring buffer is needed (see §2.6) — stderr is connected before the agent's first instruction executes. --- ## 8. Port detection (`internal/procnet`) Parse `/proc/net/tcp` and `/proc/net/tcp6` on a poll loop. The details that will bite: - **Filter on `st == 0A`** (`TCP_LISTEN`). - **IPv4 `local_address` is byte-reversed** (little-endian): `0100007F` is `127.0.0.1`, not `1.0.0.127`. IPv6 is reversed per 32-bit word. - **Dedup across both files.** A dual-stack listener bound to `::` appears only in `tcp6` but accepts IPv4 too; one bound to `0.0.0.0` appears only in `tcp`. Key the set on port and keep the most permissive bind address seen. - **Bind address determines the dial target.** `0.0.0.0` and `127.0.0.1` → dial `127.0.0.1:N` over `direct-tcpip`. A listener bound to a specific non-loopback interface IP must be dialled at that IP; `localhost` will not reach it. - **Baseline snapshot at agent startup.** Everything already listening when the agent starts (sshd, systemd-resolved, chrony, docker-proxy, …) goes on an implicit ignore list. This is what makes the tool zero-config in practice — only ports opened by actual dev work ever surface. - **Debounce removals.** A port must be absent for 2–3 consecutive polls before teardown. Dev servers flap ports constantly on restart (every Vite full reload, every `dotnet watch` rebuild) and listener churn on each one is not wanted. - **Poll interval 500 ms – 1 s.** There is no kernel notification for "a new listener appeared" short of eBPF. Netlink `sock_diag` queries faster but still requires polling. Reading a ~50-line text file at 1 Hz is free. eBPF is explicitly out of scope. --- ## 9. SSH layer (`internal/sshx`) Built on `golang.org/x/crypto/ssh` plus `github.com/pkg/sftp`. ### 9.1 Connection - Auth via `SSH_AUTH_SOCK` (`golang.org/x/crypto/ssh/agent`), falling back to a configured key file. - Host key verification via `golang.org/x/crypto/ssh/knownhosts`. Four lines — do not skip. - **`x/crypto/ssh` does not parse `~/.ssh/config`.** Options were pulling in `github.com/kevinburke/ssh_config` or defining a own config format. **Chose own config:** thumb-specific settings (filters, collision policy, log level) are needed anyway, and two sources of truth is worse than one. - Keepalives: send `keepalive@openssh.com` global requests every 20 s; a failed reply means the connection is dead. ### 9.2 Agent deployment 1. Run `uname -m` over a session channel → `x86_64` / `aarch64`. 2. SFTP-stat `~/.cache/thumb/agent-<contenthash>`; upload and `chmod 0755` if absent. Content-hash naming makes upgrades automatic and old versions trivially GC-able. 3. Exec `~/.cache/thumb/agent-<contenthash> --stdio`. 4. Attach stdin/stdout to the JSON-RPC peer; continuously drain stderr into hub logs. ### 9.3 Agent lifetime Not to survive disconnect, by design. Three independent mechanisms, all cheap: 1. **stdin EOF** when the channel closes — load-bearing, the agent's read loop ends and it exits. 2. **sshd SIGHUPs** exec'd processes on disconnect. 3. **`hub.ping` watchdog** — exit if no ping within 3× the ping interval. Belt-and-braces. --- ## 10. Data plane (`internal/forward`) **On port added** (e.g. 5173, bind `0.0.0.0`): 1. `net.Listen("tcp", "127.0.0.1:5173")`. 2. On `EADDRINUSE`, apply the collision policy: default is a fixed offset (`+10000` → `15173`), falling back to an ephemeral port. Log the mapping loudly. 3. Per accepted connection: `client.Dial("tcp", "127.0.0.1:5173")` (opens a `direct-tcpip` channel), then bidirectional copy. 4. **Handle half-close properly** — `io.Copy` in both directions with `CloseWrite()` on the SSH channel when the local side EOFs. Without this, long-lived connections (SSE, websockets, `dotnet watch` reload channels) hang on teardown. **On port removed:** close the listener immediately; let in-flight connections drain with a ~5 s grace period before force-closing. **Mapping discoverability:** because collisions remap ports, the actual mapping must be inspectable. Write `~/.local/state/thumb/forwards.json` on every change; `thumb status` reads it. No IPC needed. --- ## 11. Bazel - Standard `gazelle` + `rules_go` for all Go targets. - **No `proto_library` / `go_proto_library`** — removing protoc from the build graph is one of the concrete wins of the JSON-RPC decision. - **The one non-obvious part:** embedding cross-built agent binaries into the hub. Use `go_cross_binary` to build `//tools/thumb/cmd/thumb-agent` for `linux/amd64` and `linux/arm64`, then feed both into the hub's `embedsrcs`. The agent must be fully built before the hub compiles, and this needs **platform transitions**, not just `GOOS`/`GOARCH` environment variables. - Keep the agent **CGO-free** (pure-Go everything, stdlib only where possible) so both cross-builds are static and run on any VM regardless of libc. --- ## 12. Phasing | Phase | Deliverable | Done when | |---|---|---| | **0** | `wire` types + `procnet` parser | Golden-file tests pass against captured `/proc/net/tcp{,6}` samples including dual-stack, IPv6, and odd bind addresses | | **1** | SSH connect + auth + knownhosts, exec agent, JSON-RPC over stdio, **log streaming on both paths**, stderr drain | Agent manually `scp`'d and exec'd; `agent.hello` lands and agent logs appear interleaved in hub output | | **2** | Snapshot → reconciliation + data plane | `npm run dev` on the VM, browser on the laptop reaches `localhost:5173` | | **3** | Auto-deploy: `uname`, SFTP, `go:embed`, content-hash versioning | Agent binary never manually touched again | | **4** | Reconnect loop, keepalives, removal debounce, collision mapping, state file | Survives laptop suspend/resume and VM reboot with no intervention | | **5** | Polish: pid→process name via `/proc/*/fd`, `thumb status`, runtime log-level control, systemd user unit for the hub | You forget it exists | **Logging is deliberately in Phase 1, not Phase 5.** It is the debugging substrate for phases 2–4; building it afterwards means doing those phases blind. --- ## 13. Testing - **`procnet` and `wire`:** pure unit tests with golden files. Capture real `/proc/net/tcp{,6}` content from the dev VM in interesting states (dual-stack listener, IPv6-only, bound to a specific interface, thousands of entries) and commit them. - **SSH layer:** needs a **real sshd**. `x/crypto/ssh`'s in-process server does not exercise sshd's actual exec-channel and windowing behaviour, which is precisely what is under test. Use a containerized OpenSSH for integration tests. - Keep integration tests **out of the default test target** so `bazel test //...` stays fast. - One integration test worth writing early: fill the stderr window without draining it and assert the RPC stream stalls — locks in the §6.2 requirement so it cannot silently regress. --- ## 14. Open items 1. **JSON-RPC library:** `jrpc2` vs `sourcegraph/jsonrpc2` (§5.4). Spike before Phase 1. 2. **Collision policy default:** fixed `+10000` offset vs ephemeral-with-lookup. Offset is more predictable and more memorable; ephemeral never fails. Currently leaning offset with ephemeral fallback. 3. **Port filters:** whether the ignore-list needs anything beyond the startup baseline in practice. Defer until it's actually annoying. 4. **Multi-VM:** the design assumes one VM. Nothing forbids running several hub instances, but shared local port-space collision handling across them is unsolved. Out of scope unless it comes up.

PLAN.md

22 KiB

prskr changed title from ~~tool: portal~~ to tool: thumb

2026-08-05 19:40:39 +00:00

No labels

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

prskr/tools#24

No description provided.

Rows
Columns