Running the Engine: Bring-Up, Watch, and the Edge of the Public Story
Part VII — Running the Engine · seven builds, seven receipts
The keeper's chapter earns keeper's experiments: every build below produces a receipt an operator could hand to her relief shift — a captured 503 body, a Prometheus scrape value, a stderr line from init_app_bundles, a measured ready-flip latency. Treat the chapter's BLD-CH18-* receipts as the baselines you are matching or beating.
Run it live — GQL console
Point at the public read-only instance (no key needed) or your own engine
(start it with GIGI_CORS_ORIGIN=* for browser access — dev only).
—
E18.1Reproduce the listen-first / replay-after handshake
Build
A local rehearsal of the bring-up sequence: launch gigi-stream against a GIGI_DATA_DIR with a non-trivial WAL (write a bundle, push \(\geq\)1000 records, kill -9), restart, and instrument the window between TcpListener::bind and state.ready = true from the outside.
Receipt
A small script (or pair of curls) that records three timestamps from a single boot — first nc -z localhost 3142 success, first /v1/health returning 503 with status: "loading" and loading: true, and first /v1/health returning 200 — with the deltas tabulated against BLD-CH18-MAIN-LISTEN-FIRST and BLD-CH18-READINESS-MW. The middle row must be a 503, not a refused connection; that is the foghorn working.
Bonus
Truncate the tail of the last WAL segment by 17 bytes before the restart, repeat the measurement, and verify the ready-flip still happens. Note any new stderr line from the WAL reader; that line is the engine admitting it skipped your corruption.
E18.2Instrument the readiness gate's two layers under skew
Build
A test harness that pins the readiness middleware (BLD-CH18-READINESS-MW at gigi_stream.rs:1206) and the health handler (BLD-CH18-HEALTH-503 at gigi_stream.rs:1567) against each other during boot — two concurrent curl loops, one hitting /v1/health, one hitting /v1/query with a trivial SELECT, both started before state.ready flips.
Receipt
A log showing the exact request at which both loops cross from 503 to 200, with the request indices written to the same line. If the middleware and the health handler ever disagree in the same instant — one returning 200 while the other returns 503 — you have caught an Acquire/Release ordering bug. The chapter promises they flip together; verify it.
Bonus
Replace Ordering::Acquire with Ordering::Relaxed in a local fork, rerun, and write up whether the disagreement window opens on your hardware.
E18.3Wire the geometric column into a real Grafana panel
Build
Stand up Prometheus against http://localhost:3142/v1/metrics (Accept: text/plain) at a 15-second scrape, then build a Grafana dashboard with three panels: average holonomy_norm over the last 5 minutes, count of bundles whose k_global sits outside their published 2\(\sigma\) band, and the geometric.k_global IS NULL rate computed off the JSON /v1/bundles/{name}/health surface for a single named bundle.
Receipt
A screenshot of the dashboard with all three panels populated against a running engine that has at least one IMAGINE or flat_transport call in-flight, plus the PromQL for each panel committed alongside the dashboard JSON. The third panel must be non-zero exactly when you bypass the substrate — prove that by running a query through a non-geometric path and watching the rate climb.
Bonus
Add an alert rule that fires when k_global for any bundle exceeds its threshold_3sigma from the per-bundle health endpoint; trigger it by deliberately ingesting an out-of-band record.
E18.4Stress the seed_env footgun, then close it
Build
Reproduce the destructive path I named: deploy a GIGI_APP_BUNDLES manifest with one encrypted bundle whose seed_env points at an unset variable, write a record, restart the engine without setting the variable, and confirm the prior ciphertext is unrecoverable. Then patch init_app_bundles to gate the fallback behind a GIGI_STRICT_SEEDS=1 env flag that turns the stderr warning into a fatal boot error.
Receipt
Two stderr captures from the same machine: the first showing the original [app-bundles] warning with SET THE SECRET FOR PRODUCTION in caps, the second showing the new strict-mode boot refusal. A passing test in tests/ that asserts strict mode refuses to start when any manifest entry has seed_env naming an unset variable.
Bonus
Wire the strict-mode fork into the ci.yml.disabled pipeline (still disabled) so future-me sees the safeguard the day I turn CI back on.
E18.5Replay a volume-loss cold start using only the public surface
Build
On a staging machine, force a snapshot rotation so the background tigris_push (BLD-CH18-TIGRIS-PUSH-BG) fires; confirm the bucket contents from outside. Then destroy the local /data mount, restart the process, and ride the cold-start path I shipped (BLD-CH18-TIGRIS-PULL): Tigris pull, WAL replay, snapshot write, mmap reopen, ready-flip.
Receipt
A wall-clock table comparing warm restart (from E18.1) against this cold-start replay, with the three sub-phases broken out from /v1/health polling alone — no SSH into the machine, only /v1/health and /v1/metrics. The 13 GB heap \(\rightarrow\) 200 MB RSS drop (gigi_stream.rs:14744–14745) must show up in your scrape; record the two RSS values.
Bonus
Run E18.5 with a deliberately fat bucket and measure how close you come to the 900-second grace_period I set in fly.toml. Report the safety margin.
E18.6Time the SIGTERM graceful-shutdown contract
Build
A driver that opens \(N\) slow in-flight requests against /v1/query (use a SELECT that scans a large bundle) while a second process sends SIGTERM to gigi-stream. Verify the engine stops accepting new connections immediately, lets every in-flight request finish, flushes the WAL, closes the mmap, and exits zero — exactly the four-step contract.
Receipt
A captured exit code (must be 0), counts of completed vs. refused requests during the shutdown window (refused must be the new ones, completed must be the in-flight ones), and a final WAL byte count matching the last accepted record. No 503s should arrive between SIGTERM and exit — the engine closes the door, it does not foghorn on the way out.
Bonus
Wire this exercise behind flyctl machine stop against a staging machine and confirm the Fly rolling-deploy strategy never drops a client request mid-handoff to the new machine.
E18.7Refute the geometric column on a different substrate
Build
Pick any non-geometric database you already have (Postgres, DuckDB, a vector store, your choice) and try to populate the same eight non-null slots of GeometricFields — k_global through jensen_shannon — for a single write event. You may compute the curvature scalars however the substrate allows.
Receipt
A short writeup of what you could and could not emit, and what scaffolding you had to bolt on outside the substrate to produce the column. If you could produce all eight without leaning on a sidecar that re-implements bundle algebra, file a bug against my chapter — I claimed the column required the substrate, and you have refuted me. Otherwise, the writeup is the receipt that the column is the substrate's, not the logger's.