zygaena / zyginit
|
Files Changesets Issues Pull Requests Releases
root / docs / superpowers / specs / 2026-05-02-contract-readoption-design.md
2026-05-02-contract-readoption-design.md markdown 369 lines 25.6 KB
Raw

Contract Re-adoption (Live Replace) Design

Date: 2026-05-02
Status: Approved (brainstorm complete); implementation plan to follow
Scope: Live in-place replacement of the running zyginit PID-1 process — replacing the /sbin/init binary while preserving every running service across the swap. Operator-driven only; not a crash-recovery design.

1. Goal

Allow an operator (or automation tool) to upgrade the running zyginit binary without rebooting and without dropping any service. After the operation, every previously-RUNNING service continues to run with the same contract ID, accumulated restart_count, and continuous last_start_time (uptime). The connection between zyginit and supervised services is preserved across the exec; the only thing replaced is the program text and in-memory state of PID 1 itself.

The verb the operator uses is zygctl replace. The mechanism is execve("/sbin/init", ...) against the same proc-table slot, with a state-file handoff for in-memory bookkeeping.

2. Non-goals

  • Crash recovery. If zyginit segfaults / panics, the kernel's restart_init re-execs /sbin/init in a tight loop, but the contracts are not preserved across that path (NOORPHAN templates would have killed members already). PID 1 dying is a much larger issue than this design solves; we explicitly do not try.
  • Live binary swap. zygctl replace does not copy or stage a binary. The operator (or package manager / configuration management tool) is expected to drop the new binary at /sbin/init first, then call zygctl replace. We hard-wire the exec target.
  • Switching to a different init system. zygctl replace always execs /sbin/init with no path argument. Switching init systems is a rescue-media operation.
  • Signal-driven trigger. No SIGUSR2 handler. Socket is the only path. (Revisit only if a real demand surfaces; see §10.)
  • Crash-recovery argv-scan / proc-walk. No fallback that reconstructs identity from /proc/<pid>/psinfo. State file is the only source of truth.
  • Recovery for transient-state services (STARTING, STOPPING, WAITING-for-restart). replace refuses on transient state by default; with --wait=N it polls for stability before refusing. --force is not in scope for v1.
  • Preserving in-flight zygctl connections. Clients lose their connection at exec; that's expected. Operators reconnect for follow-up commands.

3. Why this works on Hammerhead — the key insight

On illumos / Hammerhead, process contracts are tracked on the process structure (proc_t.p_ct_process / p_ct_equiv). When a process calls execve(), the kernel replaces the program text and (re-)initializes most of proc_t, but the proc-table slot and contract ownership survive. This is what makes live replace possible: PID 1 is the same proc_t before and after the exec, so all contracts it held continue to be held by it.

Consequence: the new zyginit does not need ct_ctl_adopt. Adoption is for the SMF-startd model — a non-PID-1 supervisor dies and its successor (a fresh process with a different proc_t) needs to claim the orphan-inherited contracts. For PID-1 in-place exec, kernel-level ownership is automatic; we only need to rebuild the userspace bookkeeping (which contract belongs to which service, and what runtime fields to carry over).

The bundle fd (/system/contract/process/bundle) does close at exec (we don't try to inherit it via FD_CLOEXEC manipulation), so the new zyginit re-opens it. Events that fire during the gap between old-process exec and new-process bundle re-open are recovered via a post-startup ct_status_read check on each known contract — if the contract is already empty, we synthesize an empty event into the existing handle_contract_event path. This is idempotent regardless of whether the gap actually missed any events.

4. Architecture

                  Operator
                     |
                     v
              zygctl replace
              [--wait=N]
                     |
              +------+-------+
              |   socket     |   "replace queued"
              |   (sock.reef)|----------------> zygctl prints, exits
              +------+-------+
                     |  set replace_requested flag
                     v
            main.reef event loop
                     |
                     v
              replace_self()
                     |
   +-----------------+-----------------+
   | 1. preconditions (--wait optional)|
   | 2. drain pending bundle events    |
   | 3. write state.toml (atomic)      |
   | 4. unlink socket                  |
   | 5. close internal fds             |
   | 6. execve("/sbin/init", argv)     |
   +-----------------+-----------------+
                     |
                     v
          /// SAME proc_t, new program text ///
                     |
                     v
            new main() startup
                     |
   +-----------------+-----------------+
   | 1. setup_pid1_console (as today)  |
   | 2. recover_state(boot_time)       |
   |    — reads state.toml, validates  |
   |      boot_time, returns entries   |
   | 3. load enabled.d/ TOML (as today)|
   | 4. apply_recovered_state(table,   |
   |       recovered)                  |
   | 5. tier-by-tier start_service     |
   |    (skips entries already RUNNING)|
   +-----------------+-----------------+
                     |
                     v
              normal main loop

Module split:

File Change Why
src/replace.reef new (~200 lines) Holds the state-file format, serialization, and recovery logic. Keeps main.reef lean.
src/main.reef +60 lines replace_self, apply_recovered_state, integration with main loop and startup.
src/socket.reef +30 lines replace [--wait=N] command handler, replace_requested() / clear_replace_flag() / replace_wait_seconds() accessors.
src/contract.reef +20 lines is_contract_empty(ctid): bool helper wrapping ct_status_read + member-count check.
tools/zygctl/src/zygctl.reef +20 lines replace [--wait=N] subcommand.

No new C helpers. No FFI additions (adopt_contract already exists but is not used by this flow; it stays for potential future uses).

5. State file format

Location: /var/run/zyginit/state.toml. Same directory currently used for the socket. On the current zyginit-only BE, /var/run is persistent across real reboots (not yet tmpfs); the boot-id check (§5.2) makes stale post-reboot files harmless.

Format: TOML, parseable by the existing encoding.toml parser.

boot_time = 1746204000

[[service]]
name = "sshd"
contract_id = 23
restart_count = 0
last_start_time = 1746204051
last_exit_code = -1

[[service]]
name = "syslogd"
contract_id = 17
restart_count = 1
last_start_time = 1746204047
last_exit_code = 0

5.1 What is and isn't serialized

Serialized:

  • boot_time — PID 1's process start time, read from /proc/1/stat (Linux) or /proc/1/psinfo (Hammerhead). Survives execve but changes on real reboot. Used as a staleness sentinel — see §5.2.
  • For each service in STATE_RUNNING:
    • name — the canonical service name (matches enabled.d/ symlink basename).
    • contract_id — kernel-assigned contract ID.
    • restart_count — accumulated restart counter (for max_retries accounting).
    • last_start_time — Unix timestamp of the most recent start.
    • last_exit_code — most recent exit code (or -1 if never exited).

Not serialized:

  • pid — recoverable from contract status if needed; not needed for the contract-driven supervision path.
  • state — only RUNNING services serialize (per §6 pre-conditions); the field is implicit.
  • stop_requested_time, restart_scheduled_time — only meaningful in transient states, which we refuse.
  • def (the parsed TOML ServiceDef) — the new zyginit re-reads enabled.d/ from disk, so the def comes from current TOML on the new side. This means a TOML edit between old-start and replace takes effect on next restart. (See §7.3 for the divergence handling.)

5.2 Boot-id staleness check

The boot-id sentinel is PID 1's process start time, read from /proc/1/stat field 22 (Linux: jiffies-since-boot) or /proc/1/psinfo pr_start.tv_sec (illumos/Hammerhead: Unix timestamp) via zyginit_read_pid1_start() in helpers.c. This value survives execve (same proc_t slot) but changes on real reboot (new PID 1). If state.toml.boot_time != current.boot_time, the file is from a different system uptime — most likely a stale file left over from before a real reboot. We log "replace: stale state file (boot_time mismatch); ignoring", delete the file, and proceed as a fresh boot.

Why not utmpx BOOT_TIME? The original design used utmpx BOOT_TIME as the sentinel. This was found to be incorrect: pututxline with ut_type = BOOT_TIME updates the existing matching record (by design, to support who -b), rather than appending a new one. After a live replace, the new zyginit's startup call to zyginit_write_boot_utmpx() overwrites the original BOOT_TIME with a fresh timestamp. The new zyginit then reads its own updated timestamp as the current boot_time — which matches the new timestamp it just wrote — while state.toml has the old timestamp from before the replace. Result: mismatch, state.toml discarded, recovery fails silently. /proc/1/start avoids this entirely: no write occurs to it at startup, and it is immutable for the life of the PID 1 proc_t.

This guards against the edge case where contract IDs are re-used after a real reboot (the kernel does recycle them) and we'd otherwise adopt the wrong contract. The check is cheap (one integer comparison) and correct in the common case (replace = same boot_time; reboot = different boot_time).

The state file is deleted by the new zyginit after a successful read, regardless of whether all entries were applied successfully. This way, a panic-restart loop (kernel re-execing /sbin/init repeatedly) does not keep trying to apply long-dead entries.

5.3 Atomic write

Writing follows the standard pattern, using the existing zyginit_fsync_path C helper:

  1. open("/var/run/zyginit/state.toml.tmp", O_WRONLY|O_CREAT|O_TRUNC, 0600)
  2. Write boot_time + each service entry. (Reef-side TOML emit is straightforward; we don't need the full encoding.toml emit machinery, just print to fd.)
  3. fsync(fd), close(fd).
  4. rename("/var/run/zyginit/state.toml.tmp", "/var/run/zyginit/state.toml").
  5. zyginit_fsync_path("/var/run/zyginit") — fsync the parent directory so the rename is durable.

Any failure aborts the replace (we do not exec). At most we leave a .tmp file, which is overwritten on the next attempt and never read.

6. Operator interface

6.1 zygctl replace

zygctl replace [--wait=N]
  • Connects to /var/run/zyginit.sock.
  • Sends replace [--wait=N]\n.
  • Reads reply (replace queued (wait=N)\n or an error).
  • Prints reply, exits 0 on success, non-zero on parse error from server.
  • The TCP/Unix-socket connection itself drops mid-response if the server proceeds to exec. zygctl treats EPIPE / connection-closed-after-ack as success and prints replace: socket closed (process replaced)\n.

6.2 --wait=N

If any service is in a transient state (STARTING, STOPPING, WAITING-for-restart), the server polls once per second for up to N seconds, checking whether all services are in stable states (RUNNING, STOPPED, FAILED, MAINTENANCE, DISABLED). If stability is reached within the window, replace proceeds. If not, replace is refused with "replace blocked: <name>:<state>, ..." logged to syslog (and to stderr if attached). Default is --wait=0 (refuse immediately).

The wait happens server-side after the ack to the client, so the zygctl client doesn't have to hold a connection open — automation can zygctl replace --wait=120 and either get a clean dropped connection (success) or, if the wait times out without exec'ing, can detect the failure by reconnecting and inspecting service states.

6.3 Pre-conditions in detail

Before exec, zyginit checks:

  1. No service is in a transient state. (After --wait window if specified.)
  2. state.toml.tmp write succeeds, including fsync.
  3. rename to state.toml succeeds, including parent-dir fsync.

If any of these fail, replace aborts cleanly and the running zyginit continues. The flag is cleared, no exec happens, and the failure is logged. The operator can retry.

The socket unlink and fd cleanup happen after a successful state write — they are last-mile actions before exec, and we don't fail the replace if they error (we log and proceed).

7. Recovery flow (new zyginit)

7.1 Startup sequence

main():
    parse argv, kernel boot args
    if pid == 1: setup_pid1_console()
    boot_time = read_boot_time()       // /proc/1/start (pid1 start time) or time_now()

    recovered = replace.recover_state(boot_time)

    config = load_enabled_dir("/etc/zyginit/enabled.d")
    table = build_service_table(config)

    apply_recovered_state(table, recovered)    // <-- new step

    // Enter normal startup: tier-by-tier start_service
    // (services already in STATE_RUNNING are skipped)

    main_loop()

7.2 apply_recovered_state algorithm

For each rs in recovered.services:

idx = supervisor.find_service(table, rs.name)

if idx < 0:
    // Service was in state.toml but is no longer in enabled.d/.
    // Operator removed it dirty (manual symlink removal between old start
    // and replace). Don't kill — let the process exit on its own and the
    // kernel reap the contract. We just stop tracking it.
    contract.abandon_contract(rs.contract_id)
    log("replace: orphaned " + rs.name + " (no longer enabled)")
    continue

// Found in current table — patch in runtime fields.
rt = table.runtimes[idx]
rt.state = STATE_RUNNING
rt.contract_id = rs.contract_id
rt.pid = -1                     // unknown; not needed
rt.restart_count = rs.restart_count
rt.last_start_time = rs.last_start_time
rt.last_exit_code = rs.last_exit_code
table.runtimes[idx] = rt
table.contract_map.set(int_to_str(rs.contract_id), idx)

// Idempotency: did the contract go empty during the exec gap?
if contract.is_contract_empty(rs.contract_id):
    log("replace: " + rs.name + " contract empty post-recovery, applying restart policy")
    supervisor.handle_contract_event(table, rs.contract_id, -1)
    // existing logic resolves to STOPPED or scheduled-for-restart

7.3 Config divergence between old start and replace

Three cases the new zyginit must handle:

Case Detection Action
Service in state.toml AND in current enabled.d/ find_service returns valid idx Adopt: patch runtime fields, register in contract_map. Use current TOML for future supervision (so an edit takes effect on next restart).
Service in state.toml, NOT in current enabled.d/ find_service returns -1 Abandon contract, log warning. Member processes continue running but unsupervised; kernel reaps the contract on their exit.
Service in current enabled.d/, NOT in state.toml Not in recovered list Treated as a freshly-enabled service. Tier-by-tier startup will start_service it normally on first event-loop tick.

This means automation flows like coral install some-service-package; zygctl replace work correctly — the newly-installed service appears in enabled.d/ and gets started by the new zyginit even though the old one never knew about it.

7.4 No adopt_contract calls

As established in §3, contract ownership survives execve at the kernel level. The new zyginit's bookkeeping is entirely in userspace: rebuild contract_map, populate ServiceRuntime fields. The contract.adopt_contract FFI remains in the codebase for potential future use (e.g., a non-PID-1 zyginit "supervisor mode" recovery flow) but is not exercised by this design.

8. Error handling

8.1 Pre-exec (in old zyginit)

Every pre-exec failure aborts the replace cleanly — old zyginit returns to its main loop with the flag cleared.

Failure Behavior
Service in transient state, --wait exhausted Log "replace blocked: <name>:<state>, ...". Clear flag. Return to main loop.
state.toml.tmp open / write / fsync fails Log error with errno, unlink any partial .tmp. Return to main loop.
rename(state.toml.tmp, state.toml) fails Log error. Return to main loop. Stale state.toml from prior aborted attempt has the wrong boot_time and is harmless.
unlink("/var/run/zyginit.sock") fails Log warning, proceed to exec. New zyginit's unix_listen will overwrite the path.
process.process_exec() returns Means kernel couldn't load /sbin/init (binary missing or corrupt). Log fatal. Old zyginit is in an unrecoverable state (state.toml written, socket unlinked). Fall through to the existing PID-1 infinite-sleep handling. Operator's recourse is reboot. Should never reach this in practice.

8.2 Post-exec (in new zyginit)

Failures here are per-entry — one bad entry doesn't stop other recoveries.

Failure Behavior
state.toml missing Treated as fresh boot. Not an error; no log.
state.toml parse error / corrupt Log warning, treat as missing. Surviving contracts (if any) become orphans.
state.toml.boot_time != current Log info, delete file, treat as missing.
Service in state.toml but not in current enabled.d/ Abandon contract, log warning. (See §7.3.)
is_contract_empty(ctid) post-recovery returns true Synthesize a contract event; existing handle_contract_event applies restart policy.
ct_status_read returns error (contract gone, ENOENT) Treat as empty / missing. Service starts fresh on first tick.

After all entries are processed (success or otherwise), state.toml is deleted. This breaks panic-restart loops where the same bad state would otherwise be re-applied repeatedly.

8.3 Panic safety

recover_state and apply_recovered_state are written defensively. Any unexpected condition is logged and skipped — never panicked on. A bad state.toml entry must never prevent the new zyginit from completing startup.

9. Testing

9.1 Linux dev (no contract FFI)

The Linux contract stubs return -1 for all contract calls, so end-to-end re-adoption can't run on Linux. What we can unit-test:

  • State serialization round-trip (tests/test_replace.reef): build a synthetic ServiceTable with services in known states, call serialize_state, read back with recover_state, assert all fields match. Cover empty table, all-RUNNING, mixed states (only RUNNING entries serialize), boot-id mismatch (ignored), corrupt TOML (ignored).
  • Pre-condition check: factor the transient-state scan into an exported check_replace_preconditions(table) -> string (empty on OK, error message otherwise). Test transient states individually and combined.
  • Stitching logic: unit-test apply_recovered_state covering the three cases of §7.3. Linux contract stubs let is_contract_empty always return false; we exercise the in-state-and-enabled and in-state-not-enabled branches.
  • Integration smoke (tests/integration/run_tests.sh): start zyginit-as-supervisor (ZYGINIT_CONFIG_DIR=/tmp/...), run a few oneshot+daemon services, run zygctl replace. On Linux, contract-driven supervision is partially stubbed but the exec path itself runs. Assert the second zyginit instance starts, its log shows "replace: stale state file" or "replace: <name> contract empty post-recovery" per stub returns, and no crash.

9.2 Hammerhead (hh-prototest)

Real validation. Deploy via the iter 11+ procedure (scp source, gcc helpers.c, reefc build, mv build/zyginit /sbin/init.new && mv /sbin/init.new /sbin/init).

Test cases:

  1. Smoke — multi-user up; zygctl replace; confirm:
    • Connection drops with the documented message.
    • Within ~1s, zygctl status works again.
    • All services still running, same PIDs, same contract IDs, restart counts preserved.
    • Uptimes preserved (continuous, not reset).
    • The SSH session driving the test stays alive across the replace. This is the highest-signal check.
    • who -b and who -r still report correctly.
  2. Transient-state refusal — start a service with a slow start.sh (or watch for the WAITING window during a restart); run zygctl replace during the transient window; confirm refusal with the right error.
  3. --wait=N success — same scenario with --wait=10; confirm replace proceeds once the service stabilizes.
  4. Dirty disable — rm /etc/zyginit/enabled.d/<svc> (without zygctl reload); zygctl replace; confirm orphan path runs (contract abandoned, process keeps running, log message).
  5. Crash during exec gap — kill -9 <pid> immediately before zygctl replace; confirm new zyginit's is_contract_empty post-recovery catches the empty contract and applies restart policy.
  6. Stale state.toml after real reboot — write a state.toml file, do a real reboot, file persists in /var/run/zyginit/; confirm boot-id mismatch is detected, file deleted, fresh boot proceeds.
  7. Repeated replace — zygctl replace 5 times in succession; confirm restart counts accumulate correctly (no resets), uptimes grow continuously, no state-file leakage.

9.3 Out of scope for testing

  • Crash recovery (operator-driven only).
  • Replace under heavy load (we accept some bundle events will land post-exec; the post-recovery status check covers it).
  • Concurrent replace invocations (single-shot flag; second invocation between flag-set and exec is silently dropped — operator error).

10. Deferred / open questions

These are intentionally not in scope for v1, but flagged so we can revisit:

  • zygctl replace --force — accept transient states by re-driving them (re-issue stop, re-arm restart timer, etc.). Adds significant recovery complexity. Defer until there's demonstrated automation demand.
  • SIGUSR2 fallback — operator-side trigger if the socket is wedged. The reload path (SIGHUP) shows that signal-driven flag-setting works; we just don't yet need the additional reachable code path for replace. Defer.
  • Socket access tightening — orthogonal to this design, but the socket is currently created with default permissions. It should be 0600 root-only. File as a separate hardening item.
  • Bundle fd inheritance — clearing FD_CLOEXEC and passing the fd through exec would close the (already-handled) gap window completely. Adds complexity for an already-mitigated case. Defer indefinitely.
  • /var/run/zyginit/ to tmpfs — once fs-minimal is added back to the boot chain, state.toml automatically becomes ephemeral and the boot-id check becomes belt-and-suspenders. Tracked separately under §"Important Notes" in CLAUDE.md.
  • utmpx BOOT_TIME as sentinel (rejected) — utmpx BOOT_TIME's pututxline updates rather than appends, so the new zyginit overwrites the original timestamp at startup, defeating the staleness check. /proc/1/start naturally survives execve but changes on real reboot. See §5.2 for full rationale.

11. Summary of decisions

Decision Choice Rationale
Trigger model Operator-driven only (live upgrade) Crash recovery is a much harder problem; PID 1 dying is already catastrophic.
Trigger interface zygctl replace (socket only) Smallest blast radius; signal fallback defers cleanly.
Verb replace Operator-readable; covers upgrade + memory hygiene + state reset reasons; no overload with existing commands.
Exec target Hard-wired to /sbin/init Security: never accept attacker-controllable exec paths in PID-1.
Identity mapping State file (single TOML) Preserves runtime fields (restart_count especially); deterministic; matches deliberate-handover model.
Pre-condition check Refuse on transient state by default; --wait=N for automation Eliminates an entire class of recovery bugs; automation has a clean knob.
Bundle fd Reopen, fix gap via post-recovery is_contract_empty Idempotent, no fd-passing protocol needed.
Config divergence Stitch state.toml against current enabled.d/, abandon orphans Permissive without complexity; supports coral install + replace automation.
Adoption No adopt_contract call PID-1 in-place execve preserves kernel-level contract ownership; only userspace bookkeeping needs rebuilding.
State file location /var/run/zyginit/state.toml Existing runtime tree; boot-id check makes it safe across real reboots.

12. Why keep this in the Hammerhead-userland package?

Hammerhead's primary upgrade workflow is BE-based: pkg update → new BE → beadm activate → reboot. In that workflow, zygctl replace is bypassed — the new zyginit comes up as a fresh PID 1 in the new BE. So why ship the feature?

Linux port. zyginit is written in Reef which compiles to native C; a Linux port is realistic. Linux package managers (apt, yum, dnf) operate on an in-place upgrade model where daemons need a way to reload their own binary without restarting the system. systemd's systemctl daemon-reexec exists for exactly this reason. If zyginit lands on Linux, zygctl replace becomes a primary feature.

Live patching trajectory. A system that supports live kernel patching (kpatch-style) has implicit cultural pressure to support live userland patching for critical daemons. zyginit being live-upgradeable is a natural fit for that posture if Hammerhead develops it.

Mechanism reuse. Several pieces (contract.is_contract_empty, supervisor.adopt_runtime, the state-file pattern in utils/replace_probe) have potential consumers beyond this feature — shutdown verification, future crash-recovery work, generic state-file testing patterns.

The feature stays; the trigger is documented as Hammerhead-secondary, Linux-primary.