# PID-1 Boot Design **Date:** 2026-04-24 **Revision:** 2026-04-25 — BSD-pivot (post iter 1-5 findings) **Status:** Revised; iter 6 implementation pending against the new service set **Scope:** First boot of zyginit as PID 1 on Hammerhead (hh-prototest VM), replacing the SMF-managed boot chain with a zyginit-managed BSD-style boot. ## 1. Goal Boot hh-prototest with `/sbin/init` pointing at zyginit, bringing the system to a state where the operator can SSH in and use the box. Deliver a small **BSD-style** service set (≈15 services) that uses direct command invocation (`ifconfig`, `dhcpagent`) instead of supervising SMF-coupled control-plane daemons. Tested rollback path via ZFS Boot Environments. ## 1.1 Design revision: BSD pivot (2026-04-25) The original 21-service design (preserved in §4 history at the bottom of this doc) tried to supervise the existing illumos control-plane daemons — `dlmgmtd`, `ipmgmtd`, `inetd`, `rpcbind`, `fmd` — directly. Iter 1-5 (rev 24 through rev 50) surfaced a fundamental issue: these daemons are tightly coupled to **`svc.configd`** for their persistent property store via `libscf`. Setting `SMF_FMRI` in the environment bypasses their refusal-to-start banner, but they then silently degrade because `scf_handle_decode_fmri()` fails (no `/etc/svc/volatile/repository_door`), leaving them running but useless — `ipmgmtd` doesn't open its door socket; `ipadm` consequently fails with "Could not open handle to library"; `lo0` is never plumbed; `rpcbind` cascades into "could not find loopback transports"; the whole network stack is dead. We had three choices: - **A: Run `configd` under zyginit** — keeps the SMF property-store daemon alive, drops only `svc.startd`. Gets the daemons working but admits SMF is too entangled to fully escape. - **B: Use `/lib/svc/method/*` scripts as zyginit's exec.start** — same effective coupling to `configd` as A; just hides it behind shell wrappers. - **C: Pivot to BSD-style direct invocation** — drop the configd-coupled daemons entirely. Use `/sbin/ifconfig` directly to plumb interfaces and configure addresses. Use `/sbin/dhcpagent` directly for DHCP. Skip `inetd`/`rpcbind`/`fmd` for the default service set. We chose **C**. Hammerhead retains the BSD primitives that make this work: - `/sbin/ifconfig` exists and successfully plumbs interfaces without needing `ipmgmtd` - `/etc/hostname.` files supported (BSD-style interface naming/config) - `/sbin/dhcpagent` is callable directly - `/etc/init.d/`, `/etc/rc[0-3].d/` directories present (we won't use them, but the convention is there) Reference for the approach: IRIX 6.5's `/etc/init.d/network` script (`os_research/irix-6517-src/eoe/cmd/initpkg/init.d/network`) does exactly this — `IFCONFIG=/usr/etc/ifconfig`, `ROUTE=/usr/etc/route`, flat-file configuration in `/etc/config/`, no daemon-based registry. zyginit-on-Hammerhead becomes essentially a Reef rewrite of pre-SMF Solaris boot semantics, replacing what IRIX/Solaris-9 did with shell. Lessons from iter 1-5 also drove these other concrete fixes that survive in v2: - **fd-setup at PID 1**: kernel doesn't preopen 0/1/2; zyginit must `open(/dev/console)` and `dup2()` onto 0/1/2 before any I/O (rev 45) - **argv[0] in supervisor**: must include the program name as argv[0] when execvp'ing (rev 44) - **root-fs ZFS/UFS dispatch**: `mount -o remount,rw /` errors on ZFS root; use `zfs set readonly=off` (rev 43) - **Tier ordering enforcement**: between tiers, wait for oneshots to STOPPED and daemons to RUNNING before starting next tier (rev 47) - **Persistent log dir**: `/var/log/zyginit/` (not `/var/run`) so logs survive reboot for diagnosis (rev 48) - **Hammerhead path corrections**: `/bin/hostname` (not `/usr/bin/`), `/sbin/swapadd` (no `-a`), `/sbin/dumpadm`, etc. - **cron FIFO is at `/etc/cron.d/FIFO`**, not `/var/run/cron_fifo` A useful diagnostic capability also fell out: with the VM shut down, the BE's logs are readable from the dev host via `qemu-nbd` + zfsonlinux read-only import, then mounting `hhtest/ROOT/zyginit` to read `/var/log/zyginit/*.log`. This bypasses the GRUB-cant-pick-BE limitation on this Hammerhead version. ## 2. Non-goals - **SMF removal from the BE.** Phase C, separate milestone. First boot leaves SMF binaries installed-but-unused. We just don't *invoke* them from zyginit's service set. - **Supervising configd-coupled daemons.** `dlmgmtd`, `ipmgmtd`, `inetd`, `rpcbind`, `fmd` are explicitly out of zyginit's default service set. They depend on a running `svc.configd` that we no longer launch. They remain available on the BE for an admin who chooses to set up SMF separately. - **Modern illumos datalink/IP management semantics.** No `dladm show-link`/`ipadm show-addr` integration. Configuration is via flat files (`/etc/hostname.`, `/etc/defaultrouter`, `/etc/resolv.conf`). For laptops with WiFi roaming, link aggregation, IPMP, or VNICs for zones, this design is insufficient — those workflows need the SMF stack. - **On-demand network services (inetd-style).** No `inetd` in the default set. Services that historically ran under `inetd` (telnet, rsh, finger, etc.) are not started. SSH is direct. - **NFS / RPC.** `rpcbind` not started; no NFS in scope. - **Fault management.** `fmd` not started; FMA events are not collected. - **Single-user mode / runlevels.** `zygctl single` and `zygctl multiuser` remain stubs. - **Crash-recovery re-adoption.** `ct_ctl_adopt` requires descendant relationship; only PID-1 in-place re-exec is supported. - **`/sbin/halt`, `/sbin/reboot`, `/sbin/poweroff`, `/usr/sbin/shutdown` integration.** Continue to call `uadmin` directly. Clean shutdown goes through `zygctl`. - **Name services (NSS), NTP, performance targets.** Deferred. ## 3. Architecture ### 3.1 Components | Component | Path | Type | Purpose | |---|---|---|---| | `zyginit` | `/sbin/zyginit` | ELF binary | Init daemon (PID 1) | | `zygctl` | `/sbin/zygctl` | ELF binary | Admin CLI | | Config tree | `/etc/zyginit/` | Directory | Service TOMLs, per-service script dirs, `enabled.d/` symlink farm | | Runtime tree | `/var/run/zyginit/` | Directory | Per-service log files, socket, transient state | | Control socket | `/var/run/zyginit.sock` | Unix socket | zygctl ↔ zyginit IPC | ### 3.2 Boot sequence at runtime 1. Kernel execs `/sbin/init` (symlink to `/sbin/zyginit`) 2. zyginit detects PID (`getpid() == 1`) and enters PID-1 mode 3. zyginit scans `/etc/zyginit/enabled.d/` for symlinks 4. zyginit parses each TOML referenced; builds in-memory `ServiceDef` set 5. zyginit builds dependency graph via topological sort into tier ordering 6. zyginit starts services tier by tier: - Within a tier: sequential start (parallel start is a future optimization) - Between tiers: wait for all members of tier N to reach terminal state (RUNNING for daemons, STOPPED+exit=0 for oneshots) before starting tier N+1 7. zyginit opens event loop on: contract bundle fd, signal self-pipe, zygctl socket 8. System is "up" when all enabled services have started successfully ### 3.3 PID-1 vs non-PID-1 behavior | Condition | Trigger | Behavior | |---|---|---| | PID 1 | SIGTERM / SIGINT | Halt: stop services in reverse tier order, sync, `uadmin(A_SHUTDOWN, AD_HALT, 0)` | | PID 1 | `zygctl halt` | As above | | PID 1 | `zygctl reboot` | Stop services, sync, `uadmin(A_SHUTDOWN, AD_BOOT, 0)` | | PID 1 | `zygctl poweroff` | Stop services, sync, `uadmin(A_SHUTDOWN, AD_POWEROFF, 0)` | | not PID 1 | SIGTERM / SIGINT | Stop services, return from `main()` (normal process exit) | | not PID 1 | `zygctl halt` | Stop services, exit | | not PID 1 | `zygctl reboot` / `poweroff` | Refuse ("not PID 1") — safety rail for dev/test | ## 4. Service set (BSD-pivot, v2) 15 zyginit services in `enabled.d/` by default (counted across §4.1–§4.6). Replaces the 21-service v1 design (full v1 service table viewable in `hg log -p` for this file) by dropping the configd-coupled daemons in favor of direct command invocation. ### 4.1 Tier 0 — no dependencies | Service | Type | Script | Stop | |---|---|---|---| | `root-fs` | oneshot | `/etc/zyginit/root-fs/start.sh` (ZFS or UFS root: `zfs set readonly=off` or `mount -F ufs -o remount,rw /`) | `/etc/zyginit/root-fs/stop.sh` (matching readonly=on / remount,ro) | ### 4.2 Tier 1 — requires root-fs | Service | Type | Script | Stop | |---|---|---|---| | `devfs` | oneshot | inline: `/usr/sbin/devfsadm` | — | | `swap` | oneshot | inline: `/sbin/swapadd` (no `-a`; reads `/etc/vfstab`) | `/etc/zyginit/swap/stop.sh` (`swap -d` each active entry) | | `crypto` | oneshot | `/etc/zyginit/crypto/start.sh` (`cryptoadm refresh`) | — | ### 4.3 Tier 2 — requires tier 1 | Service | Type | Script | Stop | |---|---|---|---| | `filesystem` | oneshot | `/etc/zyginit/filesystem/start.sh` (`mountall -l` + `zpool import -a` + `zfs mount -a`) | `/etc/zyginit/filesystem/stop.sh` (`umountall -l` + `zpool sync`) | | `identity` | oneshot | `/etc/zyginit/identity/start.sh` (`hostname` from `/etc/nodename`) | — | | `sysconfig` | oneshot | `/etc/zyginit/sysconfig/start.sh` (coreadm, dumpadm, keymap, scheduler, rctl, rmtmpfiles) | — | ### 4.4 Tier 3 — requires filesystem + identity | Service | Type | Script | Stop | |---|---|---|---| | `network` | oneshot | `/etc/zyginit/network/start.sh` — **direct `ifconfig` invocation, no `ipmgmtd`**: plumb lo0, walk `/etc/hostname.` files for each physical interface, `ifconfig dhcp` (which forks `dhcpagent` directly) or static address per file content, install default route from `/etc/defaultrouter` if present | `/etc/zyginit/network/stop.sh` (kill dhcpagent, ifconfig unplumb interfaces) | **Critical change from v1:** no `dlmgmtd` or `ipmgmtd` services. `ifconfig` plumbs interfaces directly via DLPI. `dhcpagent` is invoked directly via `ifconfig dhcp start`, not via `ipadm -T dhcp`. This matches IRIX 6.5's `/etc/init.d/network` pattern (`IFCONFIG=/usr/etc/ifconfig`, `ROUTE=/usr/etc/route`). #### `/etc/hostname.` file format (OpenBSD-inspired, illumos-translated) Verified on hh-prototest: `/sbin/ifconfig` is fully operational without `ipmgmtd` (plumb, address assignment, link state — all work). The OpenBSD `hostname.if(5)` format is line-oriented with each line being ifconfig args. We adopt the same line-oriented convention but translate to illumos's ifconfig syntax, since OpenBSD-specific keywords like `inet alias` map to illumos's `addif`. Line forms supported by zyginit's `network/start.sh`: | Line | Action | |---|---| | (empty file or whitespace) | `ifconfig plumb up` — interface up, no address | | `# comment` or blank line | skipped | | `dhcp` | `ifconfig plumb`; `ifconfig dhcp start` (best effort — see DHCP caveat) | | `inet /` | `ifconfig inet / up` | | `inet netmask ` | `ifconfig inet netmask up` | | `addif /` | additional alias address (illumos `addif`); requires interface already plumbed by an earlier line | | `inet6 /` | `ifconfig inet6 / up` (deferred to post-first-boot) | | `up` / `down` | set link state explicitly | Examples: ``` # /etc/hostname.vioif0 — DHCP via dhcpagent (best effort without ipmgmtd) dhcp ``` ``` # /etc/hostname.vioif0 — static IPv4 inet 192.168.122.50/24 ``` ``` # /etc/hostname.vioif0 — static IPv4 + alias inet 192.168.122.50/24 addif 192.168.122.51/24 ``` #### Default route `/etc/defaultrouter` (one IPv4 address per line). `network/start.sh` runs `route -n add default ` for each. Optional — DHCP-acquired routes don't need this file. #### DHCP caveat `dhcpagent` is linked against `libipadm.so.1`, which talks to `ipmgmtd`'s door for persistent state. Without `ipmgmtd` running: - `dhcpagent` itself runs (self-contained DHCP state machine) - DHCP request/response on the wire works (uses raw sockets, not libipadm) - **Applying the lease to the interface** uses libipadm → ipmgmtd, which may degrade Empirical test required on hh-prototest in iter 6 boot. If DHCP-via-ifconfig fails without ipmgmtd, fallback options: 1. Manually invoke `dhcpagent -a -f` (adopt mode + foreground) and configure interface from received lease via direct DLPI ioctl 2. Static IP only for the test VM (configure libvirt to have a known static address, set `/etc/hostname.vioif0` to `inet 192.168.122.50/24`) 3. Patch dhcpagent / replace with dhclient if available The implementation plan should boot-test with static IP first to derisk the rest of the boot, then attempt DHCP as a follow-up. ### 4.5 Tier 4 — requires network (configd-free system daemons) All daemons. All use `stop.method = "contract"` (default). `restart.on = "failure"`, `delay = 5`, `max_retries = 3`. | Service | exec.start | Note | |---|---|---| | `syseventd` | `/usr/lib/sysevent/syseventd` | Kernel event delivery; doesn't read SMF properties | | `syslogd` | `/usr/sbin/syslogd` | Reads `/etc/syslog.conf` directly; SMF-property reads degrade to defaults | | `utmpd` | `/usr/lib/utmpd` | Cleans stale utmpx entries; no configd dependency | | `pfexecd` | `/usr/lib/pfexecd` | RBAC profile-exec helper; reads `/etc/security/`, no configd | ### 4.6 Tier 5 — user-facing services | Service | exec.start | Note | |---|---|---| | `cron` | `/etc/zyginit/cron/start.sh` | wrapper: `rm -f /etc/cron.d/FIFO; exec /usr/sbin/cron` | | `sshd` | `/usr/lib/ssh/sshd -D` | `-D` keeps it foreground; reads `/etc/ssh/sshd_config` | | `console-login` | `/etc/zyginit/console-login/start.sh` | wrapper exec'ing ttymon with properly-quoted prompt | ### 4.7 Optional / installable — TOMLs ship but symlinks not in `enabled.d/` by default These TOMLs exist in `services/hammerhead/` but are NOT enabled by default. An admin can enable them via `zygctl enable ` on hardware/configurations where they make sense: | Service | When to enable | |---|---| | `powerd` | Real hardware with `/dev/srn` (not VMs) | | `fmd` | When fault management is wanted (requires configd workaround or future SMF-coexistence design) | | `dlmgmtd` / `ipmgmtd` / `inetd` / `rpcbind` | Only after a separate "run minimal SMF infrastructure under zyginit" design is implemented | ### 4.8 Dropped from the v1 design (configd-coupled, requires separate design to revisit) These daemons require `svc.configd` running for `libscf` calls to succeed. They are intentionally absent from the BSD-pivot v2 service set: - `dlmgmtd` — datalink management. Replaced by direct `ifconfig` plumbing. Drawback: no `dladm show-link` queries; persistent datalink state across reboots is via `/etc/hostname.` instead. - `ipmgmtd` — IP interface management. Replaced by direct `ifconfig`/`route` invocation. Drawback: no `ipadm` queries. - `inetd` — on-demand service launching. Modern illumos `inetd` reads from configd. No replacement for first boot; can be re-added when needed (e.g., via `xinetd` or by accepting configd dependency). - `rpcbind` — only needed for NFS/RPC; deferred until NFS is in scope. - `fmd` — fault management daemon; heavy SMF coupling. Deferred. ### 4.9 SMF infrastructure (subsumed; no replacement) - `svc.startd`, `svc.configd` — replaced by zyginit (or absent in v2) - `manifest-import`, `early-manifest-import` — no XML manifests - `boot-archive`, `boot-archive-update` — SMF-specific - `logadm-upgrade`, `update-man-index` — admin/cron concerns, not init's - All 7 SMF milestones (virtual groupings) — zyginit's dependency graph replaces these ### 4.10 Currently-disabled SMF services — also not resurrected in zyginit Mirrors v1: `name-service-cache`, `rbac`, `process-security`, `auditd`, NFS/DNS/LDAP stacks all stay disabled. ## 5. TOML schema usage Uses the schema already documented in `docs/DESIGN.md` §"Service Definition Format". The following subset is in active use for the 21 services: ```toml [service] name = "..." description = "..." type = "daemon" | "oneshot" [exec] start = "command or /etc/zyginit//start.sh" [dependencies] requires = ["..."] after = ["..."] # Daemons only: [stop] method = "contract" # default; may be omitted signal = "TERM" # default; may be omitted timeout = 60 # default; may be omitted [restart] on = "failure" delay = 5 max_retries = 3 [contract] param = ["inherit", "noorphan"] fatal = ["hwerr"] # Oneshots with reversal: [exec] stop = "/etc/zyginit//stop.sh" [restart] on = "never" ``` **Defaults and omissions:** - Omitted `exec.user` / `exec.group` → inherits from zyginit → root when PID 1. Explicit and documented so there's no ambiguity. - Omitted `[stop]` block → uses contract-signal defaults (TERM, 60s timeout). - Omitted `[contract]` block → uses template defaults from `setup_template()` in `contract.reef`. **Deferred schema fields** (supported by zyginit, unused in first boot): - `exec.user`, `exec.group`, `exec.working_directory`, `exec.environment` — none of the 21 services need privilege drop or per-service env - `security.*`, `limits.*` — not implemented / not in first boot **`contract.param = ["inherit", "noorphan"]` note:** The NOORPHAN flag means that if zyginit-as-PID-1 dies without cleanly abandoning contracts, children receive SIGHUP. This is correct for PID-1 re-exec (kernel sees the same process, no SIGHUP fires). It is **incompatible** with a crash-recovery path that promotes a fresh zyginit PID — but such a path is an explicit non-goal per §2. ## 6. Method script conventions All scripts live under `/etc/zyginit//start.sh` (and optional `stop.sh`) unless the service is trivial enough for an inline TOML command. **Template:** ```zsh #!/bin/zsh # — # Invoked by zyginit via exec.start in /etc/zyginit/.toml set -e set -u # ... steps ... exit 0 ``` **Conventions:** 1. Shebang: `#!/bin/zsh` (consistent with Hammerhead's shell policy; zsh is the system default, ksh93 is being phased out) 2. Error handling: `set -e` (POSIX-style — preferred over `setopt ERR_EXIT` for cross-shell readability) + `set -u` 3. Non-interactive only: no `read`, no tty-dependent behavior 4. Self-contained: no `source`ing of SMF helper scripts (`/lib/svc/share/*.sh`) 5. No SMF API calls: no `svcprop`, `svcadm`, `svccfg`. Config input comes via `exec.environment` in the TOML. 6. Explicit `exit 0` on success (not relying on implicit last-command exit) 7. Use `|| true` deliberately per-command where a specific failure is acceptable (e.g., `dumpadm -d /path || true` if the dump device doesn't exist yet) 8. Idempotent when the action allows: prefer `mount -o remount,rw /` (idempotent) over raw `mount` commands. Use "check then do" patterns for inherently non-idempotent operations (`ipadm show-if lo0 2>/dev/null || ipadm create-if lo0`). **Execution environment zyginit provides to scripts:** - Fresh process (fork + exec) - Working dir: `/` unless TOML sets `exec.working_directory` - Environment: minimal (PATH, HOME, etc.) plus whatever `exec.environment` specifies - stdout/stderr: redirected to `/var/run/zyginit/log/.log`, truncated per start - User: inherited from zyginit (root at PID 1) unless `exec.user` set ## 7. Shutdown orchestration Shutdown is a first-class concern, not deferred. Unclean shutdown can corrupt ZFS; we do it right from the first boot. ### 7.1 Shutdown triggers | Source | Type | |---|---| | `zygctl halt` | halt | | `zygctl reboot` | reboot | | `zygctl poweroff` | poweroff | | SIGTERM / SIGINT (PID 1) | halt (conservative default) | | SIGTERM / SIGINT (not PID 1) | exit (integration test path) | Shutdown type is communicated to zyginit via the socket protocol (for zygctl commands) or inferred from signal source. ### 7.2 Shutdown walk 1. Set `shutdown_requested = true`, `shutdown_type = ` 2. Break out of main event loop 3. Walk services **in reverse tier order** (not insertion order) 4. For each service in the tier: - **Daemons** in `STATE_RUNNING`: `stop_service()` → SIGTERM via contract, wait for contract EMPTY event, mark `STATE_STOPPED` - **Oneshots** in `STATE_STOPPED` with `exec.stop` defined (or `/etc/zyginit//stop.sh` present): fork/exec the stop command; wait for exit - **Oneshots** without stop: skip (no reversal defined) 5. Wait for all services in tier N to reach `STATE_STOPPED` before moving to tier N-1 6. After tier 0 is drained: `sync()` (flush pending writes) 7. Call `uadmin(A_SHUTDOWN, , 0)`: - `AD_HALT` (0) — halt, operator decides next step physically - `AD_BOOT` (1) — reboot immediately - `AD_POWEROFF` (6) — ACPI/BMC chassis off ### 7.3 Stop scripts concretely required | Service | stop.sh action | |---|---| | `root-fs` | `mount -o remount,ro /` (called after filesystem/stop.sh has umounted everything else) | | `filesystem` | `umountall -l` | | `swap` | `swap -d` each entry | Other services either use the contract-signal daemon default or have nothing to reverse. ### 7.4 Required code changes **`supervisor.reef`:** - Extend shutdown path to walk tiers in reverse - For each tier: first send stops to all daemons, wait for them to exit, then run stop scripts for oneshots that have them - Currently `shutdown_services` skips `STATE_STOPPED` services — this must change **`main.reef`:** - Track `shutdown_type` variable (halt / reboot / poweroff) - After `shutdown_services()`: call `sync()`, then `zyginit_shutdown()` **FFI: `helpers.c` and `contract.reef` (or new `shutdown.reef`)**: - `int zyginit_shutdown(int fcn)` → wraps `uadmin(A_SHUTDOWN, fcn, 0)` - `extern "C" fn zyginit_shutdown(fcn: int): int` **Linux stub**: `contract_linux_stubs.c` gets a matching stub returning -1 (should never actually fire on Linux). **`zygctl`** gets three new subcommands: `halt`, `reboot`, `poweroff`. Each sends a specific command over the socket; zyginit sets `shutdown_type` accordingly. **`zygctl`** also gets two stub subcommands: `single`, `multiuser`. Both print "not yet implemented — reserved for future runlevel support" and exit 1. Documented in `zygctl --help` so the surface is stable. ## 8. Rollout phases ### 8.1 Phase A — non-PID-1 dry run Run zyginit on the live hh-prototest system alongside SMF, using a scratch config directory. ```sh ZYGINIT_CONFIG_DIR=/tmp/zyginit-dry \ /path/to/zyginit ``` **Goals:** - TOMLs parse cleanly - Dependency graph builds without cycles; tier order matches the spec - zyginit supervises one or two non-conflicting test daemons (e.g., a second sshd on port 2222) - Clean shutdown via SIGTERM works without uadmin (non-PID-1 path) **Out of scope for Phase A:** - Destructive oneshots (filesystem, network, devfs) — they'd conflict with SMF-managed copies. Skip these in the dry run. ### 8.2 Phase B — PID-1 boot in a fresh Boot Environment 1. `beadm create zyginit` 2. `beadm mount zyginit /mnt` 3. Install into `/mnt`: - `/mnt/sbin/zyginit`, `/mnt/sbin/zygctl` - `/mnt/etc/zyginit/*.toml` (21 files) - `/mnt/etc/zyginit//start.sh` + `stop.sh` as specified above - `/mnt/etc/zyginit/enabled.d/` symlinks for all 21 - Preserve old init: `mv /mnt/sbin/init /mnt/sbin/init.smf` - New init: `ln -s /sbin/zyginit /mnt/sbin/init` 4. `beadm umount zyginit; beadm activate zyginit; init 6` ### 8.3 Phase C — SMF removal (deferred, separate milestone) Optional follow-up once Phase B is stable: - In the zyginit BE: remove `/lib/svc/bin/svc.startd`, `/lib/svc/bin/svc.configd`, `/lib/svc/method/*`, `/etc/svc/` - Activate, reboot, prove the system still works without SMF - This is when "zyginit replaces SMF" is literally true. ### 8.4 Failure recovery ladder In order of cost: 1. `zygctl reboot` → pick `hammerhead` BE from GRUB menu (seconds) 2. If GRUB menu inaccessible: boot into rescue media / single-user from install media, mount original BE, activate it 3. If the pool is damaged: `sudo cp hh-prototest-backup-2026-04-24-pre-zyginit-pid1.qcow2 hh-prototest.qcow2` on the dev host (2-3 min with VM shut down) 4. Last resort: VM rebuild from scratch (~20 min) Disk-level backup captured at rev 21. Recovery drill is part of §9.F. ## 9. Success criteria for Phase B Literal checklist run after first boot into the zyginit BE. ### A. Boot completes - [ ] No `kernel panic: init died` - [ ] `zyginit: entering event loop` appears on console - [ ] Every tier finishes within 60s total - [ ] Console login prompt appears on `/dev/console` ### B. Expected service states - [ ] All 21 services present in `zygctl list` - [ ] Tier 0-3 oneshots: `STOPPED` with `exit=0` - [ ] Tier 3-5 daemons: `RUNNING` - [ ] No service in `MAINTENANCE` ### C. Services actually functional - [ ] `ssh zygaena@192.168.122.197` from dev host - [ ] `logger -p user.info test` → visible in `/var/adm/messages` (syslogd) - [ ] `dladm show-link` lists physical interface `up` - [ ] `ipadm show-addr` lists lo0 + physical interface with IPv4 addresses - [ ] `ping 192.168.122.1` succeeds - [ ] `crontab -l` runs for root - [ ] `zfs list` works, `hhtest` pool mounted ### D. zygctl end-to-end - [ ] `zygctl status cron` shows `RUNNING` with uptime > 0 - [ ] `zygctl stop cron` → `STOPPED`, then `zygctl start cron` → `RUNNING` - [ ] `zygctl log sshd` returns without error - [ ] `zygctl reload` succeeds (SIGHUP path) ### E. Clean shutdown - [ ] `zygctl halt`: reverse-tier-order stop output in console; system halted cleanly - [ ] `zygctl reboot`: cycles; comes back via zyginit again - [ ] `zygctl poweroff`: VM transitions to `shut off` in libvirt - [ ] After any zygctl shutdown: `zpool status` clean, no replay-required flags ### F. Rollback confirmed - [ ] 3 clean reboot cycles into the zyginit BE - [ ] Reboot into original `hammerhead` BE from GRUB → SMF comes up normally - [ ] `beadm activate hammerhead` from zyginit BE → reboot → lands on `hammerhead` BE - [ ] Disk-level restore drill executed at least once (time to recovery measured) ### G. Explicit non-requirements - No boot-time performance gate (measured for baseline only) - No stability soak - No DNS / hostname resolution testing - No NTP / time sync - No name services (NSS beyond /etc/passwd, /etc/hosts, /etc/group) - No `zygctl single` / `zygctl multiuser` functionality (stubs only) - No SMF removal (Phase C) ## 10. Observability during test - Console access via `virsh console hh-prototest` for live boot watching. - `/var/log/zyginit/.log` per-service logs (rev 48 changed default from `/var/run/zyginit/log` to `/var/log/zyginit` so logs survive reboot). - **Out-of-band log access:** with VM shut down, mount the BE's qcow2 from the dev host: ``` virsh destroy hh-prototest sudo qemu-nbd -c /dev/nbd0 -r /var/lib/libvirt/images/hh-prototest.qcow2 sudo zpool import -R /tmp/hh-mnt -o readonly=on -N -f -d /dev/nbd0p2 hhtest sudo zfs mount hhtest/ROOT/zyginit # logs at /tmp/hh-mnt/var/log/zyginit/ ``` Restore: `zfs umount`, `zpool export hhtest`, `qemu-nbd -d /dev/nbd0`. Proven across 3 iterations. - Capture console screenshot at each iter for visual cross-check. ## 11. Explicit deferrals (captured for future work) Each of the following is a known follow-up, documented here so future milestones inherit the boundary: - **configd-coupled daemons re-add**: if `dlmgmtd`/`ipmgmtd`/`inetd`/`rpcbind`/`fmd` are needed, design either a "minimal SMF infrastructure under zyginit" (configd as a zyginit-supervised daemon) or a per-daemon configd-bypass shim. - **Runlevel targets**: `zygctl single`, `zygctl multiuser` stubs shipped; implementation needs a target-set schema extension (e.g., `service.runlevels = ["single", "multiuser"]`). - **Milestone-style virtual services**: systemd-target equivalent; probably needed to implement runlevels cleanly. - **`/sbin/halt`, `/sbin/reboot`, `/sbin/poweroff`, `/usr/sbin/shutdown` wrappers**: replace with versions that route through `zygctl`. - **`init ` runlevel compatibility**: unsupported; use zygctl. - **Crash-recovery re-adoption**: architecturally not supported (illumos descendant requirement); documented. - **NOORPHAN revisit**: current template is correct for PID-1 re-exec; revisit only if a non-exec crash-recovery goal is added. - **GC tuning under sustained uptime**: track separately. - **SMF removal from disk**: Phase C — once zyginit is stable, optionally remove `/lib/svc/`, `/etc/svc/` from the BE entirely. - **Boot performance targets**: no gate. - **Additional services** (resource limits, audit, DNS, NFS, zones): add as needed post-first-boot. - **Hostid persistence**: first boot reads `/etc/hostid`; generation logic (for first-ever boot) not yet implemented. - **Modern datalink/IP semantics**: BSD-pivot loses `dladm`/`ipadm` query interfaces. For zone hosts, laptops, multi-homed servers, this design is insufficient; revisit when those use cases are in scope. ## 12. Pre-requisites completed before this spec - **Contract probe**: 16/16 FFI probes pass on hh-prototest (rev 19) - **contract.reef bugs fixed**: P_CTID 14→13, ctl fd O_WRONLY (rev 18) - **Path rename**: `/etc/zyginit.d/` → `/etc/zyginit/` (rev 20, 21) - **Signal handling fix**: dead-code flag-clearing block removed (rev 15) - **Build split**: `helpers.c` / `contract_linux_stubs.c` separated (rev 16) - **Disk backup**: `hh-prototest-backup-2026-04-24-pre-zyginit-pid1.qcow2` captured on dev host **Pre-requisites added during iter 1-5 of v1 (kept in v2):** - **fd-setup at PID 1** (rev 45): `setup_pid1_console()` opens `/dev/console` and dup2's onto 0/1/2 before any I/O - **argv[0] fix** (rev 44): supervisor.start_service includes program name as argv[0] - **uadmin/sync wiring** (rev 24-31): full PID-1 shutdown path - **Tier-ordering enforcement** (rev 47): supervisor waits for tier N to settle before starting N+1 - **Persistent log dir** (rev 48): `/var/log/zyginit/` default - **root-fs ZFS/UFS dispatch** (rev 43): correct readonly toggle per fstype - **zygctl shutdown commands** (rev 33-34): halt/reboot/poweroff + single/multiuser stubs - **Path corrections in scripts** (rev 49-50): `/bin/hostname`, `/bin/rm`, `/sbin/swapadd` (no `-a`), `/sbin/dumpadm`, etc. ## 13. Open questions / defer to implementation Most v1 open questions are now answered (uadmin constants verified; ipmgmtd path verified; ttymon argv conventions known). v2 open questions answered through 2026-04-25 hh-prototest probing: - ✅ **`ifconfig` capability without ipmgmtd**: `ifconfig lo0:99 plumb`, `ifconfig lo0:99 inet 127.99.0.1 netmask 255.0.0.0 up`, and `ifconfig lo0:99 unplumb` all worked silently while ipmgmtd was running. `ifconfig` operates at the DLPI/kernel-ioctl layer for plumbing; ipmgmtd integration is for *persistent state*, not for runtime configuration. - ✅ **`ifconfig dhcp ` syntax**: per ifconfig(8) man page on hh-prototest, supported subcommands are `start | stop | drop | extend | inform | ping | release | status`. `ifconfig vioif0 dhcp status` returned valid info while pinging the running dhcpagent on TCP 4999. - ✅ **`/etc/hostname.` parsing**: Hammerhead's `ifconfig` does NOT read this file directly. The file is consumed by SMF method scripts. zyginit's `network/start.sh` will parse the file itself (line-oriented format documented in §4.4). - ✅ **`/etc/defaultrouter`**: empty/absent on hh-prototest (DHCP supplies the route). Format: one IPv4 address per line. `network/start.sh` reads when present and DHCP isn't in use. Remaining for iter 6 to empirically verify: - ⚠ **DHCP-via-dhcpagent without ipmgmtd running**: dhcpagent links libipadm which talks to ipmgmtd. Plan: try `ifconfig dhcp start` after starting dhcpagent fresh under zyginit; if it works, ship as default; if it fails, ship static-IP-only and document. The test VM has a libvirt-supplied dnsmasq DHCP server on 192.168.122.0/24 — perfect environment to verify. - ⚠ **First-boot fallback**: if DHCP fails, our test config will use a static IP (e.g., `inet 192.168.122.50/24`) chosen outside libvirt's dnsmasq pool to avoid conflicts. # Contract Re-adoption (Live Replace) Design **Date:** 2026-05-02 **Status:** Approved (brainstorm complete); implementation plan to follow **Scope:** Live in-place replacement of the running zyginit PID-1 process — replacing the `/sbin/init` binary while preserving every running service across the swap. Operator-driven only; not a crash-recovery design. ## 1. Goal Allow an operator (or automation tool) to upgrade the running zyginit binary without rebooting and without dropping any service. After the operation, every previously-RUNNING service continues to run with the same contract ID, accumulated `restart_count`, and continuous `last_start_time` (uptime). The connection between zyginit and supervised services is preserved across the exec; the only thing replaced is the program text and in-memory state of PID 1 itself. The verb the operator uses is `zygctl replace`. The mechanism is `execve("/sbin/init", ...)` against the same proc-table slot, with a state-file handoff for in-memory bookkeeping. ## 2. Non-goals - **Crash recovery.** If zyginit segfaults / panics, the kernel's `restart_init` re-execs `/sbin/init` in a tight loop, but the contracts are not preserved across that path (NOORPHAN templates would have killed members already). PID 1 dying is a much larger issue than this design solves; we explicitly do not try. - **Live binary swap.** `zygctl replace` does not copy or stage a binary. The operator (or package manager / configuration management tool) is expected to drop the new binary at `/sbin/init` first, then call `zygctl replace`. We hard-wire the exec target. - **Switching to a different init system.** `zygctl replace` always execs `/sbin/init` with no path argument. Switching init systems is a rescue-media operation. - **Signal-driven trigger.** No SIGUSR2 handler. Socket is the only path. (Revisit only if a real demand surfaces; see §10.) - **Crash-recovery argv-scan / proc-walk.** No fallback that reconstructs identity from `/proc//psinfo`. State file is the only source of truth. - **Recovery for transient-state services** (`STARTING`, `STOPPING`, `WAITING`-for-restart). `replace` refuses on transient state by default; with `--wait=N` it polls for stability before refusing. `--force` is **not** in scope for v1. - **Preserving in-flight zygctl connections.** Clients lose their connection at exec; that's expected. Operators reconnect for follow-up commands. ## 3. Why this works on Hammerhead — the key insight On illumos / Hammerhead, process contracts are tracked on the process structure (`proc_t.p_ct_process` / `p_ct_equiv`). When a process calls `execve()`, the kernel replaces the program text and (re-)initializes most of `proc_t`, but **the proc-table slot and contract ownership survive**. This is what makes live replace possible: PID 1 is the same `proc_t` before and after the exec, so all contracts it held continue to be held by it. Consequence: the new zyginit **does not need `ct_ctl_adopt`**. Adoption is for the SMF-startd model — a non-PID-1 supervisor dies and its successor (a fresh process with a different `proc_t`) needs to claim the orphan-inherited contracts. For PID-1 in-place exec, kernel-level ownership is automatic; we only need to rebuild the userspace bookkeeping (which contract belongs to which service, and what runtime fields to carry over). The bundle fd (`/system/contract/process/bundle`) does close at exec (we don't try to inherit it via FD_CLOEXEC manipulation), so the new zyginit re-opens it. Events that fire during the gap between old-process exec and new-process bundle re-open are recovered via a post-startup `ct_status_read` check on each known contract — if the contract is already empty, we synthesize an empty event into the existing `handle_contract_event` path. This is idempotent regardless of whether the gap actually missed any events. ## 4. Architecture ``` Operator | v zygctl replace [--wait=N] | +------+-------+ | socket | "replace queued" | (sock.reef)|----------------> zygctl prints, exits +------+-------+ | set replace_requested flag v main.reef event loop | v replace_self() | +-----------------+-----------------+ | 1. preconditions (--wait optional)| | 2. drain pending bundle events | | 3. write state.toml (atomic) | | 4. unlink socket | | 5. close internal fds | | 6. execve("/sbin/init", argv) | +-----------------+-----------------+ | v /// SAME proc_t, new program text /// | v new main() startup | +-----------------+-----------------+ | 1. setup_pid1_console (as today) | | 2. recover_state(boot_time) | | — reads state.toml, validates | | boot_time, returns entries | | 3. load enabled.d/ TOML (as today)| | 4. apply_recovered_state(table, | | recovered) | | 5. tier-by-tier start_service | | (skips entries already RUNNING)| +-----------------+-----------------+ | v normal main loop ``` **Module split:** | File | Change | Why | |---|---|---| | `src/replace.reef` | new (~200 lines) | Holds the state-file format, serialization, and recovery logic. Keeps `main.reef` lean. | | `src/main.reef` | +60 lines | `replace_self`, `apply_recovered_state`, integration with main loop and startup. | | `src/socket.reef` | +30 lines | `replace [--wait=N]` command handler, `replace_requested()` / `clear_replace_flag()` / `replace_wait_seconds()` accessors. | | `src/contract.reef` | +20 lines | `is_contract_empty(ctid): bool` helper wrapping `ct_status_read` + member-count check. | | `tools/zygctl/src/zygctl.reef` | +20 lines | `replace [--wait=N]` subcommand. | No new C helpers. No FFI additions (`adopt_contract` already exists but is not used by this flow; it stays for potential future uses). ## 5. State file format **Location:** `/var/run/zyginit/state.toml`. Same directory currently used for the socket. On the current zyginit-only BE, `/var/run` is persistent across real reboots (not yet tmpfs); the boot-id check (§5.2) makes stale post-reboot files harmless. **Format:** TOML, parseable by the existing `encoding.toml` parser. ```toml boot_time = 1746204000 [[service]] name = "sshd" contract_id = 23 restart_count = 0 last_start_time = 1746204051 last_exit_code = -1 [[service]] name = "syslogd" contract_id = 17 restart_count = 1 last_start_time = 1746204047 last_exit_code = 0 ``` ### 5.1 What is and isn't serialized Serialized: - `boot_time` — PID 1's process start time, read from `/proc/1/stat` (Linux) or `/proc/1/psinfo` (Hammerhead). Survives `execve` but changes on real reboot. Used as a staleness sentinel — see §5.2. - For each service in `STATE_RUNNING`: - `name` — the canonical service name (matches `enabled.d/` symlink basename). - `contract_id` — kernel-assigned contract ID. - `restart_count` — accumulated restart counter (for `max_retries` accounting). - `last_start_time` — Unix timestamp of the most recent start. - `last_exit_code` — most recent exit code (or `-1` if never exited). Not serialized: - `pid` — recoverable from contract status if needed; not needed for the contract-driven supervision path. - `state` — only `RUNNING` services serialize (per §6 pre-conditions); the field is implicit. - `stop_requested_time`, `restart_scheduled_time` — only meaningful in transient states, which we refuse. - `def` (the parsed TOML ServiceDef) — the new zyginit re-reads `enabled.d/` from disk, so the def comes from current TOML on the new side. This means a TOML edit between old-start and replace takes effect on next restart. (See §7.3 for the divergence handling.) ### 5.2 Boot-id staleness check The boot-id sentinel is PID 1's process start time, read from `/proc/1/stat` field 22 (Linux: jiffies-since-boot) or `/proc/1/psinfo pr_start.tv_sec` (illumos/Hammerhead: Unix timestamp) via `zyginit_read_pid1_start()` in `helpers.c`. This value survives `execve` (same `proc_t` slot) but changes on real reboot (new PID 1). If `state.toml.boot_time != current.boot_time`, the file is from a different system uptime — most likely a stale file left over from before a real reboot. We log `"replace: stale state file (boot_time mismatch); ignoring"`, delete the file, and proceed as a fresh boot. **Why not utmpx BOOT_TIME?** The original design used `utmpx BOOT_TIME` as the sentinel. This was found to be incorrect: `pututxline` with `ut_type = BOOT_TIME` *updates* the existing matching record (by design, to support `who -b`), rather than appending a new one. After a live replace, the new zyginit's startup call to `zyginit_write_boot_utmpx()` overwrites the original BOOT_TIME with a fresh timestamp. The new zyginit then reads its own updated timestamp as the current boot_time — which matches the new timestamp it just wrote — while state.toml has the old timestamp from before the replace. Result: mismatch, state.toml discarded, recovery fails silently. `/proc/1/start` avoids this entirely: no write occurs to it at startup, and it is immutable for the life of the PID 1 proc_t. This guards against the edge case where contract IDs are re-used after a real reboot (the kernel does recycle them) and we'd otherwise adopt the wrong contract. The check is cheap (one integer comparison) and correct in the common case (replace = same boot_time; reboot = different boot_time). The state file is **deleted by the new zyginit after a successful read**, regardless of whether all entries were applied successfully. This way, a panic-restart loop (kernel re-execing `/sbin/init` repeatedly) does not keep trying to apply long-dead entries. ### 5.3 Atomic write Writing follows the standard pattern, using the existing `zyginit_fsync_path` C helper: 1. `open("/var/run/zyginit/state.toml.tmp", O_WRONLY|O_CREAT|O_TRUNC, 0600)` 2. Write boot_time + each service entry. (Reef-side TOML emit is straightforward; we don't need the full encoding.toml emit machinery, just print to fd.) 3. `fsync(fd)`, `close(fd)`. 4. `rename("/var/run/zyginit/state.toml.tmp", "/var/run/zyginit/state.toml")`. 5. `zyginit_fsync_path("/var/run/zyginit")` — fsync the parent directory so the rename is durable. Any failure aborts the replace (we do not exec). At most we leave a `.tmp` file, which is overwritten on the next attempt and never read. ## 6. Operator interface ### 6.1 `zygctl replace` ``` zygctl replace [--wait=N] ``` - Connects to `/var/run/zyginit.sock`. - Sends `replace [--wait=N]\n`. - Reads reply (`replace queued (wait=N)\n` or an error). - Prints reply, exits 0 on success, non-zero on parse error from server. - The TCP/Unix-socket connection itself drops mid-response if the server proceeds to exec. zygctl treats `EPIPE` / connection-closed-after-ack as success and prints `replace: socket closed (process replaced)\n`. ### 6.2 `--wait=N` If any service is in a transient state (`STARTING`, `STOPPING`, `WAITING`-for-restart), the server polls once per second for up to `N` seconds, checking whether all services are in stable states (`RUNNING`, `STOPPED`, `FAILED`, `MAINTENANCE`, `DISABLED`). If stability is reached within the window, replace proceeds. If not, replace is refused with `"replace blocked: :, ..."` logged to syslog (and to stderr if attached). Default is `--wait=0` (refuse immediately). The wait happens server-side after the ack to the client, so the zygctl client doesn't have to hold a connection open — automation can `zygctl replace --wait=120` and either get a clean dropped connection (success) or, if the wait times out without exec'ing, can detect the failure by reconnecting and inspecting service states. ### 6.3 Pre-conditions in detail Before exec, zyginit checks: 1. **No service is in a transient state.** (After `--wait` window if specified.) 2. **`state.toml.tmp` write succeeds**, including fsync. 3. **`rename` to `state.toml` succeeds**, including parent-dir fsync. If any of these fail, replace aborts cleanly and the running zyginit continues. The flag is cleared, no exec happens, and the failure is logged. The operator can retry. The socket unlink and fd cleanup happen *after* a successful state write — they are last-mile actions before exec, and we don't fail the replace if they error (we log and proceed). ## 7. Recovery flow (new zyginit) ### 7.1 Startup sequence ``` main(): parse argv, kernel boot args if pid == 1: setup_pid1_console() boot_time = read_boot_time() // /proc/1/start (pid1 start time) or time_now() recovered = replace.recover_state(boot_time) config = load_enabled_dir("/etc/zyginit/enabled.d") table = build_service_table(config) apply_recovered_state(table, recovered) // <-- new step // Enter normal startup: tier-by-tier start_service // (services already in STATE_RUNNING are skipped) main_loop() ``` ### 7.2 `apply_recovered_state` algorithm For each `rs` in `recovered.services`: ``` idx = supervisor.find_service(table, rs.name) if idx < 0: // Service was in state.toml but is no longer in enabled.d/. // Operator removed it dirty (manual symlink removal between old start // and replace). Don't kill — let the process exit on its own and the // kernel reap the contract. We just stop tracking it. contract.abandon_contract(rs.contract_id) log("replace: orphaned " + rs.name + " (no longer enabled)") continue // Found in current table — patch in runtime fields. rt = table.runtimes[idx] rt.state = STATE_RUNNING rt.contract_id = rs.contract_id rt.pid = -1 // unknown; not needed rt.restart_count = rs.restart_count rt.last_start_time = rs.last_start_time rt.last_exit_code = rs.last_exit_code table.runtimes[idx] = rt table.contract_map.set(int_to_str(rs.contract_id), idx) // Idempotency: did the contract go empty during the exec gap? if contract.is_contract_empty(rs.contract_id): log("replace: " + rs.name + " contract empty post-recovery, applying restart policy") supervisor.handle_contract_event(table, rs.contract_id, -1) // existing logic resolves to STOPPED or scheduled-for-restart ``` ### 7.3 Config divergence between old start and replace Three cases the new zyginit must handle: | Case | Detection | Action | |---|---|---| | Service in state.toml AND in current `enabled.d/` | `find_service` returns valid idx | Adopt: patch runtime fields, register in `contract_map`. Use **current** TOML for future supervision (so an edit takes effect on next restart). | | Service in state.toml, NOT in current `enabled.d/` | `find_service` returns -1 | Abandon contract, log warning. Member processes continue running but unsupervised; kernel reaps the contract on their exit. | | Service in current `enabled.d/`, NOT in state.toml | Not in recovered list | Treated as a freshly-enabled service. Tier-by-tier startup will `start_service` it normally on first event-loop tick. | This means automation flows like `coral install some-service-package; zygctl replace` work correctly — the newly-installed service appears in `enabled.d/` and gets started by the new zyginit even though the old one never knew about it. ### 7.4 No `adopt_contract` calls As established in §3, contract ownership survives execve at the kernel level. The new zyginit's bookkeeping is entirely in userspace: rebuild `contract_map`, populate `ServiceRuntime` fields. The `contract.adopt_contract` FFI remains in the codebase for potential future use (e.g., a non-PID-1 zyginit "supervisor mode" recovery flow) but is not exercised by this design. ## 8. Error handling ### 8.1 Pre-exec (in old zyginit) Every pre-exec failure aborts the replace cleanly — old zyginit returns to its main loop with the flag cleared. | Failure | Behavior | |---|---| | Service in transient state, `--wait` exhausted | Log `"replace blocked: :, ..."`. Clear flag. Return to main loop. | | `state.toml.tmp` open / write / fsync fails | Log error with errno, unlink any partial `.tmp`. Return to main loop. | | `rename(state.toml.tmp, state.toml)` fails | Log error. Return to main loop. Stale `state.toml` from prior aborted attempt has the wrong boot_time and is harmless. | | `unlink("/var/run/zyginit.sock")` fails | Log warning, **proceed to exec**. New zyginit's `unix_listen` will overwrite the path. | | `process.process_exec()` returns | Means kernel couldn't load `/sbin/init` (binary missing or corrupt). Log fatal. Old zyginit is in an unrecoverable state (state.toml written, socket unlinked). Fall through to the existing PID-1 infinite-sleep handling. Operator's recourse is reboot. Should never reach this in practice. | ### 8.2 Post-exec (in new zyginit) Failures here are per-entry — one bad entry doesn't stop other recoveries. | Failure | Behavior | |---|---| | `state.toml` missing | Treated as fresh boot. Not an error; no log. | | `state.toml` parse error / corrupt | Log warning, treat as missing. Surviving contracts (if any) become orphans. | | `state.toml.boot_time != current` | Log info, delete file, treat as missing. | | Service in state.toml but not in current `enabled.d/` | Abandon contract, log warning. (See §7.3.) | | `is_contract_empty(ctid)` post-recovery returns true | Synthesize a contract event; existing `handle_contract_event` applies restart policy. | | `ct_status_read` returns error (contract gone, `ENOENT`) | Treat as empty / missing. Service starts fresh on first tick. | After all entries are processed (success or otherwise), `state.toml` is deleted. This breaks panic-restart loops where the same bad state would otherwise be re-applied repeatedly. ### 8.3 Panic safety `recover_state` and `apply_recovered_state` are written defensively. Any unexpected condition is logged and skipped — never panicked on. A bad state.toml entry must never prevent the new zyginit from completing startup. ## 9. Testing ### 9.1 Linux dev (no contract FFI) The Linux contract stubs return -1 for all contract calls, so end-to-end re-adoption can't run on Linux. What we *can* unit-test: - **State serialization round-trip** (`tests/test_replace.reef`): build a synthetic `ServiceTable` with services in known states, call `serialize_state`, read back with `recover_state`, assert all fields match. Cover empty table, all-RUNNING, mixed states (only RUNNING entries serialize), boot-id mismatch (ignored), corrupt TOML (ignored). - **Pre-condition check**: factor the transient-state scan into an exported `check_replace_preconditions(table) -> string` (empty on OK, error message otherwise). Test transient states individually and combined. - **Stitching logic**: unit-test `apply_recovered_state` covering the three cases of §7.3. Linux contract stubs let `is_contract_empty` always return false; we exercise the in-state-and-enabled and in-state-not-enabled branches. - **Integration smoke** (`tests/integration/run_tests.sh`): start zyginit-as-supervisor (`ZYGINIT_CONFIG_DIR=/tmp/...`), run a few oneshot+daemon services, run `zygctl replace`. On Linux, contract-driven supervision is partially stubbed but the exec path itself runs. Assert the second zyginit instance starts, its log shows `"replace: stale state file"` or `"replace: contract empty post-recovery"` per stub returns, and no crash. ### 9.2 Hammerhead (`hh-prototest`) Real validation. Deploy via the iter 11+ procedure (`scp` source, `gcc helpers.c`, `reefc build`, `mv build/zyginit /sbin/init.new && mv /sbin/init.new /sbin/init`). Test cases: 1. **Smoke** — multi-user up; `zygctl replace`; confirm: - Connection drops with the documented message. - Within ~1s, `zygctl status` works again. - All services still running, **same PIDs**, **same contract IDs**, restart counts preserved. - Uptimes preserved (continuous, not reset). - **The SSH session driving the test stays alive across the replace.** This is the highest-signal check. - `who -b` and `who -r` still report correctly. 2. **Transient-state refusal** — start a service with a slow `start.sh` (or watch for the WAITING window during a restart); run `zygctl replace` during the transient window; confirm refusal with the right error. 3. **`--wait=N` success** — same scenario with `--wait=10`; confirm replace proceeds once the service stabilizes. 4. **Dirty disable** — `rm /etc/zyginit/enabled.d/` (without `zygctl reload`); `zygctl replace`; confirm orphan path runs (contract abandoned, process keeps running, log message). 5. **Crash during exec gap** — `kill -9 ` immediately before `zygctl replace`; confirm new zyginit's `is_contract_empty` post-recovery catches the empty contract and applies restart policy. 6. **Stale state.toml after real reboot** — write a state.toml file, do a real reboot, file persists in `/var/run/zyginit/`; confirm boot-id mismatch is detected, file deleted, fresh boot proceeds. 7. **Repeated replace** — `zygctl replace` 5 times in succession; confirm restart counts accumulate correctly (no resets), uptimes grow continuously, no state-file leakage. ### 9.3 Out of scope for testing - Crash recovery (operator-driven only). - Replace under heavy load (we accept some bundle events will land post-exec; the post-recovery status check covers it). - Concurrent `replace` invocations (single-shot flag; second invocation between flag-set and exec is silently dropped — operator error). ## 10. Deferred / open questions These are intentionally not in scope for v1, but flagged so we can revisit: - **`zygctl replace --force`** — accept transient states by re-driving them (re-issue stop, re-arm restart timer, etc.). Adds significant recovery complexity. Defer until there's demonstrated automation demand. - **SIGUSR2 fallback** — operator-side trigger if the socket is wedged. The reload path (SIGHUP) shows that signal-driven flag-setting works; we just don't yet need the additional reachable code path for replace. Defer. - **Socket access tightening** — orthogonal to this design, but the socket is currently created with default permissions. It should be 0600 root-only. File as a separate hardening item. - **Bundle fd inheritance** — clearing FD_CLOEXEC and passing the fd through exec would close the (already-handled) gap window completely. Adds complexity for an already-mitigated case. Defer indefinitely. - **`/var/run/zyginit/` to tmpfs** — once `fs-minimal` is added back to the boot chain, `state.toml` automatically becomes ephemeral and the boot-id check becomes belt-and-suspenders. Tracked separately under §"Important Notes" in CLAUDE.md. - **utmpx BOOT_TIME as sentinel (rejected)** — utmpx BOOT_TIME's `pututxline` updates rather than appends, so the new zyginit overwrites the original timestamp at startup, defeating the staleness check. `/proc/1/start` naturally survives execve but changes on real reboot. See §5.2 for full rationale. ## 11. Summary of decisions | Decision | Choice | Rationale | |---|---|---| | Trigger model | Operator-driven only (live upgrade) | Crash recovery is a much harder problem; PID 1 dying is already catastrophic. | | Trigger interface | `zygctl replace` (socket only) | Smallest blast radius; signal fallback defers cleanly. | | Verb | `replace` | Operator-readable; covers upgrade + memory hygiene + state reset reasons; no overload with existing commands. | | Exec target | Hard-wired to `/sbin/init` | Security: never accept attacker-controllable exec paths in PID-1. | | Identity mapping | State file (single TOML) | Preserves runtime fields (`restart_count` especially); deterministic; matches deliberate-handover model. | | Pre-condition check | Refuse on transient state by default; `--wait=N` for automation | Eliminates an entire class of recovery bugs; automation has a clean knob. | | Bundle fd | Reopen, fix gap via post-recovery `is_contract_empty` | Idempotent, no fd-passing protocol needed. | | Config divergence | Stitch state.toml against current `enabled.d/`, abandon orphans | Permissive without complexity; supports `coral install + replace` automation. | | Adoption | **No `adopt_contract` call** | PID-1 in-place execve preserves kernel-level contract ownership; only userspace bookkeeping needs rebuilding. | | State file location | `/var/run/zyginit/state.toml` | Existing runtime tree; boot-id check makes it safe across real reboots. | ## 12. Why keep this in the Hammerhead-userland package? Hammerhead's primary upgrade workflow is BE-based: `pkg update` → new BE → `beadm activate` → reboot. In that workflow, `zygctl replace` is bypassed — the new zyginit comes up as a fresh PID 1 in the new BE. So why ship the feature? **Linux port.** zyginit is written in Reef which compiles to native C; a Linux port is realistic. Linux package managers (`apt`, `yum`, `dnf`) operate on an in-place upgrade model where daemons need a way to reload their own binary without restarting the system. systemd's `systemctl daemon-reexec` exists for exactly this reason. If zyginit lands on Linux, `zygctl replace` becomes a primary feature. **Live patching trajectory.** A system that supports live kernel patching (kpatch-style) has implicit cultural pressure to support live userland patching for critical daemons. zyginit being live-upgradeable is a natural fit for that posture if Hammerhead develops it. **Mechanism reuse.** Several pieces (`contract.is_contract_empty`, `supervisor.adopt_runtime`, the state-file pattern in `utils/replace_probe`) have potential consumers beyond this feature — shutdown verification, future crash-recovery work, generic state-file testing patterns. The feature stays; the trigger is documented as Hammerhead-secondary, Linux-primary. # SMF Migration Handoff Document — Design **Date:** 2026-05-04 **Status:** Approved (brainstorm complete); implementation plan to follow **Scope:** A single Markdown document at `docs/SMF_MIGRATION.md` that hands off the remaining SMF-removal and zyginit-integration work from the zyginit project to the Hammerhead-project Claude (or Hammerhead human engineers). No tooling. No code. ## 1. Goal Hand off the remaining SMF removal work to the Hammerhead team (or a Hammerhead-project Claude agent) with everything they need to: 1. Vendor zyginit source from this repo into Hammerhead's source tree as part of `hammerhead-userland`. 2. Wire zyginit into Hammerhead's master build, install paths, and packaging. 3. Migrate the existing 18 zyginit-converted services (currently in `services/hammerhead/`) into Hammerhead's source tree as the starting point. 4. Convert the remaining ~110 SMF manifests using the patterns documented in this work, the existing converted services as references, and project judgment about what Hammerhead actually ships. 5. Remove SMF infrastructure (`svc.startd`, `svc.configd`, master restarter, libscf consumers, manifest packaging) from `hammerhead-userland`. The deliverable is **one Markdown document**. zyginit ships docs and patterns; Hammerhead does the OS-level execution. ## 2. Non-goals - **No `smf-to-toml` conversion tool.** A Claude agent doing the conversion in-place can do both the mechanical XML-to-TOML translation AND the per-service judgment that a tool can't. Avoiding the tool also keeps zyginit free of a Python runtime dependency and aligns with the structural rule "DO NOT modify Hammerhead source code" — the Hammerhead Claude does that work in its own project. - **No per-service detailed catalog with full recommendations.** zyginit doesn't know what Hammerhead ships; per-service decisions stay with the Hammerhead team. The doc provides a coarse one-line classification per manifest and lets the Hammerhead Claude finish each per-service decision with full local context. - **No format reference duplication.** TOML field-by-field details stay in `man/zyginit.5`. The migration doc references it. - **No build-system implementation.** The doc describes what the Hammerhead build needs to do; the Hammerhead Claude integrates it with the actual build conventions in their tree. - **No Coral packaging for zyginit.** zyginit is part of `hammerhead-userland` (BSD-style base system), built into the OS by Hammerhead's master build. - **No automated catalog maintenance.** The catalog is a snapshot of one Hammerhead manifest tree at one point in time; it is archival once the migration is complete. ## 3. Document structure The single Markdown file at `docs/SMF_MIGRATION.md`, ~600-800 lines. Top-level sections: ``` 1. Handoff Brief (~100 lines) 2. Decision Tree (~150 lines) 3. Worked Examples (~200 lines) 4. Anti-patterns & Gotchas (~100 lines) 5. Integration with Hammerhead (~150 lines) 6. Per-Category Guidance (~100 lines) 7. Catalog Appendix (~200 lines, table format) 8. Format Reference Pointer (~10 lines) 9. Glossary (~30 lines) ``` The doc is intended for both human engineers and AI agents working on the Hammerhead source tree. Format optimized for human readability; structured enough that a Claude agent can load it as context and act on it. The file lives at `docs/SMF_MIGRATION.md` (top-level under `docs/`, alongside `DESIGN.md`). The spec for THIS doc lives at `docs/superpowers/specs/2026-05-04-smf-migration-doc-design.md`. The migration doc itself stays at the top level to signal "this is a real artifact for the Hammerhead team," not a planning artifact. ## 4. Section content ### 4.1 Handoff Brief Sets up the context for the Hammerhead Claude. Includes: - What zyginit has done so far (the iter 1-20 work; the contract-readoption feature; the 18 converted services in the boot chain on `hh-prototest`). - What's left for the Hammerhead team to do (source vendoring, build integration, migrate the 18 existing TOMLs, convert the remaining ~110 manifests, remove SMF infrastructure). - Coordination points: when to bounce a question back to the zyginit project (e.g., "the existing patterns don't fit this service" → file a zyginit issue; we extend the format/runtime if needed). - Out-of-scope reminders ("DO NOT modify Hammerhead source from inside zyginit's tree" — this constraint is for the zyginit-project Claude, not the Hammerhead Claude; the Hammerhead Claude WILL modify Hammerhead source — that is its job). - Done criteria for "SMF removal complete": Hammerhead boots and reaches multi-user without `svc.startd` / `svc.configd` ever being invoked; libscf-only consumers gone or stubbed; manifest infrastructure removed from packaging; existing zyginit-supervised services continue to work. ### 4.2 Decision Tree The most consequential single piece of the doc — what the Hammerhead Claude leans on for every manifest. Three top-level exits per manifest: ``` Step 1: Should this service exist on Hammerhead at all? ├── No (deprecated, replaced) │ → DROP. Remove from package; do not write a TOML. ├── Yes, but as zports add-on, not base │ → DEFER. Out of scope for hammerhead-userland. └── Yes, ships in hammerhead-userland → continue to Step 2 Step 2: Is the service tightly coupled to SMF infrastructure (svc.configd / libscf / repository_door)? ├── Yes, fundamentally requires libscf at runtime │ → INVESTIGATE. Find a foreground/standalone mode, or │ coordinate with the daemon's upstream to add one. ├── Yes, but the daemon has a non-SMF mode flag (-d, -f, -D) │ → CONVERT (foreground-mode pattern). See Example 1. └── No, or coupling is incidental → continue to Step 3 Step 3: What runs the service today? ├── Direct daemon binary, no SMF method script │ → CONVERT (full-TOML pattern, simplest case) ├── /lib/svc/method/ shell script that does setup before exec │ ├── Trivial setup → INLINE into TOML's exec.start │ ├── Substantial setup (sources lib/svc/share/*.sh) → KEEP │ method script as exec.start. See Example 2. ├── Oneshot configuration step (mountall, swapadd, dumpadm) │ → CONVERT as oneshot service. See services/hammerhead/ │ root-fs, swap, filesystem. └── inetd-managed service → DROP unless strong reason to keep. We don't run inetd. ``` Within "CONVERT," a field-by-field translation table maps SMF manifest XML elements to TOML fields: | SMF concept | TOML field | Notes | |---|---|---| | `` | `[service] name = "X"` | Strip category prefix | | `