zygaena / zyginit
|
Files Changesets Issues Pull Requests Releases
root / docs / superpowers / specs / 2026-04-24-pid1-boot-design.md
2026-04-24-pid1-boot-design.md markdown 549 lines 31.3 KB
Raw

PID-1 Boot Design

Date: 2026-04-24
Revision: 2026-04-25 — BSD-pivot (post iter 1-5 findings)
Status: Revised; iter 6 implementation pending against the new service set
Scope: First boot of zyginit as PID 1 on Hammerhead (hh-prototest VM), replacing the SMF-managed boot chain with a zyginit-managed BSD-style boot.

1. Goal

Boot hh-prototest with /sbin/init pointing at zyginit, bringing the system to a state where the operator can SSH in and use the box. Deliver a small BSD-style service set (≈15 services) that uses direct command invocation (ifconfig, dhcpagent) instead of supervising SMF-coupled control-plane daemons. Tested rollback path via ZFS Boot Environments.

1.1 Design revision: BSD pivot (2026-04-25)

The original 21-service design (preserved in §4 history at the bottom of this doc) tried to supervise the existing illumos control-plane daemons — dlmgmtd, ipmgmtd, inetd, rpcbind, fmd — directly. Iter 1-5 (rev 24 through rev 50) surfaced a fundamental issue: these daemons are tightly coupled to svc.configd for their persistent property store via libscf. Setting SMF_FMRI in the environment bypasses their refusal-to-start banner, but they then silently degrade because scf_handle_decode_fmri() fails (no /etc/svc/volatile/repository_door), leaving them running but useless — ipmgmtd doesn't open its door socket; ipadm consequently fails with "Could not open handle to library"; lo0 is never plumbed; rpcbind cascades into "could not find loopback transports"; the whole network stack is dead.

We had three choices:

  • A: Run configd under zyginit — keeps the SMF property-store daemon alive, drops only svc.startd. Gets the daemons working but admits SMF is too entangled to fully escape.
  • B: Use /lib/svc/method/* scripts as zyginit's exec.start — same effective coupling to configd as A; just hides it behind shell wrappers.
  • C: Pivot to BSD-style direct invocation — drop the configd-coupled daemons entirely. Use /sbin/ifconfig directly to plumb interfaces and configure addresses. Use /sbin/dhcpagent directly for DHCP. Skip inetd/rpcbind/fmd for the default service set.

We chose C. Hammerhead retains the BSD primitives that make this work:

  • /sbin/ifconfig exists and successfully plumbs interfaces without needing ipmgmtd
  • /etc/hostname.<if> files supported (BSD-style interface naming/config)
  • /sbin/dhcpagent is callable directly
  • /etc/init.d/, /etc/rc[0-3].d/ directories present (we won't use them, but the convention is there)

Reference for the approach: IRIX 6.5's /etc/init.d/network script (os_research/irix-6517-src/eoe/cmd/initpkg/init.d/network) does exactly this — IFCONFIG=/usr/etc/ifconfig, ROUTE=/usr/etc/route, flat-file configuration in /etc/config/, no daemon-based registry. zyginit-on-Hammerhead becomes essentially a Reef rewrite of pre-SMF Solaris boot semantics, replacing what IRIX/Solaris-9 did with shell.

Lessons from iter 1-5 also drove these other concrete fixes that survive in v2:

  • fd-setup at PID 1: kernel doesn't preopen 0/1/2; zyginit must open(/dev/console) and dup2() onto 0/1/2 before any I/O (rev 45)
  • argv[0] in supervisor: must include the program name as argv[0] when execvp'ing (rev 44)
  • root-fs ZFS/UFS dispatch: mount -o remount,rw / errors on ZFS root; use zfs set readonly=off (rev 43)
  • Tier ordering enforcement: between tiers, wait for oneshots to STOPPED and daemons to RUNNING before starting next tier (rev 47)
  • Persistent log dir: /var/log/zyginit/ (not /var/run) so logs survive reboot for diagnosis (rev 48)
  • Hammerhead path corrections: /bin/hostname (not /usr/bin/), /sbin/swapadd (no -a), /sbin/dumpadm, etc.
  • cron FIFO is at /etc/cron.d/FIFO, not /var/run/cron_fifo

A useful diagnostic capability also fell out: with the VM shut down, the BE's logs are readable from the dev host via qemu-nbd + zfsonlinux read-only import, then mounting hhtest/ROOT/zyginit to read /var/log/zyginit/*.log. This bypasses the GRUB-cant-pick-BE limitation on this Hammerhead version.

2. Non-goals

  • SMF removal from the BE. Phase C, separate milestone. First boot leaves SMF binaries installed-but-unused. We just don't invoke them from zyginit's service set.
  • Supervising configd-coupled daemons. dlmgmtd, ipmgmtd, inetd, rpcbind, fmd are explicitly out of zyginit's default service set. They depend on a running svc.configd that we no longer launch. They remain available on the BE for an admin who chooses to set up SMF separately.
  • Modern illumos datalink/IP management semantics. No dladm show-link/ipadm show-addr integration. Configuration is via flat files (/etc/hostname.<if>, /etc/defaultrouter, /etc/resolv.conf). For laptops with WiFi roaming, link aggregation, IPMP, or VNICs for zones, this design is insufficient — those workflows need the SMF stack.
  • On-demand network services (inetd-style). No inetd in the default set. Services that historically ran under inetd (telnet, rsh, finger, etc.) are not started. SSH is direct.
  • NFS / RPC. rpcbind not started; no NFS in scope.
  • Fault management. fmd not started; FMA events are not collected.
  • Single-user mode / runlevels. zygctl single and zygctl multiuser remain stubs.
  • Crash-recovery re-adoption. ct_ctl_adopt requires descendant relationship; only PID-1 in-place re-exec is supported.
  • /sbin/halt, /sbin/reboot, /sbin/poweroff, /usr/sbin/shutdown integration. Continue to call uadmin directly. Clean shutdown goes through zygctl.
  • Name services (NSS), NTP, performance targets. Deferred.

3. Architecture

3.1 Components

Component Path Type Purpose
zyginit /sbin/zyginit ELF binary Init daemon (PID 1)
zygctl /sbin/zygctl ELF binary Admin CLI
Config tree /etc/zyginit/ Directory Service TOMLs, per-service script dirs, enabled.d/ symlink farm
Runtime tree /var/run/zyginit/ Directory Per-service log files, socket, transient state
Control socket /var/run/zyginit.sock Unix socket zygctl ↔ zyginit IPC

3.2 Boot sequence at runtime

  1. Kernel execs /sbin/init (symlink to /sbin/zyginit)
  2. zyginit detects PID (getpid() == 1) and enters PID-1 mode
  3. zyginit scans /etc/zyginit/enabled.d/ for symlinks
  4. zyginit parses each TOML referenced; builds in-memory ServiceDef set
  5. zyginit builds dependency graph via topological sort into tier ordering
  6. zyginit starts services tier by tier:
    • Within a tier: sequential start (parallel start is a future optimization)
    • Between tiers: wait for all members of tier N to reach terminal state (RUNNING for daemons, STOPPED+exit=0 for oneshots) before starting tier N+1
  7. zyginit opens event loop on: contract bundle fd, signal self-pipe, zygctl socket
  8. System is "up" when all enabled services have started successfully

3.3 PID-1 vs non-PID-1 behavior

Condition Trigger Behavior
PID 1 SIGTERM / SIGINT Halt: stop services in reverse tier order, sync, uadmin(A_SHUTDOWN, AD_HALT, 0)
PID 1 zygctl halt As above
PID 1 zygctl reboot Stop services, sync, uadmin(A_SHUTDOWN, AD_BOOT, 0)
PID 1 zygctl poweroff Stop services, sync, uadmin(A_SHUTDOWN, AD_POWEROFF, 0)
not PID 1 SIGTERM / SIGINT Stop services, return from main() (normal process exit)
not PID 1 zygctl halt Stop services, exit
not PID 1 zygctl reboot / poweroff Refuse ("not PID 1") — safety rail for dev/test

4. Service set (BSD-pivot, v2)

15 zyginit services in enabled.d/ by default (counted across §4.1–§4.6). Replaces the 21-service v1 design (full v1 service table viewable in hg log -p for this file) by dropping the configd-coupled daemons in favor of direct command invocation.

4.1 Tier 0 — no dependencies

Service Type Script Stop
root-fs oneshot /etc/zyginit/root-fs/start.sh (ZFS or UFS root: zfs set readonly=off or mount -F ufs -o remount,rw /) /etc/zyginit/root-fs/stop.sh (matching readonly=on / remount,ro)

4.2 Tier 1 — requires root-fs

Service Type Script Stop
devfs oneshot inline: /usr/sbin/devfsadm —
swap oneshot inline: /sbin/swapadd (no -a; reads /etc/vfstab) /etc/zyginit/swap/stop.sh (swap -d each active entry)
crypto oneshot /etc/zyginit/crypto/start.sh (cryptoadm refresh) —

4.3 Tier 2 — requires tier 1

Service Type Script Stop
filesystem oneshot /etc/zyginit/filesystem/start.sh (mountall -l + zpool import -a + zfs mount -a) /etc/zyginit/filesystem/stop.sh (umountall -l + zpool sync)
identity oneshot /etc/zyginit/identity/start.sh (hostname from /etc/nodename) —
sysconfig oneshot /etc/zyginit/sysconfig/start.sh (coreadm, dumpadm, keymap, scheduler, rctl, rmtmpfiles) —

4.4 Tier 3 — requires filesystem + identity

Service Type Script Stop
network oneshot /etc/zyginit/network/start.sh — direct ifconfig invocation, no ipmgmtd: plumb lo0, walk /etc/hostname.<if> files for each physical interface, ifconfig <if> dhcp (which forks dhcpagent directly) or static address per file content, install default route from /etc/defaultrouter if present /etc/zyginit/network/stop.sh (kill dhcpagent, ifconfig unplumb interfaces)

Critical change from v1: no dlmgmtd or ipmgmtd services. ifconfig plumbs interfaces directly via DLPI. dhcpagent is invoked directly via ifconfig <if> dhcp start, not via ipadm -T dhcp. This matches IRIX 6.5's /etc/init.d/network pattern (IFCONFIG=/usr/etc/ifconfig, ROUTE=/usr/etc/route).

/etc/hostname.<if> file format (OpenBSD-inspired, illumos-translated)

Verified on hh-prototest: /sbin/ifconfig is fully operational without ipmgmtd (plumb, address assignment, link state — all work). The OpenBSD hostname.if(5) format is line-oriented with each line being ifconfig args. We adopt the same line-oriented convention but translate to illumos's ifconfig syntax, since OpenBSD-specific keywords like inet alias map to illumos's addif.

Line forms supported by zyginit's network/start.sh:

Line Action
(empty file or whitespace) ifconfig <if> plumb up — interface up, no address
# comment or blank line skipped
dhcp ifconfig <if> plumb; ifconfig <if> dhcp start (best effort — see DHCP caveat)
inet <addr>/<prefix> ifconfig <if> inet <addr>/<prefix> up
inet <addr> netmask <mask> ifconfig <if> inet <addr> netmask <mask> up
addif <addr>/<prefix> additional alias address (illumos addif); requires interface already plumbed by an earlier line
inet6 <addr>/<prefix> ifconfig <if> inet6 <addr>/<prefix> up (deferred to post-first-boot)
up / down set link state explicitly

Examples:

# /etc/hostname.vioif0 — DHCP via dhcpagent (best effort without ipmgmtd)
dhcp

# /etc/hostname.vioif0 — static IPv4
inet 192.168.122.50/24

# /etc/hostname.vioif0 — static IPv4 + alias
inet 192.168.122.50/24
addif 192.168.122.51/24

Default route

/etc/defaultrouter (one IPv4 address per line). network/start.sh runs route -n add default <addr> for each. Optional — DHCP-acquired routes don't need this file.

DHCP caveat

dhcpagent is linked against libipadm.so.1, which talks to ipmgmtd's door for persistent state. Without ipmgmtd running:

  • dhcpagent itself runs (self-contained DHCP state machine)
  • DHCP request/response on the wire works (uses raw sockets, not libipadm)
  • Applying the lease to the interface uses libipadm → ipmgmtd, which may degrade

Empirical test required on hh-prototest in iter 6 boot. If DHCP-via-ifconfig fails without ipmgmtd, fallback options:

  1. Manually invoke dhcpagent -a -f (adopt mode + foreground) and configure interface from received lease via direct DLPI ioctl
  2. Static IP only for the test VM (configure libvirt to have a known static address, set /etc/hostname.vioif0 to inet 192.168.122.50/24)
  3. Patch dhcpagent / replace with dhclient if available

The implementation plan should boot-test with static IP first to derisk the rest of the boot, then attempt DHCP as a follow-up.

4.5 Tier 4 — requires network (configd-free system daemons)

All daemons. All use stop.method = "contract" (default). restart.on = "failure", delay = 5, max_retries = 3.

Service exec.start Note
syseventd /usr/lib/sysevent/syseventd Kernel event delivery; doesn't read SMF properties
syslogd /usr/sbin/syslogd Reads /etc/syslog.conf directly; SMF-property reads degrade to defaults
utmpd /usr/lib/utmpd Cleans stale utmpx entries; no configd dependency
pfexecd /usr/lib/pfexecd RBAC profile-exec helper; reads /etc/security/, no configd

4.6 Tier 5 — user-facing services

Service exec.start Note
cron /etc/zyginit/cron/start.sh wrapper: rm -f /etc/cron.d/FIFO; exec /usr/sbin/cron
sshd /usr/lib/ssh/sshd -D -D keeps it foreground; reads /etc/ssh/sshd_config
console-login /etc/zyginit/console-login/start.sh wrapper exec'ing ttymon with properly-quoted prompt

4.7 Optional / installable — TOMLs ship but symlinks not in enabled.d/ by default

These TOMLs exist in services/hammerhead/ but are NOT enabled by default. An admin can enable them via zygctl enable <svc> on hardware/configurations where they make sense:

Service When to enable
powerd Real hardware with /dev/srn (not VMs)
fmd When fault management is wanted (requires configd workaround or future SMF-coexistence design)
dlmgmtd / ipmgmtd / inetd / rpcbind Only after a separate "run minimal SMF infrastructure under zyginit" design is implemented

4.8 Dropped from the v1 design (configd-coupled, requires separate design to revisit)

These daemons require svc.configd running for libscf calls to succeed. They are intentionally absent from the BSD-pivot v2 service set:

  • dlmgmtd — datalink management. Replaced by direct ifconfig plumbing. Drawback: no dladm show-link queries; persistent datalink state across reboots is via /etc/hostname.<if> instead.
  • ipmgmtd — IP interface management. Replaced by direct ifconfig/route invocation. Drawback: no ipadm queries.
  • inetd — on-demand service launching. Modern illumos inetd reads from configd. No replacement for first boot; can be re-added when needed (e.g., via xinetd or by accepting configd dependency).
  • rpcbind — only needed for NFS/RPC; deferred until NFS is in scope.
  • fmd — fault management daemon; heavy SMF coupling. Deferred.

4.9 SMF infrastructure (subsumed; no replacement)

  • svc.startd, svc.configd — replaced by zyginit (or absent in v2)
  • manifest-import, early-manifest-import — no XML manifests
  • boot-archive, boot-archive-update — SMF-specific
  • logadm-upgrade, update-man-index — admin/cron concerns, not init's
  • All 7 SMF milestones (virtual groupings) — zyginit's dependency graph replaces these

4.10 Currently-disabled SMF services — also not resurrected in zyginit

Mirrors v1: name-service-cache, rbac, process-security, auditd, NFS/DNS/LDAP stacks all stay disabled.

5. TOML schema usage

Uses the schema already documented in docs/DESIGN.md §"Service Definition Format". The following subset is in active use for the 21 services:

[service]
name = "..."
description = "..."
type = "daemon" | "oneshot"

[exec]
start = "command or /etc/zyginit/<svc>/start.sh"

[dependencies]
requires = ["..."]
after = ["..."]

# Daemons only:
[stop]
method = "contract"          # default; may be omitted
signal = "TERM"              # default; may be omitted
timeout = 60                 # default; may be omitted

[restart]
on = "failure"
delay = 5
max_retries = 3

[contract]
param = ["inherit", "noorphan"]
fatal = ["hwerr"]

# Oneshots with reversal:
[exec]
stop = "/etc/zyginit/<svc>/stop.sh"

[restart]
on = "never"

Defaults and omissions:

  • Omitted exec.user / exec.group → inherits from zyginit → root when PID 1. Explicit and documented so there's no ambiguity.
  • Omitted [stop] block → uses contract-signal defaults (TERM, 60s timeout).
  • Omitted [contract] block → uses template defaults from setup_template() in contract.reef.

Deferred schema fields (supported by zyginit, unused in first boot):

  • exec.user, exec.group, exec.working_directory, exec.environment — none of the 21 services need privilege drop or per-service env
  • security.*, limits.* — not implemented / not in first boot

contract.param = ["inherit", "noorphan"] note:
The NOORPHAN flag means that if zyginit-as-PID-1 dies without cleanly abandoning contracts, children receive SIGHUP. This is correct for PID-1 re-exec (kernel sees the same process, no SIGHUP fires). It is incompatible with a crash-recovery path that promotes a fresh zyginit PID — but such a path is an explicit non-goal per §2.

6. Method script conventions

All scripts live under /etc/zyginit/<svc>/start.sh (and optional stop.sh) unless the service is trivial enough for an inline TOML command.

Template:

#!/bin/zsh
# <service> — <one-line purpose>
# Invoked by zyginit via exec.start in /etc/zyginit/<service>.toml

set -e
set -u

# ... steps ...

exit 0

Conventions:

  1. Shebang: #!/bin/zsh (consistent with Hammerhead's shell policy; zsh is the system default, ksh93 is being phased out)
  2. Error handling: set -e (POSIX-style — preferred over setopt ERR_EXIT for cross-shell readability) + set -u
  3. Non-interactive only: no read, no tty-dependent behavior
  4. Self-contained: no sourceing of SMF helper scripts (/lib/svc/share/*.sh)
  5. No SMF API calls: no svcprop, svcadm, svccfg. Config input comes via exec.environment in the TOML.
  6. Explicit exit 0 on success (not relying on implicit last-command exit)
  7. Use || true deliberately per-command where a specific failure is acceptable (e.g., dumpadm -d /path || true if the dump device doesn't exist yet)
  8. Idempotent when the action allows: prefer mount -o remount,rw / (idempotent) over raw mount commands. Use "check then do" patterns for inherently non-idempotent operations (ipadm show-if lo0 2>/dev/null || ipadm create-if lo0).

Execution environment zyginit provides to scripts:

  • Fresh process (fork + exec)
  • Working dir: / unless TOML sets exec.working_directory
  • Environment: minimal (PATH, HOME, etc.) plus whatever exec.environment specifies
  • stdout/stderr: redirected to /var/run/zyginit/log/<svc>.log, truncated per start
  • User: inherited from zyginit (root at PID 1) unless exec.user set

7. Shutdown orchestration

Shutdown is a first-class concern, not deferred. Unclean shutdown can corrupt ZFS; we do it right from the first boot.

7.1 Shutdown triggers

Source Type
zygctl halt halt
zygctl reboot reboot
zygctl poweroff poweroff
SIGTERM / SIGINT (PID 1) halt (conservative default)
SIGTERM / SIGINT (not PID 1) exit (integration test path)

Shutdown type is communicated to zyginit via the socket protocol (for zygctl commands) or inferred from signal source.

7.2 Shutdown walk

  1. Set shutdown_requested = true, shutdown_type = <type>
  2. Break out of main event loop
  3. Walk services in reverse tier order (not insertion order)
  4. For each service in the tier:
    • Daemons in STATE_RUNNING: stop_service() → SIGTERM via contract, wait for contract EMPTY event, mark STATE_STOPPED
    • Oneshots in STATE_STOPPED with exec.stop defined (or /etc/zyginit/<svc>/stop.sh present): fork/exec the stop command; wait for exit
    • Oneshots without stop: skip (no reversal defined)
  5. Wait for all services in tier N to reach STATE_STOPPED before moving to tier N-1
  6. After tier 0 is drained: sync() (flush pending writes)
  7. Call uadmin(A_SHUTDOWN, <fcn>, 0):
    • AD_HALT (0) — halt, operator decides next step physically
    • AD_BOOT (1) — reboot immediately
    • AD_POWEROFF (6) — ACPI/BMC chassis off

7.3 Stop scripts concretely required

Service stop.sh action
root-fs mount -o remount,ro / (called after filesystem/stop.sh has umounted everything else)
filesystem umountall -l
swap swap -d each entry

Other services either use the contract-signal daemon default or have nothing to reverse.

7.4 Required code changes

supervisor.reef:

  • Extend shutdown path to walk tiers in reverse
  • For each tier: first send stops to all daemons, wait for them to exit, then run stop scripts for oneshots that have them
  • Currently shutdown_services skips STATE_STOPPED services — this must change

main.reef:

  • Track shutdown_type variable (halt / reboot / poweroff)
  • After shutdown_services(): call sync(), then zyginit_shutdown(<fcn>)

FFI: helpers.c and contract.reef (or new shutdown.reef):

  • int zyginit_shutdown(int fcn) → wraps uadmin(A_SHUTDOWN, fcn, 0)
  • extern "C" fn zyginit_shutdown(fcn: int): int

Linux stub: contract_linux_stubs.c gets a matching stub returning -1 (should never actually fire on Linux).

zygctl gets three new subcommands: halt, reboot, poweroff. Each sends a specific command over the socket; zyginit sets shutdown_type accordingly.

zygctl also gets two stub subcommands: single, multiuser. Both print "not yet implemented — reserved for future runlevel support" and exit 1. Documented in zygctl --help so the surface is stable.

8. Rollout phases

8.1 Phase A — non-PID-1 dry run

Run zyginit on the live hh-prototest system alongside SMF, using a scratch config directory.

ZYGINIT_CONFIG_DIR=/tmp/zyginit-dry \
  /path/to/zyginit

Goals:

  • TOMLs parse cleanly
  • Dependency graph builds without cycles; tier order matches the spec
  • zyginit supervises one or two non-conflicting test daemons (e.g., a second sshd on port 2222)
  • Clean shutdown via SIGTERM works without uadmin (non-PID-1 path)

Out of scope for Phase A:

  • Destructive oneshots (filesystem, network, devfs) — they'd conflict with SMF-managed copies. Skip these in the dry run.

8.2 Phase B — PID-1 boot in a fresh Boot Environment

  1. beadm create zyginit
  2. beadm mount zyginit /mnt
  3. Install into /mnt:
    • /mnt/sbin/zyginit, /mnt/sbin/zygctl
    • /mnt/etc/zyginit/*.toml (21 files)
    • /mnt/etc/zyginit/<svc>/start.sh + stop.sh as specified above
    • /mnt/etc/zyginit/enabled.d/ symlinks for all 21
    • Preserve old init: mv /mnt/sbin/init /mnt/sbin/init.smf
    • New init: ln -s /sbin/zyginit /mnt/sbin/init
  4. beadm umount zyginit; beadm activate zyginit; init 6

8.3 Phase C — SMF removal (deferred, separate milestone)

Optional follow-up once Phase B is stable:

  • In the zyginit BE: remove /lib/svc/bin/svc.startd, /lib/svc/bin/svc.configd, /lib/svc/method/*, /etc/svc/
  • Activate, reboot, prove the system still works without SMF
  • This is when "zyginit replaces SMF" is literally true.

8.4 Failure recovery ladder

In order of cost:

  1. zygctl reboot → pick hammerhead BE from GRUB menu (seconds)
  2. If GRUB menu inaccessible: boot into rescue media / single-user from install media, mount original BE, activate it
  3. If the pool is damaged: sudo cp hh-prototest-backup-2026-04-24-pre-zyginit-pid1.qcow2 hh-prototest.qcow2 on the dev host (2-3 min with VM shut down)
  4. Last resort: VM rebuild from scratch (~20 min)

Disk-level backup captured at rev 21. Recovery drill is part of §9.F.

9. Success criteria for Phase B

Literal checklist run after first boot into the zyginit BE.

A. Boot completes

  • No kernel panic: init died
  • zyginit: entering event loop appears on console
  • Every tier finishes within 60s total
  • Console login prompt appears on /dev/console

B. Expected service states

  • All 21 services present in zygctl list
  • Tier 0-3 oneshots: STOPPED with exit=0
  • Tier 3-5 daemons: RUNNING
  • No service in MAINTENANCE

C. Services actually functional

  • ssh zygaena@192.168.122.197 from dev host
  • logger -p user.info test → visible in /var/adm/messages (syslogd)
  • dladm show-link lists physical interface up
  • ipadm show-addr lists lo0 + physical interface with IPv4 addresses
  • ping 192.168.122.1 succeeds
  • crontab -l runs for root
  • zfs list works, hhtest pool mounted

D. zygctl end-to-end

  • zygctl status cron shows RUNNING with uptime > 0
  • zygctl stop cron → STOPPED, then zygctl start cron → RUNNING
  • zygctl log sshd returns without error
  • zygctl reload succeeds (SIGHUP path)

E. Clean shutdown

  • zygctl halt: reverse-tier-order stop output in console; system halted cleanly
  • zygctl reboot: cycles; comes back via zyginit again
  • zygctl poweroff: VM transitions to shut off in libvirt
  • After any zygctl shutdown: zpool status clean, no replay-required flags

F. Rollback confirmed

  • 3 clean reboot cycles into the zyginit BE
  • Reboot into original hammerhead BE from GRUB → SMF comes up normally
  • beadm activate hammerhead from zyginit BE → reboot → lands on hammerhead BE
  • Disk-level restore drill executed at least once (time to recovery measured)

G. Explicit non-requirements

  • No boot-time performance gate (measured for baseline only)
  • No stability soak
  • No DNS / hostname resolution testing
  • No NTP / time sync
  • No name services (NSS beyond /etc/passwd, /etc/hosts, /etc/group)
  • No zygctl single / zygctl multiuser functionality (stubs only)
  • No SMF removal (Phase C)

10. Observability during test

  • Console access via virsh console hh-prototest for live boot watching.
  • /var/log/zyginit/<svc>.log per-service logs (rev 48 changed default from /var/run/zyginit/log to /var/log/zyginit so logs survive reboot).
  • Out-of-band log access: with VM shut down, mount the BE's qcow2 from the dev host: 
    virsh destroy hh-prototest
sudo qemu-nbd -c /dev/nbd0 -r /var/lib/libvirt/images/hh-prototest.qcow2
sudo zpool import -R /tmp/hh-mnt -o readonly=on -N -f -d /dev/nbd0p2 hhtest
sudo zfs mount hhtest/ROOT/zyginit
# logs at /tmp/hh-mnt/var/log/zyginit/
    Restore: zfs umount, zpool export hhtest, qemu-nbd -d /dev/nbd0. Proven across 3 iterations.
  • Capture console screenshot at each iter for visual cross-check.

11. Explicit deferrals (captured for future work)

Each of the following is a known follow-up, documented here so future milestones inherit the boundary:

  • configd-coupled daemons re-add: if dlmgmtd/ipmgmtd/inetd/rpcbind/fmd are needed, design either a "minimal SMF infrastructure under zyginit" (configd as a zyginit-supervised daemon) or a per-daemon configd-bypass shim.
  • Runlevel targets: zygctl single, zygctl multiuser stubs shipped; implementation needs a target-set schema extension (e.g., service.runlevels = ["single", "multiuser"]).
  • Milestone-style virtual services: systemd-target equivalent; probably needed to implement runlevels cleanly.
  • /sbin/halt, /sbin/reboot, /sbin/poweroff, /usr/sbin/shutdown wrappers: replace with versions that route through zygctl.
  • init <n> runlevel compatibility: unsupported; use zygctl.
  • Crash-recovery re-adoption: architecturally not supported (illumos descendant requirement); documented.
  • NOORPHAN revisit: current template is correct for PID-1 re-exec; revisit only if a non-exec crash-recovery goal is added.
  • GC tuning under sustained uptime: track separately.
  • SMF removal from disk: Phase C — once zyginit is stable, optionally remove /lib/svc/, /etc/svc/ from the BE entirely.
  • Boot performance targets: no gate.
  • Additional services (resource limits, audit, DNS, NFS, zones): add as needed post-first-boot.
  • Hostid persistence: first boot reads /etc/hostid; generation logic (for first-ever boot) not yet implemented.
  • Modern datalink/IP semantics: BSD-pivot loses dladm/ipadm query interfaces. For zone hosts, laptops, multi-homed servers, this design is insufficient; revisit when those use cases are in scope.

12. Pre-requisites completed before this spec

  • Contract probe: 16/16 FFI probes pass on hh-prototest (rev 19)
  • contract.reef bugs fixed: P_CTID 14→13, ctl fd O_WRONLY (rev 18)
  • Path rename: /etc/zyginit.d/ → /etc/zyginit/ (rev 20, 21)
  • Signal handling fix: dead-code flag-clearing block removed (rev 15)
  • Build split: helpers.c / contract_linux_stubs.c separated (rev 16)
  • Disk backup: hh-prototest-backup-2026-04-24-pre-zyginit-pid1.qcow2 captured on dev host

Pre-requisites added during iter 1-5 of v1 (kept in v2):

  • fd-setup at PID 1 (rev 45): setup_pid1_console() opens /dev/console and dup2's onto 0/1/2 before any I/O
  • argv[0] fix (rev 44): supervisor.start_service includes program name as argv[0]
  • uadmin/sync wiring (rev 24-31): full PID-1 shutdown path
  • Tier-ordering enforcement (rev 47): supervisor waits for tier N to settle before starting N+1
  • Persistent log dir (rev 48): /var/log/zyginit/ default
  • root-fs ZFS/UFS dispatch (rev 43): correct readonly toggle per fstype
  • zygctl shutdown commands (rev 33-34): halt/reboot/poweroff + single/multiuser stubs
  • Path corrections in scripts (rev 49-50): /bin/hostname, /bin/rm, /sbin/swapadd (no -a), /sbin/dumpadm, etc.

13. Open questions / defer to implementation

Most v1 open questions are now answered (uadmin constants verified; ipmgmtd path verified; ttymon argv conventions known). v2 open questions answered through 2026-04-25 hh-prototest probing:

  • ✅ ifconfig capability without ipmgmtd: ifconfig lo0:99 plumb, ifconfig lo0:99 inet 127.99.0.1 netmask 255.0.0.0 up, and ifconfig lo0:99 unplumb all worked silently while ipmgmtd was running. ifconfig operates at the DLPI/kernel-ioctl layer for plumbing; ipmgmtd integration is for persistent state, not for runtime configuration.
  • ✅ ifconfig <if> dhcp <subcommand> syntax: per ifconfig(8) man page on hh-prototest, supported subcommands are start | stop | drop | extend | inform | ping | release | status. ifconfig vioif0 dhcp status returned valid info while pinging the running dhcpagent on TCP 4999.
  • ✅ /etc/hostname.<if> parsing: Hammerhead's ifconfig does NOT read this file directly. The file is consumed by SMF method scripts. zyginit's network/start.sh will parse the file itself (line-oriented format documented in §4.4).
  • ✅ /etc/defaultrouter: empty/absent on hh-prototest (DHCP supplies the route). Format: one IPv4 address per line. network/start.sh reads when present and DHCP isn't in use.

Remaining for iter 6 to empirically verify:

  • ⚠ DHCP-via-dhcpagent without ipmgmtd running: dhcpagent links libipadm which talks to ipmgmtd. Plan: try ifconfig <if> dhcp start after starting dhcpagent fresh under zyginit; if it works, ship as default; if it fails, ship static-IP-only and document. The test VM has a libvirt-supplied dnsmasq DHCP server on 192.168.122.0/24 — perfect environment to verify.
  • ⚠ First-boot fallback: if DHCP fails, our test config will use a static IP (e.g., inet 192.168.122.50/24) chosen outside libvirt's dnsmasq pool to avoid conflicts.