specs - zygaena/zyginit

root / docs / superpowers / specs
specs Plain Text 2507 lines 146.8 KB
# PID-1 Boot Design

**Date:** 2026-04-24
**Revision:** 2026-04-25 — BSD-pivot (post iter 1-5 findings)
**Status:** Revised; iter 6 implementation pending against the new service set
**Scope:** First boot of zyginit as PID 1 on Hammerhead (hh-prototest VM), replacing the SMF-managed boot chain with a zyginit-managed BSD-style boot.

## 1. Goal

Boot hh-prototest with `/sbin/init` pointing at zyginit, bringing the system to a state where the operator can SSH in and use the box. Deliver a small **BSD-style** service set (≈15 services) that uses direct command invocation (`ifconfig`, `dhcpagent`) instead of supervising SMF-coupled control-plane daemons. Tested rollback path via ZFS Boot Environments.

## 1.1 Design revision: BSD pivot (2026-04-25)

The original 21-service design (preserved in §4 history at the bottom of this doc) tried to supervise the existing illumos control-plane daemons — `dlmgmtd`, `ipmgmtd`, `inetd`, `rpcbind`, `fmd` — directly. Iter 1-5 (rev 24 through rev 50) surfaced a fundamental issue: these daemons are tightly coupled to **`svc.configd`** for their persistent property store via `libscf`. Setting `SMF_FMRI` in the environment bypasses their refusal-to-start banner, but they then silently degrade because `scf_handle_decode_fmri()` fails (no `/etc/svc/volatile/repository_door`), leaving them running but useless — `ipmgmtd` doesn't open its door socket; `ipadm` consequently fails with "Could not open handle to library"; `lo0` is never plumbed; `rpcbind` cascades into "could not find loopback transports"; the whole network stack is dead.

We had three choices:

- **A: Run `configd` under zyginit** — keeps the SMF property-store daemon alive, drops only `svc.startd`. Gets the daemons working but admits SMF is too entangled to fully escape.
- **B: Use `/lib/svc/method/*` scripts as zyginit's exec.start** — same effective coupling to `configd` as A; just hides it behind shell wrappers.
- **C: Pivot to BSD-style direct invocation** — drop the configd-coupled daemons entirely. Use `/sbin/ifconfig` directly to plumb interfaces and configure addresses. Use `/sbin/dhcpagent` directly for DHCP. Skip `inetd`/`rpcbind`/`fmd` for the default service set.

We chose **C**. Hammerhead retains the BSD primitives that make this work:

- `/sbin/ifconfig` exists and successfully plumbs interfaces without needing `ipmgmtd`
- `/etc/hostname.<if>` files supported (BSD-style interface naming/config)
- `/sbin/dhcpagent` is callable directly
- `/etc/init.d/`, `/etc/rc[0-3].d/` directories present (we won't use them, but the convention is there)

Reference for the approach: IRIX 6.5's `/etc/init.d/network` script (`os_research/irix-6517-src/eoe/cmd/initpkg/init.d/network`) does exactly this — `IFCONFIG=/usr/etc/ifconfig`, `ROUTE=/usr/etc/route`, flat-file configuration in `/etc/config/`, no daemon-based registry. zyginit-on-Hammerhead becomes essentially a Reef rewrite of pre-SMF Solaris boot semantics, replacing what IRIX/Solaris-9 did with shell.

Lessons from iter 1-5 also drove these other concrete fixes that survive in v2:

- **fd-setup at PID 1**: kernel doesn't preopen 0/1/2; zyginit must `open(/dev/console)` and `dup2()` onto 0/1/2 before any I/O (rev 45)
- **argv[0] in supervisor**: must include the program name as argv[0] when execvp'ing (rev 44)
- **root-fs ZFS/UFS dispatch**: `mount -o remount,rw /` errors on ZFS root; use `zfs set readonly=off` (rev 43)
- **Tier ordering enforcement**: between tiers, wait for oneshots to STOPPED and daemons to RUNNING before starting next tier (rev 47)
- **Persistent log dir**: `/var/log/zyginit/` (not `/var/run`) so logs survive reboot for diagnosis (rev 48)
- **Hammerhead path corrections**: `/bin/hostname` (not `/usr/bin/`), `/sbin/swapadd` (no `-a`), `/sbin/dumpadm`, etc.
- **cron FIFO is at `/etc/cron.d/FIFO`**, not `/var/run/cron_fifo`

A useful diagnostic capability also fell out: with the VM shut down, the BE's logs are readable from the dev host via `qemu-nbd` + zfsonlinux read-only import, then mounting `hhtest/ROOT/zyginit` to read `/var/log/zyginit/*.log`. This bypasses the GRUB-cant-pick-BE limitation on this Hammerhead version.

## 2. Non-goals

- **SMF removal from the BE.** Phase C, separate milestone. First boot leaves SMF binaries installed-but-unused. We just don't *invoke* them from zyginit's service set.
- **Supervising configd-coupled daemons.** `dlmgmtd`, `ipmgmtd`, `inetd`, `rpcbind`, `fmd` are explicitly out of zyginit's default service set. They depend on a running `svc.configd` that we no longer launch. They remain available on the BE for an admin who chooses to set up SMF separately.
- **Modern illumos datalink/IP management semantics.** No `dladm show-link`/`ipadm show-addr` integration. Configuration is via flat files (`/etc/hostname.<if>`, `/etc/defaultrouter`, `/etc/resolv.conf`). For laptops with WiFi roaming, link aggregation, IPMP, or VNICs for zones, this design is insufficient — those workflows need the SMF stack.
- **On-demand network services (inetd-style).** No `inetd` in the default set. Services that historically ran under `inetd` (telnet, rsh, finger, etc.) are not started. SSH is direct.
- **NFS / RPC.** `rpcbind` not started; no NFS in scope.
- **Fault management.** `fmd` not started; FMA events are not collected.
- **Single-user mode / runlevels.** `zygctl single` and `zygctl multiuser` remain stubs.
- **Crash-recovery re-adoption.** `ct_ctl_adopt` requires descendant relationship; only PID-1 in-place re-exec is supported.
- **`/sbin/halt`, `/sbin/reboot`, `/sbin/poweroff`, `/usr/sbin/shutdown` integration.** Continue to call `uadmin` directly. Clean shutdown goes through `zygctl`.
- **Name services (NSS), NTP, performance targets.** Deferred.

## 3. Architecture

### 3.1 Components

| Component | Path | Type | Purpose |
|---|---|---|---|
| `zyginit` | `/sbin/zyginit` | ELF binary | Init daemon (PID 1) |
| `zygctl` | `/sbin/zygctl` | ELF binary | Admin CLI |
| Config tree | `/etc/zyginit/` | Directory | Service TOMLs, per-service script dirs, `enabled.d/` symlink farm |
| Runtime tree | `/var/run/zyginit/` | Directory | Per-service log files, socket, transient state |
| Control socket | `/var/run/zyginit.sock` | Unix socket | zygctl ↔ zyginit IPC |

### 3.2 Boot sequence at runtime

1. Kernel execs `/sbin/init` (symlink to `/sbin/zyginit`)
2. zyginit detects PID (`getpid() == 1`) and enters PID-1 mode
3. zyginit scans `/etc/zyginit/enabled.d/` for symlinks
4. zyginit parses each TOML referenced; builds in-memory `ServiceDef` set
5. zyginit builds dependency graph via topological sort into tier ordering
6. zyginit starts services tier by tier:
   - Within a tier: sequential start (parallel start is a future optimization)
   - Between tiers: wait for all members of tier N to reach terminal state (RUNNING for daemons, STOPPED+exit=0 for oneshots) before starting tier N+1
7. zyginit opens event loop on: contract bundle fd, signal self-pipe, zygctl socket
8. System is "up" when all enabled services have started successfully

### 3.3 PID-1 vs non-PID-1 behavior

| Condition | Trigger | Behavior |
|---|---|---|
| PID 1 | SIGTERM / SIGINT | Halt: stop services in reverse tier order, sync, `uadmin(A_SHUTDOWN, AD_HALT, 0)` |
| PID 1 | `zygctl halt` | As above |
| PID 1 | `zygctl reboot` | Stop services, sync, `uadmin(A_SHUTDOWN, AD_BOOT, 0)` |
| PID 1 | `zygctl poweroff` | Stop services, sync, `uadmin(A_SHUTDOWN, AD_POWEROFF, 0)` |
| not PID 1 | SIGTERM / SIGINT | Stop services, return from `main()` (normal process exit) |
| not PID 1 | `zygctl halt` | Stop services, exit |
| not PID 1 | `zygctl reboot` / `poweroff` | Refuse ("not PID 1") — safety rail for dev/test |

## 4. Service set (BSD-pivot, v2)

15 zyginit services in `enabled.d/` by default (counted across §4.1–§4.6). Replaces the 21-service v1 design (full v1 service table viewable in `hg log -p` for this file) by dropping the configd-coupled daemons in favor of direct command invocation.

### 4.1 Tier 0 — no dependencies

| Service | Type | Script | Stop |
|---|---|---|---|
| `root-fs` | oneshot | `/etc/zyginit/root-fs/start.sh` (ZFS or UFS root: `zfs set readonly=off` or `mount -F ufs -o remount,rw /`) | `/etc/zyginit/root-fs/stop.sh` (matching readonly=on / remount,ro) |

### 4.2 Tier 1 — requires root-fs

| Service | Type | Script | Stop |
|---|---|---|---|
| `devfs` | oneshot | inline: `/usr/sbin/devfsadm` | — |
| `swap` | oneshot | inline: `/sbin/swapadd` (no `-a`; reads `/etc/vfstab`) | `/etc/zyginit/swap/stop.sh` (`swap -d` each active entry) |
| `crypto` | oneshot | `/etc/zyginit/crypto/start.sh` (`cryptoadm refresh`) | — |

### 4.3 Tier 2 — requires tier 1

| Service | Type | Script | Stop |
|---|---|---|---|
| `filesystem` | oneshot | `/etc/zyginit/filesystem/start.sh` (`mountall -l` + `zpool import -a` + `zfs mount -a`) | `/etc/zyginit/filesystem/stop.sh` (`umountall -l` + `zpool sync`) |
| `identity` | oneshot | `/etc/zyginit/identity/start.sh` (`hostname` from `/etc/nodename`) | — |
| `sysconfig` | oneshot | `/etc/zyginit/sysconfig/start.sh` (coreadm, dumpadm, keymap, scheduler, rctl, rmtmpfiles) | — |

### 4.4 Tier 3 — requires filesystem + identity

| Service | Type | Script | Stop |
|---|---|---|---|
| `network` | oneshot | `/etc/zyginit/network/start.sh` — **direct `ifconfig` invocation, no `ipmgmtd`**: plumb lo0, walk `/etc/hostname.<if>` files for each physical interface, `ifconfig <if> dhcp` (which forks `dhcpagent` directly) or static address per file content, install default route from `/etc/defaultrouter` if present | `/etc/zyginit/network/stop.sh` (kill dhcpagent, ifconfig unplumb interfaces) |

**Critical change from v1:** no `dlmgmtd` or `ipmgmtd` services. `ifconfig` plumbs interfaces directly via DLPI. `dhcpagent` is invoked directly via `ifconfig <if> dhcp start`, not via `ipadm -T dhcp`. This matches IRIX 6.5's `/etc/init.d/network` pattern (`IFCONFIG=/usr/etc/ifconfig`, `ROUTE=/usr/etc/route`).

#### `/etc/hostname.<if>` file format (OpenBSD-inspired, illumos-translated)

Verified on hh-prototest: `/sbin/ifconfig` is fully operational without `ipmgmtd` (plumb, address assignment, link state — all work). The OpenBSD `hostname.if(5)` format is line-oriented with each line being ifconfig args. We adopt the same line-oriented convention but translate to illumos's ifconfig syntax, since OpenBSD-specific keywords like `inet alias` map to illumos's `addif`.

Line forms supported by zyginit's `network/start.sh`:

| Line | Action |
|---|---|
| (empty file or whitespace) | `ifconfig <if> plumb up` — interface up, no address |
| `# comment` or blank line | skipped |
| `dhcp` | `ifconfig <if> plumb`; `ifconfig <if> dhcp start` (best effort — see DHCP caveat) |
| `inet <addr>/<prefix>` | `ifconfig <if> inet <addr>/<prefix> up` |
| `inet <addr> netmask <mask>` | `ifconfig <if> inet <addr> netmask <mask> up` |
| `addif <addr>/<prefix>` | additional alias address (illumos `addif`); requires interface already plumbed by an earlier line |
| `inet6 <addr>/<prefix>` | `ifconfig <if> inet6 <addr>/<prefix> up` (deferred to post-first-boot) |
| `up` / `down` | set link state explicitly |

Examples:

```
# /etc/hostname.vioif0 — DHCP via dhcpagent (best effort without ipmgmtd)
dhcp
```

```
# /etc/hostname.vioif0 — static IPv4
inet 192.168.122.50/24
```

```
# /etc/hostname.vioif0 — static IPv4 + alias
inet 192.168.122.50/24
addif 192.168.122.51/24
```

#### Default route

`/etc/defaultrouter` (one IPv4 address per line). `network/start.sh` runs `route -n add default <addr>` for each. Optional — DHCP-acquired routes don't need this file.

#### DHCP caveat

`dhcpagent` is linked against `libipadm.so.1`, which talks to `ipmgmtd`'s door for persistent state. Without `ipmgmtd` running:
- `dhcpagent` itself runs (self-contained DHCP state machine)
- DHCP request/response on the wire works (uses raw sockets, not libipadm)
- **Applying the lease to the interface** uses libipadm → ipmgmtd, which may degrade

Empirical test required on hh-prototest in iter 6 boot. If DHCP-via-ifconfig fails without ipmgmtd, fallback options:
1. Manually invoke `dhcpagent -a -f` (adopt mode + foreground) and configure interface from received lease via direct DLPI ioctl
2. Static IP only for the test VM (configure libvirt to have a known static address, set `/etc/hostname.vioif0` to `inet 192.168.122.50/24`)
3. Patch dhcpagent / replace with dhclient if available

The implementation plan should boot-test with static IP first to derisk the rest of the boot, then attempt DHCP as a follow-up.

### 4.5 Tier 4 — requires network (configd-free system daemons)

All daemons. All use `stop.method = "contract"` (default). `restart.on = "failure"`, `delay = 5`, `max_retries = 3`.

| Service | exec.start | Note |
|---|---|---|
| `syseventd` | `/usr/lib/sysevent/syseventd` | Kernel event delivery; doesn't read SMF properties |
| `syslogd` | `/usr/sbin/syslogd` | Reads `/etc/syslog.conf` directly; SMF-property reads degrade to defaults |
| `utmpd` | `/usr/lib/utmpd` | Cleans stale utmpx entries; no configd dependency |
| `pfexecd` | `/usr/lib/pfexecd` | RBAC profile-exec helper; reads `/etc/security/`, no configd |

### 4.6 Tier 5 — user-facing services

| Service | exec.start | Note |
|---|---|---|
| `cron` | `/etc/zyginit/cron/start.sh` | wrapper: `rm -f /etc/cron.d/FIFO; exec /usr/sbin/cron` |
| `sshd` | `/usr/lib/ssh/sshd -D` | `-D` keeps it foreground; reads `/etc/ssh/sshd_config` |
| `console-login` | `/etc/zyginit/console-login/start.sh` | wrapper exec'ing ttymon with properly-quoted prompt |

### 4.7 Optional / installable — TOMLs ship but symlinks not in `enabled.d/` by default

These TOMLs exist in `services/hammerhead/` but are NOT enabled by default. An admin can enable them via `zygctl enable <svc>` on hardware/configurations where they make sense:

| Service | When to enable |
|---|---|
| `powerd` | Real hardware with `/dev/srn` (not VMs) |
| `fmd` | When fault management is wanted (requires configd workaround or future SMF-coexistence design) |
| `dlmgmtd` / `ipmgmtd` / `inetd` / `rpcbind` | Only after a separate "run minimal SMF infrastructure under zyginit" design is implemented |

### 4.8 Dropped from the v1 design (configd-coupled, requires separate design to revisit)

These daemons require `svc.configd` running for `libscf` calls to succeed. They are intentionally absent from the BSD-pivot v2 service set:

- `dlmgmtd` — datalink management. Replaced by direct `ifconfig` plumbing. Drawback: no `dladm show-link` queries; persistent datalink state across reboots is via `/etc/hostname.<if>` instead.
- `ipmgmtd` — IP interface management. Replaced by direct `ifconfig`/`route` invocation. Drawback: no `ipadm` queries.
- `inetd` — on-demand service launching. Modern illumos `inetd` reads from configd. No replacement for first boot; can be re-added when needed (e.g., via `xinetd` or by accepting configd dependency).
- `rpcbind` — only needed for NFS/RPC; deferred until NFS is in scope.
- `fmd` — fault management daemon; heavy SMF coupling. Deferred.

### 4.9 SMF infrastructure (subsumed; no replacement)

- `svc.startd`, `svc.configd` — replaced by zyginit (or absent in v2)
- `manifest-import`, `early-manifest-import` — no XML manifests
- `boot-archive`, `boot-archive-update` — SMF-specific
- `logadm-upgrade`, `update-man-index` — admin/cron concerns, not init's
- All 7 SMF milestones (virtual groupings) — zyginit's dependency graph replaces these

### 4.10 Currently-disabled SMF services — also not resurrected in zyginit

Mirrors v1: `name-service-cache`, `rbac`, `process-security`, `auditd`, NFS/DNS/LDAP stacks all stay disabled.

## 5. TOML schema usage

Uses the schema already documented in `docs/DESIGN.md` §"Service Definition Format". The following subset is in active use for the 21 services:

```toml
[service]
name = "..."
description = "..."
type = "daemon" | "oneshot"

[exec]
start = "command or /etc/zyginit/<svc>/start.sh"

[dependencies]
requires = ["..."]
after = ["..."]

# Daemons only:
[stop]
method = "contract"          # default; may be omitted
signal = "TERM"              # default; may be omitted
timeout = 60                 # default; may be omitted

[restart]
on = "failure"
delay = 5
max_retries = 3

[contract]
param = ["inherit", "noorphan"]
fatal = ["hwerr"]

# Oneshots with reversal:
[exec]
stop = "/etc/zyginit/<svc>/stop.sh"

[restart]
on = "never"
```

**Defaults and omissions:**
- Omitted `exec.user` / `exec.group` → inherits from zyginit → root when PID 1. Explicit and documented so there's no ambiguity.
- Omitted `[stop]` block → uses contract-signal defaults (TERM, 60s timeout).
- Omitted `[contract]` block → uses template defaults from `setup_template()` in `contract.reef`.

**Deferred schema fields** (supported by zyginit, unused in first boot):
- `exec.user`, `exec.group`, `exec.working_directory`, `exec.environment` — none of the 21 services need privilege drop or per-service env
- `security.*`, `limits.*` — not implemented / not in first boot

**`contract.param = ["inherit", "noorphan"]` note:**
The NOORPHAN flag means that if zyginit-as-PID-1 dies without cleanly abandoning contracts, children receive SIGHUP. This is correct for PID-1 re-exec (kernel sees the same process, no SIGHUP fires). It is **incompatible** with a crash-recovery path that promotes a fresh zyginit PID — but such a path is an explicit non-goal per §2.

## 6. Method script conventions

All scripts live under `/etc/zyginit/<svc>/start.sh` (and optional `stop.sh`) unless the service is trivial enough for an inline TOML command.

**Template:**
```zsh
#!/bin/zsh
# <service> — <one-line purpose>
# Invoked by zyginit via exec.start in /etc/zyginit/<service>.toml

set -e
set -u

# ... steps ...

exit 0
```

**Conventions:**
1. Shebang: `#!/bin/zsh` (consistent with Hammerhead's shell policy; zsh is the system default, ksh93 is being phased out)
2. Error handling: `set -e` (POSIX-style — preferred over `setopt ERR_EXIT` for cross-shell readability) + `set -u`
3. Non-interactive only: no `read`, no tty-dependent behavior
4. Self-contained: no `source`ing of SMF helper scripts (`/lib/svc/share/*.sh`)
5. No SMF API calls: no `svcprop`, `svcadm`, `svccfg`. Config input comes via `exec.environment` in the TOML.
6. Explicit `exit 0` on success (not relying on implicit last-command exit)
7. Use `|| true` deliberately per-command where a specific failure is acceptable (e.g., `dumpadm -d /path || true` if the dump device doesn't exist yet)
8. Idempotent when the action allows: prefer `mount -o remount,rw /` (idempotent) over raw `mount` commands. Use "check then do" patterns for inherently non-idempotent operations (`ipadm show-if lo0 2>/dev/null || ipadm create-if lo0`).

**Execution environment zyginit provides to scripts:**
- Fresh process (fork + exec)
- Working dir: `/` unless TOML sets `exec.working_directory`
- Environment: minimal (PATH, HOME, etc.) plus whatever `exec.environment` specifies
- stdout/stderr: redirected to `/var/run/zyginit/log/<svc>.log`, truncated per start
- User: inherited from zyginit (root at PID 1) unless `exec.user` set

## 7. Shutdown orchestration

Shutdown is a first-class concern, not deferred. Unclean shutdown can corrupt ZFS; we do it right from the first boot.

### 7.1 Shutdown triggers

| Source | Type |
|---|---|
| `zygctl halt` | halt |
| `zygctl reboot` | reboot |
| `zygctl poweroff` | poweroff |
| SIGTERM / SIGINT (PID 1) | halt (conservative default) |
| SIGTERM / SIGINT (not PID 1) | exit (integration test path) |

Shutdown type is communicated to zyginit via the socket protocol (for zygctl commands) or inferred from signal source.

### 7.2 Shutdown walk

1. Set `shutdown_requested = true`, `shutdown_type = <type>`
2. Break out of main event loop
3. Walk services **in reverse tier order** (not insertion order)
4. For each service in the tier:
   - **Daemons** in `STATE_RUNNING`: `stop_service()` → SIGTERM via contract, wait for contract EMPTY event, mark `STATE_STOPPED`
   - **Oneshots** in `STATE_STOPPED` with `exec.stop` defined (or `/etc/zyginit/<svc>/stop.sh` present): fork/exec the stop command; wait for exit
   - **Oneshots** without stop: skip (no reversal defined)
5. Wait for all services in tier N to reach `STATE_STOPPED` before moving to tier N-1
6. After tier 0 is drained: `sync()` (flush pending writes)
7. Call `uadmin(A_SHUTDOWN, <fcn>, 0)`:
   - `AD_HALT` (0) — halt, operator decides next step physically
   - `AD_BOOT` (1) — reboot immediately
   - `AD_POWEROFF` (6) — ACPI/BMC chassis off

### 7.3 Stop scripts concretely required

| Service | stop.sh action |
|---|---|
| `root-fs` | `mount -o remount,ro /` (called after filesystem/stop.sh has umounted everything else) |
| `filesystem` | `umountall -l` |
| `swap` | `swap -d` each entry |

Other services either use the contract-signal daemon default or have nothing to reverse.

### 7.4 Required code changes

**`supervisor.reef`:**
- Extend shutdown path to walk tiers in reverse
- For each tier: first send stops to all daemons, wait for them to exit, then run stop scripts for oneshots that have them
- Currently `shutdown_services` skips `STATE_STOPPED` services — this must change

**`main.reef`:**
- Track `shutdown_type` variable (halt / reboot / poweroff)
- After `shutdown_services()`: call `sync()`, then `zyginit_shutdown(<fcn>)`

**FFI: `helpers.c` and `contract.reef` (or new `shutdown.reef`)**:
- `int zyginit_shutdown(int fcn)` → wraps `uadmin(A_SHUTDOWN, fcn, 0)`
- `extern "C" fn zyginit_shutdown(fcn: int): int`

**Linux stub**: `contract_linux_stubs.c` gets a matching stub returning -1 (should never actually fire on Linux).

**`zygctl`** gets three new subcommands: `halt`, `reboot`, `poweroff`. Each sends a specific command over the socket; zyginit sets `shutdown_type` accordingly.

**`zygctl`** also gets two stub subcommands: `single`, `multiuser`. Both print "not yet implemented — reserved for future runlevel support" and exit 1. Documented in `zygctl --help` so the surface is stable.

## 8. Rollout phases

### 8.1 Phase A — non-PID-1 dry run

Run zyginit on the live hh-prototest system alongside SMF, using a scratch config directory.

```sh
ZYGINIT_CONFIG_DIR=/tmp/zyginit-dry \
  /path/to/zyginit
```

**Goals:**
- TOMLs parse cleanly
- Dependency graph builds without cycles; tier order matches the spec
- zyginit supervises one or two non-conflicting test daemons (e.g., a second sshd on port 2222)
- Clean shutdown via SIGTERM works without uadmin (non-PID-1 path)

**Out of scope for Phase A:**
- Destructive oneshots (filesystem, network, devfs) — they'd conflict with SMF-managed copies. Skip these in the dry run.

### 8.2 Phase B — PID-1 boot in a fresh Boot Environment

1. `beadm create zyginit`
2. `beadm mount zyginit /mnt`
3. Install into `/mnt`:
   - `/mnt/sbin/zyginit`, `/mnt/sbin/zygctl`
   - `/mnt/etc/zyginit/*.toml` (21 files)
   - `/mnt/etc/zyginit/<svc>/start.sh` + `stop.sh` as specified above
   - `/mnt/etc/zyginit/enabled.d/` symlinks for all 21
   - Preserve old init: `mv /mnt/sbin/init /mnt/sbin/init.smf`
   - New init: `ln -s /sbin/zyginit /mnt/sbin/init`
4. `beadm umount zyginit; beadm activate zyginit; init 6`

### 8.3 Phase C — SMF removal (deferred, separate milestone)

Optional follow-up once Phase B is stable:
- In the zyginit BE: remove `/lib/svc/bin/svc.startd`, `/lib/svc/bin/svc.configd`, `/lib/svc/method/*`, `/etc/svc/`
- Activate, reboot, prove the system still works without SMF
- This is when "zyginit replaces SMF" is literally true.

### 8.4 Failure recovery ladder

In order of cost:

1. `zygctl reboot` → pick `hammerhead` BE from GRUB menu (seconds)
2. If GRUB menu inaccessible: boot into rescue media / single-user from install media, mount original BE, activate it
3. If the pool is damaged: `sudo cp hh-prototest-backup-2026-04-24-pre-zyginit-pid1.qcow2 hh-prototest.qcow2` on the dev host (2-3 min with VM shut down)
4. Last resort: VM rebuild from scratch (~20 min)

Disk-level backup captured at rev 21. Recovery drill is part of §9.F.

## 9. Success criteria for Phase B

Literal checklist run after first boot into the zyginit BE.

### A. Boot completes
- [ ] No `kernel panic: init died`
- [ ] `zyginit: entering event loop` appears on console
- [ ] Every tier finishes within 60s total
- [ ] Console login prompt appears on `/dev/console`

### B. Expected service states
- [ ] All 21 services present in `zygctl list`
- [ ] Tier 0-3 oneshots: `STOPPED` with `exit=0`
- [ ] Tier 3-5 daemons: `RUNNING`
- [ ] No service in `MAINTENANCE`

### C. Services actually functional
- [ ] `ssh zygaena@192.168.122.197` from dev host
- [ ] `logger -p user.info test` → visible in `/var/adm/messages` (syslogd)
- [ ] `dladm show-link` lists physical interface `up`
- [ ] `ipadm show-addr` lists lo0 + physical interface with IPv4 addresses
- [ ] `ping 192.168.122.1` succeeds
- [ ] `crontab -l` runs for root
- [ ] `zfs list` works, `hhtest` pool mounted

### D. zygctl end-to-end
- [ ] `zygctl status cron` shows `RUNNING` with uptime > 0
- [ ] `zygctl stop cron` → `STOPPED`, then `zygctl start cron` → `RUNNING`
- [ ] `zygctl log sshd` returns without error
- [ ] `zygctl reload` succeeds (SIGHUP path)

### E. Clean shutdown
- [ ] `zygctl halt`: reverse-tier-order stop output in console; system halted cleanly
- [ ] `zygctl reboot`: cycles; comes back via zyginit again
- [ ] `zygctl poweroff`: VM transitions to `shut off` in libvirt
- [ ] After any zygctl shutdown: `zpool status` clean, no replay-required flags

### F. Rollback confirmed
- [ ] 3 clean reboot cycles into the zyginit BE
- [ ] Reboot into original `hammerhead` BE from GRUB → SMF comes up normally
- [ ] `beadm activate hammerhead` from zyginit BE → reboot → lands on `hammerhead` BE
- [ ] Disk-level restore drill executed at least once (time to recovery measured)

### G. Explicit non-requirements
- No boot-time performance gate (measured for baseline only)
- No stability soak
- No DNS / hostname resolution testing
- No NTP / time sync
- No name services (NSS beyond /etc/passwd, /etc/hosts, /etc/group)
- No `zygctl single` / `zygctl multiuser` functionality (stubs only)
- No SMF removal (Phase C)

## 10. Observability during test

- Console access via `virsh console hh-prototest` for live boot watching.
- `/var/log/zyginit/<svc>.log` per-service logs (rev 48 changed default from `/var/run/zyginit/log` to `/var/log/zyginit` so logs survive reboot).
- **Out-of-band log access:** with VM shut down, mount the BE's qcow2 from the dev host:
  ```
  virsh destroy hh-prototest
  sudo qemu-nbd -c /dev/nbd0 -r /var/lib/libvirt/images/hh-prototest.qcow2
  sudo zpool import -R /tmp/hh-mnt -o readonly=on -N -f -d /dev/nbd0p2 hhtest
  sudo zfs mount hhtest/ROOT/zyginit
  # logs at /tmp/hh-mnt/var/log/zyginit/
  ```
  Restore: `zfs umount`, `zpool export hhtest`, `qemu-nbd -d /dev/nbd0`. Proven across 3 iterations.
- Capture console screenshot at each iter for visual cross-check.

## 11. Explicit deferrals (captured for future work)

Each of the following is a known follow-up, documented here so future milestones inherit the boundary:

- **configd-coupled daemons re-add**: if `dlmgmtd`/`ipmgmtd`/`inetd`/`rpcbind`/`fmd` are needed, design either a "minimal SMF infrastructure under zyginit" (configd as a zyginit-supervised daemon) or a per-daemon configd-bypass shim.
- **Runlevel targets**: `zygctl single`, `zygctl multiuser` stubs shipped; implementation needs a target-set schema extension (e.g., `service.runlevels = ["single", "multiuser"]`).
- **Milestone-style virtual services**: systemd-target equivalent; probably needed to implement runlevels cleanly.
- **`/sbin/halt`, `/sbin/reboot`, `/sbin/poweroff`, `/usr/sbin/shutdown` wrappers**: replace with versions that route through `zygctl`.
- **`init <n>` runlevel compatibility**: unsupported; use zygctl.
- **Crash-recovery re-adoption**: architecturally not supported (illumos descendant requirement); documented.
- **NOORPHAN revisit**: current template is correct for PID-1 re-exec; revisit only if a non-exec crash-recovery goal is added.
- **GC tuning under sustained uptime**: track separately.
- **SMF removal from disk**: Phase C — once zyginit is stable, optionally remove `/lib/svc/`, `/etc/svc/` from the BE entirely.
- **Boot performance targets**: no gate.
- **Additional services** (resource limits, audit, DNS, NFS, zones): add as needed post-first-boot.
- **Hostid persistence**: first boot reads `/etc/hostid`; generation logic (for first-ever boot) not yet implemented.
- **Modern datalink/IP semantics**: BSD-pivot loses `dladm`/`ipadm` query interfaces. For zone hosts, laptops, multi-homed servers, this design is insufficient; revisit when those use cases are in scope.

## 12. Pre-requisites completed before this spec

- **Contract probe**: 16/16 FFI probes pass on hh-prototest (rev 19)
- **contract.reef bugs fixed**: P_CTID 14→13, ctl fd O_WRONLY (rev 18)
- **Path rename**: `/etc/zyginit.d/` → `/etc/zyginit/` (rev 20, 21)
- **Signal handling fix**: dead-code flag-clearing block removed (rev 15)
- **Build split**: `helpers.c` / `contract_linux_stubs.c` separated (rev 16)
- **Disk backup**: `hh-prototest-backup-2026-04-24-pre-zyginit-pid1.qcow2` captured on dev host

**Pre-requisites added during iter 1-5 of v1 (kept in v2):**
- **fd-setup at PID 1** (rev 45): `setup_pid1_console()` opens `/dev/console` and dup2's onto 0/1/2 before any I/O
- **argv[0] fix** (rev 44): supervisor.start_service includes program name as argv[0]
- **uadmin/sync wiring** (rev 24-31): full PID-1 shutdown path
- **Tier-ordering enforcement** (rev 47): supervisor waits for tier N to settle before starting N+1
- **Persistent log dir** (rev 48): `/var/log/zyginit/` default
- **root-fs ZFS/UFS dispatch** (rev 43): correct readonly toggle per fstype
- **zygctl shutdown commands** (rev 33-34): halt/reboot/poweroff + single/multiuser stubs
- **Path corrections in scripts** (rev 49-50): `/bin/hostname`, `/bin/rm`, `/sbin/swapadd` (no `-a`), `/sbin/dumpadm`, etc.

## 13. Open questions / defer to implementation

Most v1 open questions are now answered (uadmin constants verified; ipmgmtd path verified; ttymon argv conventions known). v2 open questions answered through 2026-04-25 hh-prototest probing:

- ✅ **`ifconfig` capability without ipmgmtd**: `ifconfig lo0:99 plumb`, `ifconfig lo0:99 inet 127.99.0.1 netmask 255.0.0.0 up`, and `ifconfig lo0:99 unplumb` all worked silently while ipmgmtd was running. `ifconfig` operates at the DLPI/kernel-ioctl layer for plumbing; ipmgmtd integration is for *persistent state*, not for runtime configuration.
- ✅ **`ifconfig <if> dhcp <subcommand>` syntax**: per ifconfig(8) man page on hh-prototest, supported subcommands are `start | stop | drop | extend | inform | ping | release | status`. `ifconfig vioif0 dhcp status` returned valid info while pinging the running dhcpagent on TCP 4999.
- ✅ **`/etc/hostname.<if>` parsing**: Hammerhead's `ifconfig` does NOT read this file directly. The file is consumed by SMF method scripts. zyginit's `network/start.sh` will parse the file itself (line-oriented format documented in §4.4).
- ✅ **`/etc/defaultrouter`**: empty/absent on hh-prototest (DHCP supplies the route). Format: one IPv4 address per line. `network/start.sh` reads when present and DHCP isn't in use.

Remaining for iter 6 to empirically verify:

- ⚠ **DHCP-via-dhcpagent without ipmgmtd running**: dhcpagent links libipadm which talks to ipmgmtd. Plan: try `ifconfig <if> dhcp start` after starting dhcpagent fresh under zyginit; if it works, ship as default; if it fails, ship static-IP-only and document. The test VM has a libvirt-supplied dnsmasq DHCP server on 192.168.122.0/24 — perfect environment to verify.
- ⚠ **First-boot fallback**: if DHCP fails, our test config will use a static IP (e.g., `inet 192.168.122.50/24`) chosen outside libvirt's dnsmasq pool to avoid conflicts.
# Contract Re-adoption (Live Replace) Design

**Date:** 2026-05-02
**Status:** Approved (brainstorm complete); implementation plan to follow
**Scope:** Live in-place replacement of the running zyginit PID-1 process — replacing the `/sbin/init` binary while preserving every running service across the swap. Operator-driven only; not a crash-recovery design.

## 1. Goal

Allow an operator (or automation tool) to upgrade the running zyginit binary without rebooting and without dropping any service. After the operation, every previously-RUNNING service continues to run with the same contract ID, accumulated `restart_count`, and continuous `last_start_time` (uptime). The connection between zyginit and supervised services is preserved across the exec; the only thing replaced is the program text and in-memory state of PID 1 itself.

The verb the operator uses is `zygctl replace`. The mechanism is `execve("/sbin/init", ...)` against the same proc-table slot, with a state-file handoff for in-memory bookkeeping.

## 2. Non-goals

- **Crash recovery.** If zyginit segfaults / panics, the kernel's `restart_init` re-execs `/sbin/init` in a tight loop, but the contracts are not preserved across that path (NOORPHAN templates would have killed members already). PID 1 dying is a much larger issue than this design solves; we explicitly do not try.
- **Live binary swap.** `zygctl replace` does not copy or stage a binary. The operator (or package manager / configuration management tool) is expected to drop the new binary at `/sbin/init` first, then call `zygctl replace`. We hard-wire the exec target.
- **Switching to a different init system.** `zygctl replace` always execs `/sbin/init` with no path argument. Switching init systems is a rescue-media operation.
- **Signal-driven trigger.** No SIGUSR2 handler. Socket is the only path. (Revisit only if a real demand surfaces; see §10.)
- **Crash-recovery argv-scan / proc-walk.** No fallback that reconstructs identity from `/proc/<pid>/psinfo`. State file is the only source of truth.
- **Recovery for transient-state services** (`STARTING`, `STOPPING`, `WAITING`-for-restart). `replace` refuses on transient state by default; with `--wait=N` it polls for stability before refusing. `--force` is **not** in scope for v1.
- **Preserving in-flight zygctl connections.** Clients lose their connection at exec; that's expected. Operators reconnect for follow-up commands.

## 3. Why this works on Hammerhead — the key insight

On illumos / Hammerhead, process contracts are tracked on the process structure (`proc_t.p_ct_process` / `p_ct_equiv`). When a process calls `execve()`, the kernel replaces the program text and (re-)initializes most of `proc_t`, but **the proc-table slot and contract ownership survive**. This is what makes live replace possible: PID 1 is the same `proc_t` before and after the exec, so all contracts it held continue to be held by it.

Consequence: the new zyginit **does not need `ct_ctl_adopt`**. Adoption is for the SMF-startd model — a non-PID-1 supervisor dies and its successor (a fresh process with a different `proc_t`) needs to claim the orphan-inherited contracts. For PID-1 in-place exec, kernel-level ownership is automatic; we only need to rebuild the userspace bookkeeping (which contract belongs to which service, and what runtime fields to carry over).

The bundle fd (`/system/contract/process/bundle`) does close at exec (we don't try to inherit it via FD_CLOEXEC manipulation), so the new zyginit re-opens it. Events that fire during the gap between old-process exec and new-process bundle re-open are recovered via a post-startup `ct_status_read` check on each known contract — if the contract is already empty, we synthesize an empty event into the existing `handle_contract_event` path. This is idempotent regardless of whether the gap actually missed any events.

## 4. Architecture

```
                  Operator
                     |
                     v
              zygctl replace
              [--wait=N]
                     |
              +------+-------+
              |   socket     |   "replace queued"
              |   (sock.reef)|----------------> zygctl prints, exits
              +------+-------+
                     |  set replace_requested flag
                     v
            main.reef event loop
                     |
                     v
              replace_self()
                     |
   +-----------------+-----------------+
   | 1. preconditions (--wait optional)|
   | 2. drain pending bundle events    |
   | 3. write state.toml (atomic)      |
   | 4. unlink socket                  |
   | 5. close internal fds             |
   | 6. execve("/sbin/init", argv)     |
   +-----------------+-----------------+
                     |
                     v
          /// SAME proc_t, new program text ///
                     |
                     v
            new main() startup
                     |
   +-----------------+-----------------+
   | 1. setup_pid1_console (as today)  |
   | 2. recover_state(boot_time)       |
   |    — reads state.toml, validates  |
   |      boot_time, returns entries   |
   | 3. load enabled.d/ TOML (as today)|
   | 4. apply_recovered_state(table,   |
   |       recovered)                  |
   | 5. tier-by-tier start_service     |
   |    (skips entries already RUNNING)|
   +-----------------+-----------------+
                     |
                     v
              normal main loop
```

**Module split:**

| File | Change | Why |
|---|---|---|
| `src/replace.reef` | new (~200 lines) | Holds the state-file format, serialization, and recovery logic. Keeps `main.reef` lean. |
| `src/main.reef` | +60 lines | `replace_self`, `apply_recovered_state`, integration with main loop and startup. |
| `src/socket.reef` | +30 lines | `replace [--wait=N]` command handler, `replace_requested()` / `clear_replace_flag()` / `replace_wait_seconds()` accessors. |
| `src/contract.reef` | +20 lines | `is_contract_empty(ctid): bool` helper wrapping `ct_status_read` + member-count check. |
| `tools/zygctl/src/zygctl.reef` | +20 lines | `replace [--wait=N]` subcommand. |

No new C helpers. No FFI additions (`adopt_contract` already exists but is not used by this flow; it stays for potential future uses).

## 5. State file format

**Location:** `/var/run/zyginit/state.toml`. Same directory currently used for the socket. On the current zyginit-only BE, `/var/run` is persistent across real reboots (not yet tmpfs); the boot-id check (§5.2) makes stale post-reboot files harmless.

**Format:** TOML, parseable by the existing `encoding.toml` parser.

```toml
boot_time = 1746204000

[[service]]
name = "sshd"
contract_id = 23
restart_count = 0
last_start_time = 1746204051
last_exit_code = -1

[[service]]
name = "syslogd"
contract_id = 17
restart_count = 1
last_start_time = 1746204047
last_exit_code = 0
```

### 5.1 What is and isn't serialized

Serialized:
- `boot_time` — PID 1's process start time, read from `/proc/1/stat` (Linux) or `/proc/1/psinfo` (Hammerhead). Survives `execve` but changes on real reboot. Used as a staleness sentinel — see §5.2.
- For each service in `STATE_RUNNING`:
  - `name` — the canonical service name (matches `enabled.d/` symlink basename).
  - `contract_id` — kernel-assigned contract ID.
  - `restart_count` — accumulated restart counter (for `max_retries` accounting).
  - `last_start_time` — Unix timestamp of the most recent start.
  - `last_exit_code` — most recent exit code (or `-1` if never exited).

Not serialized:
- `pid` — recoverable from contract status if needed; not needed for the contract-driven supervision path.
- `state` — only `RUNNING` services serialize (per §6 pre-conditions); the field is implicit.
- `stop_requested_time`, `restart_scheduled_time` — only meaningful in transient states, which we refuse.
- `def` (the parsed TOML ServiceDef) — the new zyginit re-reads `enabled.d/` from disk, so the def comes from current TOML on the new side. This means a TOML edit between old-start and replace takes effect on next restart. (See §7.3 for the divergence handling.)

### 5.2 Boot-id staleness check

The boot-id sentinel is PID 1's process start time, read from `/proc/1/stat` field 22 (Linux: jiffies-since-boot) or `/proc/1/psinfo pr_start.tv_sec` (illumos/Hammerhead: Unix timestamp) via `zyginit_read_pid1_start()` in `helpers.c`. This value survives `execve` (same `proc_t` slot) but changes on real reboot (new PID 1). If `state.toml.boot_time != current.boot_time`, the file is from a different system uptime — most likely a stale file left over from before a real reboot. We log `"replace: stale state file (boot_time mismatch); ignoring"`, delete the file, and proceed as a fresh boot.

**Why not utmpx BOOT_TIME?** The original design used `utmpx BOOT_TIME` as the sentinel. This was found to be incorrect: `pututxline` with `ut_type = BOOT_TIME` *updates* the existing matching record (by design, to support `who -b`), rather than appending a new one. After a live replace, the new zyginit's startup call to `zyginit_write_boot_utmpx()` overwrites the original BOOT_TIME with a fresh timestamp. The new zyginit then reads its own updated timestamp as the current boot_time — which matches the new timestamp it just wrote — while state.toml has the old timestamp from before the replace. Result: mismatch, state.toml discarded, recovery fails silently. `/proc/1/start` avoids this entirely: no write occurs to it at startup, and it is immutable for the life of the PID 1 proc_t.

This guards against the edge case where contract IDs are re-used after a real reboot (the kernel does recycle them) and we'd otherwise adopt the wrong contract. The check is cheap (one integer comparison) and correct in the common case (replace = same boot_time; reboot = different boot_time).

The state file is **deleted by the new zyginit after a successful read**, regardless of whether all entries were applied successfully. This way, a panic-restart loop (kernel re-execing `/sbin/init` repeatedly) does not keep trying to apply long-dead entries.

### 5.3 Atomic write

Writing follows the standard pattern, using the existing `zyginit_fsync_path` C helper:

1. `open("/var/run/zyginit/state.toml.tmp", O_WRONLY|O_CREAT|O_TRUNC, 0600)`
2. Write boot_time + each service entry. (Reef-side TOML emit is straightforward; we don't need the full encoding.toml emit machinery, just print to fd.)
3. `fsync(fd)`, `close(fd)`.
4. `rename("/var/run/zyginit/state.toml.tmp", "/var/run/zyginit/state.toml")`.
5. `zyginit_fsync_path("/var/run/zyginit")` — fsync the parent directory so the rename is durable.

Any failure aborts the replace (we do not exec). At most we leave a `.tmp` file, which is overwritten on the next attempt and never read.

## 6. Operator interface

### 6.1 `zygctl replace`

```
zygctl replace [--wait=N]
```

- Connects to `/var/run/zyginit.sock`.
- Sends `replace [--wait=N]\n`.
- Reads reply (`replace queued (wait=N)\n` or an error).
- Prints reply, exits 0 on success, non-zero on parse error from server.
- The TCP/Unix-socket connection itself drops mid-response if the server proceeds to exec. zygctl treats `EPIPE` / connection-closed-after-ack as success and prints `replace: socket closed (process replaced)\n`.

### 6.2 `--wait=N`

If any service is in a transient state (`STARTING`, `STOPPING`, `WAITING`-for-restart), the server polls once per second for up to `N` seconds, checking whether all services are in stable states (`RUNNING`, `STOPPED`, `FAILED`, `MAINTENANCE`, `DISABLED`). If stability is reached within the window, replace proceeds. If not, replace is refused with `"replace blocked: <name>:<state>, ..."` logged to syslog (and to stderr if attached). Default is `--wait=0` (refuse immediately).

The wait happens server-side after the ack to the client, so the zygctl client doesn't have to hold a connection open — automation can `zygctl replace --wait=120` and either get a clean dropped connection (success) or, if the wait times out without exec'ing, can detect the failure by reconnecting and inspecting service states.

### 6.3 Pre-conditions in detail

Before exec, zyginit checks:

1. **No service is in a transient state.** (After `--wait` window if specified.)
2. **`state.toml.tmp` write succeeds**, including fsync.
3. **`rename` to `state.toml` succeeds**, including parent-dir fsync.

If any of these fail, replace aborts cleanly and the running zyginit continues. The flag is cleared, no exec happens, and the failure is logged. The operator can retry.

The socket unlink and fd cleanup happen *after* a successful state write — they are last-mile actions before exec, and we don't fail the replace if they error (we log and proceed).

## 7. Recovery flow (new zyginit)

### 7.1 Startup sequence

```
main():
    parse argv, kernel boot args
    if pid == 1: setup_pid1_console()
    boot_time = read_boot_time()       // /proc/1/start (pid1 start time) or time_now()

    recovered = replace.recover_state(boot_time)

    config = load_enabled_dir("/etc/zyginit/enabled.d")
    table = build_service_table(config)

    apply_recovered_state(table, recovered)    // <-- new step

    // Enter normal startup: tier-by-tier start_service
    // (services already in STATE_RUNNING are skipped)

    main_loop()
```

### 7.2 `apply_recovered_state` algorithm

For each `rs` in `recovered.services`:

```
idx = supervisor.find_service(table, rs.name)

if idx < 0:
    // Service was in state.toml but is no longer in enabled.d/.
    // Operator removed it dirty (manual symlink removal between old start
    // and replace). Don't kill — let the process exit on its own and the
    // kernel reap the contract. We just stop tracking it.
    contract.abandon_contract(rs.contract_id)
    log("replace: orphaned " + rs.name + " (no longer enabled)")
    continue

// Found in current table — patch in runtime fields.
rt = table.runtimes[idx]
rt.state = STATE_RUNNING
rt.contract_id = rs.contract_id
rt.pid = -1                     // unknown; not needed
rt.restart_count = rs.restart_count
rt.last_start_time = rs.last_start_time
rt.last_exit_code = rs.last_exit_code
table.runtimes[idx] = rt
table.contract_map.set(int_to_str(rs.contract_id), idx)

// Idempotency: did the contract go empty during the exec gap?
if contract.is_contract_empty(rs.contract_id):
    log("replace: " + rs.name + " contract empty post-recovery, applying restart policy")
    supervisor.handle_contract_event(table, rs.contract_id, -1)
    // existing logic resolves to STOPPED or scheduled-for-restart
```

### 7.3 Config divergence between old start and replace

Three cases the new zyginit must handle:

| Case | Detection | Action |
|---|---|---|
| Service in state.toml AND in current `enabled.d/` | `find_service` returns valid idx | Adopt: patch runtime fields, register in `contract_map`. Use **current** TOML for future supervision (so an edit takes effect on next restart). |
| Service in state.toml, NOT in current `enabled.d/` | `find_service` returns -1 | Abandon contract, log warning. Member processes continue running but unsupervised; kernel reaps the contract on their exit. |
| Service in current `enabled.d/`, NOT in state.toml | Not in recovered list | Treated as a freshly-enabled service. Tier-by-tier startup will `start_service` it normally on first event-loop tick. |

This means automation flows like `coral install some-service-package; zygctl replace` work correctly — the newly-installed service appears in `enabled.d/` and gets started by the new zyginit even though the old one never knew about it.

### 7.4 No `adopt_contract` calls

As established in §3, contract ownership survives execve at the kernel level. The new zyginit's bookkeeping is entirely in userspace: rebuild `contract_map`, populate `ServiceRuntime` fields. The `contract.adopt_contract` FFI remains in the codebase for potential future use (e.g., a non-PID-1 zyginit "supervisor mode" recovery flow) but is not exercised by this design.

## 8. Error handling

### 8.1 Pre-exec (in old zyginit)

Every pre-exec failure aborts the replace cleanly — old zyginit returns to its main loop with the flag cleared.

| Failure | Behavior |
|---|---|
| Service in transient state, `--wait` exhausted | Log `"replace blocked: <name>:<state>, ..."`. Clear flag. Return to main loop. |
| `state.toml.tmp` open / write / fsync fails | Log error with errno, unlink any partial `.tmp`. Return to main loop. |
| `rename(state.toml.tmp, state.toml)` fails | Log error. Return to main loop. Stale `state.toml` from prior aborted attempt has the wrong boot_time and is harmless. |
| `unlink("/var/run/zyginit.sock")` fails | Log warning, **proceed to exec**. New zyginit's `unix_listen` will overwrite the path. |
| `process.process_exec()` returns | Means kernel couldn't load `/sbin/init` (binary missing or corrupt). Log fatal. Old zyginit is in an unrecoverable state (state.toml written, socket unlinked). Fall through to the existing PID-1 infinite-sleep handling. Operator's recourse is reboot. Should never reach this in practice. |

### 8.2 Post-exec (in new zyginit)

Failures here are per-entry — one bad entry doesn't stop other recoveries.

| Failure | Behavior |
|---|---|
| `state.toml` missing | Treated as fresh boot. Not an error; no log. |
| `state.toml` parse error / corrupt | Log warning, treat as missing. Surviving contracts (if any) become orphans. |
| `state.toml.boot_time != current` | Log info, delete file, treat as missing. |
| Service in state.toml but not in current `enabled.d/` | Abandon contract, log warning. (See §7.3.) |
| `is_contract_empty(ctid)` post-recovery returns true | Synthesize a contract event; existing `handle_contract_event` applies restart policy. |
| `ct_status_read` returns error (contract gone, `ENOENT`) | Treat as empty / missing. Service starts fresh on first tick. |

After all entries are processed (success or otherwise), `state.toml` is deleted. This breaks panic-restart loops where the same bad state would otherwise be re-applied repeatedly.

### 8.3 Panic safety

`recover_state` and `apply_recovered_state` are written defensively. Any unexpected condition is logged and skipped — never panicked on. A bad state.toml entry must never prevent the new zyginit from completing startup.

## 9. Testing

### 9.1 Linux dev (no contract FFI)

The Linux contract stubs return -1 for all contract calls, so end-to-end re-adoption can't run on Linux. What we *can* unit-test:

- **State serialization round-trip** (`tests/test_replace.reef`): build a synthetic `ServiceTable` with services in known states, call `serialize_state`, read back with `recover_state`, assert all fields match. Cover empty table, all-RUNNING, mixed states (only RUNNING entries serialize), boot-id mismatch (ignored), corrupt TOML (ignored).
- **Pre-condition check**: factor the transient-state scan into an exported `check_replace_preconditions(table) -> string` (empty on OK, error message otherwise). Test transient states individually and combined.
- **Stitching logic**: unit-test `apply_recovered_state` covering the three cases of §7.3. Linux contract stubs let `is_contract_empty` always return false; we exercise the in-state-and-enabled and in-state-not-enabled branches.
- **Integration smoke** (`tests/integration/run_tests.sh`): start zyginit-as-supervisor (`ZYGINIT_CONFIG_DIR=/tmp/...`), run a few oneshot+daemon services, run `zygctl replace`. On Linux, contract-driven supervision is partially stubbed but the exec path itself runs. Assert the second zyginit instance starts, its log shows `"replace: stale state file"` or `"replace: <name> contract empty post-recovery"` per stub returns, and no crash.

### 9.2 Hammerhead (`hh-prototest`)

Real validation. Deploy via the iter 11+ procedure (`scp` source, `gcc helpers.c`, `reefc build`, `mv build/zyginit /sbin/init.new && mv /sbin/init.new /sbin/init`).

Test cases:

1. **Smoke** — multi-user up; `zygctl replace`; confirm:
    - Connection drops with the documented message.
    - Within ~1s, `zygctl status` works again.
    - All services still running, **same PIDs**, **same contract IDs**, restart counts preserved.
    - Uptimes preserved (continuous, not reset).
    - **The SSH session driving the test stays alive across the replace.** This is the highest-signal check.
    - `who -b` and `who -r` still report correctly.
2. **Transient-state refusal** — start a service with a slow `start.sh` (or watch for the WAITING window during a restart); run `zygctl replace` during the transient window; confirm refusal with the right error.
3. **`--wait=N` success** — same scenario with `--wait=10`; confirm replace proceeds once the service stabilizes.
4. **Dirty disable** — `rm /etc/zyginit/enabled.d/<svc>` (without `zygctl reload`); `zygctl replace`; confirm orphan path runs (contract abandoned, process keeps running, log message).
5. **Crash during exec gap** — `kill -9 <pid>` immediately before `zygctl replace`; confirm new zyginit's `is_contract_empty` post-recovery catches the empty contract and applies restart policy.
6. **Stale state.toml after real reboot** — write a state.toml file, do a real reboot, file persists in `/var/run/zyginit/`; confirm boot-id mismatch is detected, file deleted, fresh boot proceeds.
7. **Repeated replace** — `zygctl replace` 5 times in succession; confirm restart counts accumulate correctly (no resets), uptimes grow continuously, no state-file leakage.

### 9.3 Out of scope for testing

- Crash recovery (operator-driven only).
- Replace under heavy load (we accept some bundle events will land post-exec; the post-recovery status check covers it).
- Concurrent `replace` invocations (single-shot flag; second invocation between flag-set and exec is silently dropped — operator error).

## 10. Deferred / open questions

These are intentionally not in scope for v1, but flagged so we can revisit:

- **`zygctl replace --force`** — accept transient states by re-driving them (re-issue stop, re-arm restart timer, etc.). Adds significant recovery complexity. Defer until there's demonstrated automation demand.
- **SIGUSR2 fallback** — operator-side trigger if the socket is wedged. The reload path (SIGHUP) shows that signal-driven flag-setting works; we just don't yet need the additional reachable code path for replace. Defer.
- **Socket access tightening** — orthogonal to this design, but the socket is currently created with default permissions. It should be 0600 root-only. File as a separate hardening item.
- **Bundle fd inheritance** — clearing FD_CLOEXEC and passing the fd through exec would close the (already-handled) gap window completely. Adds complexity for an already-mitigated case. Defer indefinitely.
- **`/var/run/zyginit/` to tmpfs** — once `fs-minimal` is added back to the boot chain, `state.toml` automatically becomes ephemeral and the boot-id check becomes belt-and-suspenders. Tracked separately under §"Important Notes" in CLAUDE.md.
- **utmpx BOOT_TIME as sentinel (rejected)** — utmpx BOOT_TIME's `pututxline` updates rather than appends, so the new zyginit overwrites the original timestamp at startup, defeating the staleness check. `/proc/1/start` naturally survives execve but changes on real reboot. See §5.2 for full rationale.

## 11. Summary of decisions

| Decision | Choice | Rationale |
|---|---|---|
| Trigger model | Operator-driven only (live upgrade) | Crash recovery is a much harder problem; PID 1 dying is already catastrophic. |
| Trigger interface | `zygctl replace` (socket only) | Smallest blast radius; signal fallback defers cleanly. |
| Verb | `replace` | Operator-readable; covers upgrade + memory hygiene + state reset reasons; no overload with existing commands. |
| Exec target | Hard-wired to `/sbin/init` | Security: never accept attacker-controllable exec paths in PID-1. |
| Identity mapping | State file (single TOML) | Preserves runtime fields (`restart_count` especially); deterministic; matches deliberate-handover model. |
| Pre-condition check | Refuse on transient state by default; `--wait=N` for automation | Eliminates an entire class of recovery bugs; automation has a clean knob. |
| Bundle fd | Reopen, fix gap via post-recovery `is_contract_empty` | Idempotent, no fd-passing protocol needed. |
| Config divergence | Stitch state.toml against current `enabled.d/`, abandon orphans | Permissive without complexity; supports `coral install + replace` automation. |
| Adoption | **No `adopt_contract` call** | PID-1 in-place execve preserves kernel-level contract ownership; only userspace bookkeeping needs rebuilding. |
| State file location | `/var/run/zyginit/state.toml` | Existing runtime tree; boot-id check makes it safe across real reboots. |

## 12. Why keep this in the Hammerhead-userland package?

Hammerhead's primary upgrade workflow is BE-based: `pkg update` → new BE → `beadm activate` → reboot. In that workflow, `zygctl replace` is bypassed — the new zyginit comes up as a fresh PID 1 in the new BE. So why ship the feature?

**Linux port.** zyginit is written in Reef which compiles to native C; a Linux port is realistic. Linux package managers (`apt`, `yum`, `dnf`) operate on an in-place upgrade model where daemons need a way to reload their own binary without restarting the system. systemd's `systemctl daemon-reexec` exists for exactly this reason. If zyginit lands on Linux, `zygctl replace` becomes a primary feature.

**Live patching trajectory.** A system that supports live kernel patching (kpatch-style) has implicit cultural pressure to support live userland patching for critical daemons. zyginit being live-upgradeable is a natural fit for that posture if Hammerhead develops it.

**Mechanism reuse.** Several pieces (`contract.is_contract_empty`, `supervisor.adopt_runtime`, the state-file pattern in `utils/replace_probe`) have potential consumers beyond this feature — shutdown verification, future crash-recovery work, generic state-file testing patterns.

The feature stays; the trigger is documented as Hammerhead-secondary, Linux-primary.
# SMF Migration Handoff Document — Design

**Date:** 2026-05-04
**Status:** Approved (brainstorm complete); implementation plan to follow
**Scope:** A single Markdown document at `docs/SMF_MIGRATION.md` that hands off the remaining SMF-removal and zyginit-integration work from the zyginit project to the Hammerhead-project Claude (or Hammerhead human engineers). No tooling. No code.

## 1. Goal

Hand off the remaining SMF removal work to the Hammerhead team (or a Hammerhead-project Claude agent) with everything they need to:

1. Vendor zyginit source from this repo into Hammerhead's source tree as part of `hammerhead-userland`.
2. Wire zyginit into Hammerhead's master build, install paths, and packaging.
3. Migrate the existing 18 zyginit-converted services (currently in `services/hammerhead/`) into Hammerhead's source tree as the starting point.
4. Convert the remaining ~110 SMF manifests using the patterns documented in this work, the existing converted services as references, and project judgment about what Hammerhead actually ships.
5. Remove SMF infrastructure (`svc.startd`, `svc.configd`, master restarter, libscf consumers, manifest packaging) from `hammerhead-userland`.

The deliverable is **one Markdown document**. zyginit ships docs and patterns; Hammerhead does the OS-level execution.

## 2. Non-goals

- **No `smf-to-toml` conversion tool.** A Claude agent doing the conversion in-place can do both the mechanical XML-to-TOML translation AND the per-service judgment that a tool can't. Avoiding the tool also keeps zyginit free of a Python runtime dependency and aligns with the structural rule "DO NOT modify Hammerhead source code" — the Hammerhead Claude does that work in its own project.
- **No per-service detailed catalog with full recommendations.** zyginit doesn't know what Hammerhead ships; per-service decisions stay with the Hammerhead team. The doc provides a coarse one-line classification per manifest and lets the Hammerhead Claude finish each per-service decision with full local context.
- **No format reference duplication.** TOML field-by-field details stay in `man/zyginit.5`. The migration doc references it.
- **No build-system implementation.** The doc describes what the Hammerhead build needs to do; the Hammerhead Claude integrates it with the actual build conventions in their tree.
- **No Coral packaging for zyginit.** zyginit is part of `hammerhead-userland` (BSD-style base system), built into the OS by Hammerhead's master build.
- **No automated catalog maintenance.** The catalog is a snapshot of one Hammerhead manifest tree at one point in time; it is archival once the migration is complete.

## 3. Document structure

The single Markdown file at `docs/SMF_MIGRATION.md`, ~600-800 lines. Top-level sections:

```
1. Handoff Brief                       (~100 lines)
2. Decision Tree                       (~150 lines)
3. Worked Examples                     (~200 lines)
4. Anti-patterns & Gotchas             (~100 lines)
5. Integration with Hammerhead         (~150 lines)
6. Per-Category Guidance               (~100 lines)
7. Catalog Appendix                    (~200 lines, table format)
8. Format Reference Pointer            (~10 lines)
9. Glossary                            (~30 lines)
```

The doc is intended for both human engineers and AI agents working on the Hammerhead source tree. Format optimized for human readability; structured enough that a Claude agent can load it as context and act on it.

The file lives at `docs/SMF_MIGRATION.md` (top-level under `docs/`, alongside `DESIGN.md`). The spec for THIS doc lives at `docs/superpowers/specs/2026-05-04-smf-migration-doc-design.md`. The migration doc itself stays at the top level to signal "this is a real artifact for the Hammerhead team," not a planning artifact.

## 4. Section content

### 4.1 Handoff Brief

Sets up the context for the Hammerhead Claude. Includes:

- What zyginit has done so far (the iter 1-20 work; the contract-readoption feature; the 18 converted services in the boot chain on `hh-prototest`).
- What's left for the Hammerhead team to do (source vendoring, build integration, migrate the 18 existing TOMLs, convert the remaining ~110 manifests, remove SMF infrastructure).
- Coordination points: when to bounce a question back to the zyginit project (e.g., "the existing patterns don't fit this service" → file a zyginit issue; we extend the format/runtime if needed).
- Out-of-scope reminders ("DO NOT modify Hammerhead source from inside zyginit's tree" — this constraint is for the zyginit-project Claude, not the Hammerhead Claude; the Hammerhead Claude WILL modify Hammerhead source — that is its job).
- Done criteria for "SMF removal complete": Hammerhead boots and reaches multi-user without `svc.startd` / `svc.configd` ever being invoked; libscf-only consumers gone or stubbed; manifest infrastructure removed from packaging; existing zyginit-supervised services continue to work.

### 4.2 Decision Tree

The most consequential single piece of the doc — what the Hammerhead Claude leans on for every manifest. Three top-level exits per manifest:

```
Step 1: Should this service exist on Hammerhead at all?
   ├── No (deprecated, replaced)
   │     → DROP. Remove from package; do not write a TOML.
   ├── Yes, but as zports add-on, not base
   │     → DEFER. Out of scope for hammerhead-userland.
   └── Yes, ships in hammerhead-userland → continue to Step 2

Step 2: Is the service tightly coupled to SMF infrastructure
        (svc.configd / libscf / repository_door)?
   ├── Yes, fundamentally requires libscf at runtime
   │     → INVESTIGATE. Find a foreground/standalone mode, or
   │       coordinate with the daemon's upstream to add one.
   ├── Yes, but the daemon has a non-SMF mode flag (-d, -f, -D)
   │     → CONVERT (foreground-mode pattern). See Example 1.
   └── No, or coupling is incidental → continue to Step 3

Step 3: What runs the service today?
   ├── Direct daemon binary, no SMF method script
   │     → CONVERT (full-TOML pattern, simplest case)
   ├── /lib/svc/method/<name> shell script that does setup before exec
   │     ├── Trivial setup → INLINE into TOML's exec.start
   │     ├── Substantial setup (sources lib/svc/share/*.sh) → KEEP
   │       method script as exec.start. See Example 2.
   ├── Oneshot configuration step (mountall, swapadd, dumpadm)
   │     → CONVERT as oneshot service. See services/hammerhead/
   │       root-fs, swap, filesystem.
   └── inetd-managed service
         → DROP unless strong reason to keep. We don't run inetd.
```

Within "CONVERT," a field-by-field translation table maps SMF manifest XML elements to TOML fields:

| SMF concept | TOML field | Notes |
|---|---|---|
| `<service name='X'>` | `[service] name = "X"` | Strip category prefix |
| `<template>` short_description | `[service] description` | Plain string |
| `<service_bundle type='manifest'>` | `[service] type = "daemon\|oneshot\|transient"` | Map by behavior |
| `<exec_method type='start'>` | `[exec] start = "..."` | Replace method-script if Step 2 said foreground-mode |
| `<exec_method type='stop'>` | `[stop] method = "exec"` + `[exec] stop = "..."` | `:kill` in SMF maps to `[stop] method = "contract"` |
| `<dependency type='service'>` | `[dependencies] requires = [...]` | Translate FMRI to bare name; drop SMF-internal deps |
| `<dependency type='path'>` | `[dependencies] requires = ["filesystem"]` | Path deps mostly map to "filesystem is up" |
| `<property_group name='startd'>` `duration` | `[service] type` | transient vs daemon |
| `<property_group name='general'>` `enabled` | not in TOML | Operator decides via symlink in `enabled.d/` |
| Contract behavior | `[contract] param = [...]`, `fatal = [...]` | Defaults usually fine |
| Restart-on-error | `[restart] on = "always\|failure\|never"` | SMF's `ignore_error` maps to `on = "failure"` |

Anti-patterns explicitly called out:
- "Just convert it because it exists" — Step 1 audit is mandatory.
- "Wrap the method script verbatim" — many method scripts source `lib/svc/share/smf_include.sh` and call `smf_present`; wrapping reintroduces configd coupling.
- "Skip Step 2 because it's complicated" — getting Step 2 wrong was the entire iter 1-5 dead end.

### 4.3 Worked Examples

Three examples, each ~60-80 lines, structured identically.

**Example 1: foreground-mode conversion (canonical).** Subject: `dlmgmtd` (`/lib/svc/manifest/network/dlmgmt.xml`). Walks through: original SMF manifest, decision-tree walk (Step 1 keep, Step 2 yes-coupled-but-has-`-d`-flag), the resulting `services/hammerhead/dlmgmtd.toml`, why-we-made-each-choice (the existing comment block on dlmgmtd.toml is the educational artifact), and the iter 11-15 confirmation that `dladm`/`ipadm` work.

**Example 2: SMF method-script wrap.** Subject: `system/cron`. Original manifest with the `<exec_method type='start'>/lib/svc/method/svc-cron start</exec_method>` line. Decision-tree walk (Step 1 keep, Step 2 not coupled, Step 3 method-script trivial). Two valid TOMLs side-by-side: the inlined version vs the wrap version. Show that both are correct; explain when you'd pick which. Show the existing `services/hammerhead/cron.toml` as the production choice.

**Example 3: drop.** Subject: `system/svc/restarter`. The classic must-drop. Original manifest showing FMRI (`svc:/system/svc/restarter:default`) and exec method (`/lib/svc/bin/svc.startd`). Decision-tree walk: Step 1 NO — this IS SMF; zyginit replaces it. Document the cascade: most other manifests reference this in `<dependency>` blocks; when those services get converted, those dependencies are dropped (they were SMF-internal, not real ordering constraints). Other must-drop companions: `svc.configd`, the master restarter framework as a whole.

What's not in the worked examples (but pointed to from elsewhere in the doc):
- Oneshot conversions (root-fs, swap, filesystem) → already in `services/hammerhead/`; doc points there.
- Daemonize-handshake-needing services (syslogd, utmpd) → covered in Anti-patterns rather than as a worked example.
- Network-config-style → already in `services/hammerhead/network.toml`; pointer rather than walk-through.

### 4.4 Anti-patterns & Gotchas

A short section drawn from the iter notes already in memory. Each gotcha gets ~3-4 lines: symptom, cause, fix. Cross-reference into the iter notes (`pid1_boot_iter11_15.md`, etc.) so the Hammerhead Claude can pull more context.

Topics:
1. configd-coupled daemons silently degrade (iter 1-5, BSD pivot).
2. Daemonize handshake (iter 16, supervisor handles it but expect log noise).
3. Foreground-flag names vary (`dlmgmtd -d`, `ipmgmtd -f`, `sshd -D`, `syslogd -d`, `fmd -f`).
4. soconfig at boot — without it, `socket(AF_INET)` returns EAFNOSUPPORT.
5. `/var/run` not yet tmpfs in zyginit BE — files persist across reboots; follow-up to mount it.
6. utmpx record positioning — `pututxline` updates an existing matching record, doesn't append. Live-replace boot-id sentinel must come from `/proc/1/start`, not utmpx.
7. Don't `zfs set readonly=on` on the live root during shutdown (iter 19-20).
8. uadmin must not be called from PID 1; fork a child (iter 19-20).
9. `/etc/hosts` install bug — base ships only `localhost`, not actual hostname; causes 20-second login delays via DNS timeout. Flagged today as upstream Hammerhead bug.

### 4.5 Integration with Hammerhead build

Substantive section. zyginit is part of `hammerhead-userland` (BSD-style base system); Hammerhead's master build pulls and builds zyginit alongside other userland components. zyginit repo continues as upstream development repo; periodic syncs into Hammerhead.

Subsections:

1. **Source layout decision.** Recommended target path: `usr/src/cmd/init/` (BSD-tree convention; same place `cmd/init/init.c` lives in standard illumos). zygctl to `usr/src/cmd/zygctl/`. sysv-wrapper to `usr/src/cmd/sysv-wrapper/`. Sync mechanism (git submodule, source snapshot, fetch-and-extract) is the Hammerhead team's call.
2. **File-by-file mapping** (explicit table):
   ```
   zyginit-repo/src/*.reef                  → usr/src/cmd/init/src/
   zyginit-repo/src/helpers.c               → usr/src/cmd/init/src/
   zyginit-repo/src/contract_linux_stubs.c  → DROP (Hammerhead doesn't need Linux stubs)
   zyginit-repo/services/hammerhead/*       → usr/src/cmd/init/services/
   zyginit-repo/tools/zygctl/src/           → usr/src/cmd/zygctl/src/
   zyginit-repo/tools/zygctl/src/symlink_wrapper.c → same
   zyginit-repo/tools/sysv-wrapper/         → usr/src/cmd/sysv-wrapper/
   zyginit-repo/man/man8/*                  → usr/src/man/man8/
   ```
3. **Build invocation** — what the build needs to do:
   - `reefc build -l contract --obj <helpers.o>` → produces `/sbin/init`
   - `reefc build` for zygctl → produces `/sbin/zygctl`
   - `make` for sysv-wrapper → produces `/sbin/sysv-wrapper` plus argv[0] symlinks (`/sbin/halt`, `/sbin/reboot`, `/sbin/poweroff`, `/sbin/telinit`)
   - Install service TOMLs to `/etc/zyginit/`
   - Install start.sh / stop.sh scripts to `/etc/zyginit/<svc>/`
   - Install man pages
   - Drop `svc.startd`, `svc.configd`, master restarter from the build (the actual SMF removal step)
4. **Reef compiler dependency** — Hammerhead's build needs `reefc`. Either bootstrap from Reef repo or assume a pre-built reefc. Hammerhead team's call.
5. **First migration step (concrete deliverable):** copy the existing `services/hammerhead/*.toml` (and start.sh / stop.sh subdirs) from the zyginit repo into Hammerhead's source tree. This gets the iter 18-20 boot chain into Hammerhead's source as the starting point, before any of the ~110-manifest conversion work begins. Order:
   1. Vendor zyginit source into Hammerhead tree
   2. Wire up the build to produce `/sbin/init`
   3. Vendor the existing 18 TOMLs
   4. Validate: build a Hammerhead BE; boot; verify all 18 services come up (matches current hh-prototest behavior)
   5. THEN start the per-manifest conversion of the remaining manifests using the decision tree
6. **Upstream-sync workflow** — when zyginit ships new features, the Hammerhead team:
   - Re-fetches zyginit repo at a known tag/commit
   - Refreshes the vendored source under `usr/src/cmd/init/`
   - Resolves any local Hammerhead patches that conflict
   - Rebuilds and tests
   - Documents the sync point (commit hash) in Hammerhead's changelog

### 4.6 Per-Category Guidance

Walks the SMF manifest tree top-down, gives recommendations per category. The Hammerhead Claude reads this once for orientation, then drops into the catalog appendix for per-manifest decisions.

Structure mirrors the actual filesystem at `/lib/svc/manifest/`:
- `application/` — mostly Hammerhead-irrelevant; drop or defer to zports.
- `network/` — selective: keep DNS resolver consideration, NFS only if shipped, drop NIS/iSCSI/loadbalancer/SMB.
- `platform/` — keep amd64-specific bits; drop sun4u/sun4v (SPARC; Hammerhead is amd64-only).
- `system/` — keep core (devfs, filesystem, fmd-if-FMA-wanted, ipfilter); DROP ALL `system/svc/*` (this is SMF infrastructure itself).

Each category gets ~3-5 lines: what's there, default recommendation, per-category gotchas. Cross-reference into the catalog appendix for individual manifests.

### 4.7 Catalog Appendix

The 128 manifests as a sortable, scannable Markdown table. Format:

```
| Manifest path | Category | Classification | Note |
|---|---|---|---|
| /lib/svc/manifest/system/cron.xml | system | already-converted | services/hammerhead/cron.toml |
| /lib/svc/manifest/system/svc/restarter.xml | system | drop | This IS svc.startd; we replace it |
| /lib/svc/manifest/system/fm/fmd.xml | system | foreground | fmd -f flag; needs verification |
...
```

Five classifications:
- `already-converted` — TOML exists in current `services/hammerhead/`; just migrate to Hammerhead source tree.
- `drop` — service should not be in Hammerhead at all.
- `foreground` — convert via foreground-mode pattern (decision tree Step 2).
- `method-wrap` — keep the SMF method script as exec.start (decision tree Step 3).
- `needs-investigation` — depends on a Hammerhead-team policy decision.

The "Note" column carries the why — for `already-converted` it's a path pointer; for `drop` a one-line reason; for `foreground` the candidate flag; for `method-wrap` the method script path; for `needs-investigation` the open question.

**Generation effort:** populating the catalog is the substantive zyginit-side work — open each XML, fill in classification + note. Most entries are obvious; a handful need real thought. Output is a static Markdown table.

**Maintenance posture:** the catalog is a snapshot of one Hammerhead manifest tree. As Hammerhead evolves upstream, the catalog drifts. The doc explicitly notes this is a starting-point snapshot, not a living index.

### 4.8 Format Reference Pointer

Single short section: "For TOML field-by-field reference, see `man/zyginit.5`. The format reference is authoritative there; this doc covers patterns and decisions, not syntax."

### 4.9 Glossary

Five-to-ten-line definitions of SMF concepts that show up in the doc but may be unfamiliar to non-illumos hands or AI agents whose training data is light on Solaris:
- FMRI, Manifest, Method, Profile, Repository, Restarter, Contract, libscf, libcontract, BE / Boot Environment, soconfig.

## 5. Audience and writing style

The doc is written for both human engineers and AI agents working on the Hammerhead source tree. Format:

- Markdown, GitHub-flavored, no emojis.
- Headings to two levels (`##`, `###`) for navigation; deeper structure via lists.
- Code blocks (```` ``` ````) for TOML, XML, and shell snippets.
- Cross-references to other files use repo-relative paths (e.g., `services/hammerhead/dlmgmtd.toml`, `man/zyginit.5`, `pid1_boot_iter11_15.md`).
- Each major section ends with a "see also" line pointing to related sections.
- The catalog appendix uses Markdown tables (not flat lists).
- Each subsection starts with a one-sentence purpose statement so readers can skim.

Optimizing for human readability also serves AI agents well; no separate "agent-friendly" tracks.

## 6. Implementation effort

Approximate: 4-6 hours of focused work for a Claude agent to write this doc end-to-end:

- Sections 1, 2, 4, 5, 6, 8, 9 — straightforward writeup of decisions already made; ~2-3 hours.
- Section 3 (worked examples) — requires walking through the actual XML manifests for dlmgmtd, cron, and svc.startd; ~1 hour.
- Section 7 (catalog) — open each of 128 manifests on hh-prototest, classify, fill in notes; ~2-3 hours.

Spec self-review after writing: ~30 minutes.

## 7. Decisions summary

| Decision | Choice | Rationale |
|---|---|---|
| Tooling | None (no `smf-to-toml` translator) | Claude does both mechanical and judgment work in one pass; tool would duplicate the mechanical part and add a Python dependency. |
| Format | Single Markdown doc at `docs/SMF_MIGRATION.md` | Simpler than two coupled docs; one source of truth for the handoff. |
| Depth | Middle ground (~5-7 pages prose + catalog appendix) | Decision tree + 2-3 worked examples + anti-patterns + per-category guidance. Format reference stays in `man/zyginit.5`. |
| Catalog | Coarse classification per manifest (5 classes) | One-line annotations for all 128; per-service detailed decisions stay with Hammerhead team. |
| Tool language | N/A (no tool) | — |
| Integration model | zyginit vendored into `hammerhead-userland` (OpenBSD-style base) | Per user direction; not Coral-packaged; zyginit repo continues as upstream. |
| Audience | Both human engineers and AI agents | Single doc serves both; readability optimized for humans. |
# Versioning and Release Tarball — Design

**Date:** 2026-05-05
**Status:** Approved (brainstorm complete); implementation plan to follow
**Scope:** Establish a single canonical version for the zyginit suite (zyginit + zygctl + sysv-wrapper), wire `--version` flags into all three binaries, and add scripts to bump the version and produce a source release tarball for vendoring into Hammerhead.

## 1. Goal

Hammerhead consumes zyginit as a **vendored source drop** into `hammerhead-userland` (per `docs/SMF_MIGRATION.md`). To make that drop reproducible and auditable, zyginit needs:

1. A single canonical version, tracked in source.
2. A scripted bump that updates every place the version appears, in lock-step.
3. A scripted release that produces `releases/zyginit-<version>-source.tar.xz` plus a `.sha256` companion.
4. A runtime `--version` flag on every shipped binary, so an installed Hammerhead system can answer "which zyginit am I running?".

This work intentionally mirrors Coral's `scripts/make-release.sh` pattern (single tarball, sha256, `--transform`-prefixed paths) — the Hammerhead team gets a consistent shape across Zygaena projects.

## 2. Non-goals

- **No CI/CD release automation.** Bump and release are run by hand by a maintainer; no GitHub/Hg-server hooks, no signing, no upload step.
- **No reproducible-build flags** (`--mtime`, `--owner=0`, `--sort=name`). Coral doesn't bother and Hammerhead's packaging step is what produces the OS-level reproducible artifact. Adding these is a trivial follow-up if needed.
- **No multi-tarball releases.** The whole suite (zyginit + zygctl + sysv-wrapper) ships and versions together, in one tarball.
- **No `CHANGELOG.md` integration.** The bump script doesn't touch a changelog. (We can add this later; there is no changelog file today to integrate with.)
- **No GPG signing.** Hammerhead handles signing at the OS package layer.

## 3. Versioning policy

Semver: `MAJOR.MINOR.PATCH`.

| Bump | Reason |
|---|---|
| MAJOR | Incompatible socket protocol change, incompatible service-definition (TOML) schema change, removal of a published `zygctl` subcommand |
| MINOR | Backward-compatible additions: new service-definition fields with defaults, new `zygctl` subcommands, new runlevel features |
| PATCH | Bug fixes only; no API or protocol changes |

While zyginit is pre-1.0, breaking changes may land in MINOR per semver convention. Once 1.0 ships (target: when SMF removal from Hammerhead is complete and the suite has been running unmodified for one release cycle), strict semver applies.

## 4. Source of truth

The root `reef.toml` `[package].version` is **canonical**. Every other location is **derived** and regenerated by the bump script.

Today's drift inventory (rev 82, version `0.1.0`) — the bump script must keep these all consistent:

| # | Location | Form |
|---|---|---|
| 1 | `reef.toml` (root) | `version = "0.1.0"` (canonical) |
| 2 | `tools/zygctl/reef.toml` | `version = "0.1.0"` |
| 3 | `src/version.reef` (new, generated) | `pub fn VERSION(): string` returning `"0.1.0"` |
| 4 | `tools/zygctl/src/version.reef` (new, generated) | same |
| 5 | `tools/sysv-wrapper/version.h` (new, generated) | `#define ZYGINIT_VERSION "0.1.0"` |

Generated files (`version.reef` × 2 and `version.h`) **are committed** to the repository so a tarball recipient can build without rerunning the bump script. They carry a "Generated by scripts/bump-version.sh — do not edit by hand" comment.

## 5. `scripts/bump-version.sh`

**Usage:** `scripts/bump-version.sh <new-version>`

**Behavior:**

1. Validates `<new-version>` matches `^[0-9]+\.[0-9]+\.[0-9]+$`. Refuses if not.
2. Reads current version from root `reef.toml`. Refuses if `<new-version>` equals current.
3. Rewrites all five locations listed in §4.
4. Prints a summary diff (`hg diff --stat`) and a suggested commit command:
   ```
   $ hg ci -m 'Bump version to <new-version>'
   ```
5. Does **not** auto-commit. The maintainer reviews and commits by hand.

**Generated file templates:**

`src/version.reef` (and identical-content `tools/zygctl/src/version.reef`):
```reef
/* SRCHEADER... */

module version

// Generated by scripts/bump-version.sh — do not edit by hand.
pub fn VERSION(): string
    return "0.1.0"
end VERSION

end module
```

`tools/sysv-wrapper/version.h`:
```c
/* SRCHEADER... */
/* Generated by scripts/bump-version.sh — do not edit by hand. */
#ifndef ZYGINIT_VERSION_H
#define ZYGINIT_VERSION_H
#define ZYGINIT_VERSION "0.1.0"
#endif
```

Both templates use the project's standard `SRCHEADER.txt` header. The bump script fills in `${project}` from each subproject's own `reef.toml` `[package].name` (i.e., `zyginit` for `src/version.reef`, `zygctl` for `tools/zygctl/src/version.reef`), `${file.name}` from the filename, and `${file.description}` with a fixed string ("Generated version constant").

## 6. `scripts/make-release.sh`

**Usage:** `scripts/make-release.sh [version-override]`

**Behavior** (mirrors Coral's `make-release.sh`):

1. Reads version from root `reef.toml`, or uses `[version-override]` if provided.
2. Creates `releases/` if missing.
3. Produces `releases/zyginit-<version>-source.tar.xz` via `tar --transform "s,^,zyginit-<version>-source/," -cJf ...`.
4. Generates `releases/zyginit-<version>-source.tar.xz.sha256`.
5. Prints final paths and the sha256.

**Tarball contents** (top-level paths, `--transform`-prefixed into `zyginit-<version>-source/`):

| Include | Why |
|---|---|
| `reef.toml` | Build manifest (carries the canonical version) |
| `README.md` | Project orientation |
| `ROADMAP.md` | Forward-looking roadmap |
| `CLAUDE.md` | Project context (useful even outside Claude — accurate in-tree docs) |
| `SRCHEADER.txt` | Source-header template (referenced by all source files) |
| `src/` | Main daemon Reef source |
| `docs/` | Architecture docs (`DESIGN.md`, `SMF_MIGRATION.md`, etc.) |
| `tests/` | Unit + integration tests |
| `services/` | Example + Hammerhead service TOMLs (the v2 BSD-pivot tree) |
| `tools/` | `zygctl/`, `sysv-wrapper/` (subprojects) |
| `utils/` | `contract_probe` and any other utility tools |
| `scripts/` | `bump-version.sh`, `make-release.sh`, `install-to-be.sh` |

**Tarball excludes:**

| Exclude | Why |
|---|---|
| `build/` (root and any `tools/*/build/`) | Build artifacts |
| `releases/` | Output of this very script — don't ship past tarballs |
| `.hg/`, `.hgignore`, `.hgtags` | VCS metadata |
| `*.o`, `*.a` | Compiled object files |
| `*.swp` | Editor cruft |
| `resume/` | Local scratch / resume notes (per user instruction) |
| `ss/` | Local scratch (per user instruction) |
| `stub/` | Old Rust-like design archeology, kept for reference only (per `CLAUDE.md`) |

`tar`'s `--exclude` patterns are passed in for each item. The exclusion of `tools/*/build/` is critical — the global `build/` exclude only covers the repo-root build dir.

## 7. Code changes (one-time)

These ride along with the introduction of the bump/release scripts:

| File | Change |
|---|---|
| `src/main.reef:67-69` | Drop the inline `fn VERSION(): string` definition; `import version` and use `version.VERSION()` |
| `src/socket.reef:248` | Replace literal `"zyginit 0.1.0\n"` with `"zyginit " + version.VERSION() + "\n"` |
| `tools/zygctl/src/main.reef:54-57` | Replace literal `"zygctl 0.1.0"` with `"zygctl " + version.VERSION()` |
| `tools/zygctl/src/main.reef` | Add `import version` |
| `tools/sysv-wrapper/wrapper.c` | Add `--version` / `-V` handling: prints `"<argv[0]-basename> <ZYGINIT_VERSION>\n"` and exits 0. `#include "version.h"`. |
| `tools/sysv-wrapper/Makefile` | Add `version.h` to the dependency list of `wrapper.o` |

The argv[0]-basename trick in sysv-wrapper means `halt --version` prints `halt 0.1.0`, `reboot --version` prints `reboot 0.1.0`, etc. — consistent with how each personality identifies itself in normal output.

## 8. Output format

A single line, `<binary> <version>`, exit 0. Matches Coral and the GNU/BSD common case:

```
$ zyginit --version
zyginit 0.1.0
$ zygctl --version
zygctl 0.1.0
$ halt --version
halt 0.1.0
```

`zygctl version` (no dashes — already a subcommand) continues to work and prints the same line. Both `zygctl --version` and `zygctl version` are **strictly local** — they print zygctl's compiled-in version and never round-trip the socket. This is the standard Unix convention and keeps the answer trustworthy when the daemon is wedged.

Querying the running daemon's version (useful during a partial upgrade) is an existing socket command (`socket.reef:247`) but is currently unreachable from zygctl. Wiring up a separate `zygctl daemon-version` subcommand to surface it is **out of scope** for this design — see §10.

## 9. End-to-end workflow

Maintainer cuts release `0.2.0`:

```
$ scripts/bump-version.sh 0.2.0
zyginit: bumping 0.1.0 -> 0.2.0
  rewriting reef.toml
  rewriting tools/zygctl/reef.toml
  regenerating src/version.reef
  regenerating tools/zygctl/src/version.reef
  regenerating tools/sysv-wrapper/version.h
done. review with `hg diff` and commit:
  hg ci -m 'Bump version to 0.2.0'

$ hg diff --stat
 reef.toml                          | 2 +-
 tools/zygctl/reef.toml             | 2 +-
 src/version.reef                   | 2 +-
 tools/zygctl/src/version.reef      | 2 +-
 tools/sysv-wrapper/version.h       | 2 +-
 5 files changed, 5 insertions(+), 5 deletions(-)

$ hg ci -m 'Bump version to 0.2.0'

$ scripts/make-release.sh
Creating release: zyginit-0.2.0-source
Created: releases/zyginit-0.2.0-source.tar.xz
-rw-r--r-- 1 ctusa ctusa 1234567 May  5 15:30 releases/zyginit-0.2.0-source.tar.xz
Checksum: releases/zyginit-0.2.0-source.tar.xz.sha256
abc123...  releases/zyginit-0.2.0-source.tar.xz
```

Hammerhead then consumes the tarball + sha256 into `hammerhead-userland`, per the integration steps in `docs/SMF_MIGRATION.md`.

## 10. Open questions / future work

- **`scripts/install-to-be.sh`** already exists. Out of scope for this design — it's an in-place dev install, not a release artifact path.
- **Reproducible tarballs** (`--mtime`, `--sort=name`, `--owner=0`, `--group=0`). Add when the Hammerhead build needs byte-identical tarballs across maintainer machines.
- **`CHANGELOG.md`** — add when there's enough release history to warrant one. The first one or two release notes can live in commit messages.
- **GPG signing** — Hammerhead handles this at the OS package layer; no zyginit-side signing planned.
- **Pre-release versions** (e.g., `0.2.0-rc1`). Not supported by the regex in §5. Add when needed.
- **`zygctl daemon-version` subcommand.** The socket already exposes a `version` command (`socket.reef:247`) that returns the running daemon's version, but no client surfaces it. Add a `zygctl daemon-version` (or similar) subcommand to make partial-upgrade diagnosis easy. Until then, `socket.reef`'s `version` handler stays as accessible-but-unused (an `echo version | nc -U /var/run/zyginit.sock` still works).
# zyginit Visual Pass — Design

**Date:** 2026-05-12
**Status:** Approved (brainstorm complete); implementation plan to follow
**Scope:** Unified visual identity across the boot console, `zygctl`, and shutdown. New `src/ui.reef` module owns palette, glyphs, sigil, TTY/TERM detection, ANSI rendering, rolling tape, progress bar, and spinner. Existing `println` lifecycle sites in `main.reef` / `supervisor.reef` are routed through it. Graceful fallback to ASCII glyphs on dumb terminals and to plain line-by-line output when non-TTY, `NO_COLOR`, or supervisor mode.

## 1. Goals

1. **Unified identity.** Same `[Z]` sigil, six-color palette, and four-state glyph set on every visible surface: boot console, shutdown console, `zygctl status`.
2. **Restrained brand.** No banner ASCII art, no friendly tier names, no animations beyond a single spinner glyph. The boot screen feels structured and tidy, not theatrical.
3. **Make tiers visible.** zyginit is the only init that exposes dependency tiers as a first-class concept. The boot UI uses them as the structural backbone of progress reporting.
4. **Graceful degrade.** Truecolor on Hammerhead framebuffer, 16-color baseline on serial consoles, ASCII glyphs on dumb terminals, plain machine-parseable lines when non-TTY or `NO_COLOR`.
5. **Hammerhead-tem-safe.** No animation faster than the main event loop wake-up cadence, no terminal queries (`CSI 6n` etc.), no concurrent writers to `/dev/console`. zyginit owns the screen.

## 2. Non-goals

- **No splash screen / Plymouth-equivalent.** Tem is polled I/O — text-mode only.
- **No friendly tier names.** Tiers are displayed as bare `tier N`. Adding a `tier_name` TOML field was considered and rejected.
- **No mouse, no Unicode line-drawing reliance.** Box-drawing chars (`──` divider) are the only Unicode used in the frame; everything else has an ASCII fallback.
- **No new process or binary.** Rendering happens in-process inside zyginit; `zygctl` prints the daemon's pre-rendered response verbatim.
- **No restructure of the zyginit ↔ zygctl module boundary.** They stay separate Reef projects with separate `reef.toml`.
- **No `zygctl list` banner.** `list` is the catalog command, used in scripts more than `status` — stays minimal.
- **No animation under heavy load.** Spinner ticks are event-driven (one per main-loop wake-up), not timer-based.

## 3. Architecture

### 3.1 New module: `src/ui.reef`

Single new file. Owns:

- Palette + truecolor / 16-color SGR selection
- Glyph set + ASCII fallback
- TTY / TERM / `NO_COLOR` detection
- ANSI cursor positioning helpers
- Header + rolling tape + progress bar state
- `[Z]` sigil
- Spinner frame index

### 3.2 Public surface (Reef pseudo-signatures)

```
// Lifecycle
proc ui_init(rich_mode: int)
proc ui_boot_start(num_svc: int, num_tiers: int, runlevel: string)
proc ui_tier_start(tier: int)
proc ui_tier_done(tier: int)
proc ui_event_starting(name: string)
proc ui_event_started(name: string, dur_ms: int)
proc ui_event_failed(name: string, exit_code: int, dur_ms: int)
proc ui_event_stopping(name: string)
proc ui_event_stopped(name: string, dur_ms: int)
proc ui_boot_complete(stats: BootStats)
proc ui_shutdown_start(reason: string)
proc ui_shutdown_complete(reason: string, elapsed_ms: int)

// Spinner
proc ui_tick()                          // advance spinner frame, redraw active rows
fn   ui_spinner_frame(reverse: bool): string

// On-demand rendering (for zygctl status/list)
fn   ui_render_status(table: ServiceTable, want_banner: int, plain: int): string
fn   ui_render_list(table: ServiceTable): string
```

### 3.3 Call-site changes

- **`src/main.reef`** — invoke `ui_init` immediately after `setup_pid1_console`. Wrap dep-graph load with `ui_boot_start`. Wrap each tier loop with `ui_tier_start` / `ui_tier_done`. Call `ui_boot_complete` after all tiers. Add `ui_tick` to the main poll loop. Bracket shutdown with `ui_shutdown_start` / `ui_shutdown_complete`.
- **`src/supervisor.reef`** — replace ~8 lifecycle `println` calls (started, stopped, failed, stop-timeout, scheduled-for-restart, entered-maintenance) with `ui_event_*` calls. Warning/debug prints stay as-is.
- **`src/socket.reef`** — `cmd_status` and `cmd_list` delegate to `ui_render_status` / `ui_render_list`. New `PLAIN` modifier on the wire forces banner-off + escape-free output.
- **`tools/zygctl/src/main.reef`** — pre-flight: detect `isatty(STDOUT_FILENO)` + `NO_COLOR` + `TERM`. Send `STATUS PLAIN` or `STATUS` to the daemon accordingly. Print the response verbatim.

### 3.4 What stays unchanged

- Service stdout/stderr capture to `/var/log/zyginit/<name>.log` (per-service log files) is unchanged.
- Existing socket protocol commands keep their text shape; `PLAIN` is an additive modifier.
- `zygctl list` keeps its existing minimal one-line-per-service shape.
- The supervisor state machine (8 states), restart policies, contract handling — none of it changes.

## 4. Visual elements

### 4.1 Palette

Six tokens. Truecolor on `TERM=sun-color` or `*-256color`; 16-color baseline otherwise.

| Token | Truecolor | 16-color fallback | Used for |
|-------|-----------|-------------------|----------|
| `c.frame` | `#6E7B8B` steel grey | `white` | header text, column labels |
| `c.accent` | `#006A6F` teal | `cyan` | `[Z]` sigil, dividers, "elapsed", progress-bar filled |
| `c.ok` | `#2E8B57` sea green | `green` | `●` online / running |
| `c.warn` | `#FFBF00` amber | `yellow` | `◉` / spinner — starting / stopping |
| `c.fail` | `#C8201F` signal red | `red` | `✕` failed / maintenance |
| `c.mute` | default fg + `\e[2m` (dim attr) | `dim` | `·` pending / stopped, progress-bar empty |

### 4.2 Glyphs

| State | Unicode | ASCII (TERM=dumb or `ZYGINIT_ASCII=1`) | Color |
|-------|---------|----------------------------------------|-------|
| online / running | `●` | `#` | `c.ok` |
| starting | spinner (see 4.3) | spinner | `c.warn` |
| stopping | spinner reversed | spinner reversed | `c.warn` |
| failed / maintenance | `✕` | `X` | `c.fail` |
| pending / stopped | `·` | `.` | `c.mute` |

Box-drawing divider: `──────`. Same character in rich and ASCII modes; tem renders it from the sun-color font.

### 4.3 Spinner

Frame set: `[ "/", "-", "\\", "|" ]` (same in rich and ASCII modes).

- **Starting state:** frames advance forward — `/ → - → \ → |` — index `g_tick mod 4`.
- **Stopping state:** frames advance in reverse — `| → \ → - → /` — index `(N_FRAMES - 1) - (g_tick mod 4)`.
- **`g_tick`** is incremented by `ui_tick()`, which is called once per main-loop wake-up. No separate timer thread. Spinner advances on any signal, contract event, socket activity, or poll timeout (200ms boot-mode default).
- Services that transition `STARTING → RUNNING` in under one tick never visibly spin; their row appears with `●` directly. No special case.
- Spinner is colored `c.warn`.

### 4.4 Sigil

`[Z]` rendered in `c.accent`, prefixed to every banner (boot, shutdown, `zygctl status`). No other sigil variants.

## 5. Layouts

All layouts assume 80×24 minimum. Wider terminals leave whitespace; narrower terminals trigger truncation (see §7.2).

### 5.1 Boot rolling tape — during boot

Total: 18 rows. 4 header + 12 tape + 2 progress.

```
  [Z] zyginit 0.1.2 · multi-user                       elapsed  6.2s
  17 services · tier 5 · 12 done · 1 failed
  failed: network                                       exit=1
  ──────────────────────────────────────────────────────────────
   0.49  ●  swap
   1.04  ●  filesystem
   1.08  ●  identity
   1.36  ●  sysconfig
   1.55  ●  dlmgmtd
   1.78  ●  ipmgmtd
   2.10  /  network         (starting)
   ...
  ──────────────────────────────────────────────────────────────
  █████████████░░░░░░░░░░░░░░░░░░░░░░░  12/17  70%
```

Header (rows 1–3) repainted on tier transition and on any change to fail count. The `failed:` line shows up to one currently-failed service; on a second failure it becomes `failed: 2 services` and the user reads `zygctl status` for detail. If zero failures, the line shows `failed: none`.

Tape (rows 5–16) is a 12-line rolling window. New event lines append at the bottom; oldest scrolls off the top. Each line is `<elapsed-since-boot>  <glyph>  <name>  (<state-note>)` where state-note is `(starting)` / `(stopping)` for in-progress services and absent for terminal states.

Progress bar (row 18):
- 36-cell width
- Filled `█` in `c.accent`; empty `░` in `c.mute`
- Counter: `<online + failed> / <total>` — failures DO count toward the bar's "done" tally so it can reach 100%. The header tracks failure visibility separately.
- Percent: `floor(done / total * 100)`
- ASCII fallback: `[####################          ]  12/17  70%` — 30-cell width inside brackets

### 5.2 Boot final card — boot complete → login

When the last tier completes, header and tape are erased and the final card is painted:

```
  [Z] zyginit 0.1.2 · multi-user                        boot 8.3s
  17 services — 16 online, 1 failed

  failed: network                                       exit=1
      → zygctl log network    to see why
  ──────────────────────────────────────────────────────────────
  slowest:  network 5.02s  filesystem 0.55s  sshd 0.37s
```

- "boot 8.3s" replaces "elapsed 6.2s" in the header.
- Up to 3 slowest services listed on the bottom line.
- Progress bar is removed (its information is now in the second line).
- Card stays visible until `console-login`'s ttymon paints `Login:` underneath it. No timer, no clear.
- If there are zero failures, the `failed:` and hint lines are replaced with a blank line + `all services online`.

### 5.3 Shutdown mirror — during shutdown

Mirrors §5.1 with shape adjustments:

```
  [Z] zyginit 0.1.2 · stopping for reboot              elapsed 1.4s
  17 services · tier 7 · 13 down · 0 failed
  ──────────────────────────────────────────────────────────────
   0.04  ·  sshd
   0.16  ·  console-login
   0.31  ·  cron
   0.48  |  syslogd     (stopping)
   ...
  ──────────────────────────────────────────────────────────────
  ░░░░░░░░░░░░░░░░░░░░██████████████████  13/17 down  76%
```

- Header subtitle uses `<N> down` instead of `<N> done`.
- Failures during shutdown are services that returned non-zero from their stop method or hit the stop-timeout → SIGKILL escalation; still surfaced in the `failed:` line.
- Reverse spinner: stopping services render `| \ - /`.
- Progress bar fills from right to left, signaling "winding down". Filled cells are still `c.accent`; the bar's "done" count is services that have reached the `STOPPED` state (success or kill).

### 5.4 Shutdown final card

```
  [Z] zyginit 0.1.2 · halt complete                    halt 2.1s
  17 services stopped cleanly
  ──────────────────────────────────────────────────────────────
  invoking uadmin(A_SHUTDOWN, AD_BOOT)
```

- Variant for non-clean: `17 services stopped (2 force-killed)`.
- Subtitle uses `halt` / `reboot` / `poweroff` depending on reason.
- Card stays visible until the forked uadmin child causes the kernel to reset the screen.

### 5.5 `zygctl status` — banner + table

```
  [Z] zyginit 0.1.2 · multi-user · 17 services · 16 online, 1 failed
  ──────────────────────────────────────────────────────────────
  SERVICE        STATE       PID    CTID   UPTIME    NOTES
  root-fs        ● online                            (oneshot)
  crypto         ● online                            (oneshot)
  dlmgmtd        ● running   1234   42     2h15m
  network        ✕ failed                            exit=1 restarts=3
  sshd           ● running   1280   51     2h15m
```

- Banner: same shape as boot/shutdown, no "elapsed" since this is a snapshot.
- Single-service form (`zygctl status sshd`) returns only the table row — no banner.
- Plain mode: no banner, no escapes, columns space-aligned, single trailing newline per row.
- Column widths: `SERVICE` is dynamic (max name length, clamped 8–24); `STATE` is fixed 10 chars including glyph; `PID` 6, `CTID` 6, `UPTIME` 9, `NOTES` is rest-of-line.

### 5.6 `zygctl list` — unchanged shape

Stays minimal:

```
$ zygctl list
root-fs (oneshot)
crypto (oneshot)
dlmgmtd (daemon)
network (oneshot)
sshd (daemon)
...
```

Reason: `list` is used in scripts more than `status`. Adding a banner adds noise. Color is applied to the type chip only when isatty.

## 6. Render policy

Decided once at startup by `ui_init()` via `detect_rich_mode()`. Stored in module state. No mid-run mode flips.

```
detect_rich_mode() -> int:
    if env("ZYGINIT_NO_UI") set    → MODE_PLAIN
    if env("NO_COLOR") set         → MODE_PLAIN          (https://no-color.org)
    if getpid() != 1               → MODE_PLAIN          (supervisor mode)
    if !isatty(STDOUT_FILENO)      → MODE_PLAIN
    let term = env("TERM")
    if term in {"", "dumb"}        → MODE_PLAIN
    if env("ZYGINIT_ASCII") set    → MODE_RICH_ASCII     (glyphs → /-\| ASCII)
    if term contains "256color"
       or term == "sun-color"      → MODE_RICH_TRUE      (truecolor SGR)
    else                           → MODE_RICH_16        (16-color SGR baseline)
```

| Mode | Header & tape | Color | Glyphs | Spinner | Progress bar |
|------|---------------|-------|--------|---------|--------------|
| `MODE_PLAIN` | Plain lines | none | text labels | none | none |
| `MODE_RICH_ASCII` | Drawn | 16-color | ASCII (`#`, `o`/spinner, `X`, `.`) | yes | bracket form |
| `MODE_RICH_16` | Drawn | 16-color SGR | Unicode (`●`, `◉`/spinner, `✕`, `·`) | yes | block form |
| `MODE_RICH_TRUE` | Drawn | 24-bit SGR | Unicode (`●`, `◉`/spinner, `✕`, `·`) | yes | block form |

zygctl runs its own `detect_rich_mode()` against its stdout. When client detects `MODE_PLAIN`, it sends `STATUS PLAIN` / `LIST PLAIN` to the daemon; the daemon renders banner-free, escape-free output for that one response.

Plain mode line shape (machine-parseable):

```
<elapsed-s>  <level>  <event>     <name>     [k=v ...]
       6.21  info     started     crypto     dur_ms=120
       7.14  info     starting    network
       7.20  err      failed      network    exit=1 dur_ms=4023
      11.42  info     stopping    sshd
      11.55  info     stopped     sshd       dur_ms=130
```

Level mapping: `started` / `starting` / `stopping` / `stopped` → `info`; `failed` → `err`; supervisor warnings (stop-timeout, entered-maintenance) → `warn`. Used by integration tests and any external log scraper. Same content as the rich-mode tape, no escapes.

## 7. Error handling

### 7.1 Write failures

If a `write(2)` to `/dev/console` returns an error, `ui.reef` sets a sticky `g_ui_failed` flag and downgrades to `MODE_PLAIN` for the rest of the session. PID 1 must not crash because the screen got weird.

### 7.2 Terminal width

Width is assumed 80 cols. No `CSI 6n` queries (fragile on `/dev/console`). Layouts fit 80×24 with margin.

If `COLUMNS` env var is set (zygctl client, set by the user's shell), zygctl-side rendering uses it; the daemon doesn't see it. The daemon renders for 80 cols always.

Service names longer than the `SERVICE` column width are truncated with `…` (Unicode) or `..` (ASCII).

### 7.3 Concurrent writers

Per Hammerhead's tem note ("/dev/console write path is serialized") — zyginit owns the screen during boot and shutdown. Per-service stdout/stderr already redirect to per-service log files in the supervisor's child-setup; that path is unchanged. Any rogue code that writes directly to `/dev/console` would interleave — same as today.

### 7.4 Service starts faster than one tick

The row appears with `●` directly. No spinner ever shown for that service. Correct behavior; no special case needed.

### 7.5 Boot fails before `ui_init`

If `setup_pid1_console` or kernel handover fails, `ui_init` never runs. All output falls back to whatever the pre-existing `println` path does, which is the current shape — flat lines to `/dev/console`. No regression.

## 8. Data flow

### 8.1 Boot

```
main.reef:
  setup_pid1_console()
  ui_init(detect_rich_mode())
  load services + build depgraph
  ui_boot_start(num_svc, num_tiers, runlevel)
  for each tier:
      ui_tier_start(tier)
      for each service in tier:
          supervisor.start_service(...)
              → fork/exec, set state=STARTING
              → ui_event_starting(name)        // appends spinner row to tape
      while tier has services in STARTING:
          poll() returns (signal/contract/socket/timeout 200ms)
          ui_tick()                             // advance spinner, redraw spinner rows
          handle contract events:
              service became RUNNING → ui_event_started(name, dur_ms)
              service exited 0 (oneshot) → ui_event_started(name, dur_ms)
              service failed → ui_event_failed(name, exit, dur_ms)
      ui_tier_done(tier)                        // header repaint
  ui_boot_complete(stats)                       // header + tape erased, final card painted
  enter steady-state event loop
```

Steady-state: no UI updates. Console keeps the final card frozen until `console-login`'s ttymon writes underneath it.

### 8.2 Shutdown (symmetric)

```
main.reef on SIGTERM / socket shutdown command:
  ui_shutdown_start(reason)                     // "stopping for reboot" / "halt" / "poweroff"
  for each tier in REVERSE order:
      for each service in tier (reverse):
          supervisor.stop_service(...)
              → state=STOPPING, signal sent
              → ui_event_stopping(name)         // reverse-spinner row
      poll-and-tick until tier drained
      ui_tier_done(tier)
  ui_shutdown_complete(reason, elapsed_ms)      // final card
  fork child → uadmin                           // existing flow unchanged
```

### 8.3 zygctl status

```
zygctl client:
  rich = detect_rich_mode()
  if rich == MODE_PLAIN:
      send "STATUS PLAIN"
  else:
      send "STATUS"
  print response verbatim

zyginit socket handler:
  cmd_status(table, arg, plain_flag):
      if plain_flag or arg != "":
          return ui_render_status(table, want_banner=0, plain=1)
      return ui_render_status(table, want_banner=1, plain=0)
      // when plain=0, the function reads g_render_mode from ui module state
```

The server already knows its own render mode from `ui_init`; the `PLAIN` keyword from the client just forces plain output for that one response (covers the case where zygctl is on a TTY but redirecting output, or vice versa).

## 9. Testing

| Layer | Where | How |
|-------|-------|-----|
| `detect_rich_mode()` | Linux dev | Unit test with env-var / isatty fixtures across all 5 mode outcomes |
| Rendering (plain) | Linux dev | Snapshot tests: feed event sequence, assert exact string output |
| Rendering (rich) | Linux dev | Snapshot tests: feed events, assert escape sequences against golden files |
| `ui_spinner_frame(reverse)` | Linux dev | Unit: assert frame string for `g_tick` in `{0,1,2,3,4,5,6,7}` and both reverse settings |
| Boot tape rolling | Linux dev | Feed 30 events into a 12-line tape, assert visible-window content |
| Progress bar | Linux dev | Unit: assert fill width and percent for `(done, total)` pairs |
| Integration tests | Linux dev | Existing 44 tests get `ZYGINIT_NO_UI=1` to force plain mode for diff stability |
| `zygctl STATUS PLAIN` | Linux dev | Add to existing socket integration tests |
| Framebuffer visual | `hh-prototest` | Manual smoke + capture; one golden screenshot per layout (boot tape, boot final, shutdown mirror, shutdown final, zygctl status) |
| Serial console | `hh-prototest` serial | Visual confirmation that 16-color path renders identically modulo color depth |
| Sub-tick service | Linux dev | A oneshot that exits before first `ui_tick`; assert row goes straight to `●` (no spinner glyph) |
| Write failure | Linux dev | Mock write() to return error mid-boot; assert downgrade to plain and no crash |

## 10. Open boundary cases

These were considered and resolved during brainstorming; documented here so the implementation plan inherits the decisions.

1. **Tier names** — bare `tier N`. No `tier_name` TOML field. (Rejected: too much schema and config-churn for limited value.)
2. **`zygctl replace`** — does not render the rich UI. Replace is a developer/operator action, not a lifecycle one; stays plain-line per existing behavior.
3. **Runlevel transitions** (`zygctl single` / `multi`, `init s` / `init 3`) — do not render the rich UI. These are mid-session events; spawning a rich UI mid-session would interleave with whatever is already on the console. Plain lines, as today.
4. **Reload** (`zygctl reload`) — plain lines, as today.
5. **`zygctl log <name>`** — pass-through of the per-service log file. No color, no decoration.
6. **`zygctl list`** — keeps existing minimal shape. (No banner; type chip is the only color application.)
7. **Failure visibility in progress bar** — failed services count toward the bar's "done" tally so the bar can reach 100%. Failures are visible in the header's `failed:` line, not by holding the bar back.
8. **Final card timing** — no timer; final card persists until `console-login`'s ttymon paints `Login:` underneath. If `console-login` is disabled or fails to start, the card persists indefinitely. This is fine.

## 11. Phased delivery (preview for implementation plan)

The implementation plan will likely split this into ~5 phases. Sketch:

- **Phase 1.** New `src/ui.reef` skeleton; `detect_rich_mode()`; palette / glyph / sigil primitives; plain-mode pass-through.
- **Phase 2.** Boot rolling tape + header + progress bar; integrate with `main.reef` boot path.
- **Phase 3.** Spinner + `ui_tick` integration; verify forward/reverse rotation.
- **Phase 4.** Shutdown mirror + final cards (boot + shutdown).
- **Phase 5.** `zygctl status` server-side rendering + `STATUS PLAIN` wire modifier; client-side `detect_rich_mode` and dispatch.

Each phase ships independently with the existing build/deploy procedure (Linux dev for stubs + tests, then `hh-prototest` for visual smoke).
# zyginit 0.2.0 — Design

**Date:** 2026-05-14
**Status:** Approved (brainstorm complete); implementation plan to follow
**Scope:** Three integrated changes shipped as a single MAJOR release (pre-1.0 MINOR per semver): per-service directory layout, `task` service type made functionally distinct from `daemon`, and a new `[condition]` block for declarative platform/runtime gating with a SKIPPED state.

## 1. Goals

1. **Clean `/etc/zyginit/` layout** — each service self-contained in `services/<name>/` with a canonical `service.toml` plus optional `start.sh`/`stop.sh`. Eliminates the current ~50+ item flat directory mixing `<name>.toml` files with per-service helper-script directories at the same level.
2. **`task` type made functional** — currently the supervisor only branches on `daemon` vs `oneshot`; `transient` (formerly the third constant) was a no-op alias for `daemon`. 0.2.0 renames `transient` to `task` and gives it real semantics: may run for a while, exit 0 = STOPPED (counts as online), exit ≠ 0 = FAILED, never restarts.
3. **`[condition]` block** — declarative pre-flight checks. Services with failing conditions enter a new SKIPPED state instead of restart-looping or self-disabling via `zygctl disable`.
4. **Migration tool** — `scripts/migrate-layout.sh` converts an existing 0.1.x `/etc/zyginit/` in place. Idempotent, dry-run-by-default. Required on every 0.1.x → 0.2.0 upgrade since 0.2.0 has no backward-compat reader.

## 2. Non-goals

- **No backward compat with 0.1.x layout in the daemon.** Hard-cut. zyginit 0.2.0 reads only the new layout; an unmigrated tree results in an empty boot with a loud fatal log.
- **No automatic upgrade-on-first-boot migration in PID 1.** Operators run the migration tool out of band before installing the 0.2.0 binary. PID-1 risk on a failed migration would be unacceptable.
- **No `[assert]` block** (systemd's "fail-don't-skip" variant). Defer to a future release if a real use case emerges.
- **No condition kinds beyond `exists_file`, `exists_file_any`, `command`.** Defer `env_set`, `virtualization`, `zone`, `kernel_version_min`, etc. The minimal set covers acpihpd-style "platform doesn't support this" cases.
- **No new `zygctl` subcommands.** Existing surface is sufficient.
- **No `service.toml` schema field for the service name.** Name comes from the directory name. One source of truth.

## 3. Version

**0.2.0.** Service-definition filesystem layout changing is an incompatible schema break per the project's versioning policy (`docs/superpowers/specs/2026-05-05-versioning-and-release-tarball-design.md`). Pre-1.0 MINOR is permitted to break per semver convention. Tag `v0.2.0` points at the release commit.

## 4. Filesystem layout

### 4.1 New canonical layout

```
/etc/zyginit/
├── services/
│   ├── <name>/
│   │   ├── service.toml          # canonical filename inside every dir
│   │   ├── start.sh              # optional, executable
│   │   └── stop.sh               # optional, executable
│   └── ... (one directory per service)
├── enabled.d/
│   └── <name> -> ../services/<name>/   # symlink targets the directory
├── examples/
│   └── <name>/
│       └── service.toml
└── zyginit.toml                   # reserved for future; optional
```

### 4.2 Symlink target

`enabled.d/<name>` symlinks point at `../services/<name>/` — the **directory**, not the `service.toml` inside it. Rationale:
- Enables future per-service drop-in overrides (a `services/<name>/drop-ins/` directory or similar) without changing the enable mechanism.
- One symlink per service regardless of how many files live inside.
- Consistent with the "service = directory" mental model.

### 4.3 Path resolution within a service

Paths in `service.toml` that begin with `./` resolve relative to the service's directory. So `exec.start = "./start.sh"` means `services/<name>/start.sh`. Absolute paths (`/usr/sbin/dlmgmtd`) are unchanged.

## 5. `task` type

### 5.1 Name change

The constant `SERVICE_TYPE_TRANSIENT()` is renamed to `SERVICE_TYPE_TASK()` in `src/config.reef`. Integer value is preserved (the constant was unused in any on-disk state). `service_type_name(SERVICE_TYPE_TASK())` returns `"task"`. The TOML parser accepts `type = "task"` and rejects `type = "transient"` with a clear error.

### 5.2 Semantics

A `task` service:
- May run for any duration (unlike `oneshot`, which the operator expects to exit quickly).
- Is supervised normally during its run (process contract, state RUNNING, etc.).
- On clean exit (status 0): state transitions to STOPPED, `ui_event_stopped` fires, counted as **online** by the boot stats (per the existing `STATE_STOPPED + last_exit_code == 0 → online` rule from 0.1.x).
- On error exit (status ≠ 0): state transitions to FAILED, `ui_event_failed` fires, counted as **failed**.
- **Never restarts.** The `[restart]` block, if present with `on ≠ "never"`, is ignored at runtime with a config-load-time warning.
- **Skips the daemonize handshake.** Tasks don't double-fork; the entering-RUNNING-then-exiting pattern is the task's normal completion, not a daemon ready signal.

### 5.3 Supervisor branch

`src/supervisor.reef:handle_contract_event`, inserted right after the exit-code resolution and **before** the existing daemonize-handshake check:

```reef
let svc_type = config.svc_type(def)

if svc_type == config.SERVICE_TYPE_TASK()
    let dur_ms = (time.time_now() - rt.last_start_time) * 1000
    let ec = if exit_code >= 0 then exit_code else 0 end if
    rt.pid = 0 - 1
    rt.contract_id = 0 - 1
    rt.last_exit_code = ec
    table.contract_map.remove(int_to_str(contract_id))
    if ec == 0
        rt.state = STATE_STOPPED()
        table.runtimes[idx] = rt
        ui.ui_event_stopped(name, dur_ms)
    else
        rt.state = STATE_FAILED()
        table.runtimes[idx] = rt
        ui.ui_event_failed(name, ec, dur_ms)
    end if
    return
end if
```

A `task` whose contract empties without a known exit code (the `hint_exit_code == -1` path) is best-effort treated as `exit 0` — the contract emptied cleanly and we have no signal of failure.

## 6. `[condition]` block

### 6.1 Schema

Optional block in `service.toml`:

```toml
[condition]
exists_file     = "/dev/acpihp"
exists_file_any = ["/dev/acpi", "/dev/acpihp"]
command         = "/usr/sbin/acpi_probe"
```

**Semantics:**
- All specified keys must pass (AND).
- Absent keys are no-ops.
- `exists_file_any = []` (empty list) is vacuously true.
- Missing `command` binary or exec failure: condition fails with reason `"command not found"` or `"command exec failed"`.
- `command` exit 0 = pass. Any non-zero exit = fail (reason includes the exit code).
- `command` runtime is bounded by a **5-second timeout**. Exceeded: kill the probe, condition fails with reason `"command timeout"`.

### 6.2 Evaluation timing

Conditions evaluate in `supervisor.start_service`, **before** the fork:

```reef
proc start_service(table: ServiceTable, idx: int): bool
    let rt = table.runtimes[idx]
    let def = rt.def
    let name = config.svc_name(def)

    let cond_result = evaluate_conditions(def)
    if cond_result.passed == false
        rt.state = STATE_SKIPPED()
        rt.skip_reason = cond_result.reason
        table.runtimes[idx] = rt
        ui.ui_event_skipped(name, cond_result.reason)
        return true
    end if

    // existing fork + exec path
    ...
end start_service
```

Conditions re-evaluate on every call to `start_service`. So `zygctl restart acpihpd` on a host where the condition still fails produces a fresh SKIPPED. A service SKIPPED on boot N can become RUNNING on boot N+1 if its condition becomes true.

### 6.3 Implementation location

`evaluate_conditions(def: ServiceDef): ConditionResult` lives in `src/config.reef` (or a new `src/condition.reef` if it grows beyond ~80 lines). Returns `{passed: bool, reason: string}` where reason is the human-readable cause.

### 6.4 New state

`STATE_SKIPPED()` is added to the runtime state enum in `src/supervisor.reef`. New field `skip_reason: string` on `ServiceRuntime`. State display:

| State | Glyph | Color | Label |
|-------|-------|-------|-------|
| `STATE_SKIPPED()` | `·` (Unicode) / `.` (ASCII) | `c.mute` | `" skipped"` |

The glyph is the same as STATE_STOPPED's; the label differs (`" skipped"` vs `" stopped"`). The reason text appears in the tape note and `zygctl status` NOTES column.

## 7. UI integration

### 7.1 New banner counter

`g_skipped: int` global in `ui.reef`. Reset in `ui_boot_start`. Incremented by `ui_event_skipped`. Shown in the boot/shutdown header summary line:

```
23 services · tier 5 · 19 done · 0 failed · 4 skipped
```

The boot final card:

```
[Z] zyginit 0.2.0 · multi-user                       boot 8.3s
23 services — 19 online, 0 failed, 4 skipped

all services online
─────────────────────────────────────
slowest:  dlmgmtd 1.2s  filesystem 0.55s  sshd 0.37s
```

When skipped > 0 and failed > 0: existing "failed: ..." line above the divider remains; skipped count appears in the summary headline.

### 7.2 New event proc

`ui_event_skipped(name: string, reason: string)`:
- Increments `g_skipped`.
- In rich mode: pushes a tape row with glyph `paint("mute", glyph_pending())`, name, and note `str.concat("skipped: ", reason)`. Calls `emit_redraw`.
- In plain mode: emits `<elapsed-s>  info  skipped  <name>  reason=<reason>`.

### 7.3 Progress bar

Skipped services count toward the **denominator advance** so the bar reaches 100% naturally:

```
counter = g_done + g_skipped    (where g_done already includes failed)
total   = g_num_svc
```

The label appended to the bar in shutdown mode (`" down"`) stays unchanged. In boot mode the bar's right-side counter reads `<counter>/<total>  <percent>%`.

### 7.4 `zygctl status`

Skipped services render in the table:

```
acpihpd       · skipped                              no /dev/acpihp
```

State column shows the dim glyph and `" skipped"` label. NOTES column shows the reason. Banner summary line includes the skipped count.

### 7.5 `zygctl list`

Skipped state isn't a "list" property (the list is the catalog, not the runtime state). `zygctl list` output is unchanged.

## 8. Migration

### 8.1 Tool

`scripts/migrate-layout.sh` is a portable POSIX shell script delivered in the release tarball.

**Modes:**
- `--dry-run` (default): prints the planned moves to stdout and exits 0.
- `--apply`: performs the moves.
- `--config-dir <path>`: target directory (default `/etc/zyginit/`).

**Operation:**
1. Discover top-level `<name>.toml` files in `<config_dir>/`.
2. For each, plan: `mkdir -p services/<name>/`, `mv <name>.toml services/<name>/service.toml`.
3. If `<config_dir>/<name>/` exists with helper scripts, plan: `mv <name>/* services/<name>/`.
4. For each `enabled.d/<name>` symlink: plan retarget to `../services/<name>/`.
5. Idempotent: skips any service that's already migrated (already has `services/<name>/service.toml`).

**Safety:**
- If both old (`<name>.toml`) and new (`services/<name>/service.toml`) layouts exist for the same service, refuse to proceed. Operator must resolve the ambiguity manually.
- All moves are atomic (`mv` within the same filesystem).
- The script is idempotent; safe to re-run.

### 8.2 In-repo `services/hammerhead/` tree

Gets the same migration before the 0.2.0 release tarball is built. The tree currently has 17 services in the old flat layout; the migration is a one-shot reshape in a single commit during implementation.

The `examples/` directory follows the same convention: each example becomes `examples/<name>/service.toml`. The migration tool walks `examples/` too. Operators copying an example into a real config can do a flat directory copy without renaming files.

### 8.3 Hammerhead-side overrides

`base/usr/src/zyginit/overrides/` on the Hammerhead side currently uses the same flat layout. The Hammerhead team runs the migration tool against their override tree as part of consuming the 0.2.0 source drop. Release notes call this out.

### 8.4 Release-note guidance

The 0.2.0 release notes include:

1. **Run the migration tool before swapping binaries:**
   ```sh
   /path/to/migrate-layout.sh --dry-run    # preview
   /path/to/migrate-layout.sh --apply      # apply
   ```
2. **Then swap `/sbin/init` and `/sbin/zygctl`** as in prior 0.1.x deployments.
3. **Reboot.** If something goes wrong, the operator boots from a recovery medium and reverts `/sbin/init` to the prior version (the new layout is forward-compatible-only with 0.1.x binaries on the original `<name>.toml` flat tree — operator would need to reverse the migration too).

## 9. Data flow (boot)

```
main.reef on PID 1 boot:
  setup_pid1_console()
  ui_init(detect_rich_mode_for_main())
  ui_detect_winsize()
  if !exists(<config_dir>/services/)
    println FATAL: services/ directory missing — run scripts/migrate-layout.sh
    enter idle loop
  end if
  load_services_from_layout()         // walks services/*/service.toml
  scan_enabled_dir(enabled.d/)        // dereferences directory symlinks
  build_dep_graph(...)
  ui_boot_start(num_svc, num_tiers, runlevel)
  for each tier:
    ui_tier_start(tier)
    for each service in tier:
      supervisor.start_service(...)
        // condition evaluation BEFORE fork
        evaluate_conditions(def)
        if !passed:
          state = SKIPPED
          ui_event_skipped(name, reason)
          continue
        // type dispatch
        if type == TASK:
          fork+exec; on exit handle via task branch (see §5.3)
        elif type == DAEMON:
          fork+exec; daemonize handshake path as today
        elif type == ONESHOT:
          fork+exec; oneshot completion path as today
    poll-and-tick until tier settles
    ui_tier_done(tier)
  boot_settle_period (1s, from 0.1.8)
  build_boot_stats(table) → includes online/failed/skipped/slowest
  ui_boot_complete(stats)
  enter main event loop
```

## 10. Error handling

### 10.1 Layout discovery

- **`services/<name>/` exists but no `service.toml`** — warning, skip directory, continue.
- **`services/<name>/service.toml` parse error** — fatal-tier log, omit from table, continue with other services.
- **`enabled.d/<name>` symlink target doesn't exist** — warning, ignore, continue.
- **`enabled.d/<name>` symlink target escapes `services/`** (target path doesn't start with `<config_dir>/services/`) — warning, ignore, continue. Defensive: prevents arbitrary-path read via a malicious enabled.d symlink.
- **No `services/` directory at all** — fatal log, empty boot, idle loop (operator must migrate).

### 10.2 Condition evaluation

- **`exists_file` to a missing file** — condition fails, reason `"missing /path"`.
- **`exists_file_any` with all missing** — condition fails, reason `"none of [paths] exist"`.
- **`command` binary missing** — condition fails, reason `"command not found: /path"`.
- **`command` fork failure** — condition fails, reason `"command exec failed"`.
- **`command` exit non-zero** — condition fails, reason `"command exit=<code>"`.
- **`command` exceeds 5-second timeout** — kill probe, condition fails, reason `"command timeout"`.

### 10.3 Task handling

- **Task fork fails** — `ui_event_failed`, no restart attempt. Same as daemon fork failure.
- **Task contract empties with `exit_code = -1`** — treat as exit 0 (best-effort), state STOPPED.
- **Task with `[restart] on = "always"`** — config-load-time warning logged; supervisor still doesn't restart it.

### 10.4 Migration tool

- **Both old and new layouts exist for the same service** — refuse to proceed, exit non-zero with clear message.
- **Cross-filesystem `mv` would be required** — refuse, advise operator to ensure `<config_dir>` is on a single filesystem.
- **Permission denied** — exit non-zero with the offending path.

## 11. Testing

See Brainstorm Section 6 in the chat history for the full test matrix. Highlights:

- **Layout parser unit tests** — name resolution from directory, parse-error containment, skip-directory-without-toml.
- **Symlink resolution tests** — happy path, dangling, escape rejection.
- **Migration tool tests** — dry-run, apply, idempotent, ambiguity rejection.
- **`task` type tests** — exit 0 → STOPPED+online, exit 1 → FAILED, no restart loop, fork failure handling.
- **`[condition]` block tests** — each kind in pass and fail, command timeout, binary missing, multi-kind AND.
- **UI snapshot tests** — `task_ok`, `task_failed`, `boot_with_skipped`, `final_card_with_skipped`, `status_skipped`.
- **Hammerhead VM smoke test** — manual on `hh-prototest`, includes running the migration tool against the live config before swapping binaries.

## 12. Phased delivery (preview for implementation plan)

The plan will likely break into ~6-8 phases:

- **Phase 1.** Migration tool (`scripts/migrate-layout.sh`) with dry-run + apply + idempotent + ambiguity refusal. Plus Linux-dev unit tests for the tool against fixtures.
- **Phase 2.** Layout parser changes — `scan_services_dir`, symlink-target dereference, escape rejection. Update `services/hammerhead/` tree in this repo via the migration tool.
- **Phase 3.** `task` type — rename `TRANSIENT` → `TASK`, parser update, supervisor branch.
- **Phase 4.** `[condition]` block — schema, `evaluate_conditions`, `STATE_SKIPPED()`, `skip_reason`, `ui_event_skipped`, `g_skipped` counter.
- **Phase 5.** UI integration — banner counter, final-card summary, `zygctl status` skipped row, progress-bar denominator update.
- **Phase 6.** Integration tests + snapshot tests for all the new scenarios.
- **Phase 7.** Version bump 0.1.8 → 0.2.0, tag, release tarball, release-note guidance for Hammerhead consumers.

Each phase ships as one or more focused commits; all tests must pass at the end of each phase before moving to the next.
# Progress Gauge in Header Divider — Design

**Date:** 2026-06-01
**Status:** Approved (brainstorm complete); implementation plan to follow
**Scope:** Relocate the boot/shutdown progress indicator from the last row of the screen into the existing horizontal divider in the boot header. Eliminates a long-standing "progress bar disappears and reappears" artifact and reclaims two rows of vertical space for the rolling tape. Symmetric treatment for shutdown.

## 1. Problem

The boot screen rendered by `render_boot_screen()` in `src/ui.reef` ends with a horizontal divider plus a separate progress bar row, occupying the bottom two rows of the screen. The progress bar is the very last row — the file's own comment at `src/ui.reef:517-519` notes that the layout fits exactly `g_screen_rows` and cannot afford even a trailing newline.

Between zyginit's `\e[2J\e[H` redraws, any other writer to the console — a kernel `printk`, a daemon's startup stderr, a not-yet-suppressed driver message — scrolls the screen up. The last row is the first thing to scroll off. The user sees the progress bar disappear, then reappear on the next redraw, then disappear again. The artifact is endemic to placing dynamic content on the last row of a shared console.

## 2. Goals

1. **Stability under concurrent writers.** Move the progress indicator out of the most-volatile screen row.
2. **No layout twitch at boot completion.** The transition from live gauge to final boot-time label must not shift any surrounding glyph.
3. **Reclaim vertical space.** Eliminate both the bottom cosmetic divider and the bottom progress row — give those rows to the rolling tape.
4. **Symmetric shutdown.** Apply the same treatment to the shutdown screen; same bug exists there.
5. **No regressions in plain or ASCII mode.** `MODE_PLAIN` continues to emit line-by-line events; `MODE_RICH_ASCII` continues to use only ASCII characters.

## 3. Non-goals

- **No new TOML schema fields.** This is purely a console-rendering change.
- **No `zygctl` protocol changes.** `zygctl status` is rendered by `src/ui_render.reef` and is not touched.
- **No palette, sigil, or glyph changes.** The visual pass shipped in 0.1.3 stays as-is; only layout moves.
- **No animation changes.** Spinner cadence, tape-row behavior, and tier-progression logic are unchanged.
- **No supervision or dependency-graph changes.** Pure UI work.

## 4. Visual design

### 4.1 Before

```
  [Z] zyginit 0.2.2 · multi-user   elapsed 2.4s
  20 services · tier 3 · 8 done · 0 failed · 0 skipped
  failed: none
  ─────────────────────────────────────────       (top divider — cosmetic)
  [tape row 1]
  …
  [tape row N]
  ─────────────────────────────────────────       (bottom divider — cosmetic)
  ████░░░░░░░░░░░░░  8/20  31%                    (progress bar — last row, scrolls off)
```

### 4.2 After — boot in progress

```
  [Z] zyginit 0.2.2 · multi-user   elapsed 2.4s
  20 services · tier 3 · 8 done · 0 failed · 0 skipped
  failed: none
  ─────[████████░░░░░░░░░░  31%]───────────       (gauge inside divider)
  [tape row 1]
  …
  [tape row N+2]                                  (+2 rows reclaimed)
```

### 4.3 After — boot complete

The existing `ui_boot_complete()` at `src/ui.reef:708` clears the screen and paints a separate "final card" (sigil, version, `boot Xs`, online/failed/skipped summary, optional failure hint, slowest-services list). **The final card is unchanged** by this work. The new divider gauge therefore runs only during the live boot screen; the boot post-label (`[ boot 10.0s ]`) is rendered in code but is only momentarily visible — within one frame, the final card replaces the boot screen.

The `elapsed Xs` field on header line 1 is dropped from the boot screen once `g_done + g_skipped + g_failed_count == g_num_svc` (i.e., the boot has visually completed even if the final-card transition hasn't fired yet). This avoids showing a stale `elapsed 9.8s` while the gauge reads 100% in the brief window before the final card.

### 4.4 Shutdown

Symmetric with boot. `ui_shutdown_complete()` at `src/ui.reef:818` already paints a shutdown final card after the last service stops (sigil, `reboot X.Ys`, "invoking uadmin..." line). **The shutdown final card is unchanged.** The new divider gauge runs only during the live shutdown screen as services stop:

```
  ─────[█████████████░░░░░░░  65%]──────────       (during stop sequence)
```

Header content (status lines, failure summary) above the divider is unchanged.

## 5. Rendering rules

A new function `fmt_divider_gauge(done: int, total: int): string` replaces:

- The `make_divider()` call at the end of `fmt_header()` (the top divider).
- The bottom divider line in `render_boot_screen()`.
- The `fmt_progress_bar()` line in `render_boot_screen()`.

Width: identical to today's `make_divider()` — `g_line_width - 16` cells of total divider width, with the leading two-space indent preserved. The brackets plus their inner content consume 28 cells (`[` + 26 inner + `]`); the remaining `g_line_width - 44` cells are split equally between left and right padding (odd cell goes right).

**Narrow-terminal fallback.** If `g_line_width < 46` (i.e., `total_width < 30`) the bracket pair plus at least one flanking dash on each side won't fit. In that case `fmt_divider_gauge()` degrades to a label-only divider — no bracket pair, just `<lpad>── 31% ──<rpad>`. This path is exercised only on artificially small consoles; the Hammerhead framebuffer and standard serial console widths (80+) always hit the full layout.

The layout inside the divider (full path) is:

```
  <lpad>[<content>]<rpad>
```

Where:

- **Brackets** are literal `[` and `]`, painted with the `accent` color.
- **lpad / rpad** are `─` chars in `MODE_RICH`, `-` chars in `MODE_RICH_ASCII`, painted `accent`. Padding is distributed equally on both sides; if total padding is odd, the extra cell goes to the right side.
- **content** has a fixed inner width so the bracket positions never move between live and post-complete renders. Inner width = bar(20) + two-space gap + 4-char `NNN%` field = 26 cells.

### 5.1 Live content

```
<bar>  <pct>%
```

- **bar**: 20 cells. In `MODE_RICH`: `█` (accent) for filled, `░` (mute) for empty. In `MODE_RICH_ASCII`: `#` for filled, space for empty. Filled count = `(done * 20) / total`, clamped to `[0, 20]`.
- **pct**: integer `(done * 100) / total`, clamped to `[0, 100]`, right-aligned in a 3-cell field, followed by `%`. So `"  3%"`, `" 31%"`, `"100%"` all consume 4 cells.

### 5.2 Plain mode

`MODE_PLAIN` continues to emit one line per event via `fmt_plain_event()` and ignores the divider gauge entirely. `fmt_divider_gauge()` returns `""` in plain mode, same as `fmt_progress_bar()` does today.

## 6. State changes

**None.** Both `ui_boot_complete()` (`src/ui.reef:708`) and `ui_shutdown_complete()` (`src/ui.reef:818`) already exist and paint final cards that replace the live screen. The new divider gauge is purely a transform on existing counters (`g_done`, `g_skipped`, `g_num_svc`), so no new `g_*` flags are required.

Header behavior:

- `fmt_header()` drops the `elapsed Xs` field from line 1 when `g_done + g_skipped + g_failed_count >= g_num_svc` during the boot phase. This is a pure boolean check on existing counters; no new flag. During shutdown the elapsed field is left as-is (the shutdown final card carries the timer once complete).

## 7. Files touched

- **`src/ui.reef`** — only file touched.
  - New: `fmt_divider_gauge(done, total)` — main rendering function. Includes the narrow-terminal fallback in §5.
  - Update: `fmt_header()` — conditional `elapsed Xs` (suppressed when `g_done + g_skipped + g_failed_count >= g_num_svc` and phase is boot); divider line now calls `fmt_divider_gauge(g_done + g_skipped, g_num_svc)`.
  - Update: `render_boot_screen()` at `src/ui.reef:499` — remove the bottom divider concatenation and the `fmt_progress_bar()` call. Trailing-newline accounting recomputed so total height still equals `g_screen_rows`.
  - Update: tape height calculation at `src/ui.reef:682` — `g_tape_height = g_screen_rows - 6` becomes `g_screen_rows - 4` (header is 4 rows including the gauge divider; no trailing rows).
  - Update: scenario harness in `ui_demo()` — the existing `progress_rich`, `progress_ascii`, `progress_full` scenarios at lines 950-963 currently call `fmt_progress_bar()`; convert them to exercise `fmt_divider_gauge()` instead. Add fresh `gauge_full_screen_boot` scenario for full-layout visual check.
  - Delete: `fmt_progress_bar()` (lines 393-439), `PROGRESS_WIDTH()` (line 391). No remaining callers after the harness conversion.

- **`src/main.reef`** — no changes. `ui_boot_complete()` and `ui_shutdown_complete()` already wired.
- **`src/shutdown.reef`** — no changes.

- **`src/ui_render.reef`** — no changes. `zygctl status` rendering is independent.

## 8. Backwards compatibility

- **`MODE_PLAIN` output** — unchanged. Same line-per-event format consumers parse today.
- **`MODE_RICH_ASCII` output** — gauge characters (`#`, space) and divider characters (`-`) are pure ASCII. No new Unicode dependencies.
- **`MODE_RICH` output** — gauge characters (`█`, `░`) and divider character (`─`) were already in use in the prior layout. No new code points.
- **Screen-height math** — assumes `g_screen_rows >= 5` (1 header line + 3 status lines + at least 1 tape row). Already an existing assumption; no change.
- **External consumers** — none. `zygctl status`, scripted log parsers, and the syslog stream all read `MODE_PLAIN` or daemon log output, not the framebuffer rendering.

## 9. Testing

Manual on `hh-prototest` (the Hammerhead VM at 192.168.122.50):

1. **Cold boot.** Confirm gauge fills as tiers progress; confirm post-boot label `boot Xs` appears after the last tier; confirm `elapsed Xs` disappears from header line 1.
2. **Boot under noise.** Trigger a kernel printk during boot (e.g. a `dladm` configure with verbose driver). Confirm gauge stays visible across the scroll.
3. **Shutdown.** `zygctl halt`. Confirm gauge inverse-fills; confirm `down Xs` label appears before reboot.
4. **Single-user boot.** `boot -s`. Confirm gauge still renders correctly in the smaller service set.
5. **Plain mode.** `ZYGINIT_NO_UI=1`, `NO_COLOR=1`, non-TTY redirect, or running non-PID-1. Confirm line-per-event output is unchanged.
6. **ASCII mode.** `TERM=dumb` (or any non-color-capable terminal that doesn't trip the plain-mode fallback). Confirm `#` / space gauge renders inside the divider.

No unit tests added — the boot UI has historically been validated visually on the test VM. Existing integration tests in `tests/integration/` continue to cover supervisor behavior independently.

## 10. Open questions

None. All decisions made during brainstorming:

- Gauge style: block fill (`█`/`░` in rich, `#`/space in ASCII) — matches the prior progress bar's character set.
- `elapsed Xs` on header line 1: shown only while boot is in progress; dropped once all services have resolved (done/failed/skipped sum reaches total).
- Boot completion: existing `ui_boot_complete()` final card kept as-is. The new gauge runs only during the live boot screen.
- Shutdown: same treatment as boot. Existing `ui_shutdown_complete()` final card kept as-is. Gauge fills 0→100% as services stop, then the final card replaces the screen.
# Design: `conflicts` — service mutual exclusion

**Date:** 2026-06-13
**Status:** Approved (design phase)
**Roadmap item:** Near Term — Linux Development → Dependency Ordering Improvements → `conflicts`

## Summary

Add a `conflicts` field to a service's `[dependencies]` block that declares two
services as mutually exclusive — they may never run at the same time. This is the
inverse of `requires`: where `requires` pulls a dependency *in*, `conflicts` keeps
a peer *out*.

Motivating example: two services that bind the same resource (e.g. `sendmail` vs
`postfix` on port 25) should never both be running.

The feature is fully developable and testable in Linux dev mode — it touches the
config parser, the supervisor start path, and a new boot-time pre-flight pass. It
deliberately does **not** touch `depgraph.reef`: a conflict is mutual exclusion,
not an ordering relation, so it never becomes a topo-sort edge.

## Semantics (decided)

| Question | Decision |
|----------|----------|
| Runtime: `zygctl start B` while conflicting `A` is running | **Refuse.** Fail the start with a clear message; leave `A` running. No auto-eviction. |
| Boot: both conflicting services enabled and in the active set | **Start neither; mark both `FAILED`** with reason `"conflicts with <other>"`. zyginit never auto-picks a winner. |
| Declaration symmetry | **Symmetric.** Declaring `conflicts` on *either* side makes the pair mutually exclusive in both directions. |

**Cascade (accepted consequence):** a service that `requires` a conflict-failed
service will itself fail to start (existing behavior — a required dependency that
is not running blocks its dependents). For a deterministic init this is correct:
fail loudly and visibly rather than limp along on an arbitrary choice.

## Architecture

The single most important code fact: **every service start flows through
`supervisor.start_service(table, idx)` (`src/supervisor.reef:327`)** — boot tiers
(`main.reef:start_services_by_tier`), `zygctl start` (`socket.reef:cmd_start`),
runlevel transitions, and restarts all call it. That gives us one choke point for
the runtime guard, and a separate one-shot pre-flight for the boot case.

Two enforcement points, cleanly scoped and non-overlapping:

1. **Boot-time resolution** (`resolve_conflicts`) — a pre-flight pass that runs
   once, before the tier-start loop. Detects pairs where *both* members are active
   and fails both up front.
2. **Runtime guard** — inside `start_service`, refuses to start a service when a
   conflicting peer is already `RUNNING` or `STARTING`.

After the pre-flight runs at boot, no conflicting pair both survive into the active
set, so the runtime guard never trips spuriously during boot. The two mechanisms
do not overlap.

### Component 1: Schema & parsing (`src/config.reef`)

- Add to `ServiceDef`:
  - `conflicts: [string]`
  - `conflicts_count: int`
- `new_service_def()` initializes `conflicts: new [string](16)`, `conflicts_count: 0`
  (mirrors `requires`).
- Accessors `svc_conflicts(svc): [string]` and `svc_conflicts_count(svc): int`.
- Parse branch mirroring the `dependencies.requires` block at `config.reef:606`:
  ```
  if toml.toml_has_key(keys, vals, count, "dependencies.conflicts")
      let conf_str = toml.toml_get(keys, vals, count, "dependencies.conflicts")
      svc.conflicts_count = parse_toml_array(conf_str, svc.conflicts, 16)
  end if
  ```

### Component 2: The symmetric relation (helper)

A single helper expresses the mutual relation so both enforcement points agree:

```
fn conflicts_with(table, a_idx, b_idx): bool
```

Returns true if `a` lists `b` in its `conflicts`, **or** `b` lists `a` in its
`conflicts` (symmetry). Comparison is by service name. Self-reference
(`a == b`) returns false. Names that do not resolve to a known service are
ignored (warned once during the pre-flight pass, same spirit as
`depgraph`'s "requires unknown service" warning).

Lives in `supervisor.reef` (it operates over the runtime table). Both
`resolve_conflicts` and the `start_service` guard call it.

### Component 3: Boot-time resolution (`resolve_conflicts`)

- New proc in `supervisor.reef`, invoked from `main.reef` immediately before
  `start_services_by_tier`.
- For each unordered pair `(i, j)` of services that are **both in the active set**
  and `conflicts_with(table, i, j)`: set both `runtimes[i]` and `runtimes[j]` to
  `STATE_FAILED()` with a stored reason `"conflicts with <other-name>"`.
- A service already failed by an earlier pair stays failed (three-way mutual
  conflicts → all involved fail). Order-independent.
- The tier-start loop must skip services that are not in a startable state
  (already-`FAILED`/`SKIPPED` services are not forked). Verify
  `start_services_by_tier` honors this; if it currently starts unconditionally,
  add the state guard.

### Component 4: Runtime guard (inside `start_service`)

- Insert immediately after the `[condition]` evaluation block
  (`supervisor.reef:341`), before contract template setup / fork.
- Scan the table: if any service `j` with `conflicts_with(table, idx, j)` is in
  `STATE_RUNNING()` or `STATE_STARTING()`, abort:
  - do **not** change `idx`'s state (leave it `STOPPED`/`WAITING`),
  - surface `"cannot start <name>: conflicts with running service <other>"`
    (println in plain mode; returned to the socket caller for `zygctl`),
  - `return false`.
- Because all start paths funnel through `start_service`, this covers `zygctl
  start` and runlevel transitions with no extra code at those call sites.

### Component 5: Reporting

- The failure/refusal reason is stored on the service runtime and shown in
  `zygctl status`. Reuse the existing status-reason mechanism rather than adding
  a parallel field — the exact field name (`skip_reason` vs. a new
  `status_reason`) is confirmed against the `ServiceRuntime` struct during
  implementation; if `skip_reason` is `SKIPPED`-specific, a small rename to a
  generic reason field is acceptable and preferred over a second parallel field.
- Boot case: two `FAILED` rows, each reading `conflicts with <other>`.
- Runtime case: the refusal message is returned on the `zygctl start` connection;
  the target service's status is unchanged.

## Error handling & edge cases

- **Unknown conflict target** (names a service that doesn't exist): warn once
  during `resolve_conflicts`, then ignore. Never fatal.
- **Self-conflict** (`A` lists `A`): ignored.
- **Duplicate / both-sided declaration** (`A` lists `B` and `B` lists `A`):
  harmless — the symmetric helper already treats it as one relation.
- **Three+ way mutual conflict:** every service involved in any active conflicting
  pair is failed. Deterministic and order-independent.
- **Conflict with a disabled / non-active service:** no effect — only services in
  the active set participate in boot resolution, and only `RUNNING`/`STARTING`
  peers trip the runtime guard.

## Testing (Linux dev mode, `tests/integration/`)

New suite plus a parser check:

1. **Boot resolution:** enable two services that conflict; assert both end in
   `FAILED` with a `conflicts with` reason, and neither process ran.
2. **Runtime refusal:** start `A`; `zygctl start B`; assert the refusal message,
   that `B` did not start, and that `A` is still `RUNNING`.
3. **Symmetry:** declare `conflicts` on one side only; assert enforcement holds
   regardless of which service starts first.
4. **Parser** (`config.reef` test path): a TOML with
   `[dependencies] conflicts = ["x", "y"]` parses to `conflicts_count == 2` with
   the right names.

## Documentation

- `docs/DESIGN.md` — add `conflicts` to the `[dependencies]` schema reference.
- `zyginit.5` man page — document the field and its three semantics.
- `CLAUDE.md` — add `conflicts` to the `[dependencies]` block in the Service
  Definition Schema quick reference.
- `ROADMAP.md` — check the `conflicts` box under Dependency Ordering Improvements.

No new file added to `examples/` — the 4 canonical examples stay as-is; the
schema docs carry the `conflicts` reference.

## Out of scope (YAGNI)

- systemd-style auto-eviction (stopping the running peer to start a new one) —
  explicitly rejected in favor of "refuse."
- Conflicts as graph edges / ordering — a conflict is exclusion, not order.
- `conflicts` participating in dependency resolution or milestones.