# zyginit Roadmap **Last updated:** 2026-05-12 For architecture and design details, see `docs/DESIGN.md`. For the original implementation plan, see `docs/IMPLEMENTATION_PLAN.md`. For the PID-1 boot story (iter 1-20), see the per-iter notes under `docs/superpowers/plans/` and the auto-memory directory. --- ## 0.2.2 — Engine/policy split (2026-05-21) zyginit no longer ships the in-repo `services/hammerhead/` tree. Service definitions are entirely distro-owned (Hammerhead's overlay tree at `base/usr/src/zyginit/overrides/`). zyginit ships the engine, four minimal canonical examples (`examples/`), and the schema docs. This split removes the per-release coordination dance — layout changes (like 0.2.0's per-service-directory refactor) no longer require synchronized migration of two trees. --- ## Completed All 7 implementation phases are done and working on Linux (dev mode). Integration test suite passes (57 tests across 9 suites). ### Core Implementation (Phases 1-7) - [x] **TOML config parser** — parse service definitions, validate schema, scan `enabled.d/`, inline array parsing - [x] **Dependency graph** — topological sort (Kahn's algorithm), cycle detection, tiered boot order - [x] **Process contract FFI** — `extern "C"` bindings to libcontract, `[pointer]` array pattern for `void**` params, Linux stub file - [x] **Service supervisor** — fork/exec, state machine (8 states), restart policies, stop timeouts, SIGKILL escalation - [x] **Event loop** — poll(2) on signal self-pipe, contract bundle fd, Unix domain socket; 1-second periodic tick - [x] **Unix socket server** — newline-delimited text protocol, command dispatch (status, start, stop, restart, enable, disable, reload, log, list, version, help) - [x] **zygctl CLI** — local commands (help, version, enable, disable), socket commands (everything else) ### Post-Phase Enhancements - [x] **Enable/disable** — symlink creation/removal in `enabled.d/`, works from both zygctl (local) and socket (server-side) - [x] **SIGHUP config reload** — re-scan `enabled.d/`, add new services, disable removed services; triggered by signal or `zygctl reload` - [x] **Enhanced status output** — uptime, exit code, restart count, human-readable durations - [x] **Integration test harness** — 57 tests across 9 suites covering CLI, daemon lifecycle, restart policies, reload, oneshot services, working directory, environment variables, privilege separation, service output capture - [x] **Documentation** — man pages (zyginit.8, zygctl.8, zyginit.5), administration guide, README, 19 example service definitions - [x] **Privilege separation** — `exec.user`/`exec.group` fields, child drops privileges via `setgid()` → `initgroups()` → `setuid()` - [x] **Environment variables** — `exec.environment` array of `KEY=value` strings, set in child before exec via `set_env()` - [x] **Working directory** — `exec.working_directory` field, child calls `chdir()` before exec - [x] **Service output capture** — per-service log files in `/var/run/zyginit/log/.log`, truncated on each start, accessible via `zygctl log ` ### PID-1 Boot on Hammerhead (iter 1-20, complete 2026-05-02) zyginit boots Hammerhead as PID 1 with full multi-user functionality and clean shutdown / reboot. Reached on `hh-prototest` (192.168.122.50). - [x] **PID-1 fd setup** — `setup_pid1_console()` opens `/dev/console` with `O_NOCTTY` and pushes `ldterm`+`ttcompat` STREAMS modules so userspace println output aligns with kernel writes - [x] **soconfig at boot** — `devfs/start.sh` runs `soconfig -d /etc/sock2path.d` to populate kernel sockfs's protocol map. Without it `socket(AF_INET)` returns EAFNOSUPPORT - [x] **Foreground daemon modes** — dlmgmtd `-d`, ipmgmtd `-f` — bypass each daemon's SMF_FMRI startup gate AND avoid the double-fork daemonize, simplifying contract supervision. Pattern likely applies to other illumos daemons - [x] **`dladm init-phys`** — registers physical NICs with dlmgmtd at network startup so `ifconfig plumb` resolves names - [x] **Network bring-up** — BSD-style `/etc/hostname.` parser (no ipmgmtd-CLI use; ipmgmtd-DAEMON still needed for libipadm) - [x] **Console login chain** — small C helper writes `LOGIN_PROCESS` utmpx entry before `exec ttymon` so `login(1)` doesn't bail with "No utmpx entry" - [x] **Daemonize-caveat fix** — supervisor recognizes the parent- exits-0 daemonize handshake and keeps services in RUNNING state instead of mis-marking them STOPPED - [x] **Catch-all zombie reap** — `process_wait_any_nohang()` loop after per-service reap iteration - [x] **Contract bundle non-blocking** — `O_NONBLOCK` on `/system/contract/process/bundle` so `ct_event_read` returns EAGAIN instead of wedging the main loop - [x] **Default child PATH** — supervisor sets `PATH=/usr/sbin:/usr/bin:/sbin` for service children (zyginit inherits a sparse env from the kernel) - [x] **utmpx integration** — `BOOT_TIME` and `RUN_LVL` records on boot and on every runlevel transition, format matching `svc.startd`'s `utmpx_set_runlevel()` so `who -b` and `who -r` report correctly - [x] **Per-service `[runlevel] mode`** — `"always"` | `"single"` | `"multi"` (default `"multi"`). Drives initial active-set filtering and runtime transition stop/start sets - [x] **Single-user mode** — kernel boot arg `-s` parsed in `main.reef`; `/sbin/sh` recovery shell on `/dev/console` via a regular zyginit service with `mode = "single"` - [x] **Runtime runlevel transitions** — `transition_runlevel()` stops services not in the target's active set in reverse tier order, then starts services newly in the active set - [x] **`zygctl single` / `zygctl multi`** — socket protocol handlers; legacy `multiuser` alias kept - [x] **sysv-init wrapper binary** — `tools/sysv-wrapper/wrapper.c` installed as `/sbin/halt`, `/sbin/reboot`, `/sbin/poweroff`, `/sbin/telinit` (argv[0] dispatch translates legacy commands to `zygctl` invocations). `/sbin/init` itself stays the zyginit daemon binary (kernel exec target) - [x] **Clean shutdown sequence** — `shutdown_services` (reverse tier) → `destroy_socket` (with parent-dir fsync) → `do_sync` → explicit `zpool sync` + `umountall -l` → fork child → child calls `uadmin(A_SHUTDOWN, AD_BOOT)` from non-PID-1 context. Mirrors `cmd/svc/startd/graph.c:do_uadmin`. Without the fork, kernel's `restart_init` interferes with `mdboot`'s hardware reset - [x] **No more readonly-on-shutdown** — `root-fs/stop.sh` no longer sets `zfs set readonly=on` mid-shutdown. That was silently dropping every subsequent write (destroy_socket unlink, sync, anything else) — a real data-integrity bug, not just a missing socket file. ZFS is transactionally consistent and doesn't need a "freeze" before reboot --- ## Near Term — Linux Development Items that can be developed and tested on any system without Hammerhead. ### Dependency Ordering Improvements - [ ] **`wants` (soft dependencies)** — like `requires` but don't fail if the dependency is not enabled. Start it if available, proceed without it if not. - [x] **`conflicts`** — prevent two services from running simultaneously (e.g., sendmail vs postfix). Symmetric; both fail if both enabled at boot, runtime start refused if a conflicting peer is running. - [ ] **Milestones** — virtual grouping targets (`multi-user`, `network`, `single-user`) that aggregate sets of services. A milestone is "reached" when all its member services are online. ### Improved Reload (V2) The current reload (V1) only detects added/removed services. - [ ] Detect in-place TOML changes by comparing config checksums or modification times - [ ] Support restarting services whose definitions changed - [ ] Log what changed on each reload for auditability ### Test Coverage - [ ] Unit tests for `config.reef` (TOML parsing edge cases) - [ ] Unit tests for `depgraph.reef` (cycle detection, missing deps) - [ ] Extend integration tests with dependency ordering verification - [ ] Test concurrent zygctl connections --- ## Medium Term — Requires Hammerhead VM These items need the Hammerhead test VM (`hh-prototest` at 192.168.122.50). The older standalone VM (192.168.122.17) is deprecated. ### Contract Validation on Hammerhead — DONE (rev 18, 2026-04) The process contract code is exercised end-to-end on `hh-prototest`. 9/9 probes in `utils/contract_probe` pass. Found and fixed P_CTID = 14 → 13 and ctl fd O_RDWR → O_WRONLY along the way. - [x] Build with `reefc build -l contract` on Hammerhead - [x] Verify `setup_template()` creates contracts successfully - [x] Verify `get_latest_contract()` returns valid contract IDs - [x] Verify `read_event()` receives `CT_PR_EV_EMPTY` on process exit - [x] Verify `kill_contract()` signals all processes in a contract - [x] Verify `abandon_contract()` works correctly - [ ] Contract re-adoption after zyginit restart (still pending — see "Contract Re-adoption" below) ### PID 1 Boot Testing — DONE (rev 82, 2026-05-02) zyginit runs as the actual init process on `hh-prototest`. Cold boot to multi-user works; `zygctl` operates correctly; clean shutdown via `zygctl halt`/`reboot`/`poweroff` and via `init 0`/`6`/`s`/`3`. - [x] Hammerhead boot-chain TOML definitions: root-fs → crypto → devfs → swap → filesystem → identity → sysconfig → dlmgmtd → ipmgmtd → network → utmpd/pfexecd/syseventd/syslogd → cron/console-login/sshd → single-user-shell - [x] `/sbin/init` deployed to `hh-prototest` as the zyginit binary - [x] Cold boot to multi-user with SSH accessible - [x] `zygctl` operates correctly during live boot (status, start, stop, enable, disable, reload, single, multi, halt, reboot, poweroff) - [x] Graceful shutdown — `zygctl halt` / `init 0` / `init 6` etc. mirror SMF's `cmd/svc/startd/graph.c:do_uadmin` sequence - [x] Boot chain documented in `services/hammerhead/` (TOML files + start/stop scripts) and in the iter 11-20 memory notes ### Contract Re-adoption When zyginit restarts (e.g., after a crash or upgrade), existing process contracts survive in the kernel. zyginit must re-adopt them to resume supervision of running services. - [ ] On startup, scan `/system/contract/process/` for existing contracts - [ ] Match contracts to enabled service definitions - [ ] Adopt orphaned contracts and resume monitoring - [ ] Handle the case where a service was removed while zyginit was down ### GC Tuning for PID 1 Reef's garbage collector uses signals for stop-the-world coordination. As PID 1, zyginit must not pause during critical signal handling. - [ ] Benchmark with `--no-gc` flag vs default GC - [ ] Test with `--gc-signals rtmin` to avoid signal conflicts - [ ] Profile memory usage over extended uptime (days) - [ ] Determine if `--no-gc` is acceptable (bounded working set) ### SMF Service Manifest Conversion — DONE (2026-05-12, hh-alpha5) The Hammerhead boot chain in this repo (`services/hammerhead/`) covers the core multi-user set: root-fs, crypto, devfs, swap, filesystem, identity, sysconfig, dlmgmtd, ipmgmtd, network, utmpd, pfexecd, syseventd, syslogd, cron, console-login, sshd, single-user-shell. The rest of the SMF surface (NFS, ZFS automount, ipfilter, ipsec, IPMP, fmd, inetd, rpcbind, etc.) was converted as Hammerhead-side **override TOMLs** under `base/usr/src/zyginit/overrides/` alongside the Phase D SMF removal. The foreground-mode-of-the-daemon pattern (dlmgmtd `-d`, ipmgmtd `-f`) applied to most. - [x] Core boot chain converted (~17 services in this repo) - [x] NFS server / client services (Hammerhead overrides) - [x] ZFS automount + share daemons (Hammerhead overrides) - [x] inetd / rpcbind (Hammerhead overrides) - [x] fmd (Fault Management Daemon) (Hammerhead overrides) - [x] ipfilter, ipsec, IPMP (Hammerhead overrides) - [ ] Phase B for single-user — swap `start.sh` from `/sbin/sh` to `/sbin/sulogin` so single-user requires the root password --- ## Long Term Larger features for after zyginit is stable and booting Hammerhead. ### Resource Controls Integrate with Hammerhead resource controls (projects, tasks, rctls) to enforce memory limits, CPU caps, and file descriptor limits per service. ```toml [limits] max_fds = 1024 max_memory = "512M" project = "service.sshd" ``` - [ ] Research Hammerhead `rctlblk` APIs and `setprojrg()` / `settaskid()` - [ ] Add `[limits]` section to service schema - [ ] Apply resource controls in child before exec - [ ] Display resource usage in `zygctl status` ### Zone-Aware Service Management Support running zyginit inside Hammerhead zones (non-global zones). Some services are zone-specific; others are global-only. ```toml [service] name = "zoneadmd" zone = "global" # only run in the global zone ``` - [ ] Detect whether running in global or non-global zone - [ ] Filter services based on `zone` field - [ ] Handle zone boot/halt events ### Parallel Service Startup Currently services within a tier start sequentially. True parallel startup would improve boot time. - [ ] Fork all services in a tier simultaneously - [ ] Track readiness per service (PID exists, contract active) - [ ] Only advance to next tier when all services in current tier have started (or failed and are not required) - [ ] Measure boot time improvement ### Service Notifications / Hooks Allow services to declare hooks that fire on state transitions. Useful for health checks, monitoring integration, and dependent service coordination. ```toml [hooks] on_start = "/etc/zyginit/sshd/on_start.sh" on_stop = "/etc/zyginit/sshd/on_stop.sh" on_failure = "/etc/zyginit/sshd/on_failure.sh" ``` - [ ] Add `[hooks]` section to service schema - [ ] Execute hooks asynchronously (don't block the event loop) - [ ] Pass service name, state, exit code as environment variables ### Diagnostic State Dump Write runtime state to `/var/run/zyginit/` for crash recovery analysis and monitoring tool consumption. - [ ] Write JSON or TOML state file on each state change - [ ] Include: service name, state, pid, contract ID, uptime, restarts - [ ] `zygctl dump` command to write state snapshot on demand - [ ] Document format for monitoring integrations ### SMF Removal from Hammerhead — DONE (Phase D, 2026-05-12, hh-alpha5) The Hammerhead team completed SMF removal on `main`, verified on the `hh-alpha5` greenfield build. Hammerhead now boots and runs entirely under zyginit, with no SMF master daemons anywhere on disk. The zyginit-side work for this milestone was zero — the vendored zyginit 0.1.2 already had everything needed (PID 1, Unix socket protocol, override layering, self-disable). - [x] Extended zyginit-only boot verification on `hh-prototest` - [x] Remove `svc.startd`, `svc.configd`, master restarter from build (also `svcadm`, `svccfg`, `svcs`, `svcprop`, `cmd/svc/`, `tools/svc/`, mdb svc modules, SMF man pages) - [x] Remove SMF manifest infrastructure from packaging (`/var/svc/`, `repository.db`, manifest packaging — all gone) - [x] Verify daemons we keep (dlmgmtd, ipmgmtd, etc.) still build without their SMF method scripts - [x] Update Hammerhead documentation - [x] Coordinate with Coral for package dependency updates - [~] `libscf` — **intentionally retained** as a permanent 54KB ABI stub (`base/usr/src/lib/libscf/common/libscf-stub.c`, ~1828 LOC). ~100 base consumers still `-lscf`; the stub returns `SCF_ERROR_NO_SERVER` so they degrade gracefully without a mass refactor. This is a deliberate Phase D design decision, not an oversight. - [~] `libxml2` — **stays.** SMF was never the only consumer. Remaining base consumers: `cmd/isns/isnsd` + `isnsadm`, `cmd/fm/modules/common/fabric-xlate`, `cmd/cmd-crypto/kmfcfg` + `pktool`, `cmd/dfs.cmds/sharemgr`, `cmd/zonestat/zonestatd`. Hammerhead-side overrides for this push (packaging overlays, not zyginit changes) live in `base/usr/src/zyginit/overrides/`. ~9 commits added Hammerhead-specific TOML definitions: `ipsec`, `audit`, `resource-mgmt`, `rcapd`, `syslogd`, `console-login`, `acpihpd`, plus a 24-service batch. --- ## Non-Goals Things zyginit intentionally does not do: - **Socket activation** — adds complexity; services should manage their own listeners - **Cgroup/namespace isolation** — not applicable to Hammerhead (use zones) - **Journal/binary logging** — use syslog; zyginit is not a log manager - **Timer units** — use cron; zyginit manages services, not schedules - **User-level service management** — zyginit is PID 1 only; user services can use a separate supervisor if needed - **D-Bus integration** — Hammerhead does not use D-Bus - **Automatic dependency resolution** — services must explicitly declare dependencies; zyginit does not infer them