# zyginit — Init System for Zygaena/Hammerhead ## Project Overview zyginit is a replacement for SMF (Service Management Facility) in the Hammerhead operating system. It is a single-process init daemon (PID 1) and service supervisor written in Reef, with an administrative CLI tool (`zygctl`). **Contact:** Chris Tusa **Repository:** Hosted on the Isurus Mercurial forge — . Clone: `hg clone https://leafscale.isurus.dev/zygaena/zyginit` (or SSH: `hg clone ssh://hg@leafscale.isurus.dev:2222/zygaena/zyginit`). VCS is Mercurial (`hg`), not git. ### Parent Projects | Project | Repo | Description | |---------|------|-------------| | **Hammerhead** | `~/repos/hammerhead` | Hammerhead-gate fork — the OS kernel and base system | | **Reef** | `~/repos/reef-lang` | Systems programming language — compiles to C/GCC native | | **Coral** | `~/repos/coral` | Package manager for Zygaena (written in Reef) | | **Zygaena** | — | The parent distribution combining all of the above | ### Key Design Documents - `docs/DESIGN.md` — Full architecture, design decisions, and phased plan - `ROADMAP.md` — Forward-looking roadmap with task checklists - `examples/` — Canonical example TOML service definitions (4 examples) ## Rules - **DO NOT modify Hammerhead, Coral, or Reef source code.** Each project is maintained separately. File bug reports or feature requests as needed. - zyginit depends on Reef 0.4.0 features. The Reef team has confirmed all 8 RFE items are implemented. See `docs/DESIGN.md` for details. ## Architecture Summary zyginit replaces three SMF daemons (svc.startd, svc.configd, master restarter) with a single Reef process. Key design choices: - **TOML service definitions** in `/etc/zyginit/*.toml` - **Symlink-based enable/disable** in `/etc/zyginit/enabled.d/` (no database) - **In-memory dependency graph** built at boot via topological sort - **Hammerhead process contracts** for service lifecycle tracking (kernel-level) - **poll(2) event loop** on: contract bundle fd, Unix domain socket fd, signal pipe fd - **Single-threaded** — no Active Objects, no threading complexity - **Unix domain socket** (`/var/run/zyginit.sock`) for zygctl communication ### Components | Component | Language | Binary | Description | |-----------|----------|--------|-------------| | `zyginit` | Reef | `/sbin/init` (kernel exec target on Hammerhead) | PID 1 init daemon | | `zygctl` | Reef | `/sbin/zygctl` | Admin CLI tool | | `sysv-wrapper` | C | `/sbin/sysv-wrapper` (+ symlinks `halt`, `reboot`, `poweroff`, `telinit`) | Translate legacy commands to `zygctl` calls | ### Runlevel + single-user model (iter 18+) Each service has a `[runlevel] mode` field (default `"multi"`): - `"always"` — runs in single-user AND multi-user (essentials: root-fs, devfs, swap, crypto, filesystem) - `"single"` — runs only in single-user (the recovery shell) - `"multi"` — runs only in multi-user (default for everything else) zyginit reads kernel boot args at startup; `-s` selects single-user. Runtime transitions via `zygctl single` / `zygctl multi` (or `init s`/`init 3`). `/sbin/sysv-wrapper` translates `halt`/`reboot`/`poweroff`/`init ` to the right `zygctl` calls. ### Foreground daemon modes (iter 11-15+) Hammerhead daemons that have an SMF_FMRI startup gate (dlmgmtd, ipmgmtd, etc.) also have a non-SMF foreground/debug flag (`-d`, `-f`, etc.) that bypasses the gate AND skips daemonize. zyginit uses these: - `dlmgmtd -d` (foreground; bypasses SMF_FMRI; door at `/etc/svc/volatile/dladm/dlmgmt_door`) - `ipmgmtd -f` (foreground; bypasses SMF_FMRI; door at `/etc/svc/volatile/ipadm/ipmgmt_door`) Pattern likely applies to other illumos daemons (sshd `-D` already uses this; syslogd `-d`, fmd `-f`). ### Shutdown sequence (iter 19-20) Two non-obvious things had to be right: 1. **uadmin must not be called from PID 1.** The kernel's `restart_init` path interferes — PID 1 dies during uadmin's resource cleanup, kernel re-execs init before `mdboot()` fires. So zyginit forks a child to call `uadmin(A_SHUTDOWN, AD_BOOT, 0)` from a non-PID-1 context. Mirrors how `svc.startd` does it. 2. **Don't `zfs set readonly=on` on the live root during shutdown.** Doing so silently drops every subsequent write — destroy_socket's unlink, sync, anything else. Boot 2 sees pre-shutdown state. ZFS is transactionally consistent; it doesn't need a "freeze". `zpool sync` + `umountall -l` + kernel's `vfs_syncall` (inside `mdboot`) is enough. The full sequence (mirrors `cmd/svc/startd/graph.c:do_uadmin`): ``` shutdown_services // reverse-tier stop, drain destroy_socket // unlink + fsync parent dir do_sync // sync(2) — async zpool sync // synchronous ZFS commit umountall -l // unmount non-root, force flush do_sync // belt + suspenders fork → child runs uadmin // non-PID-1 context; mdboot reboots cleanly parent infinite-sleep // never returns from main() while PID 1 ``` ### What zyginit Does NOT Do - No XML parsing (that's the whole point of replacing SMF) - No SQLite database - No door-based IPC (uses Unix domain sockets instead) - No multi-daemon architecture - No 32-bit support (Hammerhead is 64-bit only) ## Development Environment ### Reef Language - **Compiler:** `reef` command (compiles `.reef` → C → native via GCC) - **Repo:** `~/repos/reef-lang/` - **Version required:** Reef 0.4.0+ - **TOML parser:** `import encoding.toml` (built into stdlib, ~560 lines) - **FFI:** `extern "C"` blocks for calling C libraries (libcontract, POSIX) ### Building zyginit is built with the Reef compiler. The project is initialized via `reefc init` and follows Coral layout conventions (entry = `src/main.reef`). **Linux dev** (stubs the contract code, useful for parser/depgraph work): ```bash clang -c src/helpers.c -o build/helpers.o clang -c src/contract_linux_stubs.c -o build/contract_linux_stubs.o reefc build --obj build/helpers.o --obj build/contract_linux_stubs.o ``` **Hammerhead** (real contract bindings, real PID-1 binary): ```bash gcc -c src/helpers.c -o build/helpers.o reefc build -l contract --obj build/helpers.o ``` `helpers.o` is REQUIRED — without it the link fails on `zyginit_shutdown`, `zyginit_sync`, `zyginit_symlink`, `zyginit_getuid_by_name`, `zyginit_drop_privileges`, `zyginit_push_ldterm`, `zyginit_write_boot_utmpx`, `zyginit_write_runlvl_utmpx`, `zyginit_run_cmd`, `zyginit_fsync_path`. `zygctl` builds separately under `tools/zygctl/`: ```bash cd tools/zygctl && gcc -c src/symlink_wrapper.c -o build/symlink_wrapper.o && reefc build --obj build/symlink_wrapper.o ``` The sysv-init wrapper builds with plain `make` under `tools/sysv-wrapper/`. - Env vars: `ZYGINIT_CONFIG_DIR` (default `/etc/zyginit`), `ZYGINIT_LOG_DIR` (default `/var/log/zyginit`), `ZYGINIT_SOCKET` (default `/var/run/zyginit.sock`), `ZYGINIT_RUN_DIR` (default `/var/run/zyginit`, used for state.toml) ### Testing Testing happens on the Hammerhead test VM `hh-prototest`: | Host | IP | Purpose | |------|----|---------| | hh-prototest | 192.168.122.50 | Hammerhead OS test VM (libvirt host reservation: MAC `52:54:00:91:c9:d2`) | | hammerhead-build | 192.168.122.66 | Build VM (OI-based, for compiling Hammerhead itself) | The older standalone VM (192.168.122.17) is deprecated. **Deploy procedure** (current iter 11+ pattern): ```bash # From dev host: scp src/.reef root@192.168.122.50:/root/zyginit/src/ ssh root@192.168.122.50 'cd /root/zyginit && \ gcc -c src/helpers.c -o build/helpers.o && \ reefc build -l contract --obj build/helpers.o && \ cp build/zyginit /sbin/init.new && mv /sbin/init.new /sbin/init' # Reboot via virsh (init 6 from non-PID-1 zyginit doesn't actually # reboot — it spawns a second zyginit-as-supervisor, NOT what you want): virsh --connect=qemu:///system reset hh-prototest ``` `/sbin/init` IS the zyginit daemon binary on Hammerhead — it's the kernel's exec target at boot. There's no separate `/sbin/zyginit` binary in production deployment. To run zyginit non-PID-1 for development (without replacing init): ```bash ZYGINIT_CONFIG_DIR=/tmp/zygsvc /path/to/zyginit --supervisor-test ``` It detects non-PID-1 via `getpid() != 1` and runs in "supervisor mode" (no PID-1-specific setup, normal exit on shutdown signal). The `--supervisor-test` flag is **required** for a non-PID-1 run (bug 004): without it a bare `zyginit` on a host where zyginit is already PID 1 refuses to start, so it can't clobber the live control socket. Use a distinct `ZYGINIT_SOCKET` for dev runs too — and even with the flag, `create_socket` refuses to unlink a socket a live server is listening on. `zyginit -v` / `--version` print the version and exit (only when not PID 1; the kernel passes `-v` as a verbose boot-arg to PID 1, so that path never short-circuits). ## Source Layout ``` zyginit/ ├── CLAUDE.md # This file — project context for Claude ├── ROADMAP.md # Forward-looking roadmap with task checklists ├── docs/ │ └── DESIGN.md # Full architecture and design document ├── src/ │ ├── zyginit.reef # Main init daemon │ ├── zygctl.reef # Admin CLI tool │ ├── config.reef # TOML config parser / service definition types │ ├── depgraph.reef # Dependency graph and topological sort │ ├── contract.reef # Hammerhead process contract FFI bindings │ ├── supervisor.reef # Service lifecycle management │ └── ctlsocket.reef # Unix domain socket server for zygctl (module `ctlsocket`; renamed from `socket` to avoid a name clash with stdlib `net.socket`, which `net.unix` now imports as of Reef 0.7.6) ├── services/ │ └── examples/ # Example TOML service definitions ├── tests/ │ ├── test_config.reef # Config parsing tests │ ├── test_depgraph.reef # Dependency graph tests │ └── test_toml/ # Test TOML files └── Makefile # Build rules ``` ## Hammerhead Process Contracts — Quick Reference Process contracts are the kernel mechanism zyginit uses to track service processes. They are NOT available on Linux — this is Hammerhead-specific. ### Key APIs (C FFI) ```c // Template setup (before fork) int tmpl_fd = open("/system/contract/process/template", O_RDWR); ct_tmpl_set_informative(tmpl_fd, CT_PR_EV_EMPTY); // want "empty" events ct_tmpl_set_critical(tmpl_fd, CT_PR_EV_EMPTY | CT_PR_EV_HWERR); ct_pr_tmpl_set_fatal(tmpl_fd, CT_PR_EV_HWERR); ct_pr_tmpl_set_param(tmpl_fd, CT_PR_INHERIT | CT_PR_NOORPHAN); ct_tmpl_activate(tmpl_fd); // After fork — get the new contract ID int latest_fd = open("/system/contract/process/latest", O_RDONLY); ct_status_read(latest_fd, CTD_COMMON, &status); ctid_t contract_id = status->ctst_id; ct_tmpl_clear(tmpl_fd); // deactivate template close(tmpl_fd); // Event monitoring (poll on this fd) int bundle_fd = open("/system/contract/process/bundle", O_RDONLY); // Kill all processes in a contract sigsend(P_CTID, contract_id, SIGTERM); ``` ### Key Header Files ```c #include // ct_tmpl_*, ct_pr_tmpl_*, ct_status_* #include // CT_PR_*, CTD_* #include // CT_PR_EV_*, CT_PR_INHERIT, etc. #include // CTFS paths ``` ### Contract Lifecycle 1. Open template → configure parameters → activate 2. fork() — child inherits active template, new contract created 3. exec() the service binary in the child 4. Parent reads contract ID from `/system/contract/process/latest` 5. poll() on bundle fd → receive events (empty, core, hwerr, signal) 6. On service exit: contract fires "empty" event → zyginit decides restart policy 7. On zyginit restart: re-adopt existing contracts (they survive init restart) ## Service Definition Schema See `docs/DESIGN.md` for full details. Quick reference: ```toml [service] name = "string" # required, unique description = "string" # human-readable type = "daemon|oneshot|transient" # required [exec] start = "command" # required — command or path to script stop = "command" # optional — defaults to contract signal working_directory = "/path" # optional — chdir before exec user = "username" # optional — drop privileges to this user group = "groupname" # optional — override primary group environment = ["K=V"] # optional — set env vars before exec [stop] method = "contract|exec" # default: contract signal = "TERM" # default: TERM timeout = 60 # seconds before KILL [dependencies] requires = ["svc1", "svc2"] # must be running after = ["svc3"] # ordering only (not hard dependency) conflicts = ["svc4"] # mutually exclusive — both fail if both enabled; start refused if peer running [contract] param = ["inherit", "noorphan"] fatal = ["empty", "hwerr"] [restart] on = "always|failure|never" # default: failure delay = 5 # seconds between restarts max_retries = 3 # -1 for unlimited ``` ## Hammerhead Kernel → Init Handover (PID 1) The kernel-to-init handoff has subtleties that materially affect zyginit when it runs as PID 1 (vs. the integration-test path running as a child of a shell). Key facts pulled from `base/usr/src/uts/common/os/main.c` (`exec_init`, `start_init_common`, `start_init`) and `os/exit.c` (`restart_init`, `zone_init_exit`): 1. **Init path is per-zone, default `/sbin/init`.** Kernel uses `p->p_zone->zone_initname` and `p->p_zone->zone_bootargs`. Global zone defaults to `/sbin/init` and the bootargs from `/boot/loader.conf` `boot-args=` (e.g., `-v -m verbose`). Per-zone init paths can be set via `zone_setattr(2)`. 2. **`p->p_model = DATAMODEL_ILP32` is just initial state.** `exec_common` re-sets it based on the actual ELF type — a 64-bit init works fine. 3. **There is NO SMF-specific kernel handoff.** Kernel exec's whatever binary `zone_initname` points at. All SMF wiring (`-m smf-options`, starting `svc.startd`, `/etc/inittab`) is in userland `init.c`. 4. **fd 0/1/2 are NOT pre-opened by the kernel.** The canonical `/sbin/init` opens `/dev/syscon` (or `/dev/console`) every time it wants to print (see `console()` in `cmd/init/init.c`). zyginit's `setup_pid1_console()` does this once at startup, dup2()ing onto 0/1/2. 5. **If init exits, kernel re-execs it.** `restart_init()` in `exit.c` runs in a tight loop and logs `"init(8) exited with status 0: restarting automatically"`. zyginit's PID-1 path MUST NOT return from `main()` — it ends in `sync()` + `uadmin(A_SHUTDOWN, ...)` or an infinite sleep loop. The combination of #4 and the no-output-on-failure mode meant our first PID-1 boot attempt failed silently (filling the console with the kernel's restart-warning). Diagnosed by tracing main.c → exit.c, then comparing zyginit's startup against `cmd/init/init.c`. The fd-setup fix is at `src/main.reef:setup_pid1_console`. ## Zones Support Hammerhead inherits illumos zone (container) support. Each zone has its own `zone_initname` and PID 1, and zones are isolated by the kernel — the global zone init (zyginit) does NOT manage non-global zone init processes; the kernel does, per zone. What zyginit needs to provide for zones (future milestone, NOT first boot): - A `zones` oneshot service that walks `/etc/zones/index` at boot to invoke `zoneadm boot` for each `autoboot=true` zone, mirroring the SMF `system/zones:default` method (`/lib/svc/method/svc-zones`). - A matching stop in reverse-tier shutdown that halts zones before filesystem teardown (zones may have ZFS datasets). - Do NOT supervise `zoneadmd` directly — it's spawned per-zone by `zoneadm boot`, not as a singleton daemon. Currently disabled on hh-prototest (`svc:/system/zones:default disabled`), so first PID-1 boot is unaffected. Add the `zones` service alongside SMF removal in Phase C. ## Status (as of rev 122 / zyginit 0.1.2, 2026-05-12) All 7 implementation phases are complete and PID-1 boot on Hammerhead is working end-to-end on `hh-prototest`. **Phase D (SMF removal from Hammerhead) is complete as of 2026-05-12 on the `hh-alpha5` greenfield build** — Hammerhead now boots and runs entirely under zyginit, with no SMF master daemons anywhere on disk. See `ROADMAP.md` for the full status; the abbreviated state: - **Multi-user boot** works (vioif0 configured, sshd reachable at 192.168.122.50, all daemons running) - **Single-user mode** works (boot arg `-s` OR runtime `zygctl single` / `init s` — drops to /sbin/sh on /dev/console) - **Runtime runlevel transitions** (single ↔ multi) work via `zygctl` - **Clean shutdown** (`zygctl halt` / `reboot` / `poweroff`, or via `init`/`halt`/`reboot`/`poweroff` legacy commands through `/sbin/sysv-wrapper`) - **utmpx integration** — `who -b`, `who -r`, `uptime` report correctly - **SMF gone from Hammerhead** — `svc.startd`, `svc.configd`, `svcadm`, `svccfg`, `svcs`, `svcprop`, `cmd/svc/`, manifests, `repository.db` all deleted from source and packaging. `libscf.so.1` deliberately retained as a 54KB ABI stub (returns `SCF_ERROR_NO_SERVER`) so the ~100 base consumers that still `-lscf` degrade gracefully without a mass refactor. `libxml2` stays — it has non-SMF consumers (iSNS, FMA fabric-xlate, KMF crypto, sharemgr, zonestatd) zyginit ships the engine (init binary + zygctl + sysv-wrapper) plus minimal canonical examples in `examples/`. Distributions (Hammerhead and any future distro) own all service-definition policy in their own overlay trees. Layout/schema changes in zyginit can ship independently of distro service-set churn. The full Hammerhead service set lives in `base/usr/src/zyginit/overrides/` on the Hammerhead side. The remaining SMF surface (NFS, ZFS automount, ipfilter, ipsec, IPMP, fmd, inetd, rpcbind, etc.) was converted as Hammerhead-side override TOMLs alongside Phase D — that ROADMAP item is done. ## Important Notes - `/var/run` is NOT currently tmpfs in our zyginit-only BE — files persist across reboots. Standard illumos has it as tmpfs (mounted via fs-minimal). Adding it back is a follow-up; not strictly needed for current functionality but would prevent stale lock files from accumulating. - Hammerhead is 64-bit only — no 32-bit considerations. - Hammerhead uses flat library paths (`/usr/lib/` not `/usr/lib/amd64/`). - The kernel exec target IS `/sbin/init` — zyginit binary is deployed directly there. There's no separate `/sbin/zyginit` in production. - GC signals: use SIGRTMIN+0 and SIGRTMIN+1 (no collision with standard signals). Reef's `--no-gc` flag can disable GC entirely if needed for PID 1 reliability (not currently used; default GC works). - poll(2) is preferred over event ports for simplicity (only 3-4 fds to monitor). - The contract bundle fd MUST be opened with O_NONBLOCK (per iter 17), otherwise `ct_event_read` blocks the main loop after draining events. ## Hammerhead Boot Chain (rev 82) Currently enabled on `hh-prototest` (`/etc/zyginit/enabled.d/`): | Tier | Services | Notes | |------|----------|-------| | 0 | root-fs | Asserts ZFS root readonly=off | | 1 | crypto, devfs, swap | `devfs` runs `soconfig` + `devfsadm` | | 2 | filesystem | mountall + zpool import + zfs mount | | 3 | identity, sysconfig, dlmgmtd | `dlmgmtd -d` foreground mode | | 4 | ipmgmtd | `ipmgmtd -f` foreground mode | | 5 | network | `dladm init-phys` + ifconfig from /etc/hostname.* | | 6 | utmpd, pfexecd, syseventd, syslogd | daemon tier | | 7 | cron, console-login, sshd | login chain + scheduling; single-user-shell here too | The remaining ~50 SMF services on standard Hammerhead (NFS, ZFS auto- mount infrastructure, ipfilter, ipsec, fmd, inetd, rpcbind, etc.) are not yet converted. For most, the foreground-mode-of-the-daemon pattern likely applies — see `ROADMAP.md` "SMF Service Manifest Conversion".