zyginit — Init System for Zygaena/Hammerhead

Project Overview

zyginit is a replacement for SMF (Service Management Facility) in the Hammerhead
operating system. It is a single-process init daemon (PID 1) and service supervisor
written in Reef, with an administrative CLI tool (zygctl).

Contact: Chris Tusa chris.tusa@leafscale.com

Repository: Hosted on the Isurus Mercurial forge —
https://leafscale.isurus.dev/zygaena/zyginit.
Clone: hg clone https://leafscale.isurus.dev/zygaena/zyginit
(or SSH: hg clone ssh://hg@leafscale.isurus.dev:2222/zygaena/zyginit).
VCS is Mercurial (hg), not git.

Parent Projects

Project	Repo	Description
Hammerhead	~/repos/hammerhead	Hammerhead-gate fork — the OS kernel and base system
Reef	~/repos/reef-lang	Systems programming language — compiles to C/GCC native
Coral	~/repos/coral	Package manager for Zygaena (written in Reef)
Zygaena	—	The parent distribution combining all of the above

Key Design Documents

• docs/DESIGN.md — Full architecture, design decisions, and phased plan
• ROADMAP.md — Forward-looking roadmap with task checklists
• examples/ — Canonical example TOML service definitions (4 examples)

Rules

• DO NOT modify Hammerhead, Coral, or Reef source code. Each project is maintained
  separately. File bug reports or feature requests as needed.
• zyginit depends on Reef 0.4.0 features. The Reef team has confirmed all 8 RFE items
  are implemented. See docs/DESIGN.md for details.

Architecture Summary

zyginit replaces three SMF daemons (svc.startd, svc.configd, master restarter) with
a single Reef process. Key design choices:

• TOML service definitions in /etc/zyginit/*.toml
• Symlink-based enable/disable in /etc/zyginit/enabled.d/ (no database)
• In-memory dependency graph built at boot via topological sort
• Hammerhead process contracts for service lifecycle tracking (kernel-level)
• poll(2) event loop on: contract bundle fd, Unix domain socket fd, signal pipe fd
• Single-threaded — no Active Objects, no threading complexity
• Unix domain socket (/var/run/zyginit.sock) for zygctl communication

Components

Component	Language	Binary	Description
zyginit	Reef	/sbin/init (kernel exec target on Hammerhead)	PID 1 init daemon
zygctl	Reef	/sbin/zygctl	Admin CLI tool
sysv-wrapper	C	/sbin/sysv-wrapper (+ symlinks halt, reboot, poweroff, telinit)	Translate legacy commands to zygctl calls

Runlevel + single-user model (iter 18+)

Each service has a [runlevel] mode field (default "multi"):

• "always" — runs in single-user AND multi-user (essentials: root-fs, devfs, swap, crypto, filesystem)
• "single" — runs only in single-user (the recovery shell)
• "multi" — runs only in multi-user (default for everything else)

zyginit reads kernel boot args at startup; -s selects single-user.
Runtime transitions via zygctl single / zygctl multi (or init s/init 3).
/sbin/sysv-wrapper translates halt/reboot/poweroff/init <N> to the right zygctl calls.

Foreground daemon modes (iter 11-15+)

Hammerhead daemons that have an SMF_FMRI startup gate (dlmgmtd, ipmgmtd, etc.)
also have a non-SMF foreground/debug flag (-d, -f, etc.) that bypasses
the gate AND skips daemonize. zyginit uses these:

• dlmgmtd -d (foreground; bypasses SMF_FMRI; door at /etc/svc/volatile/dladm/dlmgmt_door)
• ipmgmtd -f (foreground; bypasses SMF_FMRI; door at /etc/svc/volatile/ipadm/ipmgmt_door)

Pattern likely applies to other illumos daemons (sshd -D already uses this; syslogd -d, fmd -f).

Shutdown sequence (iter 19-20)

Two non-obvious things had to be right:

1. uadmin must not be called from PID 1. The kernel's restart_init
   path interferes — PID 1 dies during uadmin's resource cleanup, kernel
   re-execs init before mdboot() fires. So zyginit forks a child to
   call uadmin(A_SHUTDOWN, AD_BOOT, 0) from a non-PID-1 context. Mirrors
   how svc.startd does it.

2. Don't zfs set readonly=on on the live root during shutdown. Doing
   so silently drops every subsequent write — destroy_socket's unlink,
   sync, anything else. Boot 2 sees pre-shutdown state. ZFS is
   transactionally consistent; it doesn't need a "freeze". zpool sync

   • umountall -l + kernel's vfs_syncall (inside mdboot) is enough.

The full sequence (mirrors cmd/svc/startd/graph.c:do_uadmin):

shutdown_services           // reverse-tier stop, drain
destroy_socket              // unlink + fsync parent dir
do_sync                     // sync(2) — async
zpool sync                  // synchronous ZFS commit
umountall -l                // unmount non-root, force flush
do_sync                     // belt + suspenders
fork → child runs uadmin    // non-PID-1 context; mdboot reboots cleanly
parent infinite-sleep       // never returns from main() while PID 1

What zyginit Does NOT Do

• No XML parsing (that's the whole point of replacing SMF)
• No SQLite database
• No door-based IPC (uses Unix domain sockets instead)
• No multi-daemon architecture
• No 32-bit support (Hammerhead is 64-bit only)

Development Environment

Reef Language

• Compiler: reef command (compiles .reef → C → native via GCC)
• Repo: ~/repos/reef-lang/
• Version required: Reef 0.4.0+
• TOML parser: import encoding.toml (built into stdlib, ~560 lines)
• FFI: extern "C" blocks for calling C libraries (libcontract, POSIX)

Building

zyginit is built with the Reef compiler. The project is initialized via
reefc init and follows Coral layout conventions (entry = src/main.reef).

Linux dev (stubs the contract code, useful for parser/depgraph work):

clang -c src/helpers.c -o build/helpers.o
clang -c src/contract_linux_stubs.c -o build/contract_linux_stubs.o
reefc build --obj build/helpers.o --obj build/contract_linux_stubs.o

Hammerhead (real contract bindings, real PID-1 binary):

gcc -c src/helpers.c -o build/helpers.o
reefc build -l contract --obj build/helpers.o

helpers.o is REQUIRED — without it the link fails on
zyginit_shutdown, zyginit_sync, zyginit_symlink,
zyginit_getuid_by_name, zyginit_drop_privileges,
zyginit_push_ldterm, zyginit_write_boot_utmpx,
zyginit_write_runlvl_utmpx, zyginit_run_cmd, zyginit_fsync_path.

zygctl builds separately under tools/zygctl/:

cd tools/zygctl && gcc -c src/symlink_wrapper.c -o build/symlink_wrapper.o && reefc build --obj build/symlink_wrapper.o

The sysv-init wrapper builds with plain make under tools/sysv-wrapper/.

• Env vars: ZYGINIT_CONFIG_DIR (default /etc/zyginit), ZYGINIT_LOG_DIR (default /var/log/zyginit), ZYGINIT_SOCKET (default /var/run/zyginit.sock), ZYGINIT_RUN_DIR (default /var/run/zyginit, used for state.toml)

Testing

Testing happens on the Hammerhead test VM hh-prototest:

Host	IP	Purpose
hh-prototest	192.168.122.50	Hammerhead OS test VM (libvirt host reservation: MAC 52:54:00:91:c9:d2)
hammerhead-build	192.168.122.66	Build VM (OI-based, for compiling Hammerhead itself)

The older standalone VM (192.168.122.17) is deprecated.

Deploy procedure (current iter 11+ pattern):

# From dev host:
scp src/<changed>.reef root@192.168.122.50:/root/zyginit/src/
ssh root@192.168.122.50 'cd /root/zyginit && \
    gcc -c src/helpers.c -o build/helpers.o && \
    reefc build -l contract --obj build/helpers.o && \
    cp build/zyginit /sbin/init.new && mv /sbin/init.new /sbin/init'

# Reboot via virsh (init 6 from non-PID-1 zyginit doesn't actually
# reboot — it spawns a second zyginit-as-supervisor, NOT what you want):
virsh --connect=qemu:///system reset hh-prototest

/sbin/init IS the zyginit daemon binary on Hammerhead — it's the
kernel's exec target at boot. There's no separate /sbin/zyginit
binary in production deployment.

To run zyginit non-PID-1 for development (without replacing init):

ZYGINIT_CONFIG_DIR=/tmp/zygsvc /path/to/zyginit

It detects non-PID-1 via getpid() != 1 and runs in "supervisor
mode" (no PID-1-specific setup, normal exit on shutdown signal).

Source Layout

zyginit/
├── CLAUDE.md              # This file — project context for Claude
├── ROADMAP.md             # Forward-looking roadmap with task checklists
├── docs/
│   └── DESIGN.md          # Full architecture and design document
├── src/
│   ├── zyginit.reef       # Main init daemon
│   ├── zygctl.reef        # Admin CLI tool
│   ├── config.reef        # TOML config parser / service definition types
│   ├── depgraph.reef      # Dependency graph and topological sort
│   ├── contract.reef      # Hammerhead process contract FFI bindings
│   ├── supervisor.reef    # Service lifecycle management
│   └── ctlsocket.reef     # Unix domain socket server for zygctl (module `ctlsocket`; renamed from `socket` to avoid a name clash with stdlib `net.socket`, which `net.unix` now imports as of Reef 0.7.6)
├── services/
│   └── examples/          # Example TOML service definitions
├── tests/
│   ├── test_config.reef   # Config parsing tests
│   ├── test_depgraph.reef # Dependency graph tests
│   └── test_toml/         # Test TOML files
└── Makefile               # Build rules

Hammerhead Process Contracts — Quick Reference

Process contracts are the kernel mechanism zyginit uses to track service processes.
They are NOT available on Linux — this is Hammerhead-specific.

Key APIs (C FFI)

// Template setup (before fork)
int tmpl_fd = open("/system/contract/process/template", O_RDWR);
ct_tmpl_set_informative(tmpl_fd, CT_PR_EV_EMPTY);  // want "empty" events
ct_tmpl_set_critical(tmpl_fd, CT_PR_EV_EMPTY | CT_PR_EV_HWERR);
ct_pr_tmpl_set_fatal(tmpl_fd, CT_PR_EV_HWERR);
ct_pr_tmpl_set_param(tmpl_fd, CT_PR_INHERIT | CT_PR_NOORPHAN);
ct_tmpl_activate(tmpl_fd);

// After fork — get the new contract ID
int latest_fd = open("/system/contract/process/latest", O_RDONLY);
ct_status_read(latest_fd, CTD_COMMON, &status);
ctid_t contract_id = status->ctst_id;

ct_tmpl_clear(tmpl_fd);   // deactivate template
close(tmpl_fd);

// Event monitoring (poll on this fd)
int bundle_fd = open("/system/contract/process/bundle", O_RDONLY);

// Kill all processes in a contract
sigsend(P_CTID, contract_id, SIGTERM);

Key Header Files

#include <libcontract.h>       // ct_tmpl_*, ct_pr_tmpl_*, ct_status_*
#include <sys/contract.h>      // CT_PR_*, CTD_*
#include <sys/contract/process.h>  // CT_PR_EV_*, CT_PR_INHERIT, etc.
#include <sys/ctfs.h>          // CTFS paths

Contract Lifecycle

1. Open template → configure parameters → activate
2. fork() — child inherits active template, new contract created
3. exec() the service binary in the child
4. Parent reads contract ID from /system/contract/process/latest
5. poll() on bundle fd → receive events (empty, core, hwerr, signal)
6. On service exit: contract fires "empty" event → zyginit decides restart policy
7. On zyginit restart: re-adopt existing contracts (they survive init restart)

Service Definition Schema

See docs/DESIGN.md for full details. Quick reference:

[service]
name = "string"           # required, unique
description = "string"    # human-readable
type = "daemon|oneshot|transient"  # required

[exec]
start = "command"         # required — command or path to script
stop = "command"          # optional — defaults to contract signal
working_directory = "/path"  # optional — chdir before exec
user = "username"         # optional — drop privileges to this user
group = "groupname"       # optional — override primary group
environment = ["K=V"]     # optional — set env vars before exec

[stop]
method = "contract|exec"  # default: contract
signal = "TERM"           # default: TERM
timeout = 60              # seconds before KILL

[dependencies]
requires = ["svc1", "svc2"]  # must be running
after = ["svc3"]             # ordering only (not hard dependency)
conflicts = ["svc4"]         # mutually exclusive — both fail if both enabled; start refused if peer running

[contract]
param = ["inherit", "noorphan"]
fatal = ["empty", "hwerr"]

[restart]
on = "always|failure|never"  # default: failure
delay = 5                    # seconds between restarts
max_retries = 3              # -1 for unlimited

Hammerhead Kernel → Init Handover (PID 1)

The kernel-to-init handoff has subtleties that materially affect zyginit
when it runs as PID 1 (vs. the integration-test path running as a child
of a shell). Key facts pulled from base/usr/src/uts/common/os/main.c
(exec_init, start_init_common, start_init) and os/exit.c
(restart_init, zone_init_exit):

1. Init path is per-zone, default /sbin/init. Kernel uses
   p->p_zone->zone_initname and p->p_zone->zone_bootargs. Global
   zone defaults to /sbin/init and the bootargs from /boot/loader.conf
boot-args= (e.g., -v -m verbose). Per-zone init paths can be set
   via zone_setattr(2).
2. p->p_model = DATAMODEL_ILP32 is just initial state. exec_common
   re-sets it based on the actual ELF type — a 64-bit init works fine.
3. There is NO SMF-specific kernel handoff. Kernel exec's whatever
   binary zone_initname points at. All SMF wiring (-m smf-options,
   starting svc.startd, /etc/inittab) is in userland init.c.
4. fd 0/1/2 are NOT pre-opened by the kernel. The canonical
   /sbin/init opens /dev/syscon (or /dev/console) every time it
   wants to print (see console() in cmd/init/init.c). zyginit's
   setup_pid1_console() does this once at startup, dup2()ing onto
   0/1/2.
5. If init exits, kernel re-execs it. restart_init() in exit.c
   runs in a tight loop and logs "init(8) exited with status 0: restarting automatically". zyginit's PID-1 path MUST NOT return
   from main() — it ends in sync() + uadmin(A_SHUTDOWN, ...) or
   an infinite sleep loop.

The combination of #4 and the no-output-on-failure mode meant our first
PID-1 boot attempt failed silently (filling the console with the kernel's
restart-warning). Diagnosed by tracing main.c → exit.c, then comparing
zyginit's startup against cmd/init/init.c. The fd-setup fix is at
src/main.reef:setup_pid1_console.

Zones Support

Hammerhead inherits illumos zone (container) support. Each zone has its
own zone_initname and PID 1, and zones are isolated by the kernel —
the global zone init (zyginit) does NOT manage non-global zone init
processes; the kernel does, per zone.

What zyginit needs to provide for zones (future milestone, NOT first
boot):

• A zones oneshot service that walks /etc/zones/index at boot to
  invoke zoneadm boot for each autoboot=true zone, mirroring the
  SMF system/zones:default method (/lib/svc/method/svc-zones).
• A matching stop in reverse-tier shutdown that halts zones before
  filesystem teardown (zones may have ZFS datasets).
• Do NOT supervise zoneadmd directly — it's spawned per-zone by
  zoneadm boot, not as a singleton daemon.

Currently disabled on hh-prototest (svc:/system/zones:default disabled),
so first PID-1 boot is unaffected. Add the zones service alongside
SMF removal in Phase C.

Status (as of rev 122 / zyginit 0.1.2, 2026-05-12)

All 7 implementation phases are complete and PID-1 boot on Hammerhead is
working end-to-end on hh-prototest. Phase D (SMF removal from
Hammerhead) is complete as of 2026-05-12 on the hh-alpha5 greenfield
build — Hammerhead now boots and runs entirely under zyginit, with
no SMF master daemons anywhere on disk. See ROADMAP.md for the full
status; the abbreviated state:

• Multi-user boot works (vioif0 configured, sshd reachable at
  192.168.122.50, all daemons running)
• Single-user mode works (boot arg -s OR runtime zygctl single
  / init s — drops to /sbin/sh on /dev/console)
• Runtime runlevel transitions (single ↔ multi) work via zygctl
• Clean shutdown (zygctl halt / reboot / poweroff, or via
  init/halt/reboot/poweroff legacy commands through
  /sbin/sysv-wrapper)
• utmpx integration — who -b, who -r, uptime report correctly
• SMF gone from Hammerhead — svc.startd, svc.configd, svcadm,
  svccfg, svcs, svcprop, cmd/svc/, manifests, repository.db
  all deleted from source and packaging. libscf.so.1 deliberately
  retained as a 54KB ABI stub (returns SCF_ERROR_NO_SERVER) so the
  ~100 base consumers that still -lscf degrade gracefully without a
  mass refactor. libxml2 stays — it has non-SMF consumers (iSNS,
  FMA fabric-xlate, KMF crypto, sharemgr, zonestatd)

zyginit ships the engine (init binary + zygctl + sysv-wrapper) plus
minimal canonical examples in examples/. Distributions (Hammerhead
and any future distro) own all service-definition policy in their
own overlay trees. Layout/schema changes in zyginit can ship
independently of distro service-set churn.

The full Hammerhead service set lives in base/usr/src/zyginit/overrides/
on the Hammerhead side. The remaining SMF surface (NFS, ZFS automount,
ipfilter, ipsec, IPMP, fmd, inetd, rpcbind, etc.) was converted as
Hammerhead-side override TOMLs alongside Phase D — that ROADMAP item is done.

Important Notes

• /var/run is NOT currently tmpfs in our zyginit-only BE — files
  persist across reboots. Standard illumos has it as tmpfs (mounted
  via fs-minimal). Adding it back is a follow-up; not strictly
  needed for current functionality but would prevent stale lock
  files from accumulating.
• Hammerhead is 64-bit only — no 32-bit considerations.
• Hammerhead uses flat library paths (/usr/lib/ not /usr/lib/amd64/).
• The kernel exec target IS /sbin/init — zyginit binary is deployed
  directly there. There's no separate /sbin/zyginit in production.
• GC signals: use SIGRTMIN+0 and SIGRTMIN+1 (no collision with standard
  signals). Reef's --no-gc flag can disable GC entirely if needed
  for PID 1 reliability (not currently used; default GC works).
• poll(2) is preferred over event ports for simplicity (only 3-4 fds
  to monitor).
• The contract bundle fd MUST be opened with O_NONBLOCK (per iter 17),
  otherwise ct_event_read blocks the main loop after draining events.

Hammerhead Boot Chain (rev 82)

Currently enabled on hh-prototest (/etc/zyginit/enabled.d/):

Tier	Services	Notes
0	root-fs	Asserts ZFS root readonly=off
1	crypto, devfs, swap	devfs runs soconfig + devfsadm
2	filesystem	mountall + zpool import + zfs mount
3	identity, sysconfig, dlmgmtd	dlmgmtd -d foreground mode
4	ipmgmtd	ipmgmtd -f foreground mode
5	network	dladm init-phys + ifconfig from /etc/hostname.*
6	utmpd, pfexecd, syseventd, syslogd	daemon tier
7	cron, console-login, sshd	login chain + scheduling; single-user-shell here too

The remaining ~50 SMF services on standard Hammerhead (NFS, ZFS auto-
mount infrastructure, ipfilter, ipsec, fmd, inetd, rpcbind, etc.) are
not yet converted. For most, the foreground-mode-of-the-daemon pattern
likely applies — see ROADMAP.md "SMF Service Manifest Conversion".