zygaena / zyginit
|
Files Changesets Issues Pull Requests Releases
root / ROADMAP.md
ROADMAP.md markdown 375 lines 16.3 KB
Raw

zyginit Roadmap

Last updated: 2026-05-12

For architecture and design details, see docs/DESIGN.md.
For the original implementation plan, see docs/IMPLEMENTATION_PLAN.md.
For the PID-1 boot story (iter 1-20), see the per-iter notes under
docs/superpowers/plans/ and the auto-memory directory.

Completed

All 7 implementation phases are done and working on Linux (dev mode).
Integration test suite passes (57 tests across 9 suites).

Core Implementation (Phases 1-7)

  • TOML config parser — parse service definitions, validate schema,
    scan enabled.d/, inline array parsing
  • Dependency graph — topological sort (Kahn's algorithm), cycle
    detection, tiered boot order
  • Process contract FFI — extern "C" bindings to libcontract,
    [pointer] array pattern for void** params, Linux stub file
  • Service supervisor — fork/exec, state machine (8 states),
    restart policies, stop timeouts, SIGKILL escalation
  • Event loop — poll(2) on signal self-pipe, contract bundle fd,
    Unix domain socket; 1-second periodic tick
  • Unix socket server — newline-delimited text protocol, command
    dispatch (status, start, stop, restart, enable, disable, reload,
    log, list, version, help)
  • zygctl CLI — local commands (help, version, enable, disable),
    socket commands (everything else)

Post-Phase Enhancements

  • Enable/disable — symlink creation/removal in enabled.d/,
    works from both zygctl (local) and socket (server-side)
  • SIGHUP config reload — re-scan enabled.d/, add new services,
    disable removed services; triggered by signal or zygctl reload
  • Enhanced status output — uptime, exit code, restart count,
    human-readable durations
  • Integration test harness — 57 tests across 9 suites covering
    CLI, daemon lifecycle, restart policies, reload, oneshot services,
    working directory, environment variables, privilege separation,
    service output capture
  • Documentation — man pages (zyginit.8, zygctl.8, zyginit.5),
    administration guide, README, 19 example service definitions
  • Privilege separation — exec.user/exec.group fields, child
    drops privileges via setgid() → initgroups() → setuid()
  • Environment variables — exec.environment array of KEY=value
    strings, set in child before exec via set_env()
  • Working directory — exec.working_directory field, child calls
    chdir() before exec
  • Service output capture — per-service log files in
    /var/run/zyginit/log/<name>.log, truncated on each start,
    accessible via zygctl log <name>

PID-1 Boot on Hammerhead (iter 1-20, complete 2026-05-02)

zyginit boots Hammerhead as PID 1 with full multi-user functionality
and clean shutdown / reboot. Reached on hh-prototest (192.168.122.50).

  • PID-1 fd setup — setup_pid1_console() opens /dev/console
    with O_NOCTTY and pushes ldterm+ttcompat STREAMS modules
    so userspace println output aligns with kernel writes
  • soconfig at boot — devfs/start.sh runs soconfig -d /etc/sock2path.d to populate kernel sockfs's protocol map.
    Without it socket(AF_INET) returns EAFNOSUPPORT
  • Foreground daemon modes — dlmgmtd -d, ipmgmtd -f —
    bypass each daemon's SMF_FMRI startup gate AND avoid the
    double-fork daemonize, simplifying contract supervision.
    Pattern likely applies to other illumos daemons
  • dladm init-phys — registers physical NICs with dlmgmtd
    at network startup so ifconfig <link> plumb resolves names
  • Network bring-up — BSD-style /etc/hostname.<if> parser
    (no ipmgmtd-CLI use; ipmgmtd-DAEMON still needed for libipadm)
  • Console login chain — small C helper writes LOGIN_PROCESS
    utmpx entry before exec ttymon so login(1) doesn't bail
    with "No utmpx entry"
  • Daemonize-caveat fix — supervisor recognizes the parent-
    exits-0 daemonize handshake and keeps services in RUNNING
    state instead of mis-marking them STOPPED
  • Catch-all zombie reap — process_wait_any_nohang() loop
    after per-service reap iteration
  • Contract bundle non-blocking — O_NONBLOCK on
    /system/contract/process/bundle so ct_event_read returns
    EAGAIN instead of wedging the main loop
  • Default child PATH — supervisor sets
    PATH=/usr/sbin:/usr/bin:/sbin for service children (zyginit
    inherits a sparse env from the kernel)
  • utmpx integration — BOOT_TIME and RUN_LVL records on
    boot and on every runlevel transition, format matching
    svc.startd's utmpx_set_runlevel() so who -b and who -r
    report correctly
  • Per-service [runlevel] mode — "always" | "single" |
    "multi" (default "multi"). Drives initial active-set
    filtering and runtime transition stop/start sets
  • Single-user mode — kernel boot arg -s parsed in
    main.reef; /sbin/sh recovery shell on /dev/console via
    a regular zyginit service with mode = "single"
  • Runtime runlevel transitions — transition_runlevel()
    stops services not in the target's active set in reverse
    tier order, then starts services newly in the active set
  • zygctl single / zygctl multi — socket protocol
    handlers; legacy multiuser alias kept
  • sysv-init wrapper binary — tools/sysv-wrapper/wrapper.c
    installed as /sbin/halt, /sbin/reboot, /sbin/poweroff,
    /sbin/telinit (argv[0] dispatch translates legacy commands
    to zygctl invocations). /sbin/init itself stays the
    zyginit daemon binary (kernel exec target)
  • Clean shutdown sequence — shutdown_services (reverse
    tier) → destroy_socket (with parent-dir fsync) → do_sync
    → explicit zpool sync + umountall -l → fork child →
    child calls uadmin(A_SHUTDOWN, AD_BOOT) from non-PID-1
    context. Mirrors cmd/svc/startd/graph.c:do_uadmin. Without
    the fork, kernel's restart_init interferes with mdboot's
    hardware reset
  • No more readonly-on-shutdown — root-fs/stop.sh no
    longer sets zfs set readonly=on mid-shutdown. That was
    silently dropping every subsequent write (destroy_socket
    unlink, sync, anything else) — a real data-integrity bug,
    not just a missing socket file. ZFS is transactionally
    consistent and doesn't need a "freeze" before reboot

Near Term — Linux Development

Items that can be developed and tested on any system without Hammerhead.

Dependency Ordering Improvements

  • wants (soft dependencies) — like requires but don't fail if
    the dependency is not enabled. Start it if available, proceed without
    it if not.
  • conflicts — prevent two services from running simultaneously
    (e.g., sendmail vs postfix)
  • Milestones — virtual grouping targets (multi-user, network,
    single-user) that aggregate sets of services. A milestone is
    "reached" when all its member services are online.

Improved Reload (V2)

The current reload (V1) only detects added/removed services.

  • Detect in-place TOML changes by comparing config checksums or
    modification times
  • Support restarting services whose definitions changed
  • Log what changed on each reload for auditability

Test Coverage

  • Unit tests for config.reef (TOML parsing edge cases)
  • Unit tests for depgraph.reef (cycle detection, missing deps)
  • Extend integration tests with dependency ordering verification
  • Test concurrent zygctl connections

Medium Term — Requires Hammerhead VM

These items need the Hammerhead test VM (hh-prototest at
192.168.122.50). The older standalone VM (192.168.122.17) is
deprecated.

Contract Validation on Hammerhead — DONE (rev 18, 2026-04)

The process contract code is exercised end-to-end on hh-prototest.
9/9 probes in utils/contract_probe pass. Found and fixed P_CTID =
14 → 13 and ctl fd O_RDWR → O_WRONLY along the way.

  • Build with reefc build -l contract on Hammerhead
  • Verify setup_template() creates contracts successfully
  • Verify get_latest_contract() returns valid contract IDs
  • Verify read_event() receives CT_PR_EV_EMPTY on process exit
  • Verify kill_contract() signals all processes in a contract
  • Verify abandon_contract() works correctly
  • Contract re-adoption after zyginit restart (still pending —
    see "Contract Re-adoption" below)

PID 1 Boot Testing — DONE (rev 82, 2026-05-02)

zyginit runs as the actual init process on hh-prototest. Cold boot
to multi-user works; zygctl operates correctly; clean shutdown via
zygctl halt/reboot/poweroff and via init 0/6/s/3.

  • Hammerhead boot-chain TOML definitions: root-fs → crypto →
    devfs → swap → filesystem → identity → sysconfig → dlmgmtd →
    ipmgmtd → network → utmpd/pfexecd/syseventd/syslogd →
    cron/console-login/sshd → single-user-shell
  • /sbin/init deployed to hh-prototest as the zyginit binary
  • Cold boot to multi-user with SSH accessible
  • zygctl operates correctly during live boot (status, start,
    stop, enable, disable, reload, single, multi, halt, reboot,
    poweroff)
  • Graceful shutdown — zygctl halt / init 0 / init 6 etc.
    mirror SMF's cmd/svc/startd/graph.c:do_uadmin sequence
  • Boot chain documented in services/hammerhead/ (TOML files
    + start/stop scripts) and in the iter 11-20 memory notes

Contract Re-adoption

When zyginit restarts (e.g., after a crash or upgrade), existing process
contracts survive in the kernel. zyginit must re-adopt them to resume
supervision of running services.

  • On startup, scan /system/contract/process/ for existing contracts
  • Match contracts to enabled service definitions
  • Adopt orphaned contracts and resume monitoring
  • Handle the case where a service was removed while zyginit was down

GC Tuning for PID 1

Reef's garbage collector uses signals for stop-the-world coordination.
As PID 1, zyginit must not pause during critical signal handling.

  • Benchmark with --no-gc flag vs default GC
  • Test with --gc-signals rtmin to avoid signal conflicts
  • Profile memory usage over extended uptime (days)
  • Determine if --no-gc is acceptable (bounded working set)

SMF Service Manifest Conversion — DONE (2026-05-12, hh-alpha5)

The Hammerhead boot chain in this repo (services/hammerhead/)
covers the core multi-user set: root-fs, crypto, devfs, swap,
filesystem, identity, sysconfig, dlmgmtd, ipmgmtd, network, utmpd,
pfexecd, syseventd, syslogd, cron, console-login, sshd,
single-user-shell.

The rest of the SMF surface (NFS, ZFS automount, ipfilter, ipsec,
IPMP, fmd, inetd, rpcbind, etc.) was converted as Hammerhead-side
override TOMLs under base/usr/src/zyginit/overrides/ alongside
the Phase D SMF removal. The foreground-mode-of-the-daemon pattern
(dlmgmtd -d, ipmgmtd -f) applied to most.

  • Core boot chain converted (~17 services in this repo)
  • NFS server / client services (Hammerhead overrides)
  • ZFS automount + share daemons (Hammerhead overrides)
  • inetd / rpcbind (Hammerhead overrides)
  • fmd (Fault Management Daemon) (Hammerhead overrides)
  • ipfilter, ipsec, IPMP (Hammerhead overrides)
  • Phase B for single-user — swap start.sh from /sbin/sh to
    /sbin/sulogin so single-user requires the root password

Long Term

Larger features for after zyginit is stable and booting Hammerhead.

Resource Controls

Integrate with Hammerhead resource controls (projects, tasks, rctls) to
enforce memory limits, CPU caps, and file descriptor limits per service.

[limits]
max_fds = 1024
max_memory = "512M"
project = "service.sshd"
  • Research Hammerhead rctlblk APIs and setprojrg() / settaskid()
  • Add [limits] section to service schema
  • Apply resource controls in child before exec
  • Display resource usage in zygctl status

Zone-Aware Service Management

Support running zyginit inside Hammerhead zones (non-global zones).
Some services are zone-specific; others are global-only.

[service]
name = "zoneadmd"
zone = "global"    # only run in the global zone
  • Detect whether running in global or non-global zone
  • Filter services based on zone field
  • Handle zone boot/halt events

Parallel Service Startup

Currently services within a tier start sequentially. True parallel
startup would improve boot time.

  • Fork all services in a tier simultaneously
  • Track readiness per service (PID exists, contract active)
  • Only advance to next tier when all services in current tier
    have started (or failed and are not required)
  • Measure boot time improvement

Service Notifications / Hooks

Allow services to declare hooks that fire on state transitions.
Useful for health checks, monitoring integration, and dependent
service coordination.

[hooks]
on_start = "/etc/zyginit/sshd/on_start.sh"
on_stop = "/etc/zyginit/sshd/on_stop.sh"
on_failure = "/etc/zyginit/sshd/on_failure.sh"
  • Add [hooks] section to service schema
  • Execute hooks asynchronously (don't block the event loop)
  • Pass service name, state, exit code as environment variables

Diagnostic State Dump

Write runtime state to /var/run/zyginit/ for crash recovery analysis
and monitoring tool consumption.

  • Write JSON or TOML state file on each state change
  • Include: service name, state, pid, contract ID, uptime, restarts
  • zygctl dump command to write state snapshot on demand
  • Document format for monitoring integrations

SMF Removal from Hammerhead — DONE (Phase D, 2026-05-12, hh-alpha5)

The Hammerhead team completed SMF removal on main, verified on the
hh-alpha5 greenfield build. Hammerhead now boots and runs entirely
under zyginit, with no SMF master daemons anywhere on disk. The
zyginit-side work for this milestone was zero — the vendored zyginit
0.1.2 already had everything needed (PID 1, Unix socket protocol,
override layering, self-disable).

  • Extended zyginit-only boot verification on hh-prototest
  • Remove svc.startd, svc.configd, master restarter from build
    (also svcadm, svccfg, svcs, svcprop, cmd/svc/,
    tools/svc/, mdb svc modules, SMF man pages)
  • Remove SMF manifest infrastructure from packaging (/var/svc/,
    repository.db, manifest packaging — all gone)
  • Verify daemons we keep (dlmgmtd, ipmgmtd, etc.) still build
    without their SMF method scripts
  • Update Hammerhead documentation
  • Coordinate with Coral for package dependency updates
  • [~] libscf — intentionally retained as a permanent 54KB ABI
    stub (base/usr/src/lib/libscf/common/libscf-stub.c, ~1828 LOC).
    ~100 base consumers still -lscf; the stub returns
    SCF_ERROR_NO_SERVER so they degrade gracefully without a mass
    refactor. This is a deliberate Phase D design decision, not an
    oversight.
  • [~] libxml2 — stays. SMF was never the only consumer.
    Remaining base consumers: cmd/isns/isnsd + isnsadm,
    cmd/fm/modules/common/fabric-xlate,
    cmd/cmd-crypto/kmfcfg + pktool, cmd/dfs.cmds/sharemgr,
    cmd/zonestat/zonestatd.

Hammerhead-side overrides for this push (packaging overlays, not
zyginit changes) live in base/usr/src/zyginit/overrides/. ~9
commits added Hammerhead-specific TOML definitions: ipsec,
audit, resource-mgmt, rcapd, syslogd, console-login,
acpihpd, plus a 24-service batch.

Non-Goals

Things zyginit intentionally does not do:

  • Socket activation — adds complexity; services should manage their
    own listeners
  • Cgroup/namespace isolation — not applicable to Hammerhead (use zones)
  • Journal/binary logging — use syslog; zyginit is not a log manager
  • Timer units — use cron; zyginit manages services, not schedules
  • User-level service management — zyginit is PID 1 only; user
    services can use a separate supervisor if needed
  • D-Bus integration — Hammerhead does not use D-Bus
  • Automatic dependency resolution — services must explicitly declare
    dependencies; zyginit does not infer them