zyginit Administration Guide

Overview

zyginit is the init daemon and service supervisor for Zygaena/Hammerhead.
It manages system services from boot to shutdown using TOML-based service
definitions, symlink-based enable/disable, and Hammerhead process contracts
for reliable service tracking.

This guide covers day-to-day administration: writing service definitions,
managing services with zygctl, understanding the boot sequence, and
troubleshooting.

Quick Start

Check the status of all services:

$ zygctl status
sshd running pid=1234 ctid=7 uptime=2h30m restarts=0
cron running pid=1235 ctid=8 uptime=2h30m restarts=0
filesystem stopped exit=0
network stopped exit=0

Start or stop a service:

# zygctl start sshd
sshd started

# zygctl stop sshd
sshd stopping

Enable a service to start at boot:

# zygctl enable sshd
# zygctl disable sshd

List all known service definitions:

$ zygctl list
sshd (daemon)
cron (daemon)
network (oneshot)
filesystem (oneshot)

Service Definitions

Service definitions are TOML files in /etc/zyginit/. Each file
describes a single service.

Directory Layout

/etc/zyginit/
├── sshd.toml               # Service definition
├── cron.toml
├── syslog.toml
├── network.toml
├── network/                 # Scripts for complex startup
│   └── start.sh
├── filesystem.toml
├── filesystem/
│   └── start.sh
└── enabled.d/               # Symlinks = what starts at boot
    ├── sshd -> ../sshd.toml
    ├── cron -> ../cron.toml
    ├── syslog -> ../syslog.toml
    ├── network -> ../network.toml
    └── filesystem -> ../filesystem.toml

Enabling and Disabling

A service starts at boot only if it has a symlink in enabled.d/. The
symlink name must match the service name (without .toml):

# Enable:
ln -s ../sshd.toml /etc/zyginit/enabled.d/sshd

# Disable:
rm /etc/zyginit/enabled.d/sshd

Or use zygctl:

# zygctl enable sshd
# zygctl disable sshd

Enabling does not start the service immediately. Disabling does not stop
it. Use zygctl start and zygctl stop for immediate control.

Reloading Configuration

After enabling or disabling services, tell zyginit to re-scan the
enabled.d/ directory:

# zygctl reload

Newly-enabled services are started; disabled services are stopped. This
is equivalent to sending SIGHUP to the zyginit process.

Viewing Service Logs

zyginit captures stdout and stderr from each service to per-service log
files under /var/run/zyginit/log/. Log files are truncated on each
service start, so only the most recent output is available.

$ zygctl log sshd

This is useful for diagnosing startup failures. For long-term logging,
services should use syslog or their own logging mechanism.

Writing Service Definitions

Service Types

Type	Description	Boot behavior	Restart
daemon	Long-running process	Started and monitored	Per restart policy
oneshot	Runs once at boot	Runs, then "done"	Never
transient	On-demand only	Not started at boot	Per restart policy

Complete TOML Schema

[service]
name = "string"           # Required. Unique identifier.
description = "string"    # Optional. Human-readable.
type = "daemon"           # Required. daemon | oneshot | transient

[exec]
start = "command"         # Required. Command to start the service.
stop = "command"          # Optional. Only used with method = "exec".
working_directory = "/path"  # Optional. chdir before exec.
user = "username"         # Optional. Drop privileges to this user.
group = "groupname"       # Optional. Override primary group.
environment = ["K=V"]     # Optional. Set env vars before exec.

[stop]
method = "contract"       # Optional. contract (default) | exec
signal = "TERM"           # Optional. Signal name without SIG prefix.
timeout = 60              # Optional. Seconds before SIGKILL.

[dependencies]
requires = ["svc1"]       # Optional. Must be running first.
after = ["svc2"]          # Optional. Ordering only, not hard dep.

[contract]
param = ["inherit", "noorphan"]   # Optional. Contract parameters.
fatal = ["empty", "hwerr"]        # Optional. Fatal event types.

[restart]
on = "failure"            # Optional. always | failure (default) | never
delay = 5                 # Optional. Seconds between restarts.
max_retries = 3           # Optional. -1 = unlimited.

Simple Daemon

Most daemons need only a few lines. Defaults handle the rest:

# /etc/zyginit/cron.toml
[service]
name = "cron"
description = "Clock Daemon"
type = "daemon"

[exec]
start = "/usr/sbin/cron"

[dependencies]
requires = ["filesystem"]
after = ["syslog"]

This uses all defaults: contract-based stop with SIGTERM, 60-second
timeout, restart on failure with 5-second delay and 3 retries.

Foreground Daemons

zyginit tracks service processes via contracts. Daemons should run in the
foreground (not fork into the background) so that zyginit can monitor the
actual process. Most daemons support a foreground flag:

Daemon	Foreground flag
sshd	-D
ntpd	-n
syslogd	-d (debug/foreground)

Example:

[exec]
start = "/usr/lib/ssh/sshd -D"

If a daemon does not support foreground mode, zyginit's contract mechanism
still tracks all descendant processes, including the backgrounded child.

Oneshot Services

Oneshot services run a command once at boot. They are useful for setup
tasks like mounting filesystems, configuring network interfaces, or
setting the hostname.

# /etc/zyginit/hostname.toml
[service]
name = "hostname"
description = "Set System Hostname"
type = "oneshot"

[exec]
start = "/usr/bin/hostname myhost"

[dependencies]
requires = ["root-fs"]

[restart]
on = "never"

A oneshot service with exit code 0 enters the stopped state. A non-zero
exit puts it in the failed state.

Script Escape Hatch

For complex startup logic, put scripts in a subdirectory matching the
service name:

/etc/zyginit/
├── network.toml
└── network/
    └── start.sh

# network.toml
[service]
name = "network"
description = "Network Interface Configuration"
type = "oneshot"

[exec]
start = "/etc/zyginit/network/start.sh"

The start script can use ipadm, dladm, and other tools to
configure interfaces. Keep scripts simple and idempotent where possible.

Custom Stop Commands

Some services need a specific shutdown procedure (e.g., databases):

[exec]
start = "/etc/zyginit/postgresql/start.sh"
stop = "/etc/zyginit/postgresql/stop.sh"

[stop]
method = "exec"
timeout = 120

When method = "exec", zyginit runs the stop command instead of
signaling the process contract. The timeout still applies; if the stop
command does not complete in time, zyginit sends SIGKILL.

Working Directory

Use working_directory to change to a specific directory before starting
the service. Useful for applications that expect to run from a particular
path:

[exec]
start = "/usr/lib/myapp/bin/server"
working_directory = "/var/lib/myapp"

Environment Variables

Set environment variables for the service process using the environment
array. Each entry has the form KEY=value:

[exec]
start = "/usr/lib/postgresql/bin/postgres -D /var/lib/pgsql/data"
environment = ["PGDATA=/var/lib/pgsql/data", "LANG=en_US.UTF-8", "TZ=US/Pacific"]

These variables are set in the child process before exec and do not
affect zyginit itself or other services.

Privilege Separation

Run services as a non-root user with the user and group fields.
Before executing the start command, zyginit drops privileges by calling
setgid(), initgroups(), and setuid() in the correct order:

[exec]
start = "/usr/lib/ssh/sshd -D"
user = "sshd"
group = "sshd"

If only user is specified, the primary group is determined from the
user's passwd entry. If both user and group are specified, the
group overrides the user's primary group.

If the user or group lookup fails, the service start fails and the
error is logged.

Dependencies

Dependencies control boot order. There are two kinds:

requires — Hard dependencies. The listed services must be running
before this service starts. If a required service is not enabled, zyginit
prints a warning.

after — Soft ordering. The listed services should start first, but
if they are not enabled, this service starts anyway.

[dependencies]
requires = ["network", "filesystem"]
after = ["name-services", "syslog"]

zyginit performs a topological sort to group services into parallel boot
tiers. Services within the same tier can start concurrently.

Restart Policies

For daemon services, the restart policy determines what happens when the
process exits:

Policy	Behavior
always	Restart regardless of exit code
failure	Restart only on non-zero exit (default)
never	Do not restart

The delay field sets the wait time (seconds) between restarts. The
max_retries field sets the maximum consecutive restart attempts before
the service enters maintenance state. Set max_retries = -1 for
unlimited.

[restart]
on = "always"
delay = 10
max_retries = -1

Process Contract Parameters

The [contract] section controls Hammerhead kernel process contracts.
Most services should use the defaults:

[contract]
param = ["inherit", "noorphan"]
fatal = ["empty", "hwerr"]

Parameter	Description
inherit	Child processes inherit the contract
noorphan	Kill orphans when contract is abandoned
pgrponly	Track only the process group leader
regent	Contract owner acts as regent

Fatal event	Description
empty	All processes in the contract exited
hwerr	Hardware error detected
core	A process produced a core dump
signal	A process was killed by a signal

Boot Sequence

When zyginit starts as PID 1, the boot proceeds as follows:

1. Scan /etc/zyginit/enabled.d/ for symlinks
2. Parse each linked TOML file to load service definitions
3. Build a dependency graph from requires and after declarations
4. Sort the graph into parallel boot tiers (topological sort)
5. Start services tier by tier
6. Open the control socket at /var/run/zyginit.sock
7. Enter the main event loop

The event loop monitors three sources via poll(2):

• Process contracts — service exit events from the kernel
• Signal pipe — SIGCHLD, SIGTERM, SIGHUP
• Control socket — commands from zygctl

Every loop iteration also checks for stop timeouts (escalate to SIGKILL)
and restart delays (restart services whose delay has elapsed).

Shutdown

When zyginit receives SIGTERM or SIGINT, it performs a graceful shutdown:

1. Stop accepting new zygctl connections
2. Stop all running services in reverse boot order
3. Wait for each service to exit (respecting stop timeouts)
4. Clean up the control socket
5. Exit

Troubleshooting

Service in maintenance state

A service enters maintenance when it has exceeded max_retries. To
recover:

# zygctl restart myservice

Investigate the cause of repeated failures before restarting. Check
service logs, file permissions, and dependency availability.

Service won't start

Check the service output log for error messages:

$ zygctl log myservice

Also check that:

• The TOML file exists in /etc/zyginit/
• The TOML syntax is valid
• The start command path exists and is executable
• All requires dependencies are enabled and running
• The service name matches the filename (without .toml)
• If user/group is specified, the user and group exist on the system

Cannot connect to zyginit

If zygctl reports "cannot connect":

• Verify zyginit is running: ps -ef | grep zyginit
• Check the socket exists: ls -la /var/run/zyginit.sock
• If using a custom socket path, set ZYGINIT_SOCKET

Orphaned processes

On Hammerhead, process contracts prevent orphans. The noorphan contract
parameter (default) ensures all descendant processes are killed when a
service's contract is abandoned.

On systems without contract support, zyginit tracks only the direct child
PID and cannot guarantee cleanup of grandchild processes.

Environment Variables

Variable	Default	Description
ZYGINIT_CONFIG_DIR	/etc/zyginit	Service definition directory
ZYGINIT_SOCKET	/var/run/zyginit.sock	Control socket path
ZYGINIT_LOG_DIR	/var/run/zyginit/log	Per-service log directory

These are primarily useful for development and testing. In production,
the defaults should be used.