# zyginit Administration Guide

## Overview

zyginit is the init daemon and service supervisor for Zygaena/Hammerhead.
It manages system services from boot to shutdown using TOML-based service
definitions, symlink-based enable/disable, and Hammerhead process contracts
for reliable service tracking.

This guide covers day-to-day administration: writing service definitions,
managing services with `zygctl`, understanding the boot sequence, and
troubleshooting.

## Quick Start

Check the status of all services:

```
$ zygctl status
sshd running pid=1234 ctid=7 uptime=2h30m restarts=0
cron running pid=1235 ctid=8 uptime=2h30m restarts=0
filesystem stopped exit=0
network stopped exit=0
```

Start or stop a service:

```
# zygctl start sshd
sshd started

# zygctl stop sshd
sshd stopping
```

Enable a service to start at boot:

```
# zygctl enable sshd
# zygctl disable sshd
```

List all known service definitions:

```
$ zygctl list
sshd (daemon)
cron (daemon)
network (oneshot)
filesystem (oneshot)
```

## Service Definitions

Service definitions are TOML files in `/etc/zyginit/`. Each file
describes a single service.

### Directory Layout

```
/etc/zyginit/
├── sshd.toml               # Service definition
├── cron.toml
├── syslog.toml
├── network.toml
├── network/                 # Scripts for complex startup
│   └── start.sh
├── filesystem.toml
├── filesystem/
│   └── start.sh
└── enabled.d/               # Symlinks = what starts at boot
    ├── sshd -> ../sshd.toml
    ├── cron -> ../cron.toml
    ├── syslog -> ../syslog.toml
    ├── network -> ../network.toml
    └── filesystem -> ../filesystem.toml
```

### Enabling and Disabling

A service starts at boot only if it has a symlink in `enabled.d/`. The
symlink name must match the service name (without `.toml`):

```
# Enable:
ln -s ../sshd.toml /etc/zyginit/enabled.d/sshd

# Disable:
rm /etc/zyginit/enabled.d/sshd
```

Or use `zygctl`:

```
# zygctl enable sshd
# zygctl disable sshd
```

Enabling does not start the service immediately. Disabling does not stop
it. Use `zygctl start` and `zygctl stop` for immediate control.

### Reloading Configuration

After enabling or disabling services, tell zyginit to re-scan the
`enabled.d/` directory:

```
# zygctl reload
```

Newly-enabled services are started; disabled services are stopped. This
is equivalent to sending SIGHUP to the zyginit process.

### Viewing Service Logs

zyginit captures stdout and stderr from each service to per-service log
files under `/var/run/zyginit/log/`. Log files are truncated on each
service start, so only the most recent output is available.

```
$ zygctl log sshd
```

This is useful for diagnosing startup failures. For long-term logging,
services should use syslog or their own logging mechanism.

## Writing Service Definitions

### Service Types

| Type | Description | Boot behavior | Restart |
|------|-------------|---------------|---------|
| `daemon` | Long-running process | Started and monitored | Per restart policy |
| `oneshot` | Runs once at boot | Runs, then "done" | Never |
| `transient` | On-demand only | Not started at boot | Per restart policy |

### Complete TOML Schema

```toml
[service]
name = "string"           # Required. Unique identifier.
description = "string"    # Optional. Human-readable.
type = "daemon"           # Required. daemon | oneshot | transient

[exec]
start = "command"         # Required. Command to start the service.
stop = "command"          # Optional. Only used with method = "exec".
working_directory = "/path"  # Optional. chdir before exec.
user = "username"         # Optional. Drop privileges to this user.
group = "groupname"       # Optional. Override primary group.
environment = ["K=V"]     # Optional. Set env vars before exec.

[stop]
method = "contract"       # Optional. contract (default) | exec
signal = "TERM"           # Optional. Signal name without SIG prefix.
timeout = 60              # Optional. Seconds before SIGKILL.

[dependencies]
requires = ["svc1"]       # Optional. Must be running first.
after = ["svc2"]          # Optional. Ordering only, not hard dep.

[contract]
param = ["inherit", "noorphan"]   # Optional. Contract parameters.
fatal = ["empty", "hwerr"]        # Optional. Fatal event types.

[restart]
on = "failure"            # Optional. always | failure (default) | never
delay = 5                 # Optional. Seconds between restarts.
max_retries = 3           # Optional. -1 = unlimited.
```

### Simple Daemon

Most daemons need only a few lines. Defaults handle the rest:

```toml
# /etc/zyginit/cron.toml
[service]
name = "cron"
description = "Clock Daemon"
type = "daemon"

[exec]
start = "/usr/sbin/cron"

[dependencies]
requires = ["filesystem"]
after = ["syslog"]
```

This uses all defaults: contract-based stop with SIGTERM, 60-second
timeout, restart on failure with 5-second delay and 3 retries.

### Foreground Daemons

zyginit tracks service processes via contracts. Daemons should run in the
foreground (not fork into the background) so that zyginit can monitor the
actual process. Most daemons support a foreground flag:

| Daemon | Foreground flag |
|--------|-----------------|
| sshd | `-D` |
| ntpd | `-n` |
| syslogd | `-d` (debug/foreground) |

Example:

```toml
[exec]
start = "/usr/lib/ssh/sshd -D"
```

If a daemon does not support foreground mode, zyginit's contract mechanism
still tracks all descendant processes, including the backgrounded child.

### Oneshot Services

Oneshot services run a command once at boot. They are useful for setup
tasks like mounting filesystems, configuring network interfaces, or
setting the hostname.

```toml
# /etc/zyginit/hostname.toml
[service]
name = "hostname"
description = "Set System Hostname"
type = "oneshot"

[exec]
start = "/usr/bin/hostname myhost"

[dependencies]
requires = ["root-fs"]

[restart]
on = "never"
```

A oneshot service with exit code 0 enters the `stopped` state. A non-zero
exit puts it in the `failed` state.

### Script Escape Hatch

For complex startup logic, put scripts in a subdirectory matching the
service name:

```
/etc/zyginit/
├── network.toml
└── network/
    └── start.sh
```

```toml
# network.toml
[service]
name = "network"
description = "Network Interface Configuration"
type = "oneshot"

[exec]
start = "/etc/zyginit/network/start.sh"
```

The start script can use `ipadm`, `dladm`, and other tools to
configure interfaces. Keep scripts simple and idempotent where possible.

### Custom Stop Commands

Some services need a specific shutdown procedure (e.g., databases):

```toml
[exec]
start = "/etc/zyginit/postgresql/start.sh"
stop = "/etc/zyginit/postgresql/stop.sh"

[stop]
method = "exec"
timeout = 120
```

When `method = "exec"`, zyginit runs the stop command instead of
signaling the process contract. The timeout still applies; if the stop
command does not complete in time, zyginit sends SIGKILL.

### Working Directory

Use `working_directory` to change to a specific directory before starting
the service. Useful for applications that expect to run from a particular
path:

```toml
[exec]
start = "/usr/lib/myapp/bin/server"
working_directory = "/var/lib/myapp"
```

### Environment Variables

Set environment variables for the service process using the `environment`
array. Each entry has the form `KEY=value`:

```toml
[exec]
start = "/usr/lib/postgresql/bin/postgres -D /var/lib/pgsql/data"
environment = ["PGDATA=/var/lib/pgsql/data", "LANG=en_US.UTF-8", "TZ=US/Pacific"]
```

These variables are set in the child process before exec and do not
affect zyginit itself or other services.

### Privilege Separation

Run services as a non-root user with the `user` and `group` fields.
Before executing the start command, zyginit drops privileges by calling
`setgid()`, `initgroups()`, and `setuid()` in the correct order:

```toml
[exec]
start = "/usr/lib/ssh/sshd -D"
user = "sshd"
group = "sshd"
```

If only `user` is specified, the primary group is determined from the
user's passwd entry. If both `user` and `group` are specified, the
group overrides the user's primary group.

If the user or group lookup fails, the service start fails and the
error is logged.

### Dependencies

Dependencies control boot order. There are two kinds:

**`requires`** — Hard dependencies. The listed services must be running
before this service starts. If a required service is not enabled, zyginit
prints a warning.

**`after`** — Soft ordering. The listed services should start first, but
if they are not enabled, this service starts anyway.

```toml
[dependencies]
requires = ["network", "filesystem"]
after = ["name-services", "syslog"]
```

zyginit performs a topological sort to group services into parallel boot
tiers. Services within the same tier can start concurrently.

### Restart Policies

For daemon services, the restart policy determines what happens when the
process exits:

| Policy | Behavior |
|--------|----------|
| `always` | Restart regardless of exit code |
| `failure` | Restart only on non-zero exit (default) |
| `never` | Do not restart |

The `delay` field sets the wait time (seconds) between restarts. The
`max_retries` field sets the maximum consecutive restart attempts before
the service enters `maintenance` state. Set `max_retries = -1` for
unlimited.

```toml
[restart]
on = "always"
delay = 10
max_retries = -1
```

### Process Contract Parameters

The `[contract]` section controls Hammerhead kernel process contracts.
Most services should use the defaults:

```toml
[contract]
param = ["inherit", "noorphan"]
fatal = ["empty", "hwerr"]
```

| Parameter | Description |
|-----------|-------------|
| `inherit` | Child processes inherit the contract |
| `noorphan` | Kill orphans when contract is abandoned |
| `pgrponly` | Track only the process group leader |
| `regent` | Contract owner acts as regent |

| Fatal event | Description |
|-------------|-------------|
| `empty` | All processes in the contract exited |
| `hwerr` | Hardware error detected |
| `core` | A process produced a core dump |
| `signal` | A process was killed by a signal |

## Boot Sequence

When zyginit starts as PID 1, the boot proceeds as follows:

1. **Scan** `/etc/zyginit/enabled.d/` for symlinks
2. **Parse** each linked TOML file to load service definitions
3. **Build** a dependency graph from `requires` and `after` declarations
4. **Sort** the graph into parallel boot tiers (topological sort)
5. **Start** services tier by tier
6. **Open** the control socket at `/var/run/zyginit.sock`
7. **Enter** the main event loop

The event loop monitors three sources via `poll(2)`:

- **Process contracts** — service exit events from the kernel
- **Signal pipe** — SIGCHLD, SIGTERM, SIGHUP
- **Control socket** — commands from `zygctl`

Every loop iteration also checks for stop timeouts (escalate to SIGKILL)
and restart delays (restart services whose delay has elapsed).

## Shutdown

When zyginit receives SIGTERM or SIGINT, it performs a graceful shutdown:

1. Stop accepting new `zygctl` connections
2. Stop all running services in reverse boot order
3. Wait for each service to exit (respecting stop timeouts)
4. Clean up the control socket
5. Exit

## Troubleshooting

### Service in maintenance state

A service enters maintenance when it has exceeded `max_retries`. To
recover:

```
# zygctl restart myservice
```

Investigate the cause of repeated failures before restarting. Check
service logs, file permissions, and dependency availability.

### Service won't start

Check the service output log for error messages:

```
$ zygctl log myservice
```

Also check that:
- The TOML file exists in `/etc/zyginit/`
- The TOML syntax is valid
- The `start` command path exists and is executable
- All `requires` dependencies are enabled and running
- The service name matches the filename (without `.toml`)
- If `user`/`group` is specified, the user and group exist on the system

### Cannot connect to zyginit

If `zygctl` reports "cannot connect":
- Verify zyginit is running: `ps -ef | grep zyginit`
- Check the socket exists: `ls -la /var/run/zyginit.sock`
- If using a custom socket path, set `ZYGINIT_SOCKET`

### Orphaned processes

On Hammerhead, process contracts prevent orphans. The `noorphan` contract
parameter (default) ensures all descendant processes are killed when a
service's contract is abandoned.

On systems without contract support, zyginit tracks only the direct child
PID and cannot guarantee cleanup of grandchild processes.

## Environment Variables

| Variable | Default | Description |
|----------|---------|-------------|
| `ZYGINIT_CONFIG_DIR` | `/etc/zyginit` | Service definition directory |
| `ZYGINIT_SOCKET` | `/var/run/zyginit.sock` | Control socket path |
| `ZYGINIT_LOG_DIR` | `/var/run/zyginit/log` | Per-service log directory |

These are primarily useful for development and testing. In production,
the defaults should be used.