bugs: archive completed 001 (fd-0 ctty) and 002 (group-type RFE)
Author:
Chris Tusa <chris.tusa@leafscale.com>
Date:
Jul 20, 2026 21:07
Changeset:
71b8fd133c85748877eb86d795dec8b914f86b8c
Branch:
default
Changed files:
D
bugs/001-supervisor-leaves-console-as-fd0-in-children.md
D
bugs/002-rfe-group-target-service-type.md
Diff
diff -r c0018f136328 -r 71b8fd133c85 bugs/001-supervisor-leaves-console-as-fd0-in-children.md --- a/bugs/001-supervisor-leaves-console-as-fd0-in-children.md Mon Jul 20 19:49:45 2026 -0500 +++ /dev/null Thu Jan 01 00:00:00 1970 +0000 @@ -1,150 +0,0 @@ -> Note (2026-05-21): The `console-login.toml` referenced below is -> now in Hammerhead's overlay tree at -> `base/usr/src/zyginit/overrides/console-login/service.toml`. - -# Bug 001: supervisor leaves /dev/console as inherited fd 0 in service children - -## Summary - -`src/supervisor.reef`'s child-setup path redirects fd 1 and fd 2 to the -per-service log file but leaves fd 0 inherited from PID 1. PID 1 (zyginit) -runs `setup_pid1_console()` which opens `/dev/console` and dup2's the fd -onto 0/1/2 (with `O_NOCTTY`, correctly), so any service child inherits -fd 0 = `/dev/console`. The child then calls `setsid()`, becoming a session -leader with a `/dev/console` fd open. - -That state — session leader with a tty fd inherited from before the -session was created — appears to be enough on Hammerhead for the kernel -to associate `/dev/console` as the new session's controlling tty even -though no explicit `open()` (with or without `O_NOCTTY`) was made by -the child for that fd. Symptom: `ps -o tty` shows `TT=console` for the -service, and any later daemon trying to claim ctty on `/dev/console` -(typically `ttymon` started by the `console-login` service in a higher -tier) fails with **"cannot allocate controlling tty on /dev/console"** -and login(1) cannot proceed → `console-login` enters a restart loop and -the system is unreachable from the physical console. - -This was hit on hh-alpha1 immediately after first boot. SSH worked, -console login did not. - -## Severity - -**Medium.** Affects any deployment where the kernel auto-claims ctty -under the inherited-fd-0-then-setsid pattern. The triggering daemons -are kernel-version- and toolchain-build-dependent (we observed it with -a newly-built `syslogd` on hh-alpha1; an older syslogd on hh-prototest -does not trip the same kernel path). Recoverable via SSH but blocks -the physical console use case. - -## Affected Version - -zyginit 0.1.1 (as shipped in `zyginit-0.1.1-source.tar.xz`). - -The defect is in `src/supervisor.reef`, the post-fork child setup near -line 333 in the version we vendored. We did not check 0.1.0. - -## Reproducer - -1. Boot Hammerhead with zyginit as PID 1. -2. Have `setup_pid1_console()` open `/dev/console` and dup it onto 0/1/2. -3. Start any daemon that calls `setsid()` and continues running with - the inherited fd 0 in scope (e.g. `/usr/sbin/syslogd`). -4. Inspect: `ps -ef -o pid,sid,tty,comm | grep <daemon>`. - The TT column shows `console` even though the daemon never explicitly - opened `/dev/console` without `O_NOCTTY`. -5. Start `console-login` (or any service running `ttymon -d /dev/console`) - in a higher tier. ttymon prints the login prompt but `login(1)` then - fails to claim ctty: "cannot allocate controlling tty on /dev/console - — There may be another session active on this port." -6. zyginit restarts `console-login`; same failure repeats; restart loop. - -## Root cause (from the supervisor side) - -`src/supervisor.reef` post-fork child setup (snippet from the version -we vendored): - -```reef -if pid == 0 - // CHILD PROCESS - process.process_setsid() - - // Redirect stdout/stderr to per-service log file - let log_path = ... - let log_fd = fd.fd_open(log_path, log_flags, 420) - if log_fd >= 0 - fd.fd_dup2(log_fd, 1) - fd.fd_dup2(log_fd, 2) - fd.fd_close(log_fd) - end if - ... -``` - -Only fd 1 and fd 2 get redirected. fd 0 keeps PID 1's `/dev/console` -across the `setsid()`. From the kernel's perspective the new session is -born with an open fd referencing a tty that has no controlling session, -which under at least some illumos variants is enough to associate it -as the new session's ctty. - -## Suggested fix - -In the child-setup path, before `setsid()`, redirect fd 0 to `/dev/null` -(or to the per-service log file, or close it entirely). For example: - -```reef -if pid == 0 - // CHILD PROCESS - - // Redirect fd 0 to /dev/null BEFORE setsid so the new session is - // never born holding an inherited /dev/console fd. Without this, - // illumos may auto-claim /dev/console as the new session's ctty, - // which then blocks ttymon (in console-login) from claiming it. - let null_fd = fd.fd_open("/dev/null", fd.O_RDWR(), 0) - if null_fd >= 0 - fd.fd_dup2(null_fd, 0) - fd.fd_close(null_fd) - end if - - process.process_setsid() - - // existing fd 1/2 → log file redirection - ... -``` - -Optionally, add an explicit defensive `TIOCNOTTY` after `setsid()` — -open `/dev/tty` (which refers to the current ctty if any) and ioctl -`TIOCNOTTY` to revoke the association. This is a no-op when there's -no ctty, and a fix when one was unexpectedly inherited. - -The fd-0-to-/dev/null change alone is the cheaper of the two and -matches the contract most service authors expect: services that need -stdin go open it explicitly. - -## Workarounds while pending - -1. **Per-service `start.sh` wrapper**: prepend `exec </dev/null` to each - daemon's exec line. This redirects fd 0 to `/dev/null` from inside - the service script. Tested on alpha1 — did *not* eliminate the - ctty grab in our specific syslogd build, suggesting the kernel - path is more subtle than just fd-0-inherited (or our particular - syslogd has a separate code path). See companion Hammerhead-side - fix (TIOCNOTTY in `untty()`) which addressed our concrete symptom. - -2. **Reorder so console-login starts before any potential ctty-grabber**: - change the dependency in `services/hammerhead/console-login.toml` so - it runs in an earlier tier than syslogd. ttymon claims ctty first; - later daemons can't steal it. - -## Cross-references - -- `src/supervisor.reef` — post-fork child setup -- `src/main.reef` — `setup_pid1_console()` (correctly uses `O_NOCTTY`) -- Hammerhead bug filed in parallel: syslogd defensive `TIOCNOTTY` in - `untty()` to handle the kernel-side symptom from the cmd-side. That - fix masks the specific syslogd case but doesn't address the broader - class of any-daemon-may-trip-this — hence this RFE. - -## Reporter - -Hammerhead build team (Chris Tusa, christusa@gmail.com) -Hit on hh-alpha1 first boot, 2026-05-06. -Recovery: SSH still worked; console login was the only blocked path. diff -r c0018f136328 -r 71b8fd133c85 bugs/002-rfe-group-target-service-type.md --- a/bugs/002-rfe-group-target-service-type.md Mon Jul 20 19:49:45 2026 -0500 +++ /dev/null Thu Jan 01 00:00:00 1970 +0000 @@ -1,79 +0,0 @@ -# RFE 002: `type = "group"` service for grouping/targeting - -## Summary - -Add a `type = "group"` (or `"target"`) service kind to zyginit. A -group service has no `[exec]` block; it exists only as a named -synchronization/enable point. Its `[dependencies]` list expands into -the set of services it represents. - -This is the equivalent of systemd's `.target` units. It lets admins -enable/disable a feature set with one command without consolidating -unrelated daemons under a single supervisor. - -## Motivation - -Hammerhead is converting the NFS, IPsec, and zones SMF service sets -to zyginit. NFS in particular spreads across multiple independent -daemons (`nfsmapid`, `statd`, `lockd`, `nfs4cbd`, plus the `client` -mount oneshot) — each with its own lifecycle, restart policy, and -log file. - -The natural design is one zyginit service per daemon. But that loses -the SMF-era ergonomics of "enable nfs-client and the dependents come -along". Without a group abstraction we have two bad options: - -1. **Combine multiple daemons into one zyginit service.** Cluttered: - one contract becomes N contracts, restart semantics get fuzzy, - log routing needs splitting, and the supervisor's child-death - handling no longer maps 1:1 to a service. - -2. **Document the bundle and tell admins to enable each member.** - Error-prone and ugly: - ``` - zygctl enable nfs-cbd nfs-mapid nfs-statd nfs-lockd nfs-client - ``` - -A group service neatly avoids both: -```toml -[service] -name = "nfs-client" -description = "NFSv4 client services" -type = "group" -[dependencies] -requires = ["nfs-cbd", "nfs-mapid", "nfs-statd", "nfs-lockd", - "nfs-client-mount"] -``` -Then `zygctl enable nfs-client` is sufficient. `zygctl status nfs-client` -reports composite up/down derived from the members. - -## Suggested semantics - -- `type = "group"` services have no `[exec]` block. The supervisor - never forks anything for them. -- Status: `running` iff all `requires` members are `running`; - `stopped` iff none are; `partial` (or `degraded`) if some. -- `zygctl enable <group>` symlinks the group's TOML in `enabled.d/` - *and* recursively enables each member that is not already enabled - (or, simpler: refuses unless all members are also enabled — - consistent with how SMF dependencies block enable today). -- `zygctl disable <group>` is symmetric. -- `zygctl start <group>` / `stop <group>` operate on the member set. -- Cycle detection treats group-edges the same as service-edges. - -## Use cases - -| Group | Members | -|---|---| -| `nfs-client` | nfs-cbd, nfs-mapid, nfs-statd, nfs-lockd, nfs-client-mount | -| `nfs-server` | nfsd, mountd, nfs-statd, nfs-lockd | -| `zones` | zones, zonestat | -| `audit` | (just `auditd` after the auditset fold-in — not very useful for audit, but harmless) | -| `network-routing` | forwarding, routing/ndp, network-iptun | - -Hammerhead would land this in roughly 4–6 group definitions. - -## Reporter - -Hammerhead build team (Chris Tusa, christusa@gmail.com) -Filed during SMF-to-zyginit migration policy work, 2026-05-08. diff -r c0018f136328 -r 71b8fd133c85 bugs/archive/001-supervisor-leaves-console-as-fd0-in-children.md --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/bugs/archive/001-supervisor-leaves-console-as-fd0-in-children.md Mon Jul 20 21:07:33 2026 -0500 @@ -0,0 +1,150 @@ +> Note (2026-05-21): The `console-login.toml` referenced below is +> now in Hammerhead's overlay tree at +> `base/usr/src/zyginit/overrides/console-login/service.toml`. + +# Bug 001: supervisor leaves /dev/console as inherited fd 0 in service children + +## Summary + +`src/supervisor.reef`'s child-setup path redirects fd 1 and fd 2 to the +per-service log file but leaves fd 0 inherited from PID 1. PID 1 (zyginit) +runs `setup_pid1_console()` which opens `/dev/console` and dup2's the fd +onto 0/1/2 (with `O_NOCTTY`, correctly), so any service child inherits +fd 0 = `/dev/console`. The child then calls `setsid()`, becoming a session +leader with a `/dev/console` fd open. + +That state — session leader with a tty fd inherited from before the +session was created — appears to be enough on Hammerhead for the kernel +to associate `/dev/console` as the new session's controlling tty even +though no explicit `open()` (with or without `O_NOCTTY`) was made by +the child for that fd. Symptom: `ps -o tty` shows `TT=console` for the +service, and any later daemon trying to claim ctty on `/dev/console` +(typically `ttymon` started by the `console-login` service in a higher +tier) fails with **"cannot allocate controlling tty on /dev/console"** +and login(1) cannot proceed → `console-login` enters a restart loop and +the system is unreachable from the physical console. + +This was hit on hh-alpha1 immediately after first boot. SSH worked, +console login did not. + +## Severity + +**Medium.** Affects any deployment where the kernel auto-claims ctty +under the inherited-fd-0-then-setsid pattern. The triggering daemons +are kernel-version- and toolchain-build-dependent (we observed it with +a newly-built `syslogd` on hh-alpha1; an older syslogd on hh-prototest +does not trip the same kernel path). Recoverable via SSH but blocks +the physical console use case. + +## Affected Version + +zyginit 0.1.1 (as shipped in `zyginit-0.1.1-source.tar.xz`). + +The defect is in `src/supervisor.reef`, the post-fork child setup near +line 333 in the version we vendored. We did not check 0.1.0. + +## Reproducer + +1. Boot Hammerhead with zyginit as PID 1. +2. Have `setup_pid1_console()` open `/dev/console` and dup it onto 0/1/2. +3. Start any daemon that calls `setsid()` and continues running with + the inherited fd 0 in scope (e.g. `/usr/sbin/syslogd`). +4. Inspect: `ps -ef -o pid,sid,tty,comm | grep <daemon>`. + The TT column shows `console` even though the daemon never explicitly + opened `/dev/console` without `O_NOCTTY`. +5. Start `console-login` (or any service running `ttymon -d /dev/console`) + in a higher tier. ttymon prints the login prompt but `login(1)` then + fails to claim ctty: "cannot allocate controlling tty on /dev/console + — There may be another session active on this port." +6. zyginit restarts `console-login`; same failure repeats; restart loop. + +## Root cause (from the supervisor side) + +`src/supervisor.reef` post-fork child setup (snippet from the version +we vendored): + +```reef +if pid == 0 + // CHILD PROCESS + process.process_setsid() + + // Redirect stdout/stderr to per-service log file + let log_path = ... + let log_fd = fd.fd_open(log_path, log_flags, 420) + if log_fd >= 0 + fd.fd_dup2(log_fd, 1) + fd.fd_dup2(log_fd, 2) + fd.fd_close(log_fd) + end if + ... +``` + +Only fd 1 and fd 2 get redirected. fd 0 keeps PID 1's `/dev/console` +across the `setsid()`. From the kernel's perspective the new session is +born with an open fd referencing a tty that has no controlling session, +which under at least some illumos variants is enough to associate it +as the new session's ctty. + +## Suggested fix + +In the child-setup path, before `setsid()`, redirect fd 0 to `/dev/null` +(or to the per-service log file, or close it entirely). For example: + +```reef +if pid == 0 + // CHILD PROCESS + + // Redirect fd 0 to /dev/null BEFORE setsid so the new session is + // never born holding an inherited /dev/console fd. Without this, + // illumos may auto-claim /dev/console as the new session's ctty, + // which then blocks ttymon (in console-login) from claiming it. + let null_fd = fd.fd_open("/dev/null", fd.O_RDWR(), 0) + if null_fd >= 0 + fd.fd_dup2(null_fd, 0) + fd.fd_close(null_fd) + end if + + process.process_setsid() + + // existing fd 1/2 → log file redirection + ... +``` + +Optionally, add an explicit defensive `TIOCNOTTY` after `setsid()` — +open `/dev/tty` (which refers to the current ctty if any) and ioctl +`TIOCNOTTY` to revoke the association. This is a no-op when there's +no ctty, and a fix when one was unexpectedly inherited. + +The fd-0-to-/dev/null change alone is the cheaper of the two and +matches the contract most service authors expect: services that need +stdin go open it explicitly. + +## Workarounds while pending + +1. **Per-service `start.sh` wrapper**: prepend `exec </dev/null` to each + daemon's exec line. This redirects fd 0 to `/dev/null` from inside + the service script. Tested on alpha1 — did *not* eliminate the + ctty grab in our specific syslogd build, suggesting the kernel + path is more subtle than just fd-0-inherited (or our particular + syslogd has a separate code path). See companion Hammerhead-side + fix (TIOCNOTTY in `untty()`) which addressed our concrete symptom. + +2. **Reorder so console-login starts before any potential ctty-grabber**: + change the dependency in `services/hammerhead/console-login.toml` so + it runs in an earlier tier than syslogd. ttymon claims ctty first; + later daemons can't steal it. + +## Cross-references + +- `src/supervisor.reef` — post-fork child setup +- `src/main.reef` — `setup_pid1_console()` (correctly uses `O_NOCTTY`) +- Hammerhead bug filed in parallel: syslogd defensive `TIOCNOTTY` in + `untty()` to handle the kernel-side symptom from the cmd-side. That + fix masks the specific syslogd case but doesn't address the broader + class of any-daemon-may-trip-this — hence this RFE. + +## Reporter + +Hammerhead build team (Chris Tusa, christusa@gmail.com) +Hit on hh-alpha1 first boot, 2026-05-06. +Recovery: SSH still worked; console login was the only blocked path. diff -r c0018f136328 -r 71b8fd133c85 bugs/archive/002-rfe-group-target-service-type.md --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/bugs/archive/002-rfe-group-target-service-type.md Mon Jul 20 21:07:33 2026 -0500 @@ -0,0 +1,79 @@ +# RFE 002: `type = "group"` service for grouping/targeting + +## Summary + +Add a `type = "group"` (or `"target"`) service kind to zyginit. A +group service has no `[exec]` block; it exists only as a named +synchronization/enable point. Its `[dependencies]` list expands into +the set of services it represents. + +This is the equivalent of systemd's `.target` units. It lets admins +enable/disable a feature set with one command without consolidating +unrelated daemons under a single supervisor. + +## Motivation + +Hammerhead is converting the NFS, IPsec, and zones SMF service sets +to zyginit. NFS in particular spreads across multiple independent +daemons (`nfsmapid`, `statd`, `lockd`, `nfs4cbd`, plus the `client` +mount oneshot) — each with its own lifecycle, restart policy, and +log file. + +The natural design is one zyginit service per daemon. But that loses +the SMF-era ergonomics of "enable nfs-client and the dependents come +along". Without a group abstraction we have two bad options: + +1. **Combine multiple daemons into one zyginit service.** Cluttered: + one contract becomes N contracts, restart semantics get fuzzy, + log routing needs splitting, and the supervisor's child-death + handling no longer maps 1:1 to a service. + +2. **Document the bundle and tell admins to enable each member.** + Error-prone and ugly: + ``` + zygctl enable nfs-cbd nfs-mapid nfs-statd nfs-lockd nfs-client + ``` + +A group service neatly avoids both: +```toml +[service] +name = "nfs-client" +description = "NFSv4 client services" +type = "group" +[dependencies] +requires = ["nfs-cbd", "nfs-mapid", "nfs-statd", "nfs-lockd", + "nfs-client-mount"] +``` +Then `zygctl enable nfs-client` is sufficient. `zygctl status nfs-client` +reports composite up/down derived from the members. + +## Suggested semantics + +- `type = "group"` services have no `[exec]` block. The supervisor + never forks anything for them. +- Status: `running` iff all `requires` members are `running`; + `stopped` iff none are; `partial` (or `degraded`) if some. +- `zygctl enable <group>` symlinks the group's TOML in `enabled.d/` + *and* recursively enables each member that is not already enabled + (or, simpler: refuses unless all members are also enabled — + consistent with how SMF dependencies block enable today). +- `zygctl disable <group>` is symmetric. +- `zygctl start <group>` / `stop <group>` operate on the member set. +- Cycle detection treats group-edges the same as service-edges. + +## Use cases + +| Group | Members | +|---|---| +| `nfs-client` | nfs-cbd, nfs-mapid, nfs-statd, nfs-lockd, nfs-client-mount | +| `nfs-server` | nfsd, mountd, nfs-statd, nfs-lockd | +| `zones` | zones, zonestat | +| `audit` | (just `auditd` after the auditset fold-in — not very useful for audit, but harmless) | +| `network-routing` | forwarding, routing/ndp, network-iptun | + +Hammerhead would land this in roughly 4–6 group definitions. + +## Reporter + +Hammerhead build team (Chris Tusa, christusa@gmail.com) +Filed during SMF-to-zyginit migration policy work, 2026-05-08.