Contract Re-adoption (Live Replace) Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Spec: docs/superpowers/specs/2026-05-02-contract-readoption-design.md (rev 84).

Goal: Allow zygctl replace to upgrade the running zyginit PID-1 process via in-place execve("/sbin/init", ...) while preserving every running service across the swap, with state-file handoff for in-memory bookkeeping.

Architecture: New src/replace.reef module owns the state-file format, serialization, recovery, and pre-condition check. main.reef gains a replace_self proc (outgoing flow: drain, write state, unlink socket, exec) and an apply_recovered_state proc (incoming flow: stitch state.toml against current enabled.d/, register contract_map, idempotent post-recovery empty-contract check). socket.reef adds a replace [--wait=N] command. contract.reef adds an is_contract_empty helper. zygctl.reef adds the replace subcommand. No ct_ctl_adopt call is needed: PID-1 in-place execve preserves proc_t.p_ct_process so kernel-level contract ownership survives automatically; only userspace bookkeeping needs rebuilding.

Tech Stack: Reef 0.4.0+, Hammerhead libcontract, POSIX FFI, TOML (existing encoding.toml parser), Mercurial (hg) for VCS.

Conventions — pulled from CLAUDE.md / SRCHEADER.txt. All new source files MUST start with the SRCHEADER.txt template with ${project}=zyginit, ${file.name} = the filename, ${file.description} filled in. VCS is hg not git. License is CDDL-1.0. "Hammerhead" not "illumos" in code/comments. No emojis in source.

Build commands (used in test/verify steps throughout):

# Linux dev (stubs the contract bindings)
clang -c src/helpers.c -o build/helpers.o
clang -c src/contract_linux_stubs.c -o build/contract_linux_stubs.o
reefc build --obj build/helpers.o --obj build/contract_linux_stubs.o

# zygctl Linux
cd tools/zygctl
clang -c src/symlink_wrapper.c -o build/symlink_wrapper.o
reefc build --obj build/symlink_wrapper.o
cd ../..

File structure

File	Action	Responsibility
`src/helpers.c`	Modify	Add `zyginit_read_boot_time` C helper.
`src/replace.reef`	Create (~250 lines)	State-file format, serialize, recover, pre-condition check.
`src/contract.reef`	Modify (+~25 lines)	Add `is_contract_empty(ctid): bool` helper.
`src/contract_linux_stubs.c`	Modify (+5 lines)	Linux stub for `is_contract_empty`.
`src/socket.reef`	Modify (+~50 lines)	Add `replace [--wait=N]` command, flag, accessors.
`src/main.reef`	Modify (+~120 lines)	Add `apply_recovered_state`, `replace_self`, integrate into startup + event loop.
`tools/zygctl/src/zygctl.reef`	Modify (+~30 lines)	Add `replace [--wait=N]` subcommand.
`utils/replace_probe/`	Create	Standalone Reef program for state-file format unit tests, modeled on `utils/contract_probe/`.
`tests/integration/run_tests.sh`	Modify	Add Linux integration scenarios (smoke + pre-condition refusal).
`tests/integration/test_replace_hammerhead.md`	Create	Manual checklist for Hammerhead `hh-prototest` validation.
`man/zygctl.8`	Modify	Document `zygctl replace [--wait=N]`.
`man/zyginit.8`	Modify	Document state.toml file path + replace flow at a high level.

Task 1: Add `zyginit_read_boot_time` C helper

Files:

Modify: src/helpers.c (append a new function near the existing utmpx writers around line 150)

The helper reads BOOT_TIME from /var/adm/utmpx and returns its ut_tv.tv_sec as int, or -1 if not found / not readable. This becomes the boot-id sentinel that survives execve and lets the new zyginit detect a stale state.toml from before a real reboot.

Step 1: Read existing helpers.c around the BOOT_TIME writer to confirm placement and headers

Run: grep -n 'BOOT_TIME\|getutxent\|setutxent\|endutxent' src/helpers.c
Expected: see BOOT_TIME referenced in zyginit_write_boot_utmpx, header utmpx.h already included.

Step 2: Add the read helper after zyginit_write_runlvl_utmpx

Append to src/helpers.c:

/* Read the BOOT_TIME utmpx record's tv_sec value, used as the boot-id
 * sentinel for the live-replace state file. PID-1 in-place execve
 * preserves the same proc_t but the kernel does not preserve a "boot
 * id" anywhere obvious — utmpx BOOT_TIME survives because it lives in
 * /var/adm/utmpx, written by zyginit_write_boot_utmpx() at boot. A
 * real reboot writes a new BOOT_TIME with a different tv_sec, so
 * comparing against it lets the new zyginit detect a stale state file.
 *
 * Returns the BOOT_TIME tv_sec on success, or -1 if the file is not
 * readable, has no BOOT_TIME entry, or any utmpx call fails.
 */
int zyginit_read_boot_time(void) {
    struct utmpx *up;
    int result = -1;

    setutxent();
    while ((up = getutxent()) != NULL) {
        if (up->ut_type == BOOT_TIME) {
            result = (int) up->ut_tv.tv_sec;
            break;
        }
    }
    endutxent();
    return result;
}

Step 3: Build helpers.o on Linux to verify it compiles

Run: clang -c src/helpers.c -o build/helpers.o
Expected: no errors, no warnings. (utmpx.h is in libc; on Linux it's glibc's utmpx.)

Step 4: Commit

hg add src/helpers.c
hg commit -m "helpers: add zyginit_read_boot_time for live-replace boot-id check"

(Note: hg add is a no-op for already-tracked files, but harmless.)

Task 2: Skeleton `src/replace.reef` module

Files:

Create: src/replace.reef

Empty-bodied module with SRCHEADER, FFI extern, and the export block that subsequent tasks will fill in. Establishing the surface up front lets later tasks be small and incremental.

Step 1: Create the file with header and module declaration

Write src/replace.reef:

/******************************************************************************
                __               ____                __
               / /   ___  ____ _/ __/_____________ _/ /__
              / /   / _ \/ __ `/ /_/ ___/ ___/ __ `/ / _ \
             / /___/  __/ /_/ / __(__  ) /__/ /_/ / /  __/
            /_____/\___/\__,_/_/ /____/\___/\__,_/_/\___/

    (C)opyright 2025-2026, Leafscale, LLC -  https://www.leafscale.com

    Project: zyginit
   Filename: replace.reef
    Authors: Chris Tusa <chris.tusa@leafscale.com>
    License: <see LICENSE file included with this source code>
Description: Live-replace state-file handoff (operator-driven PID-1 upgrade)

******************************************************************************/

// Live-replace operator flow: zygctl replace -> old zyginit serializes
// runtime state to /var/run/zyginit/state.toml, execs /sbin/init in
// place; new zyginit reads state.toml, validates boot_time matches
// current utmpx BOOT_TIME (else stale across a real reboot), and
// rebuilds the contract_map / runtime fields in supervisor.ServiceTable.
//
// PID-1 in-place execve preserves proc_t.p_ct_process at the kernel
// level, so contract ownership survives automatically — no
// ct_ctl_adopt call is needed. This module owns userspace bookkeeping.

module replace

import config
import supervisor
import io.file
import sys.fd as fd
import sys.env as sysenv
import core.str
import encoding.toml
import time.time as time

export
    type RecoveredService
    type RecoveredState

    // Accessors — RecoveredState
    fn rs_count(s: RecoveredState): int
    fn rs_service(s: RecoveredState, idx: int): RecoveredService
    fn rs_boot_time(s: RecoveredState): int
    fn rs_is_empty(s: RecoveredState): bool

    // Accessors — RecoveredService
    fn svc_name(rs: RecoveredService): string
    fn svc_contract_id(rs: RecoveredService): int
    fn svc_restart_count(rs: RecoveredService): int
    fn svc_last_start_time(rs: RecoveredService): int
    fn svc_last_exit_code(rs: RecoveredService): int

    // Outgoing flow — write state, return true on success
    fn serialize_state(table: supervisor.ServiceTable, boot_time: int): bool

    // Incoming flow — read state file. Returns an empty RecoveredState
    // if the file is missing, corrupt, or has a mismatched boot_time.
    fn recover_state(current_boot_time: int): RecoveredState

    // Pre-condition check. Returns "" if all services are stable, else
    // a comma-separated "name:state" list suitable for logging.
    fn check_preconditions(table: supervisor.ServiceTable): string

    // FFI for boot-time sentinel (helpers.c)
    fn read_boot_time(): int

    // Constants
    fn STATE_FILE_PATH(): string
    fn STATE_TMP_PATH(): string
    fn STATE_DIR(): string
end export

// FFI — implemented in helpers.c
extern "C" fn zyginit_read_boot_time(): int
extern "C" fn zyginit_fsync_path(path: string): int

// FFI — POSIX
extern "C" fn open(path: string, flags: int, mode: int): int
extern "C" fn close(fd: int): int
extern "C" fn rename(oldpath: string, newpath: string): int
extern "C" fn unlink(path: string): int

// Reads utmpx BOOT_TIME via the C helper. Returns -1 if unavailable.
fn read_boot_time(): int
    return zyginit_read_boot_time()
end read_boot_time

// File-system layout. Keep grouped here so a future tmpfs migration
// only edits one place.
fn STATE_DIR(): string
    return sysenv.get_env_or("ZYGINIT_RUN_DIR", "/var/run/zyginit")
end STATE_DIR

fn STATE_FILE_PATH(): string
    return str.concat(STATE_DIR(), "/state.toml")
end STATE_FILE_PATH

fn STATE_TMP_PATH(): string
    return str.concat(STATE_DIR(), "/state.toml.tmp")
end STATE_TMP_PATH

end module

Step 2: Add to reef.toml if it lists modules explicitly (it doesn't currently — source_dirs = ["src"]); confirm with build

Run: clang -c src/helpers.c -o build/helpers.o && clang -c src/contract_linux_stubs.c -o build/contract_linux_stubs.o && reefc build --obj build/helpers.o --obj build/contract_linux_stubs.o
Expected: build fails with errors about missing function bodies (rs_count, serialize_state, etc. are exported but not yet implemented). This is expected — subsequent tasks fill them in.

If reefc complains about exporting types/fns without bodies in the same module, comment out the export block temporarily by wrapping it in /* ... */ (Reef does not support /* */ comments — use line // per item) until enough is in place. The skeleton below in Task 3+ will populate types and fn bodies in the order needed to keep the build happy.

If reefc allows undefined exports as forward declarations, the build will fail on the first reference site instead — that's fine, fix it as the next task adds bodies.

Step 3: Commit even though build is incomplete

hg add src/replace.reef
hg commit -m "replace: skeleton module with FFI declarations and exports"

Task 3: Define `RecoveredService` and `RecoveredState` types

Files:

Modify: src/replace.reef

Add the types and trivial accessors. After this task the module compiles even with stub bodies for the larger fns.

Step 1: Add type definitions before the end module at the bottom of src/replace.reef

// ============================================================================
// Types
// ============================================================================

type RecoveredService = struct
    name: string
    contract_id: int
    restart_count: int
    last_start_time: int
    last_exit_code: int
end RecoveredService

type RecoveredState = struct
    boot_time: int
    services: [RecoveredService]
    count: int
end RecoveredState

fn new_recovered_state(): RecoveredState
    return RecoveredState{
        boot_time: 0 - 1,
        services: new [RecoveredService](0),
        count: 0
    }
end new_recovered_state

// ============================================================================
// Accessors
// ============================================================================

fn rs_count(s: RecoveredState): int
    return s.count
end rs_count

fn rs_service(s: RecoveredState, idx: int): RecoveredService
    return s.services[idx]
end rs_service

fn rs_boot_time(s: RecoveredState): int
    return s.boot_time
end rs_boot_time

fn rs_is_empty(s: RecoveredState): bool
    return s.count == 0
end rs_is_empty

fn svc_name(rs: RecoveredService): string
    return rs.name
end svc_name

fn svc_contract_id(rs: RecoveredService): int
    return rs.contract_id
end svc_contract_id

fn svc_restart_count(rs: RecoveredService): int
    return rs.restart_count
end svc_restart_count

fn svc_last_start_time(rs: RecoveredService): int
    return rs.last_start_time
end svc_last_start_time

fn svc_last_exit_code(rs: RecoveredService): int
    return rs.last_exit_code
end svc_last_exit_code

Step 2: Add stub bodies for the three larger fns so the module compiles. They are filled in by Tasks 4, 5, and 6.

// ============================================================================
// Outgoing — stub (filled by Task 4)
// ============================================================================

fn serialize_state(table: supervisor.ServiceTable, boot_time: int): bool
    return false
end serialize_state

// ============================================================================
// Incoming — stub (filled by Task 5)
// ============================================================================

fn recover_state(current_boot_time: int): RecoveredState
    return new_recovered_state()
end recover_state

// ============================================================================
// Pre-condition check — stub (filled by Task 6)
// ============================================================================

fn check_preconditions(table: supervisor.ServiceTable): string
    return ""
end check_preconditions

Step 3: Build

Run: clang -c src/helpers.c -o build/helpers.o && clang -c src/contract_linux_stubs.c -o build/contract_linux_stubs.o && reefc build --obj build/helpers.o --obj build/contract_linux_stubs.o
Expected: build succeeds. zyginit binary produced at build/zyginit. The replace module is compiled but unused.

Step 4: Commit

hg commit -m "replace: types, accessors, and stub bodies for serialize/recover/check"

Task 4: Implement `serialize_state` (TDD via `utils/replace_probe`)

Files:

Create: utils/replace_probe/reef.toml
Create: utils/replace_probe/src/main.reef
Modify: src/replace.reef (replace the stub serialize_state body)

Round-trip is verified in Task 5 (after recover_state is implemented). For now, the probe just calls serialize_state against a synthetic table and dumps the resulting file to stdout for visual inspection.

Step 1: Scaffold utils/replace_probe/

Create utils/replace_probe/reef.toml:

[package]
name = "replace_probe"
version = "0.1.0"
author = "Chris Tusa <chris.tusa@leafscale.com>"
description = "Unit-test probe for zyginit live-replace state-file format"
license = "CDDL-1.0"

[build]
entry = "src/main.reef"
output = "replace_probe"
output_dir = "build"
source_dirs = ["src", "../../src"]

# This probe round-trips state.toml serialization without needing a
# running zyginit. Models utils/contract_probe.
#
# Linux: clang -c ../../src/helpers.c -o build/helpers.o && \
#        clang -c ../../src/contract_linux_stubs.c -o build/contract_linux_stubs.o && \
#        reefc build --obj build/helpers.o --obj build/contract_linux_stubs.o

Create utils/replace_probe/src/main.reef (test harness — exercises serialize and, in Task 5, also recover):

/******************************************************************************
                __               ____                __
               / /   ___  ____ _/ __/_____________ _/ /__
              / /   / _ \/ __ `/ /_/ ___/ ___/ __ `/ / _ \
             / /___/  __/ /_/ / __(__  ) /__/ /_/ / /  __/
            /_____/\___/\__,_/_/ /____/\___/\__,_/_/\___/

    (C)opyright 2025-2026, Leafscale, LLC -  https://www.leafscale.com

    Project: zyginit
   Filename: main.reef
    Authors: Chris Tusa <chris.tusa@leafscale.com>
    License: <see LICENSE file included with this source code>
Description: Probe — round-trip the state.toml format through replace.reef

******************************************************************************/

import config
import supervisor
import replace
import core.str
import sys.env as sysenv

mut g_pass: int = 0
mut g_fail: int = 0

proc check(label: string, ok: bool)
    if ok
        g_pass = g_pass + 1
        println("  PASS: " + label)
    else
        g_fail = g_fail + 1
        println("  FAIL: " + label)
    end if
end check

// Build a synthetic ServiceTable with known data. Uses minimal
// ServiceDef fields — enough for serialize/recover round-trip.
fn build_synthetic_table(): supervisor.ServiceTable
    let table = supervisor.new_service_table(8)

    // sshd: RUNNING with restart_count=2 and last_exit_code=0
    let def_sshd = config.new_service_def_minimal("sshd")
    let idx_sshd = supervisor.add_service(table, def_sshd)
    supervisor.set_runtime_for_test(table, idx_sshd, supervisor.STATE_RUNNING(),
        100, 23, 2, 0, 1746204051)

    // syslogd: RUNNING with no restarts
    let def_syslog = config.new_service_def_minimal("syslogd")
    let idx_syslog = supervisor.add_service(table, def_syslog)
    supervisor.set_runtime_for_test(table, idx_syslog, supervisor.STATE_RUNNING(),
        101, 17, 0, 0 - 1, 1746204047)

    // cron: STOPPED — must NOT be serialized
    let def_cron = config.new_service_def_minimal("cron")
    let idx_cron = supervisor.add_service(table, def_cron)
    supervisor.set_runtime_for_test(table, idx_cron, supervisor.STATE_STOPPED(),
        0 - 1, 0 - 1, 0, 0, 0)

    return table
end build_synthetic_table

proc test_serialize()
    println("== serialize_state ==")

    // Use $TMPDIR to avoid touching /var/run during dev tests.
    let tmpdir = sysenv.get_env_or("TMPDIR", "/tmp")
    sysenv.set_env("ZYGINIT_RUN_DIR", str.concat(tmpdir, "/replace_probe"))

    let table = build_synthetic_table()
    let ok = replace.serialize_state(table, 1746204000)
    check("serialize returns true on writable dir", ok)

    // (Round-trip assertions added in Task 5.)
end test_serialize

proc main()
    test_serialize()

    println("")
    println("--- " + int_to_str(g_pass) + " passed, " + int_to_str(g_fail) + " failed ---")
    if g_fail > 0
        sysenv.exit(1)
    end if
end main

fn int_to_str(n: int): string
    if n == 0
        return "0"
    end if
    mut value = n
    mut neg = false
    if n < 0
        value = 0 - n
        neg = true
    end if
    mut result = ""
    while value > 0
        let digit = value % 10
        result = str.concat(str.substring("0123456789", digit, 1), result)
        value = value / 10
    end while
    if neg
        result = str.concat("-", result)
    end if
    return result
end int_to_str

The probe references three new helpers in supervisor and config that don't exist yet:

config.new_service_def_minimal(name): ServiceDef — constructs a ServiceDef with name set and other fields at default. Add it as a tiny #[cfg(test)]-equivalent helper at the bottom of config.reef's export block, but unconditionally exported (Reef has no cfg-test gate).
supervisor.set_runtime_for_test(table, idx, state, pid, ctid, restarts, last_exit, last_start) — directly writes the runtime fields on a ServiceRuntime, bypassing the normal start/stop lifecycle. Add to supervisor.reef's export block.
Step 2: Add config.new_service_def_minimal to src/config.reef

Locate the export block in src/config.reef, add the line fn new_service_def_minimal(name: string): ServiceDef to it. Find the place where ServiceDef is constructed (search for ServiceDef{) and add this fn body near it:

// Test/probe helper — minimal ServiceDef used by utils/replace_probe.
// Not used by production zyginit code paths.
fn new_service_def_minimal(name: string): ServiceDef
    return ServiceDef{
        name: name,
        description: "",
        type_: SERVICE_TYPE_DAEMON(),
        start_cmd: "/bin/true",
        stop_cmd: "",
        working_dir: "",
        user: "",
        group: "",
        environment: new [string](0),
        environment_count: 0,
        stop_method: STOP_METHOD_CONTRACT(),
        stop_signal: "TERM",
        stop_timeout: 60,
        requires: new [string](0),
        requires_count: 0,
        after: new [string](0),
        after_count: 0,
        contract_params: 0,
        contract_fatal: 0,
        restart_policy: RESTART_FAILURE(),
        restart_delay: 5,
        max_retries: 0 - 1,
        runlevel_mode: RUNLEVEL_MODE_MULTI()
    }
end new_service_def_minimal

Field names and constants in this constructor must match the actual ServiceDef struct in config.reef. Open the file, find type ServiceDef = struct, and copy the field list verbatim — adjust the constructor above if names differ. (Some fields may not exist in the struct as written here; that's OK — adapt to the real fields.)

Step 3: Add supervisor.set_runtime_for_test to src/supervisor.reef

Add to the export block: proc set_runtime_for_test(table: ServiceTable, idx: int, state: int, pid: int, contract_id: int, restart_count: int, last_exit_code: int, last_start_time: int). Then add the body near the other accessors:

// Test/probe helper — directly write runtime fields on a ServiceRuntime,
// bypassing the normal start/stop lifecycle. Used only by
// utils/replace_probe; production code goes through start_service /
// handle_contract_event.
proc set_runtime_for_test(table: ServiceTable, idx: int, state: int, pid: int,
                          contract_id: int, restart_count: int,
                          last_exit_code: int, last_start_time: int)
    let rt = table.runtimes[idx]
    rt.state = state
    rt.pid = pid
    rt.contract_id = contract_id
    rt.restart_count = restart_count
    rt.last_exit_code = last_exit_code
    rt.last_start_time = last_start_time
    table.runtimes[idx] = rt

    // Also register contract mapping so apply_recovered_state can find
    // it later if the test exercises recovery.
    if contract_id >= 0
        table.contract_map.set(int_to_str_helper(contract_id), idx)
    end if
end set_runtime_for_test

// Local helper — int_to_str is private in this module; if not already
// present scan for it. (It exists at line ~870 as a private fn — reuse.)
fn int_to_str_helper(n: int): string
    return int_to_str(n)
end int_to_str_helper

Step 4: Implement serialize_state in src/replace.reef

Replace the stub. The body opens a tmp file, writes the boot_time line then a [[service]] block per RUNNING service, fsyncs, closes, renames to the final path, fsyncs the parent dir.

fn serialize_state(table: supervisor.ServiceTable, boot_time: int): bool
    let tmp_path = STATE_TMP_PATH()
    let final_path = STATE_FILE_PATH()
    let dir_path = STATE_DIR()

    // O_WRONLY|O_CREAT|O_TRUNC = 0x301 on Hammerhead/Linux for the values
    // we use in fd.O_*. Using fd module's helpers keeps this portable.
    let flags = fd.O_WRONLY() + fd.O_CREAT() + fd.O_TRUNC()
    let out = fd.fd_open(tmp_path, flags, 384)   // mode 0600

    if out < 0
        println("replace: serialize: cannot open " + tmp_path)
        return false
    end if

    // Write boot_time header
    let header = str.concat("boot_time = ", int_to_str(boot_time))
    fd.fd_write(out, str.concat(header, "\n"))

    // Iterate services — only RUNNING ones go in. Skip the rest.
    let count = supervisor.service_count(table)
    mut i = 0
    while i < count
        let rt = supervisor.get_runtime(table, i)
        if supervisor.rt_state(rt) == supervisor.STATE_RUNNING()
            let def = supervisor.rt_def(rt)
            fd.fd_write(out, "\n[[service]]\n")
            fd.fd_write(out, str.concat("name = \"", str.concat(config.svc_name(def), "\"\n")))
            fd.fd_write(out, str.concat("contract_id = ", str.concat(int_to_str(supervisor.rt_contract_id(rt)), "\n")))
            fd.fd_write(out, str.concat("restart_count = ", str.concat(int_to_str(supervisor.rt_restart_count(rt)), "\n")))
            fd.fd_write(out, str.concat("last_start_time = ", str.concat(int_to_str(supervisor.rt_last_start_time(rt)), "\n")))
            fd.fd_write(out, str.concat("last_exit_code = ", str.concat(int_to_str(supervisor.rt_last_exit_code(rt)), "\n")))
        end if
        i = i + 1
    end while

    // fsync + close
    if fd.fd_fsync(out) < 0
        fd.fd_close(out)
        unlink(tmp_path)
        return false
    end if
    fd.fd_close(out)

    // Atomic rename
    if rename(tmp_path, final_path) != 0
        unlink(tmp_path)
        return false
    end if

    // fsync the parent dir so the rename is durable
    let _ = zyginit_fsync_path(dir_path)

    return true
end serialize_state

// Local int_to_str — same shape as in supervisor.reef. Replicate to
// keep the module standalone.
fn int_to_str(n: int): string
    if n == 0
        return "0"
    end if
    mut value = n
    mut neg = false
    if n < 0
        value = 0 - n
        neg = true
    end if
    mut result = ""
    while value > 0
        let digit = value % 10
        result = str.concat(str.substring("0123456789", digit, 1), result)
        value = value / 10
    end while
    if neg
        result = str.concat("-", result)
    end if
    return result
end int_to_str

The fd.fd_fsync call may not exist if Reef's sys.fd doesn't expose it — if reefc errors on that name, replace with a direct extern "C" fn fsync(fd: int): int declaration in this module and call it as fsync(out).

Step 5: Build the main project to verify replace.reef compiles

Step 6: Build the probe

Run:

cd utils/replace_probe
mkdir -p build
clang -c ../../src/helpers.c -o build/helpers.o
clang -c ../../src/contract_linux_stubs.c -o build/contract_linux_stubs.o
reefc build --obj build/helpers.o --obj build/contract_linux_stubs.o
cd ../..

Expected: success.

Step 7: Run the probe and inspect the file

Run:

mkdir -p /tmp/replace_probe
./utils/replace_probe/build/replace_probe
cat /tmp/replace_probe/state.toml

Expected:

"PASS: serialize returns true on writable dir"

File contents:

boot_time = 1746204000

[[service]]
name = "sshd"
contract_id = 23
restart_count = 2
last_start_time = 1746204051
last_exit_code = 0

[[service]]
name = "syslogd"
contract_id = 17
restart_count = 0
last_start_time = 1746204047
last_exit_code = -1

cron (STATE_STOPPED) is not in the output.
Step 8: Commit

hg add utils/replace_probe/reef.toml utils/replace_probe/src/main.reef
hg commit -m "replace: implement serialize_state + replace_probe round-trip harness"

Task 5: Implement `recover_state` (TDD via `utils/replace_probe`)

Files:

Modify: src/replace.reef (replace the stub recover_state body)
Modify: utils/replace_probe/src/main.reef (add round-trip + edge-case assertions)

recover_state reads the state.toml file via the existing encoding.toml parser, validates boot_time, and returns either an empty RecoveredState (if file missing/corrupt/stale) or a populated one. The empty/stale path also deletes the file.

Step 1: Implement recover_state

Replace the stub in src/replace.reef:

fn recover_state(current_boot_time: int): RecoveredState
    let path = STATE_FILE_PATH()

    if not file.fileExists(path)
        return new_recovered_state()
    end if

    let content = file.readFile(path)
    if str.length(content) == 0
        // Treat empty as missing
        let _ = unlink(path)
        return new_recovered_state()
    end if

    // Parse via encoding.toml. Section-array entries flatten to
    // dotted-key arrays; the parser exposes them via toml.get_array.
    // The helper below wraps a parse + extraction.
    let parsed = toml.parse(content)
    if not toml.parse_ok(parsed)
        println("replace: recover: parse error in " + path + " — ignoring")
        let _ = unlink(path)
        return new_recovered_state()
    end if

    let saved_boot = toml.get_int_or(parsed, "boot_time", 0 - 1)
    if saved_boot != current_boot_time or current_boot_time < 0
        println("replace: recover: stale state file (boot_time " + int_to_str(saved_boot) +
                " != current " + int_to_str(current_boot_time) + "); ignoring")
        let _ = unlink(path)
        return new_recovered_state()
    end if

    let n = toml.array_count(parsed, "service")
    let arr = new [RecoveredService](n)
    mut i = 0
    while i < n
        let prefix = str.concat("service.", int_to_str(i))
        arr[i] = RecoveredService{
            name: toml.get_string_or(parsed, str.concat(prefix, ".name"), ""),
            contract_id: toml.get_int_or(parsed, str.concat(prefix, ".contract_id"), 0 - 1),
            restart_count: toml.get_int_or(parsed, str.concat(prefix, ".restart_count"), 0),
            last_start_time: toml.get_int_or(parsed, str.concat(prefix, ".last_start_time"), 0),
            last_exit_code: toml.get_int_or(parsed, str.concat(prefix, ".last_exit_code"), 0 - 1)
        }
        i = i + 1
    end while

    // State file consumed — delete to avoid panic-restart loops.
    let _ = unlink(path)

    return RecoveredState{
        boot_time: saved_boot,
        services: arr,
        count: n
    }
end recover_state

The toml.parse_ok, toml.get_int_or, toml.get_string_or, toml.array_count helper names are placeholders — open ~/repos/reef-lang/stdlib/encoding/toml.reef (or wherever the project's encoding.toml lives — try grep -rn 'module toml' ~/repos/reef-lang/) and use the actual function names. The existing config.load_enabled_services in src/config.reef is a working example of how to walk a parsed TOML array — copy its access pattern verbatim if the helper names above don't exist.

Step 2: Extend the probe with round-trip + edge-case assertions

Replace test_serialize() in utils/replace_probe/src/main.reef with:

proc test_round_trip()
    println("== round-trip ==")
    let tmpdir = sysenv.get_env_or("TMPDIR", "/tmp")
    sysenv.set_env("ZYGINIT_RUN_DIR", str.concat(tmpdir, "/replace_probe"))

    // Make sure the state dir exists
    let _ = io.dir.create_dir_all(str.concat(tmpdir, "/replace_probe"))

    let table = build_synthetic_table()
    let serialized = replace.serialize_state(table, 1746204000)
    check("serialize returns true", serialized)

    let recovered = replace.recover_state(1746204000)
    check("recover boot_time matches", replace.rs_boot_time(recovered) == 1746204000)
    check("recover count == 2 (running only)", replace.rs_count(recovered) == 2)

    // Find services by name (order is implementation-defined)
    mut found_sshd = false
    mut found_syslog = false
    mut i = 0
    while i < replace.rs_count(recovered)
        let svc = replace.rs_service(recovered, i)
        let name = replace.svc_name(svc)
        if name == "sshd"
            found_sshd = true
            check("sshd contract_id = 23", replace.svc_contract_id(svc) == 23)
            check("sshd restart_count = 2", replace.svc_restart_count(svc) == 2)
            check("sshd last_exit_code = 0", replace.svc_last_exit_code(svc) == 0)
        elif name == "syslogd"
            found_syslog = true
            check("syslogd contract_id = 17", replace.svc_contract_id(svc) == 17)
            check("syslogd last_exit_code = -1", replace.svc_last_exit_code(svc) == 0 - 1)
        end if
        i = i + 1
    end while
    check("found sshd", found_sshd)
    check("found syslogd", found_syslog)

    // After recover_state, state.toml is deleted
    check("state.toml deleted after recovery", not io.file.fileExists(replace.STATE_FILE_PATH()))
end test_round_trip

proc test_stale_boot_time()
    println("== stale state file ==")
    let tmpdir = sysenv.get_env_or("TMPDIR", "/tmp")
    sysenv.set_env("ZYGINIT_RUN_DIR", str.concat(tmpdir, "/replace_probe"))

    let table = build_synthetic_table()
    let _ = replace.serialize_state(table, 1746204000)

    // Recover with DIFFERENT boot_time -> stale -> empty + deleted
    let recovered = replace.recover_state(1746999999)
    check("stale file -> empty state", replace.rs_is_empty(recovered))
    check("stale file deleted", not io.file.fileExists(replace.STATE_FILE_PATH()))
end test_stale_boot_time

proc test_missing_file()
    println("== missing state file ==")
    let tmpdir = sysenv.get_env_or("TMPDIR", "/tmp")
    sysenv.set_env("ZYGINIT_RUN_DIR", str.concat(tmpdir, "/replace_probe_missing"))

    // Make sure file definitely doesn't exist
    let _ = io.dir.create_dir_all(str.concat(tmpdir, "/replace_probe_missing"))
    let _ = unlink(replace.STATE_FILE_PATH())

    let recovered = replace.recover_state(1746204000)
    check("missing file -> empty state", replace.rs_is_empty(recovered))
end test_missing_file

extern "C" fn unlink(path: string): int

proc main()
    test_round_trip()
    test_stale_boot_time()
    test_missing_file()

    println("")
    println("--- " + int_to_str(g_pass) + " passed, " + int_to_str(g_fail) + " failed ---")
    if g_fail > 0
        sysenv.exit(1)
    end if
end main

Add import io.dir and import io.file near the top of the probe.

Step 3: Build and run

Run:

cd utils/replace_probe
reefc build --obj build/helpers.o --obj build/contract_linux_stubs.o
./build/replace_probe
cd ../..

Expected output ends with --- 11 passed, 0 failed --- (1 from serialize + 10 from round_trip / stale / missing — adjust if asserts differ).

If it fails, the most likely cause is the TOML parser helper names. Read encoding.toml's actual API and adjust recover_state.

Step 4: Commit

hg commit -m "replace: implement recover_state with round-trip + edge-case probe tests"

Task 6: Implement `check_preconditions` (TDD)

Files:

Modify: src/replace.reef (replace the stub check_preconditions)
Modify: utils/replace_probe/src/main.reef (add precondition tests)
Step 1: Implement

Replace the stub:

fn check_preconditions(table: supervisor.ServiceTable): string
    let count = supervisor.service_count(table)
    mut report = ""
    mut found = 0
    mut i = 0
    while i < count
        let rt = supervisor.get_runtime(table, i)
        let st = supervisor.rt_state(rt)
        if st == supervisor.STATE_STARTING() or st == supervisor.STATE_STOPPING() or st == supervisor.STATE_WAITING()
            let entry = str.concat(config.svc_name(supervisor.rt_def(rt)),
                str.concat(":", supervisor.state_name(st)))
            if found > 0
                report = str.concat(report, ", ")
            end if
            report = str.concat(report, entry)
            found = found + 1
        end if
        i = i + 1
    end while
    return report
end check_preconditions

Note: STATE_WAITING is currently used both for "not yet started" services at boot and for "scheduled-for-restart". For the precondition check we want only the latter (a transient mid-restart state) — but the state machine doesn't distinguish them. Conservative: treat all WAITING as transient. Operators using replace on a fresh boot before services have started would otherwise see "blocked"; that's an acceptable false positive.

Step 2: Add probe assertions

Append to utils/replace_probe/src/main.reef:

proc test_preconditions()
    println("== preconditions ==")
    let table = supervisor.new_service_table(8)

    // sshd RUNNING — stable
    let def_sshd = config.new_service_def_minimal("sshd")
    let idx_sshd = supervisor.add_service(table, def_sshd)
    supervisor.set_runtime_for_test(table, idx_sshd, supervisor.STATE_RUNNING(),
        100, 23, 0, 0 - 1, 1746204051)

    let r1 = replace.check_preconditions(table)
    check("all-RUNNING -> empty report", str.length(r1) == 0)

    // Add a STOPPING service
    let def_cron = config.new_service_def_minimal("cron")
    let idx_cron = supervisor.add_service(table, def_cron)
    supervisor.set_runtime_for_test(table, idx_cron, supervisor.STATE_STOPPING(),
        101, 24, 0, 0 - 1, 1746204060)

    let r2 = replace.check_preconditions(table)
    check("STOPPING -> non-empty report", str.length(r2) > 0)
    check("report mentions cron", str.index_of(r2, "cron") >= 0)
    check("report mentions stopping", str.index_of(r2, "stopping") >= 0)
end test_preconditions

Add test_preconditions() to proc main().

Step 3: Build and run

Run:

cd utils/replace_probe
reefc build --obj build/helpers.o --obj build/contract_linux_stubs.o
./build/replace_probe
cd ../..

Expected: tests pass.

Step 4: Commit

hg commit -m "replace: implement check_preconditions with transient-state detection"

Task 7: Add `contract.is_contract_empty` helper

Files:

Modify: src/contract.reef
Modify: src/contract_linux_stubs.c

is_contract_empty(ctid) reads contract status via ct_status_read on /system/contract/process/<id>/status, then checks the member-count via ct_status_get_nmembers. Returns true if the contract has zero members (or the status read fails — caller treats unknown as "assume empty for safety").

Step 1: Add the FFI for nmembers and the helper

In src/contract.reef, add to the FFI declarations near the other ct_status_* lines:

extern "C" fn ct_status_get_nmembers(stathdl: pointer): int

Add to the export block:

fn is_contract_empty(contract_id: int): bool

Add the body near adopt_contract:

// Check whether a contract has zero member processes. Used by the
// post-recovery idempotency step in main.apply_recovered_state — if a
// contract was reported in state.toml but the kernel says it's empty,
// the service exited during the exec gap and we should apply restart
// policy as if a contract-empty event arrived.
//
// Returns true if the contract has zero members OR if the status
// read fails (treat unknown as empty so the recovery path applies
// restart policy rather than leaving a phantom service in RUNNING).
fn is_contract_empty(contract_id: int): bool
    let status_path = str.concat(str.concat("/system/contract/process/", int_to_str(contract_id)), "/status")
    let st_fd = open(status_path, O_RDONLY())
    if st_fd < 0
        return true
    end if

    unsafe
        let stathdl_buf = new [pointer](1)
        let rc = ct_status_read(st_fd, CTD_COMMON(), stathdl_buf)
        close(st_fd)

        if rc != 0
            return true
        end if

        let stathdl = stathdl_buf[0]
        let nmembers = ct_status_get_nmembers(stathdl)
        ct_status_free(stathdl)

        return nmembers == 0
    end unsafe
end is_contract_empty

Step 2: Add the Linux stub

In src/contract_linux_stubs.c, append:

int ct_status_get_nmembers(void *stathdl) { (void)stathdl; return 0; }

Step 3: Build

Step 4: Commit

hg commit -m "contract: add is_contract_empty helper for post-recovery empty-contract check"

Task 8: Add socket `replace` command + flag plumbing

Files:

Modify: src/socket.reef

Add a g_replace_requested flag and a g_replace_wait int, accessors, and the command handler. Mirror the existing g_reload_requested pattern.

Step 1: Add accessors to the export block

In src/socket.reef, add to the existing export block (near reload_requested line 37):

    fn replace_requested(): bool
    proc clear_replace_flag()
    fn replace_wait_seconds(): int

Step 2: Add the globals and accessors

Near the existing mut g_reload_requested: bool = false declaration (around line 49), add:

// Replace flag — set by "replace" command, checked by main event loop
mut g_replace_requested: bool = false
mut g_replace_wait: int = 0

fn replace_requested(): bool
    return g_replace_requested
end replace_requested

proc clear_replace_flag()
    g_replace_requested = false
    g_replace_wait = 0
end clear_replace_flag

fn replace_wait_seconds(): int
    return g_replace_wait
end replace_wait_seconds

Step 3: Add the command handler in dispatch_command

Insert in dispatch_command (around line 184, after the reload branch):

    elif command == "replace"
        // Optional --wait=N argument
        mut wait_seconds = 0
        if str.length(arg) > 0
            // Parse --wait=N
            if str.starts_with(arg, "--wait=")
                let val = str.substring(arg, 7, str.length(arg) - 7)
                wait_seconds = parse_int_or(val, 0)
            else
                return "error: replace: unknown argument: " + arg + "\n"
            end if
        end if
        g_replace_requested = true
        g_replace_wait = wait_seconds
        return str.concat("replace queued (wait=", str.concat(int_to_str(wait_seconds), ")\n"))

Step 4: Add the parse_int_or helper if not present

Search for it: grep -n 'parse_int_or\|fn parse_int' src/socket.reef. If absent, add at the bottom of the file:

fn parse_int_or(s: string, default_val: int): int
    let n = str.length(s)
    if n == 0
        return default_val
    end if
    mut result = 0
    mut i = 0
    while i < n
        let c = str.char_at(s, i)
        if c < '0' or c > '9'
            return default_val
        end if
        result = result * 10 + (c - '0')
        i = i + 1
    end while
    return result
end parse_int_or

If str.starts_with doesn't exist, write it inline using str.substring(arg, 0, 7) == "--wait=".

Step 5: Update the cmd_help line to mention replace

Find cmd_help (around line 413) and add replace [--wait=N] to the comma-separated command list.

Step 6: Build

Run: full Linux build command.
Expected: success.

Step 7: Smoke test the socket command

Run:

TEST=$(mktemp -d)
ZYGINIT_CONFIG_DIR=$TEST/cfg ZYGINIT_SOCKET=$TEST/sock ZYGINIT_LOG_DIR=$TEST/log mkdir -p $TEST/cfg/enabled.d
ZYGINIT_CONFIG_DIR=$TEST/cfg ZYGINIT_SOCKET=$TEST/sock ZYGINIT_LOG_DIR=$TEST/log ./build/zyginit &
ZPID=$!
sleep 0.5
echo 'replace --wait=10' | nc -U $TEST/sock
kill $ZPID 2>/dev/null
rm -rf $TEST

Expected: replace queued (wait=10). (Don't worry that the daemon doesn't actually replace — the actual flow lands in Task 11 once replace_self is wired up. This test only verifies the command parses and the flag is set.)

Step 8: Commit

hg commit -m "socket: add replace [--wait=N] command + flag plumbing"

Task 9: Add `apply_recovered_state` proc in `main.reef`

Files:

Modify: src/main.reef

This is the incoming-side stitching logic. For each entry in the recovered state, find the service by name, populate runtime fields, register the contract_map entry, and run the post-recovery is_contract_empty check.

Step 1: Add the proc near reload_services

Find reload_services (around line 328) and insert before it:

// Apply state from a previous zyginit instance (live-replace).
// For each entry in `recovered`, locate the service in `table` and
// patch in runtime fields. Services in state.toml that are no longer
// in enabled.d/ get their contract abandoned but their member
// processes left running. After patching, run the post-recovery
// empty-contract check to catch services that exited during the
// exec gap.
proc apply_recovered_state(table: supervisor.ServiceTable,
                           recovered: replace.RecoveredState)
    if replace.rs_is_empty(recovered)
        return
    end if

    let n = replace.rs_count(recovered)
    println("zyginit: replace: applying " + int_to_str(n) + " recovered services")

    mut i = 0
    while i < n
        let rs = replace.rs_service(recovered, i)
        let name = replace.svc_name(rs)
        let ctid = replace.svc_contract_id(rs)

        let idx = supervisor.find_service(table, name)
        if idx < 0
            // Operator removed the symlink between old start and replace.
            // Don't kill — abandon the contract; member processes
            // continue running unsupervised.
            let abrc = contract.abandon_contract(ctid)
            println("zyginit: replace: orphaned " + name +
                    " (no longer enabled, contract " + int_to_str(ctid) +
                    " abandon rc=" + int_to_str(abrc) + ")")
            i = i + 1
            continue
        end if

        // Patch runtime fields. PID is unknown (not serialized); contract
        // path is sufficient for supervision.
        supervisor.set_runtime_for_test(table, idx, supervisor.STATE_RUNNING(),
            0 - 1, ctid,
            replace.svc_restart_count(rs),
            replace.svc_last_exit_code(rs),
            replace.svc_last_start_time(rs))

        println("zyginit: replace: re-attached " + name +
                " (ctid=" + int_to_str(ctid) +
                ", restarts=" + int_to_str(replace.svc_restart_count(rs)) + ")")

        // Idempotency: did the contract go empty during the exec gap?
        if contract.is_contract_empty(ctid)
            println("zyginit: replace: " + name +
                    " contract empty post-recovery, applying restart policy")
            supervisor.handle_contract_event(table, ctid, 0 - 1)
        end if

        i = i + 1
    end while
end apply_recovered_state

supervisor.set_runtime_for_test is the same helper added in Task 4 — its name communicates "test/probe usage" but it does exactly what apply_recovered_state needs (write the runtime fields directly). Acceptable. If the name bothers you, rename to supervisor.adopt_runtime and update both call sites.

Step 2: Add the import at the top of main.reef

Find the imports block (top of file). Add:

import replace

Step 3: Build

Run: full Linux build.
Expected: success.

Step 4: Commit

hg commit -m "main: add apply_recovered_state proc for live-replace stitching"

Task 10: Add `replace_self` proc in `main.reef`

Files:

Modify: src/main.reef

The outgoing-side flow. Pre-condition check (with optional --wait poll), state-file write, socket unlink, exec.

Step 1: Add FFI and helpers near the top of main.reef

If not already present, add to the extern block:

extern "C" fn zyginit_fsync_path(path: string): int
extern "C" fn unlink(path: string): int

Step 2: Add the proc near apply_recovered_state

// Operator-driven live replace. Triggered by zygctl replace; the
// socket handler sets g_replace_requested and we pick it up on the
// next event-loop tick. After this returns successfully, control
// transfers to the new /sbin/init process; this fn does not return
// on success.
//
// On precondition failure or any I/O error before exec, we log and
// return false; the caller clears the flag and continues normally.
proc replace_self(table: supervisor.ServiceTable, boot_time: int, wait_seconds: int)
    println("zyginit: replace: requested (wait=" + int_to_str(wait_seconds) + ")")

    // Pre-condition check, with optional wait window
    mut elapsed = 0
    mut report = replace.check_preconditions(table)
    while str.length(report) > 0 and elapsed < wait_seconds
        clock.sleep_seconds(1)
        elapsed = elapsed + 1
        report = replace.check_preconditions(table)
    end while

    if str.length(report) > 0
        println("zyginit: replace: blocked: " + report)
        return
    end if

    // Final draining of pending contract events — services may have just
    // exited and their state hasn't been recorded yet.
    // (Reuse the existing handle_contract_events drain logic if exposed;
    // otherwise the post-recovery is_contract_empty check on the new
    // side covers any miss.)

    // Serialize state
    if not replace.serialize_state(table, boot_time)
        println("zyginit: replace: state serialization failed; aborting")
        return
    end if
    println("zyginit: replace: state written to " + replace.STATE_FILE_PATH())

    // Unlink socket. Proceed even on failure — new zyginit will overwrite.
    // zyginit_fsync_path falls back to fsyncing the parent dir when
    // the path itself doesn't exist, which is exactly what we want
    // after unlink — the socket's parent dir gets the rename committed.
    let sock_path = SOCKET_PATH()
    let urc = unlink(sock_path)
    if urc != 0
        println("zyginit: replace: warning: socket unlink failed (rc=" + int_to_str(urc) + ")")
    end if
    let _ = zyginit_fsync_path(sock_path)

    // Exec /sbin/init. argv[0] should match what the kernel originally
    // exec'd us with; we pass "/sbin/init" verbatim.
    let target = "/sbin/init"
    let argv = new [string](1)
    argv[0] = target

    println("zyginit: replace: execve " + target)
    process.process_exec(target, argv)

    // If we reach here, exec failed. We have already written state.toml
    // and unlinked the socket — recovery requires reboot.
    println("zyginit: replace: FATAL: execve returned (kernel could not load " + target + ")")
end replace_self

The "drain pending contract events" comment block is intentional — the current code path for handling bundle events lives in the main loop. If you want to drain them once more here, factor that out into a proc drain_bundle_events(table, bundle_fd) and call it; otherwise rely on the post-recovery is_contract_empty check (which is sufficient per the spec).

Step 3: Build

Expected: success.

Step 4: Commit

hg commit -m "main: add replace_self proc for live-replace outgoing flow"

Task 11: Wire `main.reef` startup + event loop

Files:

Modify: src/main.reef

Two integration points:

Startup: call replace.recover_state and apply_recovered_state between Phase 3 (table built) and Phase 6 (start_services_by_tier).
Event loop: check socket.replace_requested() and call replace_self.

Also: capture g_boot_time once during PID-1 init so it's available to both call sites.

Step 1: Add g_boot_time global

Near the top of main.reef with the other mut g_* globals, add:

mut g_boot_time: int = 0 - 1

Step 2: Capture boot_time at the start of main() after writing the BOOT_TIME utmpx record

Right after the existing let _ = zyginit_write_boot_utmpx() at line ~725:

        // Cache the BOOT_TIME we just wrote so live-replace can use
        // it as the boot-id sentinel. Read it back from utmpx (rather
        // than time_now() locally) so the value matches what the new
        // zyginit will read after exec.
        g_boot_time = replace.read_boot_time()
        if g_boot_time < 0
            // Fallback: use time_now(). This degrades the staleness
            // check (a real reboot might collide if seconds-resolution
            // happens to repeat), but it's better than refusing all
            // replaces.
            g_boot_time = time.time_now()
            println("zyginit: warning: utmpx BOOT_TIME read failed; using time_now() = " + int_to_str(g_boot_time))
        end if

For non-PID-1 supervisor mode, g_boot_time stays at -1; replace.serialize_state still writes "-1" and the new instance reads "-1" — they match, and the path is intentionally permissive in supervisor mode.

Actually — set it for non-PID-1 too:

    // (After the if is_pid_1() block)
    if g_boot_time < 0
        g_boot_time = time.time_now()
    end if

Step 3: Insert recovery between Phase 3 and Phase 6

After the i + 1 add_service loop (around line 778) and before signal init:

    // ---- Phase 3.5: Apply state from previous zyginit (live replace) ----

    let recovered = replace.recover_state(g_boot_time)
    if not replace.rs_is_empty(recovered)
        apply_recovered_state(table, recovered)
    end if

This lands AFTER add_service so all services are registered, BEFORE start_services_by_tier so re-attached services are skipped (they're in STATE_RUNNING and start_service only acts on STATE_WAITING).

Step 4: Add the replace check in the main event loop

Find the existing reload check (around line 858-862):

        // Check for reload requested via socket
        if socket.reload_requested()
            socket.clear_reload_flag()
            println("zyginit: reload requested via socket")
            reload_services(table)
        end if

Add immediately after:

        // Check for replace requested via socket
        if socket.replace_requested()
            let wait = socket.replace_wait_seconds()
            socket.clear_replace_flag()
            replace_self(table, g_boot_time, wait)
            // If replace_self exec'd successfully, we never get here.
            // If it returned, log and continue running normally.
        end if

Step 5: Build the project AND zygctl

Run:

clang -c src/helpers.c -o build/helpers.o
clang -c src/contract_linux_stubs.c -o build/contract_linux_stubs.o
reefc build --obj build/helpers.o --obj build/contract_linux_stubs.o

Expected: success, build/zyginit produced.

Step 6: Smoke test in supervisor mode

TEST=$(mktemp -d)
mkdir -p $TEST/cfg/enabled.d $TEST/log
ZYGINIT_CONFIG_DIR=$TEST/cfg ZYGINIT_SOCKET=$TEST/sock ZYGINIT_LOG_DIR=$TEST/log ZYGINIT_RUN_DIR=$TEST/run ./build/zyginit > $TEST/zyginit.log 2>&1 &
ZPID=$!
sleep 0.5
echo 'replace' | nc -U $TEST/sock
sleep 0.5
# zyginit should have logged "zyginit: replace: state written..." and
# attempted execve /sbin/init (which fails on Linux dev because
# /sbin/init isn't this binary). Acceptable failure.
cat $TEST/zyginit.log
kill $ZPID 2>/dev/null
rm -rf $TEST

Expected log lines:

zyginit: replace: requested (wait=0)
zyginit: replace: state written to /tmp/.../state.toml (or the ZYGINIT_RUN_DIR you set)
zyginit: replace: execve /sbin/init
zyginit: replace: FATAL: execve returned ... (only if /sbin/init isn't usable as a Linux binary in this dev environment)

The integration test will refine this in Task 13.

Step 7: Commit

hg commit -m "main: wire live-replace into startup recovery and event loop"

Task 12: Add `zygctl replace` subcommand

Files:

Modify: tools/zygctl/src/zygctl.reef
Step 1: Locate the command dispatch

Open tools/zygctl/src/zygctl.reef and find the dispatch (search for "reload" — there'll be a similar pattern for the existing socket-relayed commands).

Step 2: Add replace to the dispatch

Pattern: build the command line (replace + optional --wait=N), connect to socket, send, read response, print, handle EPIPE / connection-closed cleanly.

    elif cmd == "replace"
        // Optional --wait=N
        mut payload = "replace"
        let argc = sys.args.count()
        if argc >= 3
            let wait_arg = sys.args.get(2)
            if str.starts_with(wait_arg, "--wait=")
                payload = str.concat(payload, str.concat(" ", wait_arg))
            else
                println("zygctl: replace: unknown argument: " + wait_arg)
                println("zygctl: usage: zygctl replace [--wait=N]")
                process.exit_now(2)
            end if
        end if

        let response = send_command(payload)
        if str.length(response) == 0
            // Connection closed without a response — the server proceeded
            // to exec immediately. Treat as success.
            println("replace: socket closed (process replaced)")
        else
            print(response)
        end if

send_command is the existing helper that connects to the socket and reads the response — adapt to whatever name is in use. If the existing pattern doesn't gracefully handle "connection closed without bytes received", treat that as a successful replace and print the documented message.

Step 3: Update help text

Find the help text in zygctl.reef (likely a cmd_help proc) and add replace [--wait=N] to the list.

Step 4: Build zygctl

Run:

cd tools/zygctl
clang -c src/symlink_wrapper.c -o build/symlink_wrapper.o
reefc build --obj build/symlink_wrapper.o
cd ../..

Expected: success, tools/zygctl/build/zygctl produced.

Step 5: Smoke test

TEST=$(mktemp -d)
mkdir -p $TEST/cfg/enabled.d $TEST/log
ZYGINIT_CONFIG_DIR=$TEST/cfg ZYGINIT_SOCKET=$TEST/sock ZYGINIT_LOG_DIR=$TEST/log ZYGINIT_RUN_DIR=$TEST/run ./build/zyginit > $TEST/zyginit.log 2>&1 &
ZPID=$!
sleep 0.5
ZYGINIT_SOCKET=$TEST/sock ./tools/zygctl/build/zygctl replace --wait=5
sleep 0.5
cat $TEST/zyginit.log
kill $ZPID 2>/dev/null
rm -rf $TEST

Expected zygctl output: replace queued (wait=5) OR replace: socket closed (process replaced).

Step 6: Commit

hg commit -m "zygctl: add replace [--wait=N] subcommand"

Task 13: Linux integration smoke tests

Files:

Modify: tests/integration/run_tests.sh

Add new test scenarios after the existing tests. Linux can't exercise actual contract adoption (stubs return -1), but it CAN verify:

replace command parses and acks
Pre-condition failures are reported
State file writes correctly
The exec attempt fires (it'll fail because /sbin/init isn't this binary on a Linux dev box, but that's a known Linux-only failure)
Step 1: Locate where new test sections are appended in run_tests.sh

grep -n '^section ' tests/integration/run_tests.sh | tail will show the existing section markers. Insert a new section block before the final summary.

Step 2: Add the replace command-parse test

Append before the final summary:

section "replace command — basic parsing"

# Start a fresh zyginit
start_zyginit
mkdir -p "$TEST_DIR/run"
export ZYGINIT_RUN_DIR="$TEST_DIR/run"
restart_zyginit_with_run_dir() {
    stop_zyginit
    "$ZYGINIT" >> "$DAEMON_LOG" 2>&1 &
    DAEMON_PID=$!
    wait_for_socket
}
restart_zyginit_with_run_dir

# zygctl replace with no args
out=$("$ZYGCTL" replace 2>&1 || true)
assert_contains "$out" "replace queued" "replace acks with no args"

# zygctl replace --wait=10
out=$("$ZYGCTL" replace --wait=10 2>&1 || true)
assert_contains "$out" "replace queued (wait=10)" "replace acks with --wait=10"

# Check state.toml was written before exec attempt
sleep 0.2
if [ -f "$TEST_DIR/run/state.toml" ]; then
    pass "state.toml present after replace"
else
    # Either still waiting OR exec succeeded and the new instance deleted it.
    # Since /sbin/init exec fails on Linux dev, the file should still be there
    # if the test sequence was fast.
    fail "state.toml missing after replace" "file present" "missing"
fi

# Verify state.toml contents
if [ -f "$TEST_DIR/run/state.toml" ]; then
    out=$(cat "$TEST_DIR/run/state.toml")
    assert_contains "$out" "boot_time =" "state.toml has boot_time"
fi

You may need to adapt start_zyginit / stop_zyginit / wait_for_socket to use ZYGINIT_RUN_DIR. Check the existing helpers in run_tests.sh for the pattern.

Step 3: Add the precondition refusal test

Append:

section "replace — refuses on transient state"

# Set up a service whose start.sh hangs for 10s, so the service stays
# in STATE_STARTING long enough for us to issue a replace.
mkdir -p "$BIN_DIR"
cat > "$BIN_DIR/slow-start.sh" <<'EOF'
#!/bin/sh
sleep 10
EOF
chmod +x "$BIN_DIR/slow-start.sh"

cat > "$ENABLED_DIR/slow.toml" <<EOF
[service]
name = "slow"
type = "daemon"

[exec]
start = "$BIN_DIR/slow-start.sh"

[restart]
on = "never"
EOF

ln -sf "$ENABLED_DIR/slow.toml" "$ENABLED_DIR/../slow.toml" 2>/dev/null || true

# Reload to pick up the new service
"$ZYGCTL" reload >/dev/null 2>&1
sleep 0.2

# Issue replace — should ack but be refused on the server side
out=$("$ZYGCTL" replace 2>&1 || true)
assert_contains "$out" "replace queued" "replace acks even when refused"

# Server should log "replace: blocked" since slow service is in STARTING
sleep 0.5
log=$(cat "$DAEMON_LOG")
assert_contains "$log" "replace: blocked" "server logs blocked refusal"

Step 4: Run the test suite

Run: tests/integration/run_tests.sh
Expected: all existing tests pass (regression check) plus the new replace tests pass.

Step 5: Commit

hg commit -m "tests: add Linux integration coverage for zygctl replace"

Task 14: Hammerhead test plan + man page updates

Files:

Create: tests/integration/test_replace_hammerhead.md
Modify: man/zygctl.8
Modify: man/zyginit.8

Hammerhead testing is manual — it requires hh-prototest and exercises the actual contract-driven supervision path. Document the test cases as a checklist the operator runs by hand.

Step 1: Create the Hammerhead test checklist

Write tests/integration/test_replace_hammerhead.md:

# Live Replace — Hammerhead Test Checklist

Manual checklist for validating `zygctl replace` on `hh-prototest`
(192.168.122.50). Companion to the spec at
`docs/superpowers/specs/2026-05-02-contract-readoption-design.md`.

## Build + deploy

```bash
# From dev host
scp src/replace.reef src/main.reef src/contract.reef src/socket.reef \
    src/contract_linux_stubs.c src/helpers.c \
    root@192.168.122.50:/root/zyginit/src/
scp tools/zygctl/src/zygctl.reef \
    root@192.168.122.50:/root/zyginit/tools/zygctl/src/

# On hh-prototest
ssh root@192.168.122.50 'cd /root/zyginit && \
    gcc -c src/helpers.c -o build/helpers.o && \
    reefc build -l contract --obj build/helpers.o && \
    cp build/zyginit /sbin/init.new && mv /sbin/init.new /sbin/init && \
    cd tools/zygctl && \
    gcc -c src/symlink_wrapper.c -o build/symlink_wrapper.o && \
    reefc build --obj build/symlink_wrapper.o && \
    cp build/zygctl /sbin/zygctl'
```

After the binary swap, the running zyginit is still the OLD one. The
first `zygctl replace` will swap to the new binary.

## Test cases

### 1. Smoke: replace into the same binary

- [ ] `zygctl status` — note all PIDs and ctids; capture uptimes
- [ ] `zygctl replace` — should print `replace queued (wait=0)` then close
- [ ] Wait ~1 second; `zygctl status` again
- [ ] **All services have the same PIDs** (kernel-level continuity)
- [ ] **All services have the same contract IDs**
- [ ] **Uptimes are continuous** (delta from last_start_time, not reset)
- [ ] **The SSH session driving this test is still alive**
- [ ] `who -b` and `who -r` still report the same boot_time / runlevel

### 2. Pre-condition refusal

- [ ] `zygctl restart sshd` (puts sshd briefly in STOPPING/STARTING)
- [ ] Immediately: `zygctl replace`
- [ ] Expect server log line: `zyginit: replace: blocked: sshd:stopping`
- [ ] Verify the running zyginit did NOT exec (check uptime via /proc/1/start)

### 3. `--wait=N` success

- [ ] `zygctl restart syslogd && zygctl replace --wait=10`
- [ ] Replace should succeed once syslogd stabilizes
- [ ] Confirm syslogd's restart_count incremented and is preserved post-replace

### 4. Dirty disable

- [ ] `rm /etc/zyginit/enabled.d/cron.toml`  (no `zygctl reload`)
- [ ] `zygctl replace`
- [ ] After replace, `zygctl status` does NOT list cron
- [ ] But: `pgrep cron` still shows the cron pid running (process not killed)
- [ ] Server log shows `zyginit: replace: orphaned cron`

### 5. Crash during exec gap

- [ ] Pick a service with `restart.on = "always"`, e.g. sshd
- [ ] `pkill -9 -f "sshd -D"` (or whatever the daemon's argv looks like)
- [ ] Within ~0.5 seconds: `zygctl replace`
- [ ] After replace, server log includes `zyginit: replace: sshd contract empty post-recovery, applying restart policy`
- [ ] sshd is restarted with restart_count incremented

### 6. Stale state.toml across real reboot

- [ ] `cp /var/run/zyginit/state.toml /tmp/saved-state.toml` while a replace
      hasn't been issued (file may not exist; create one by issuing replace
      first then copy quickly before the new zyginit deletes it)
- [ ] `zygctl reboot` (real reboot)
- [ ] After boot: `cp /tmp/saved-state.toml /var/run/zyginit/state.toml`
- [ ] Stop+start zyginit somehow OR observe that the next replace handles it
- [ ] Server log includes `replace: stale state file (boot_time ...)`
- [ ] state.toml is deleted after that read

### 7. Repeated replace

- [ ] Note sshd's restart_count and uptime
- [ ] `zygctl replace` × 5 in a row, ~5 seconds apart
- [ ] After all 5: restart_count is unchanged (replace doesn't bump it)
- [ ] Uptime is continuous from the original start (no resets)
- [ ] No leftover state.toml.tmp files in /var/run/zyginit/

## Reverting

If a replace puts the system in a bad state and SSH is still alive:

```bash
mv /sbin/init.bak /sbin/init  # if you saved the old binary
zygctl replace                # swap back to known-good
```

If SSH is dead but the VM is still running, `virsh reset hh-prototest`
will reboot. The new binary at /sbin/init persists (filesystem-level
swap), so a reboot uses the new binary cleanly — re-test from cold boot.

Step 2: Update man/zygctl.8

Add a replace entry in the COMMANDS section. Example pattern (adapt to existing groff macros):

.TP
.B replace [--wait=N]
Replace the running zyginit process via in-place
.BR execve (2)
of
.IR /sbin/init .
Preserves all running services across the swap. With
.BR --wait=N ,
poll up to N seconds for any transient-state service to stabilize
before refusing. The operator must drop the new binary at
.I /sbin/init
before issuing this command. See
.BR zyginit (8)
for the live-replace mechanism.

Step 3: Update man/zyginit.8

Add a "LIVE REPLACE" section describing:

The state-file path (/var/run/zyginit/state.toml)
The exec target is hard-wired to /sbin/init
The boot-id sentinel (utmpx BOOT_TIME)
That the design is operator-driven and does NOT cover crash recovery

Refer to the spec for full details.

Step 4: Verify man pages render

Run:

man -l man/zygctl.8 | head -50
man -l man/zyginit.8 | head -50

Expected: no troff errors, the replace entry appears in zygctl.8.

Step 5: Commit

hg add tests/integration/test_replace_hammerhead.md
hg commit -m "docs: Hammerhead replace test checklist + man page updates"

Done — final verification

After all 14 tasks land:

Linux build clean:

clang -c src/helpers.c -o build/helpers.o
clang -c src/contract_linux_stubs.c -o build/contract_linux_stubs.o
reefc build --obj build/helpers.o --obj build/contract_linux_stubs.o
cd tools/zygctl && reefc build --obj build/symlink_wrapper.o && cd ../..

All Linux integration tests pass: tests/integration/run_tests.sh
Replace-probe tests pass: cd utils/replace_probe && ./build/replace_probe && cd ../..
Spec coverage check — re-read the spec, point at a task for each requirement:
- §3 "no adopt_contract" → Task 9 (no adopt call in apply_recovered_state) ✓
- §4 module split → Tasks 1-12 cover all five files ✓
- §5 state file format → Tasks 4, 5 ✓
- §5.2 boot-id staleness check → Tasks 1, 5 ✓
- §5.3 atomic write → Task 4 ✓
- §6.1 zygctl interface → Task 12 ✓
- §6.2 --wait=N → Tasks 8, 10 ✓
- §6.3 pre-conditions → Tasks 6, 10 ✓
- §7.2 algorithm → Task 9 ✓
- §7.3 config divergence → Task 9 (orphan abandon path) ✓
- §8 error handling → Tasks 4, 5, 9, 10 (defensive throughout) ✓
- §9.1 Linux tests → Task 13 ✓
- §9.2 Hammerhead tests → Task 14 ✓
- §11 decisions table → spec already mapped to design choices, reflected in implementation ✓
Hammerhead deploy + run the manual checklist in tests/integration/test_replace_hammerhead.md. Capture results in a memory note (pid1_replace_iter1.md or similar).

Non-trivial gotchas to watch for

Reef stdlib API names in this plan (toml.parse_ok, toml.get_int_or, fd.fd_fsync, str.starts_with, str.char_at, process.exit_now) are best-effort. The implementer should grep the actual Reef stdlib (~/repos/reef-lang/) and use the real names, adjusting code locally. The shapes are right; only the spellings may differ.
set_runtime_for_test naming. The proc is used by both the probe (legitimate test usage) and apply_recovered_state (production path). If the name bothers a reviewer, rename to supervisor.adopt_runtime and update both call sites. The function does what the production path needs; the test-y name is cosmetic.
STATE_WAITING ambiguity. Conservative interpretation: refuse replace whenever any service is in WAITING. This includes the boot-time "not-yet-started" case, which can't actually happen at the point a replace is issued (since the table is fully started before the event loop entry that processes the replace flag). False positives only land in pathological timing. Document in commit message if concerns surface.
utmpx BOOT_TIME is read-once on Hammerhead if the file is on a not-yet-mounted filesystem. The fallback to time.time_now() covers this gap, but means a panic-restart followed by a real reboot followed by another zyginit start will see different boot_times — that's CORRECT behavior (real reboot must invalidate state.toml).
process.process_exec argv handling. Reef's process_exec may or may not require argv terminated by a sentinel. Match the pattern used elsewhere in the codebase (e.g., start_service in supervisor.reef). The existing pattern there uses [string] arrays — same pattern in replace_self.
socket.reef cmd parsing. The existing dispatch only takes one string argument; for --wait=N we need to parse it from the arg string. Confirm by reading the existing handler shape; if arg is the rest of the line after the verb, the impl above is correct.

Contract Re-adoption (Live Replace) Implementation Plan

File structure

Task 1: Add zyginit_read_boot_time C helper

Task 2: Skeleton src/replace.reef module

Task 3: Define RecoveredService and RecoveredState types

Task 4: Implement serialize_state (TDD via utils/replace_probe)

Task 5: Implement recover_state (TDD via utils/replace_probe)

Task 6: Implement check_preconditions (TDD)

Task 7: Add contract.is_contract_empty helper

Task 8: Add socket replace command + flag plumbing

Task 9: Add apply_recovered_state proc in main.reef

Task 10: Add replace_self proc in main.reef

Task 11: Wire main.reef startup + event loop

Task 12: Add zygctl replace subcommand

Task 13: Linux integration smoke tests

Task 14: Hammerhead test plan + man page updates

Done — final verification

Non-trivial gotchas to watch for

Task 1: Add `zyginit_read_boot_time` C helper

Task 2: Skeleton `src/replace.reef` module

Task 3: Define `RecoveredService` and `RecoveredState` types

Task 4: Implement `serialize_state` (TDD via `utils/replace_probe`)

Task 5: Implement `recover_state` (TDD via `utils/replace_probe`)

Task 6: Implement `check_preconditions` (TDD)

Task 7: Add `contract.is_contract_empty` helper

Task 8: Add socket `replace` command + flag plumbing

Task 9: Add `apply_recovered_state` proc in `main.reef`

Task 10: Add `replace_self` proc in `main.reef`

Task 11: Wire `main.reef` startup + event loop

Task 12: Add `zygctl replace` subcommand