# Contract Re-adoption (Live Replace) Implementation Plan

> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.

**Spec:** `docs/superpowers/specs/2026-05-02-contract-readoption-design.md` (rev 84).

**Goal:** Allow `zygctl replace` to upgrade the running zyginit PID-1 process via in-place `execve("/sbin/init", ...)` while preserving every running service across the swap, with state-file handoff for in-memory bookkeeping.

**Architecture:** New `src/replace.reef` module owns the state-file format, serialization, recovery, and pre-condition check. `main.reef` gains a `replace_self` proc (outgoing flow: drain, write state, unlink socket, exec) and an `apply_recovered_state` proc (incoming flow: stitch state.toml against current `enabled.d/`, register `contract_map`, idempotent post-recovery empty-contract check). `socket.reef` adds a `replace [--wait=N]` command. `contract.reef` adds an `is_contract_empty` helper. `zygctl.reef` adds the `replace` subcommand. **No `ct_ctl_adopt` call is needed**: PID-1 in-place execve preserves `proc_t.p_ct_process` so kernel-level contract ownership survives automatically; only userspace bookkeeping needs rebuilding.

**Tech Stack:** Reef 0.4.0+, Hammerhead libcontract, POSIX FFI, TOML (existing `encoding.toml` parser), Mercurial (`hg`) for VCS.

**Conventions** — pulled from CLAUDE.md / SRCHEADER.txt. **All new source files MUST start with the SRCHEADER.txt template** with `${project}=zyginit`, `${file.name}` = the filename, `${file.description}` filled in. **VCS is `hg` not `git`.** **License is CDDL-1.0.** **"Hammerhead" not "illumos"** in code/comments. **No emojis** in source.

**Build commands** (used in test/verify steps throughout):

```bash
# Linux dev (stubs the contract bindings)
clang -c src/helpers.c -o build/helpers.o
clang -c src/contract_linux_stubs.c -o build/contract_linux_stubs.o
reefc build --obj build/helpers.o --obj build/contract_linux_stubs.o

# zygctl Linux
cd tools/zygctl
clang -c src/symlink_wrapper.c -o build/symlink_wrapper.o
reefc build --obj build/symlink_wrapper.o
cd ../..
```

---

## File structure

| File | Action | Responsibility |
|---|---|---|
| `src/helpers.c` | Modify | Add `zyginit_read_boot_time` C helper. |
| `src/replace.reef` | Create (~250 lines) | State-file format, serialize, recover, pre-condition check. |
| `src/contract.reef` | Modify (+~25 lines) | Add `is_contract_empty(ctid): bool` helper. |
| `src/contract_linux_stubs.c` | Modify (+5 lines) | Linux stub for `is_contract_empty`. |
| `src/socket.reef` | Modify (+~50 lines) | Add `replace [--wait=N]` command, flag, accessors. |
| `src/main.reef` | Modify (+~120 lines) | Add `apply_recovered_state`, `replace_self`, integrate into startup + event loop. |
| `tools/zygctl/src/zygctl.reef` | Modify (+~30 lines) | Add `replace [--wait=N]` subcommand. |
| `utils/replace_probe/` | Create | Standalone Reef program for state-file format unit tests, modeled on `utils/contract_probe/`. |
| `tests/integration/run_tests.sh` | Modify | Add Linux integration scenarios (smoke + pre-condition refusal). |
| `tests/integration/test_replace_hammerhead.md` | Create | Manual checklist for Hammerhead `hh-prototest` validation. |
| `man/zygctl.8` | Modify | Document `zygctl replace [--wait=N]`. |
| `man/zyginit.8` | Modify | Document state.toml file path + replace flow at a high level. |

---

## Task 1: Add `zyginit_read_boot_time` C helper

**Files:**
- Modify: `src/helpers.c` (append a new function near the existing utmpx writers around line 150)

The helper reads `BOOT_TIME` from `/var/adm/utmpx` and returns its `ut_tv.tv_sec` as `int`, or `-1` if not found / not readable. This becomes the boot-id sentinel that survives execve and lets the new zyginit detect a stale state.toml from before a real reboot.

- [ ] **Step 1: Read existing helpers.c around the BOOT_TIME writer to confirm placement and headers**

Run: `grep -n 'BOOT_TIME\|getutxent\|setutxent\|endutxent' src/helpers.c`
Expected: see `BOOT_TIME` referenced in `zyginit_write_boot_utmpx`, header `utmpx.h` already included.

- [ ] **Step 2: Add the read helper after `zyginit_write_runlvl_utmpx`**

Append to `src/helpers.c`:

```c
/* Read the BOOT_TIME utmpx record's tv_sec value, used as the boot-id
 * sentinel for the live-replace state file. PID-1 in-place execve
 * preserves the same proc_t but the kernel does not preserve a "boot
 * id" anywhere obvious — utmpx BOOT_TIME survives because it lives in
 * /var/adm/utmpx, written by zyginit_write_boot_utmpx() at boot. A
 * real reboot writes a new BOOT_TIME with a different tv_sec, so
 * comparing against it lets the new zyginit detect a stale state file.
 *
 * Returns the BOOT_TIME tv_sec on success, or -1 if the file is not
 * readable, has no BOOT_TIME entry, or any utmpx call fails.
 */
int zyginit_read_boot_time(void) {
    struct utmpx *up;
    int result = -1;

    setutxent();
    while ((up = getutxent()) != NULL) {
        if (up->ut_type == BOOT_TIME) {
            result = (int) up->ut_tv.tv_sec;
            break;
        }
    }
    endutxent();
    return result;
}
```

- [ ] **Step 3: Build helpers.o on Linux to verify it compiles**

Run: `clang -c src/helpers.c -o build/helpers.o`
Expected: no errors, no warnings. (`utmpx.h` is in libc; on Linux it's `glibc`'s utmpx.)

- [ ] **Step 4: Commit**

```bash
hg add src/helpers.c
hg commit -m "helpers: add zyginit_read_boot_time for live-replace boot-id check"
```

(Note: `hg add` is a no-op for already-tracked files, but harmless.)

---

## Task 2: Skeleton `src/replace.reef` module

**Files:**
- Create: `src/replace.reef`

Empty-bodied module with SRCHEADER, FFI extern, and the export block that subsequent tasks will fill in. Establishing the surface up front lets later tasks be small and incremental.

- [ ] **Step 1: Create the file with header and module declaration**

Write `src/replace.reef`:

```reef
/******************************************************************************
                __               ____                __
               / /   ___  ____ _/ __/_____________ _/ /__
              / /   / _ \/ __ `/ /_/ ___/ ___/ __ `/ / _ \
             / /___/  __/ /_/ / __(__  ) /__/ /_/ / /  __/
            /_____/\___/\__,_/_/ /____/\___/\__,_/_/\___/

    (C)opyright 2025-2026, Leafscale, LLC -  https://www.leafscale.com

    Project: zyginit
   Filename: replace.reef
    Authors: Chris Tusa <chris.tusa@leafscale.com>
    License: <see LICENSE file included with this source code>
Description: Live-replace state-file handoff (operator-driven PID-1 upgrade)

******************************************************************************/

// Live-replace operator flow: zygctl replace -> old zyginit serializes
// runtime state to /var/run/zyginit/state.toml, execs /sbin/init in
// place; new zyginit reads state.toml, validates boot_time matches
// current utmpx BOOT_TIME (else stale across a real reboot), and
// rebuilds the contract_map / runtime fields in supervisor.ServiceTable.
//
// PID-1 in-place execve preserves proc_t.p_ct_process at the kernel
// level, so contract ownership survives automatically — no
// ct_ctl_adopt call is needed. This module owns userspace bookkeeping.

module replace

import config
import supervisor
import io.file
import sys.fd as fd
import sys.env as sysenv
import core.str
import encoding.toml
import time.time as time

export
    type RecoveredService
    type RecoveredState

    // Accessors — RecoveredState
    fn rs_count(s: RecoveredState): int
    fn rs_service(s: RecoveredState, idx: int): RecoveredService
    fn rs_boot_time(s: RecoveredState): int
    fn rs_is_empty(s: RecoveredState): bool

    // Accessors — RecoveredService
    fn svc_name(rs: RecoveredService): string
    fn svc_contract_id(rs: RecoveredService): int
    fn svc_restart_count(rs: RecoveredService): int
    fn svc_last_start_time(rs: RecoveredService): int
    fn svc_last_exit_code(rs: RecoveredService): int

    // Outgoing flow — write state, return true on success
    fn serialize_state(table: supervisor.ServiceTable, boot_time: int): bool

    // Incoming flow — read state file. Returns an empty RecoveredState
    // if the file is missing, corrupt, or has a mismatched boot_time.
    fn recover_state(current_boot_time: int): RecoveredState

    // Pre-condition check. Returns "" if all services are stable, else
    // a comma-separated "name:state" list suitable for logging.
    fn check_preconditions(table: supervisor.ServiceTable): string

    // FFI for boot-time sentinel (helpers.c)
    fn read_boot_time(): int

    // Constants
    fn STATE_FILE_PATH(): string
    fn STATE_TMP_PATH(): string
    fn STATE_DIR(): string
end export

// FFI — implemented in helpers.c
extern "C" fn zyginit_read_boot_time(): int
extern "C" fn zyginit_fsync_path(path: string): int

// FFI — POSIX
extern "C" fn open(path: string, flags: int, mode: int): int
extern "C" fn close(fd: int): int
extern "C" fn rename(oldpath: string, newpath: string): int
extern "C" fn unlink(path: string): int

// Reads utmpx BOOT_TIME via the C helper. Returns -1 if unavailable.
fn read_boot_time(): int
    return zyginit_read_boot_time()
end read_boot_time

// File-system layout. Keep grouped here so a future tmpfs migration
// only edits one place.
fn STATE_DIR(): string
    return sysenv.get_env_or("ZYGINIT_RUN_DIR", "/var/run/zyginit")
end STATE_DIR

fn STATE_FILE_PATH(): string
    return str.concat(STATE_DIR(), "/state.toml")
end STATE_FILE_PATH

fn STATE_TMP_PATH(): string
    return str.concat(STATE_DIR(), "/state.toml.tmp")
end STATE_TMP_PATH

end module
```

- [ ] **Step 2: Add to `reef.toml` if it lists modules explicitly (it doesn't currently — `source_dirs = ["src"]`); confirm with build**

Run: `clang -c src/helpers.c -o build/helpers.o && clang -c src/contract_linux_stubs.c -o build/contract_linux_stubs.o && reefc build --obj build/helpers.o --obj build/contract_linux_stubs.o`
Expected: build fails with errors about missing function bodies (`rs_count`, `serialize_state`, etc. are exported but not yet implemented). This is expected — subsequent tasks fill them in.

If reefc complains about exporting types/fns without bodies in the same module, comment out the export block temporarily by wrapping it in `/* ... */` (Reef does not support `/* */` comments — use line `//` per item) until enough is in place. The skeleton below in Task 3+ will populate types and fn bodies in the order needed to keep the build happy.

If `reefc` allows undefined exports as forward declarations, the build will fail on the first reference site instead — that's fine, fix it as the next task adds bodies.

- [ ] **Step 3: Commit even though build is incomplete**

```bash
hg add src/replace.reef
hg commit -m "replace: skeleton module with FFI declarations and exports"
```

---

## Task 3: Define `RecoveredService` and `RecoveredState` types

**Files:**
- Modify: `src/replace.reef`

Add the types and trivial accessors. After this task the module compiles even with stub bodies for the larger fns.

- [ ] **Step 1: Add type definitions before the `end module` at the bottom of `src/replace.reef`**

```reef
// ============================================================================
// Types
// ============================================================================

type RecoveredService = struct
    name: string
    contract_id: int
    restart_count: int
    last_start_time: int
    last_exit_code: int
end RecoveredService

type RecoveredState = struct
    boot_time: int
    services: [RecoveredService]
    count: int
end RecoveredState

fn new_recovered_state(): RecoveredState
    return RecoveredState{
        boot_time: 0 - 1,
        services: new [RecoveredService](0),
        count: 0
    }
end new_recovered_state

// ============================================================================
// Accessors
// ============================================================================

fn rs_count(s: RecoveredState): int
    return s.count
end rs_count

fn rs_service(s: RecoveredState, idx: int): RecoveredService
    return s.services[idx]
end rs_service

fn rs_boot_time(s: RecoveredState): int
    return s.boot_time
end rs_boot_time

fn rs_is_empty(s: RecoveredState): bool
    return s.count == 0
end rs_is_empty

fn svc_name(rs: RecoveredService): string
    return rs.name
end svc_name

fn svc_contract_id(rs: RecoveredService): int
    return rs.contract_id
end svc_contract_id

fn svc_restart_count(rs: RecoveredService): int
    return rs.restart_count
end svc_restart_count

fn svc_last_start_time(rs: RecoveredService): int
    return rs.last_start_time
end svc_last_start_time

fn svc_last_exit_code(rs: RecoveredService): int
    return rs.last_exit_code
end svc_last_exit_code
```

- [ ] **Step 2: Add stub bodies for the three larger fns so the module compiles. They are filled in by Tasks 4, 5, and 6.**

```reef
// ============================================================================
// Outgoing — stub (filled by Task 4)
// ============================================================================

fn serialize_state(table: supervisor.ServiceTable, boot_time: int): bool
    return false
end serialize_state

// ============================================================================
// Incoming — stub (filled by Task 5)
// ============================================================================

fn recover_state(current_boot_time: int): RecoveredState
    return new_recovered_state()
end recover_state

// ============================================================================
// Pre-condition check — stub (filled by Task 6)
// ============================================================================

fn check_preconditions(table: supervisor.ServiceTable): string
    return ""
end check_preconditions
```

- [ ] **Step 3: Build**

Run: `clang -c src/helpers.c -o build/helpers.o && clang -c src/contract_linux_stubs.c -o build/contract_linux_stubs.o && reefc build --obj build/helpers.o --obj build/contract_linux_stubs.o`
Expected: build succeeds. zyginit binary produced at `build/zyginit`. The replace module is compiled but unused.

- [ ] **Step 4: Commit**

```bash
hg commit -m "replace: types, accessors, and stub bodies for serialize/recover/check"
```

---

## Task 4: Implement `serialize_state` (TDD via `utils/replace_probe`)

**Files:**
- Create: `utils/replace_probe/reef.toml`
- Create: `utils/replace_probe/src/main.reef`
- Modify: `src/replace.reef` (replace the stub `serialize_state` body)

Round-trip is verified in Task 5 (after `recover_state` is implemented). For now, the probe just calls `serialize_state` against a synthetic table and dumps the resulting file to stdout for visual inspection.

- [ ] **Step 1: Scaffold `utils/replace_probe/`**

Create `utils/replace_probe/reef.toml`:

```toml
[package]
name = "replace_probe"
version = "0.1.0"
author = "Chris Tusa <chris.tusa@leafscale.com>"
description = "Unit-test probe for zyginit live-replace state-file format"
license = "CDDL-1.0"

[build]
entry = "src/main.reef"
output = "replace_probe"
output_dir = "build"
source_dirs = ["src", "../../src"]

# This probe round-trips state.toml serialization without needing a
# running zyginit. Models utils/contract_probe.
#
# Linux: clang -c ../../src/helpers.c -o build/helpers.o && \
#        clang -c ../../src/contract_linux_stubs.c -o build/contract_linux_stubs.o && \
#        reefc build --obj build/helpers.o --obj build/contract_linux_stubs.o
```

Create `utils/replace_probe/src/main.reef` (test harness — exercises serialize and, in Task 5, also recover):

```reef
/******************************************************************************
                __               ____                __
               / /   ___  ____ _/ __/_____________ _/ /__
              / /   / _ \/ __ `/ /_/ ___/ ___/ __ `/ / _ \
             / /___/  __/ /_/ / __(__  ) /__/ /_/ / /  __/
            /_____/\___/\__,_/_/ /____/\___/\__,_/_/\___/

    (C)opyright 2025-2026, Leafscale, LLC -  https://www.leafscale.com

    Project: zyginit
   Filename: main.reef
    Authors: Chris Tusa <chris.tusa@leafscale.com>
    License: <see LICENSE file included with this source code>
Description: Probe — round-trip the state.toml format through replace.reef

******************************************************************************/

import config
import supervisor
import replace
import core.str
import sys.env as sysenv

mut g_pass: int = 0
mut g_fail: int = 0

proc check(label: string, ok: bool)
    if ok
        g_pass = g_pass + 1
        println("  PASS: " + label)
    else
        g_fail = g_fail + 1
        println("  FAIL: " + label)
    end if
end check

// Build a synthetic ServiceTable with known data. Uses minimal
// ServiceDef fields — enough for serialize/recover round-trip.
fn build_synthetic_table(): supervisor.ServiceTable
    let table = supervisor.new_service_table(8)

    // sshd: RUNNING with restart_count=2 and last_exit_code=0
    let def_sshd = config.new_service_def_minimal("sshd")
    let idx_sshd = supervisor.add_service(table, def_sshd)
    supervisor.set_runtime_for_test(table, idx_sshd, supervisor.STATE_RUNNING(),
        100, 23, 2, 0, 1746204051)

    // syslogd: RUNNING with no restarts
    let def_syslog = config.new_service_def_minimal("syslogd")
    let idx_syslog = supervisor.add_service(table, def_syslog)
    supervisor.set_runtime_for_test(table, idx_syslog, supervisor.STATE_RUNNING(),
        101, 17, 0, 0 - 1, 1746204047)

    // cron: STOPPED — must NOT be serialized
    let def_cron = config.new_service_def_minimal("cron")
    let idx_cron = supervisor.add_service(table, def_cron)
    supervisor.set_runtime_for_test(table, idx_cron, supervisor.STATE_STOPPED(),
        0 - 1, 0 - 1, 0, 0, 0)

    return table
end build_synthetic_table

proc test_serialize()
    println("== serialize_state ==")

    // Use $TMPDIR to avoid touching /var/run during dev tests.
    let tmpdir = sysenv.get_env_or("TMPDIR", "/tmp")
    sysenv.set_env("ZYGINIT_RUN_DIR", str.concat(tmpdir, "/replace_probe"))

    let table = build_synthetic_table()
    let ok = replace.serialize_state(table, 1746204000)
    check("serialize returns true on writable dir", ok)

    // (Round-trip assertions added in Task 5.)
end test_serialize

proc main()
    test_serialize()

    println("")
    println("--- " + int_to_str(g_pass) + " passed, " + int_to_str(g_fail) + " failed ---")
    if g_fail > 0
        sysenv.exit(1)
    end if
end main

fn int_to_str(n: int): string
    if n == 0
        return "0"
    end if
    mut value = n
    mut neg = false
    if n < 0
        value = 0 - n
        neg = true
    end if
    mut result = ""
    while value > 0
        let digit = value % 10
        result = str.concat(str.substring("0123456789", digit, 1), result)
        value = value / 10
    end while
    if neg
        result = str.concat("-", result)
    end if
    return result
end int_to_str
```

The probe references three new helpers in `supervisor` and `config` that don't exist yet:
- `config.new_service_def_minimal(name): ServiceDef` — constructs a ServiceDef with `name` set and other fields at default. Add it as a tiny `#[cfg(test)]`-equivalent helper at the bottom of `config.reef`'s export block, but unconditionally exported (Reef has no cfg-test gate).
- `supervisor.set_runtime_for_test(table, idx, state, pid, ctid, restarts, last_exit, last_start)` — directly writes the runtime fields on a `ServiceRuntime`, bypassing the normal start/stop lifecycle. Add to `supervisor.reef`'s export block.

- [ ] **Step 2: Add `config.new_service_def_minimal` to `src/config.reef`**

Locate the export block in `src/config.reef`, add the line `fn new_service_def_minimal(name: string): ServiceDef` to it. Find the place where `ServiceDef` is constructed (search for `ServiceDef{`) and add this fn body near it:

```reef
// Test/probe helper — minimal ServiceDef used by utils/replace_probe.
// Not used by production zyginit code paths.
fn new_service_def_minimal(name: string): ServiceDef
    return ServiceDef{
        name: name,
        description: "",
        type_: SERVICE_TYPE_DAEMON(),
        start_cmd: "/bin/true",
        stop_cmd: "",
        working_dir: "",
        user: "",
        group: "",
        environment: new [string](0),
        environment_count: 0,
        stop_method: STOP_METHOD_CONTRACT(),
        stop_signal: "TERM",
        stop_timeout: 60,
        requires: new [string](0),
        requires_count: 0,
        after: new [string](0),
        after_count: 0,
        contract_params: 0,
        contract_fatal: 0,
        restart_policy: RESTART_FAILURE(),
        restart_delay: 5,
        max_retries: 0 - 1,
        runlevel_mode: RUNLEVEL_MODE_MULTI()
    }
end new_service_def_minimal
```

Field names and constants in this constructor must match the actual `ServiceDef` struct in `config.reef`. Open the file, find `type ServiceDef = struct`, and copy the field list verbatim — adjust the constructor above if names differ. (Some fields may not exist in the struct as written here; that's OK — adapt to the real fields.)

- [ ] **Step 3: Add `supervisor.set_runtime_for_test` to `src/supervisor.reef`**

Add to the export block: `proc set_runtime_for_test(table: ServiceTable, idx: int, state: int, pid: int, contract_id: int, restart_count: int, last_exit_code: int, last_start_time: int)`. Then add the body near the other accessors:

```reef
// Test/probe helper — directly write runtime fields on a ServiceRuntime,
// bypassing the normal start/stop lifecycle. Used only by
// utils/replace_probe; production code goes through start_service /
// handle_contract_event.
proc set_runtime_for_test(table: ServiceTable, idx: int, state: int, pid: int,
                          contract_id: int, restart_count: int,
                          last_exit_code: int, last_start_time: int)
    let rt = table.runtimes[idx]
    rt.state = state
    rt.pid = pid
    rt.contract_id = contract_id
    rt.restart_count = restart_count
    rt.last_exit_code = last_exit_code
    rt.last_start_time = last_start_time
    table.runtimes[idx] = rt

    // Also register contract mapping so apply_recovered_state can find
    // it later if the test exercises recovery.
    if contract_id >= 0
        table.contract_map.set(int_to_str_helper(contract_id), idx)
    end if
end set_runtime_for_test

// Local helper — int_to_str is private in this module; if not already
// present scan for it. (It exists at line ~870 as a private fn — reuse.)
fn int_to_str_helper(n: int): string
    return int_to_str(n)
end int_to_str_helper
```

- [ ] **Step 4: Implement `serialize_state` in `src/replace.reef`**

Replace the stub. The body opens a tmp file, writes the boot_time line then a `[[service]]` block per RUNNING service, fsyncs, closes, renames to the final path, fsyncs the parent dir.

```reef
fn serialize_state(table: supervisor.ServiceTable, boot_time: int): bool
    let tmp_path = STATE_TMP_PATH()
    let final_path = STATE_FILE_PATH()
    let dir_path = STATE_DIR()

    // O_WRONLY|O_CREAT|O_TRUNC = 0x301 on Hammerhead/Linux for the values
    // we use in fd.O_*. Using fd module's helpers keeps this portable.
    let flags = fd.O_WRONLY() + fd.O_CREAT() + fd.O_TRUNC()
    let out = fd.fd_open(tmp_path, flags, 384)   // mode 0600

    if out < 0
        println("replace: serialize: cannot open " + tmp_path)
        return false
    end if

    // Write boot_time header
    let header = str.concat("boot_time = ", int_to_str(boot_time))
    fd.fd_write(out, str.concat(header, "\n"))

    // Iterate services — only RUNNING ones go in. Skip the rest.
    let count = supervisor.service_count(table)
    mut i = 0
    while i < count
        let rt = supervisor.get_runtime(table, i)
        if supervisor.rt_state(rt) == supervisor.STATE_RUNNING()
            let def = supervisor.rt_def(rt)
            fd.fd_write(out, "\n[[service]]\n")
            fd.fd_write(out, str.concat("name = \"", str.concat(config.svc_name(def), "\"\n")))
            fd.fd_write(out, str.concat("contract_id = ", str.concat(int_to_str(supervisor.rt_contract_id(rt)), "\n")))
            fd.fd_write(out, str.concat("restart_count = ", str.concat(int_to_str(supervisor.rt_restart_count(rt)), "\n")))
            fd.fd_write(out, str.concat("last_start_time = ", str.concat(int_to_str(supervisor.rt_last_start_time(rt)), "\n")))
            fd.fd_write(out, str.concat("last_exit_code = ", str.concat(int_to_str(supervisor.rt_last_exit_code(rt)), "\n")))
        end if
        i = i + 1
    end while

    // fsync + close
    if fd.fd_fsync(out) < 0
        fd.fd_close(out)
        unlink(tmp_path)
        return false
    end if
    fd.fd_close(out)

    // Atomic rename
    if rename(tmp_path, final_path) != 0
        unlink(tmp_path)
        return false
    end if

    // fsync the parent dir so the rename is durable
    let _ = zyginit_fsync_path(dir_path)

    return true
end serialize_state

// Local int_to_str — same shape as in supervisor.reef. Replicate to
// keep the module standalone.
fn int_to_str(n: int): string
    if n == 0
        return "0"
    end if
    mut value = n
    mut neg = false
    if n < 0
        value = 0 - n
        neg = true
    end if
    mut result = ""
    while value > 0
        let digit = value % 10
        result = str.concat(str.substring("0123456789", digit, 1), result)
        value = value / 10
    end while
    if neg
        result = str.concat("-", result)
    end if
    return result
end int_to_str
```

The `fd.fd_fsync` call may not exist if Reef's `sys.fd` doesn't expose it — if reefc errors on that name, replace with a direct `extern "C" fn fsync(fd: int): int` declaration in this module and call it as `fsync(out)`.

- [ ] **Step 5: Build the main project to verify replace.reef compiles**

Run: `clang -c src/helpers.c -o build/helpers.o && clang -c src/contract_linux_stubs.c -o build/contract_linux_stubs.o && reefc build --obj build/helpers.o --obj build/contract_linux_stubs.o`
Expected: success.

- [ ] **Step 6: Build the probe**

Run:
```bash
cd utils/replace_probe
mkdir -p build
clang -c ../../src/helpers.c -o build/helpers.o
clang -c ../../src/contract_linux_stubs.c -o build/contract_linux_stubs.o
reefc build --obj build/helpers.o --obj build/contract_linux_stubs.o
cd ../..
```
Expected: success.

- [ ] **Step 7: Run the probe and inspect the file**

Run:
```bash
mkdir -p /tmp/replace_probe
./utils/replace_probe/build/replace_probe
cat /tmp/replace_probe/state.toml
```

Expected:
- "PASS: serialize returns true on writable dir"
- File contents:
  ```
  boot_time = 1746204000

  [[service]]
  name = "sshd"
  contract_id = 23
  restart_count = 2
  last_start_time = 1746204051
  last_exit_code = 0

  [[service]]
  name = "syslogd"
  contract_id = 17
  restart_count = 0
  last_start_time = 1746204047
  last_exit_code = -1
  ```
- `cron` (STATE_STOPPED) is **not** in the output.

- [ ] **Step 8: Commit**

```bash
hg add utils/replace_probe/reef.toml utils/replace_probe/src/main.reef
hg commit -m "replace: implement serialize_state + replace_probe round-trip harness"
```

---

## Task 5: Implement `recover_state` (TDD via `utils/replace_probe`)

**Files:**
- Modify: `src/replace.reef` (replace the stub `recover_state` body)
- Modify: `utils/replace_probe/src/main.reef` (add round-trip + edge-case assertions)

`recover_state` reads the state.toml file via the existing `encoding.toml` parser, validates `boot_time`, and returns either an empty `RecoveredState` (if file missing/corrupt/stale) or a populated one. The empty/stale path also deletes the file.

- [ ] **Step 1: Implement `recover_state`**

Replace the stub in `src/replace.reef`:

```reef
fn recover_state(current_boot_time: int): RecoveredState
    let path = STATE_FILE_PATH()

    if not file.fileExists(path)
        return new_recovered_state()
    end if

    let content = file.readFile(path)
    if str.length(content) == 0
        // Treat empty as missing
        let _ = unlink(path)
        return new_recovered_state()
    end if

    // Parse via encoding.toml. Section-array entries flatten to
    // dotted-key arrays; the parser exposes them via toml.get_array.
    // The helper below wraps a parse + extraction.
    let parsed = toml.parse(content)
    if not toml.parse_ok(parsed)
        println("replace: recover: parse error in " + path + " — ignoring")
        let _ = unlink(path)
        return new_recovered_state()
    end if

    let saved_boot = toml.get_int_or(parsed, "boot_time", 0 - 1)
    if saved_boot != current_boot_time or current_boot_time < 0
        println("replace: recover: stale state file (boot_time " + int_to_str(saved_boot) +
                " != current " + int_to_str(current_boot_time) + "); ignoring")
        let _ = unlink(path)
        return new_recovered_state()
    end if

    let n = toml.array_count(parsed, "service")
    let arr = new [RecoveredService](n)
    mut i = 0
    while i < n
        let prefix = str.concat("service.", int_to_str(i))
        arr[i] = RecoveredService{
            name: toml.get_string_or(parsed, str.concat(prefix, ".name"), ""),
            contract_id: toml.get_int_or(parsed, str.concat(prefix, ".contract_id"), 0 - 1),
            restart_count: toml.get_int_or(parsed, str.concat(prefix, ".restart_count"), 0),
            last_start_time: toml.get_int_or(parsed, str.concat(prefix, ".last_start_time"), 0),
            last_exit_code: toml.get_int_or(parsed, str.concat(prefix, ".last_exit_code"), 0 - 1)
        }
        i = i + 1
    end while

    // State file consumed — delete to avoid panic-restart loops.
    let _ = unlink(path)

    return RecoveredState{
        boot_time: saved_boot,
        services: arr,
        count: n
    }
end recover_state
```

The `toml.parse_ok`, `toml.get_int_or`, `toml.get_string_or`, `toml.array_count` helper names are placeholders — open `~/repos/reef-lang/stdlib/encoding/toml.reef` (or wherever the project's encoding.toml lives — try `grep -rn 'module toml' ~/repos/reef-lang/`) and use the actual function names. The existing `config.load_enabled_services` in `src/config.reef` is a working example of how to walk a parsed TOML array — copy its access pattern verbatim if the helper names above don't exist.

- [ ] **Step 2: Extend the probe with round-trip + edge-case assertions**

Replace `test_serialize()` in `utils/replace_probe/src/main.reef` with:

```reef
proc test_round_trip()
    println("== round-trip ==")
    let tmpdir = sysenv.get_env_or("TMPDIR", "/tmp")
    sysenv.set_env("ZYGINIT_RUN_DIR", str.concat(tmpdir, "/replace_probe"))

    // Make sure the state dir exists
    let _ = io.dir.create_dir_all(str.concat(tmpdir, "/replace_probe"))

    let table = build_synthetic_table()
    let serialized = replace.serialize_state(table, 1746204000)
    check("serialize returns true", serialized)

    let recovered = replace.recover_state(1746204000)
    check("recover boot_time matches", replace.rs_boot_time(recovered) == 1746204000)
    check("recover count == 2 (running only)", replace.rs_count(recovered) == 2)

    // Find services by name (order is implementation-defined)
    mut found_sshd = false
    mut found_syslog = false
    mut i = 0
    while i < replace.rs_count(recovered)
        let svc = replace.rs_service(recovered, i)
        let name = replace.svc_name(svc)
        if name == "sshd"
            found_sshd = true
            check("sshd contract_id = 23", replace.svc_contract_id(svc) == 23)
            check("sshd restart_count = 2", replace.svc_restart_count(svc) == 2)
            check("sshd last_exit_code = 0", replace.svc_last_exit_code(svc) == 0)
        elif name == "syslogd"
            found_syslog = true
            check("syslogd contract_id = 17", replace.svc_contract_id(svc) == 17)
            check("syslogd last_exit_code = -1", replace.svc_last_exit_code(svc) == 0 - 1)
        end if
        i = i + 1
    end while
    check("found sshd", found_sshd)
    check("found syslogd", found_syslog)

    // After recover_state, state.toml is deleted
    check("state.toml deleted after recovery", not io.file.fileExists(replace.STATE_FILE_PATH()))
end test_round_trip

proc test_stale_boot_time()
    println("== stale state file ==")
    let tmpdir = sysenv.get_env_or("TMPDIR", "/tmp")
    sysenv.set_env("ZYGINIT_RUN_DIR", str.concat(tmpdir, "/replace_probe"))

    let table = build_synthetic_table()
    let _ = replace.serialize_state(table, 1746204000)

    // Recover with DIFFERENT boot_time -> stale -> empty + deleted
    let recovered = replace.recover_state(1746999999)
    check("stale file -> empty state", replace.rs_is_empty(recovered))
    check("stale file deleted", not io.file.fileExists(replace.STATE_FILE_PATH()))
end test_stale_boot_time

proc test_missing_file()
    println("== missing state file ==")
    let tmpdir = sysenv.get_env_or("TMPDIR", "/tmp")
    sysenv.set_env("ZYGINIT_RUN_DIR", str.concat(tmpdir, "/replace_probe_missing"))

    // Make sure file definitely doesn't exist
    let _ = io.dir.create_dir_all(str.concat(tmpdir, "/replace_probe_missing"))
    let _ = unlink(replace.STATE_FILE_PATH())

    let recovered = replace.recover_state(1746204000)
    check("missing file -> empty state", replace.rs_is_empty(recovered))
end test_missing_file

extern "C" fn unlink(path: string): int

proc main()
    test_round_trip()
    test_stale_boot_time()
    test_missing_file()

    println("")
    println("--- " + int_to_str(g_pass) + " passed, " + int_to_str(g_fail) + " failed ---")
    if g_fail > 0
        sysenv.exit(1)
    end if
end main
```

Add `import io.dir` and `import io.file` near the top of the probe.

- [ ] **Step 3: Build and run**

Run:
```bash
cd utils/replace_probe
reefc build --obj build/helpers.o --obj build/contract_linux_stubs.o
./build/replace_probe
cd ../..
```

Expected output ends with `--- 11 passed, 0 failed ---` (1 from serialize + 10 from round_trip / stale / missing — adjust if asserts differ).

If it fails, the most likely cause is the TOML parser helper names. Read `encoding.toml`'s actual API and adjust `recover_state`.

- [ ] **Step 4: Commit**

```bash
hg commit -m "replace: implement recover_state with round-trip + edge-case probe tests"
```

---

## Task 6: Implement `check_preconditions` (TDD)

**Files:**
- Modify: `src/replace.reef` (replace the stub `check_preconditions`)
- Modify: `utils/replace_probe/src/main.reef` (add precondition tests)

- [ ] **Step 1: Implement**

Replace the stub:

```reef
fn check_preconditions(table: supervisor.ServiceTable): string
    let count = supervisor.service_count(table)
    mut report = ""
    mut found = 0
    mut i = 0
    while i < count
        let rt = supervisor.get_runtime(table, i)
        let st = supervisor.rt_state(rt)
        if st == supervisor.STATE_STARTING() or st == supervisor.STATE_STOPPING() or st == supervisor.STATE_WAITING()
            let entry = str.concat(config.svc_name(supervisor.rt_def(rt)),
                str.concat(":", supervisor.state_name(st)))
            if found > 0
                report = str.concat(report, ", ")
            end if
            report = str.concat(report, entry)
            found = found + 1
        end if
        i = i + 1
    end while
    return report
end check_preconditions
```

Note: `STATE_WAITING` is currently used both for "not yet started" services at boot and for "scheduled-for-restart". For the precondition check we want only the latter (a transient mid-restart state) — but the state machine doesn't distinguish them. Conservative: treat all WAITING as transient. Operators using `replace` on a fresh boot before services have started would otherwise see "blocked"; that's an acceptable false positive.

- [ ] **Step 2: Add probe assertions**

Append to `utils/replace_probe/src/main.reef`:

```reef
proc test_preconditions()
    println("== preconditions ==")
    let table = supervisor.new_service_table(8)

    // sshd RUNNING — stable
    let def_sshd = config.new_service_def_minimal("sshd")
    let idx_sshd = supervisor.add_service(table, def_sshd)
    supervisor.set_runtime_for_test(table, idx_sshd, supervisor.STATE_RUNNING(),
        100, 23, 0, 0 - 1, 1746204051)

    let r1 = replace.check_preconditions(table)
    check("all-RUNNING -> empty report", str.length(r1) == 0)

    // Add a STOPPING service
    let def_cron = config.new_service_def_minimal("cron")
    let idx_cron = supervisor.add_service(table, def_cron)
    supervisor.set_runtime_for_test(table, idx_cron, supervisor.STATE_STOPPING(),
        101, 24, 0, 0 - 1, 1746204060)

    let r2 = replace.check_preconditions(table)
    check("STOPPING -> non-empty report", str.length(r2) > 0)
    check("report mentions cron", str.index_of(r2, "cron") >= 0)
    check("report mentions stopping", str.index_of(r2, "stopping") >= 0)
end test_preconditions
```

Add `test_preconditions()` to `proc main()`.

- [ ] **Step 3: Build and run**

Run:
```bash
cd utils/replace_probe
reefc build --obj build/helpers.o --obj build/contract_linux_stubs.o
./build/replace_probe
cd ../..
```

Expected: tests pass.

- [ ] **Step 4: Commit**

```bash
hg commit -m "replace: implement check_preconditions with transient-state detection"
```

---

## Task 7: Add `contract.is_contract_empty` helper

**Files:**
- Modify: `src/contract.reef`
- Modify: `src/contract_linux_stubs.c`

`is_contract_empty(ctid)` reads contract status via `ct_status_read` on `/system/contract/process/<id>/status`, then checks the member-count via `ct_status_get_nmembers`. Returns `true` if the contract has zero members (or the status read fails — caller treats unknown as "assume empty for safety").

- [ ] **Step 1: Add the FFI for nmembers and the helper**

In `src/contract.reef`, add to the FFI declarations near the other `ct_status_*` lines:

```reef
extern "C" fn ct_status_get_nmembers(stathdl: pointer): int
```

Add to the export block:

```reef
fn is_contract_empty(contract_id: int): bool
```

Add the body near `adopt_contract`:

```reef
// Check whether a contract has zero member processes. Used by the
// post-recovery idempotency step in main.apply_recovered_state — if a
// contract was reported in state.toml but the kernel says it's empty,
// the service exited during the exec gap and we should apply restart
// policy as if a contract-empty event arrived.
//
// Returns true if the contract has zero members OR if the status
// read fails (treat unknown as empty so the recovery path applies
// restart policy rather than leaving a phantom service in RUNNING).
fn is_contract_empty(contract_id: int): bool
    let status_path = str.concat(str.concat("/system/contract/process/", int_to_str(contract_id)), "/status")
    let st_fd = open(status_path, O_RDONLY())
    if st_fd < 0
        return true
    end if

    unsafe
        let stathdl_buf = new [pointer](1)
        let rc = ct_status_read(st_fd, CTD_COMMON(), stathdl_buf)
        close(st_fd)

        if rc != 0
            return true
        end if

        let stathdl = stathdl_buf[0]
        let nmembers = ct_status_get_nmembers(stathdl)
        ct_status_free(stathdl)

        return nmembers == 0
    end unsafe
end is_contract_empty
```

- [ ] **Step 2: Add the Linux stub**

In `src/contract_linux_stubs.c`, append:

```c
int ct_status_get_nmembers(void *stathdl) { (void)stathdl; return 0; }
```

- [ ] **Step 3: Build**

Run: `clang -c src/helpers.c -o build/helpers.o && clang -c src/contract_linux_stubs.c -o build/contract_linux_stubs.o && reefc build --obj build/helpers.o --obj build/contract_linux_stubs.o`
Expected: success.

- [ ] **Step 4: Commit**

```bash
hg commit -m "contract: add is_contract_empty helper for post-recovery empty-contract check"
```

---

## Task 8: Add socket `replace` command + flag plumbing

**Files:**
- Modify: `src/socket.reef`

Add a `g_replace_requested` flag and a `g_replace_wait` int, accessors, and the command handler. Mirror the existing `g_reload_requested` pattern.

- [ ] **Step 1: Add accessors to the export block**

In `src/socket.reef`, add to the existing export block (near `reload_requested` line 37):

```reef
    fn replace_requested(): bool
    proc clear_replace_flag()
    fn replace_wait_seconds(): int
```

- [ ] **Step 2: Add the globals and accessors**

Near the existing `mut g_reload_requested: bool = false` declaration (around line 49), add:

```reef
// Replace flag — set by "replace" command, checked by main event loop
mut g_replace_requested: bool = false
mut g_replace_wait: int = 0

fn replace_requested(): bool
    return g_replace_requested
end replace_requested

proc clear_replace_flag()
    g_replace_requested = false
    g_replace_wait = 0
end clear_replace_flag

fn replace_wait_seconds(): int
    return g_replace_wait
end replace_wait_seconds
```

- [ ] **Step 3: Add the command handler in `dispatch_command`**

Insert in `dispatch_command` (around line 184, after the reload branch):

```reef
    elif command == "replace"
        // Optional --wait=N argument
        mut wait_seconds = 0
        if str.length(arg) > 0
            // Parse --wait=N
            if str.starts_with(arg, "--wait=")
                let val = str.substring(arg, 7, str.length(arg) - 7)
                wait_seconds = parse_int_or(val, 0)
            else
                return "error: replace: unknown argument: " + arg + "\n"
            end if
        end if
        g_replace_requested = true
        g_replace_wait = wait_seconds
        return str.concat("replace queued (wait=", str.concat(int_to_str(wait_seconds), ")\n"))
```

- [ ] **Step 4: Add the `parse_int_or` helper if not present**

Search for it: `grep -n 'parse_int_or\|fn parse_int' src/socket.reef`. If absent, add at the bottom of the file:

```reef
fn parse_int_or(s: string, default_val: int): int
    let n = str.length(s)
    if n == 0
        return default_val
    end if
    mut result = 0
    mut i = 0
    while i < n
        let c = str.char_at(s, i)
        if c < '0' or c > '9'
            return default_val
        end if
        result = result * 10 + (c - '0')
        i = i + 1
    end while
    return result
end parse_int_or
```

If `str.starts_with` doesn't exist, write it inline using `str.substring(arg, 0, 7) == "--wait="`.

- [ ] **Step 5: Update the `cmd_help` line to mention replace**

Find `cmd_help` (around line 413) and add `replace [--wait=N]` to the comma-separated command list.

- [ ] **Step 6: Build**

Run: full Linux build command.
Expected: success.

- [ ] **Step 7: Smoke test the socket command**

Run:
```bash
TEST=$(mktemp -d)
ZYGINIT_CONFIG_DIR=$TEST/cfg ZYGINIT_SOCKET=$TEST/sock ZYGINIT_LOG_DIR=$TEST/log mkdir -p $TEST/cfg/enabled.d
ZYGINIT_CONFIG_DIR=$TEST/cfg ZYGINIT_SOCKET=$TEST/sock ZYGINIT_LOG_DIR=$TEST/log ./build/zyginit &
ZPID=$!
sleep 0.5
echo 'replace --wait=10' | nc -U $TEST/sock
kill $ZPID 2>/dev/null
rm -rf $TEST
```

Expected: `replace queued (wait=10)`. (Don't worry that the daemon doesn't actually replace — the actual flow lands in Task 11 once `replace_self` is wired up. This test only verifies the command parses and the flag is set.)

- [ ] **Step 8: Commit**

```bash
hg commit -m "socket: add replace [--wait=N] command + flag plumbing"
```

---

## Task 9: Add `apply_recovered_state` proc in `main.reef`

**Files:**
- Modify: `src/main.reef`

This is the incoming-side stitching logic. For each entry in the recovered state, find the service by name, populate runtime fields, register the contract_map entry, and run the post-recovery `is_contract_empty` check.

- [ ] **Step 1: Add the proc near `reload_services`**

Find `reload_services` (around line 328) and insert before it:

```reef
// Apply state from a previous zyginit instance (live-replace).
// For each entry in `recovered`, locate the service in `table` and
// patch in runtime fields. Services in state.toml that are no longer
// in enabled.d/ get their contract abandoned but their member
// processes left running. After patching, run the post-recovery
// empty-contract check to catch services that exited during the
// exec gap.
proc apply_recovered_state(table: supervisor.ServiceTable,
                           recovered: replace.RecoveredState)
    if replace.rs_is_empty(recovered)
        return
    end if

    let n = replace.rs_count(recovered)
    println("zyginit: replace: applying " + int_to_str(n) + " recovered services")

    mut i = 0
    while i < n
        let rs = replace.rs_service(recovered, i)
        let name = replace.svc_name(rs)
        let ctid = replace.svc_contract_id(rs)

        let idx = supervisor.find_service(table, name)
        if idx < 0
            // Operator removed the symlink between old start and replace.
            // Don't kill — abandon the contract; member processes
            // continue running unsupervised.
            let abrc = contract.abandon_contract(ctid)
            println("zyginit: replace: orphaned " + name +
                    " (no longer enabled, contract " + int_to_str(ctid) +
                    " abandon rc=" + int_to_str(abrc) + ")")
            i = i + 1
            continue
        end if

        // Patch runtime fields. PID is unknown (not serialized); contract
        // path is sufficient for supervision.
        supervisor.set_runtime_for_test(table, idx, supervisor.STATE_RUNNING(),
            0 - 1, ctid,
            replace.svc_restart_count(rs),
            replace.svc_last_exit_code(rs),
            replace.svc_last_start_time(rs))

        println("zyginit: replace: re-attached " + name +
                " (ctid=" + int_to_str(ctid) +
                ", restarts=" + int_to_str(replace.svc_restart_count(rs)) + ")")

        // Idempotency: did the contract go empty during the exec gap?
        if contract.is_contract_empty(ctid)
            println("zyginit: replace: " + name +
                    " contract empty post-recovery, applying restart policy")
            supervisor.handle_contract_event(table, ctid, 0 - 1)
        end if

        i = i + 1
    end while
end apply_recovered_state
```

`supervisor.set_runtime_for_test` is the same helper added in Task 4 — its name communicates "test/probe usage" but it does exactly what `apply_recovered_state` needs (write the runtime fields directly). Acceptable. If the name bothers you, rename to `supervisor.adopt_runtime` and update both call sites.

- [ ] **Step 2: Add the import at the top of `main.reef`**

Find the imports block (top of file). Add:

```reef
import replace
```

- [ ] **Step 3: Build**

Run: full Linux build.
Expected: success.

- [ ] **Step 4: Commit**

```bash
hg commit -m "main: add apply_recovered_state proc for live-replace stitching"
```

---

## Task 10: Add `replace_self` proc in `main.reef`

**Files:**
- Modify: `src/main.reef`

The outgoing-side flow. Pre-condition check (with optional `--wait` poll), state-file write, socket unlink, exec.

- [ ] **Step 1: Add FFI and helpers near the top of `main.reef`**

If not already present, add to the extern block:

```reef
extern "C" fn zyginit_fsync_path(path: string): int
extern "C" fn unlink(path: string): int
```

- [ ] **Step 2: Add the proc near `apply_recovered_state`**

```reef
// Operator-driven live replace. Triggered by zygctl replace; the
// socket handler sets g_replace_requested and we pick it up on the
// next event-loop tick. After this returns successfully, control
// transfers to the new /sbin/init process; this fn does not return
// on success.
//
// On precondition failure or any I/O error before exec, we log and
// return false; the caller clears the flag and continues normally.
proc replace_self(table: supervisor.ServiceTable, boot_time: int, wait_seconds: int)
    println("zyginit: replace: requested (wait=" + int_to_str(wait_seconds) + ")")

    // Pre-condition check, with optional wait window
    mut elapsed = 0
    mut report = replace.check_preconditions(table)
    while str.length(report) > 0 and elapsed < wait_seconds
        clock.sleep_seconds(1)
        elapsed = elapsed + 1
        report = replace.check_preconditions(table)
    end while

    if str.length(report) > 0
        println("zyginit: replace: blocked: " + report)
        return
    end if

    // Final draining of pending contract events — services may have just
    // exited and their state hasn't been recorded yet.
    // (Reuse the existing handle_contract_events drain logic if exposed;
    // otherwise the post-recovery is_contract_empty check on the new
    // side covers any miss.)

    // Serialize state
    if not replace.serialize_state(table, boot_time)
        println("zyginit: replace: state serialization failed; aborting")
        return
    end if
    println("zyginit: replace: state written to " + replace.STATE_FILE_PATH())

    // Unlink socket. Proceed even on failure — new zyginit will overwrite.
    // zyginit_fsync_path falls back to fsyncing the parent dir when
    // the path itself doesn't exist, which is exactly what we want
    // after unlink — the socket's parent dir gets the rename committed.
    let sock_path = SOCKET_PATH()
    let urc = unlink(sock_path)
    if urc != 0
        println("zyginit: replace: warning: socket unlink failed (rc=" + int_to_str(urc) + ")")
    end if
    let _ = zyginit_fsync_path(sock_path)

    // Exec /sbin/init. argv[0] should match what the kernel originally
    // exec'd us with; we pass "/sbin/init" verbatim.
    let target = "/sbin/init"
    let argv = new [string](1)
    argv[0] = target

    println("zyginit: replace: execve " + target)
    process.process_exec(target, argv)

    // If we reach here, exec failed. We have already written state.toml
    // and unlinked the socket — recovery requires reboot.
    println("zyginit: replace: FATAL: execve returned (kernel could not load " + target + ")")
end replace_self
```

The "drain pending contract events" comment block is intentional — the current code path for handling bundle events lives in the main loop. If you want to drain them once more here, factor that out into a `proc drain_bundle_events(table, bundle_fd)` and call it; otherwise rely on the post-recovery `is_contract_empty` check (which is sufficient per the spec).

- [ ] **Step 3: Build**

Expected: success.

- [ ] **Step 4: Commit**

```bash
hg commit -m "main: add replace_self proc for live-replace outgoing flow"
```

---

## Task 11: Wire `main.reef` startup + event loop

**Files:**
- Modify: `src/main.reef`

Two integration points:
- **Startup:** call `replace.recover_state` and `apply_recovered_state` between Phase 3 (table built) and Phase 6 (start_services_by_tier).
- **Event loop:** check `socket.replace_requested()` and call `replace_self`.

Also: capture `g_boot_time` once during PID-1 init so it's available to both call sites.

- [ ] **Step 1: Add `g_boot_time` global**

Near the top of `main.reef` with the other `mut g_*` globals, add:

```reef
mut g_boot_time: int = 0 - 1
```

- [ ] **Step 2: Capture boot_time at the start of `main()` after writing the BOOT_TIME utmpx record**

Right after the existing `let _ = zyginit_write_boot_utmpx()` at line ~725:

```reef
        // Cache the BOOT_TIME we just wrote so live-replace can use
        // it as the boot-id sentinel. Read it back from utmpx (rather
        // than time_now() locally) so the value matches what the new
        // zyginit will read after exec.
        g_boot_time = replace.read_boot_time()
        if g_boot_time < 0
            // Fallback: use time_now(). This degrades the staleness
            // check (a real reboot might collide if seconds-resolution
            // happens to repeat), but it's better than refusing all
            // replaces.
            g_boot_time = time.time_now()
            println("zyginit: warning: utmpx BOOT_TIME read failed; using time_now() = " + int_to_str(g_boot_time))
        end if
```

For non-PID-1 supervisor mode, `g_boot_time` stays at -1; replace.serialize_state still writes "-1" and the new instance reads "-1" — they match, and the path is intentionally permissive in supervisor mode.

Actually — set it for non-PID-1 too:

```reef
    // (After the if is_pid_1() block)
    if g_boot_time < 0
        g_boot_time = time.time_now()
    end if
```

- [ ] **Step 3: Insert recovery between Phase 3 and Phase 6**

After the `i + 1` add_service loop (around line 778) and before signal init:

```reef
    // ---- Phase 3.5: Apply state from previous zyginit (live replace) ----

    let recovered = replace.recover_state(g_boot_time)
    if not replace.rs_is_empty(recovered)
        apply_recovered_state(table, recovered)
    end if
```

This lands AFTER `add_service` so all services are registered, BEFORE `start_services_by_tier` so re-attached services are skipped (they're in `STATE_RUNNING` and `start_service` only acts on `STATE_WAITING`).

- [ ] **Step 4: Add the replace check in the main event loop**

Find the existing reload check (around line 858-862):

```reef
        // Check for reload requested via socket
        if socket.reload_requested()
            socket.clear_reload_flag()
            println("zyginit: reload requested via socket")
            reload_services(table)
        end if
```

Add immediately after:

```reef
        // Check for replace requested via socket
        if socket.replace_requested()
            let wait = socket.replace_wait_seconds()
            socket.clear_replace_flag()
            replace_self(table, g_boot_time, wait)
            // If replace_self exec'd successfully, we never get here.
            // If it returned, log and continue running normally.
        end if
```

- [ ] **Step 5: Build the project AND zygctl**

Run:
```bash
clang -c src/helpers.c -o build/helpers.o
clang -c src/contract_linux_stubs.c -o build/contract_linux_stubs.o
reefc build --obj build/helpers.o --obj build/contract_linux_stubs.o
```
Expected: success, `build/zyginit` produced.

- [ ] **Step 6: Smoke test in supervisor mode**

```bash
TEST=$(mktemp -d)
mkdir -p $TEST/cfg/enabled.d $TEST/log
ZYGINIT_CONFIG_DIR=$TEST/cfg ZYGINIT_SOCKET=$TEST/sock ZYGINIT_LOG_DIR=$TEST/log ZYGINIT_RUN_DIR=$TEST/run ./build/zyginit > $TEST/zyginit.log 2>&1 &
ZPID=$!
sleep 0.5
echo 'replace' | nc -U $TEST/sock
sleep 0.5
# zyginit should have logged "zyginit: replace: state written..." and
# attempted execve /sbin/init (which fails on Linux dev because
# /sbin/init isn't this binary). Acceptable failure.
cat $TEST/zyginit.log
kill $ZPID 2>/dev/null
rm -rf $TEST
```

Expected log lines:
- `zyginit: replace: requested (wait=0)`
- `zyginit: replace: state written to /tmp/.../state.toml` (or the ZYGINIT_RUN_DIR you set)
- `zyginit: replace: execve /sbin/init`
- `zyginit: replace: FATAL: execve returned ...` (only if /sbin/init isn't usable as a Linux binary in this dev environment)

The integration test will refine this in Task 13.

- [ ] **Step 7: Commit**

```bash
hg commit -m "main: wire live-replace into startup recovery and event loop"
```

---

## Task 12: Add `zygctl replace` subcommand

**Files:**
- Modify: `tools/zygctl/src/zygctl.reef`

- [ ] **Step 1: Locate the command dispatch**

Open `tools/zygctl/src/zygctl.reef` and find the dispatch (search for `"reload"` — there'll be a similar pattern for the existing socket-relayed commands).

- [ ] **Step 2: Add `replace` to the dispatch**

Pattern: build the command line (`replace` + optional `--wait=N`), connect to socket, send, read response, print, handle `EPIPE` / connection-closed cleanly.

```reef
    elif cmd == "replace"
        // Optional --wait=N
        mut payload = "replace"
        let argc = sys.args.count()
        if argc >= 3
            let wait_arg = sys.args.get(2)
            if str.starts_with(wait_arg, "--wait=")
                payload = str.concat(payload, str.concat(" ", wait_arg))
            else
                println("zygctl: replace: unknown argument: " + wait_arg)
                println("zygctl: usage: zygctl replace [--wait=N]")
                process.exit_now(2)
            end if
        end if

        let response = send_command(payload)
        if str.length(response) == 0
            // Connection closed without a response — the server proceeded
            // to exec immediately. Treat as success.
            println("replace: socket closed (process replaced)")
        else
            print(response)
        end if
```

`send_command` is the existing helper that connects to the socket and reads the response — adapt to whatever name is in use. If the existing pattern doesn't gracefully handle "connection closed without bytes received", treat that as a successful replace and print the documented message.

- [ ] **Step 3: Update help text**

Find the help text in zygctl.reef (likely a `cmd_help` proc) and add `replace [--wait=N]` to the list.

- [ ] **Step 4: Build zygctl**

Run:
```bash
cd tools/zygctl
clang -c src/symlink_wrapper.c -o build/symlink_wrapper.o
reefc build --obj build/symlink_wrapper.o
cd ../..
```

Expected: success, `tools/zygctl/build/zygctl` produced.

- [ ] **Step 5: Smoke test**

```bash
TEST=$(mktemp -d)
mkdir -p $TEST/cfg/enabled.d $TEST/log
ZYGINIT_CONFIG_DIR=$TEST/cfg ZYGINIT_SOCKET=$TEST/sock ZYGINIT_LOG_DIR=$TEST/log ZYGINIT_RUN_DIR=$TEST/run ./build/zyginit > $TEST/zyginit.log 2>&1 &
ZPID=$!
sleep 0.5
ZYGINIT_SOCKET=$TEST/sock ./tools/zygctl/build/zygctl replace --wait=5
sleep 0.5
cat $TEST/zyginit.log
kill $ZPID 2>/dev/null
rm -rf $TEST
```

Expected zygctl output: `replace queued (wait=5)` OR `replace: socket closed (process replaced)`.

- [ ] **Step 6: Commit**

```bash
hg commit -m "zygctl: add replace [--wait=N] subcommand"
```

---

## Task 13: Linux integration smoke tests

**Files:**
- Modify: `tests/integration/run_tests.sh`

Add new test scenarios after the existing tests. Linux can't exercise actual contract adoption (stubs return -1), but it CAN verify:
- `replace` command parses and acks
- Pre-condition failures are reported
- State file writes correctly
- The exec attempt fires (it'll fail because `/sbin/init` isn't this binary on a Linux dev box, but that's a known Linux-only failure)

- [ ] **Step 1: Locate where new test sections are appended in run_tests.sh**

`grep -n '^section ' tests/integration/run_tests.sh | tail` will show the existing section markers. Insert a new section block before the final summary.

- [ ] **Step 2: Add the replace command-parse test**

Append before the final summary:

```bash
section "replace command — basic parsing"

# Start a fresh zyginit
start_zyginit
mkdir -p "$TEST_DIR/run"
export ZYGINIT_RUN_DIR="$TEST_DIR/run"
restart_zyginit_with_run_dir() {
    stop_zyginit
    "$ZYGINIT" >> "$DAEMON_LOG" 2>&1 &
    DAEMON_PID=$!
    wait_for_socket
}
restart_zyginit_with_run_dir

# zygctl replace with no args
out=$("$ZYGCTL" replace 2>&1 || true)
assert_contains "$out" "replace queued" "replace acks with no args"

# zygctl replace --wait=10
out=$("$ZYGCTL" replace --wait=10 2>&1 || true)
assert_contains "$out" "replace queued (wait=10)" "replace acks with --wait=10"

# Check state.toml was written before exec attempt
sleep 0.2
if [ -f "$TEST_DIR/run/state.toml" ]; then
    pass "state.toml present after replace"
else
    # Either still waiting OR exec succeeded and the new instance deleted it.
    # Since /sbin/init exec fails on Linux dev, the file should still be there
    # if the test sequence was fast.
    fail "state.toml missing after replace" "file present" "missing"
fi

# Verify state.toml contents
if [ -f "$TEST_DIR/run/state.toml" ]; then
    out=$(cat "$TEST_DIR/run/state.toml")
    assert_contains "$out" "boot_time =" "state.toml has boot_time"
fi
```

You may need to adapt `start_zyginit` / `stop_zyginit` / `wait_for_socket` to use `ZYGINIT_RUN_DIR`. Check the existing helpers in run_tests.sh for the pattern.

- [ ] **Step 3: Add the precondition refusal test**

Append:

```bash
section "replace — refuses on transient state"

# Set up a service whose start.sh hangs for 10s, so the service stays
# in STATE_STARTING long enough for us to issue a replace.
mkdir -p "$BIN_DIR"
cat > "$BIN_DIR/slow-start.sh" <<'EOF'
#!/bin/sh
sleep 10
EOF
chmod +x "$BIN_DIR/slow-start.sh"

cat > "$ENABLED_DIR/slow.toml" <<EOF
[service]
name = "slow"
type = "daemon"

[exec]
start = "$BIN_DIR/slow-start.sh"

[restart]
on = "never"
EOF

ln -sf "$ENABLED_DIR/slow.toml" "$ENABLED_DIR/../slow.toml" 2>/dev/null || true

# Reload to pick up the new service
"$ZYGCTL" reload >/dev/null 2>&1
sleep 0.2

# Issue replace — should ack but be refused on the server side
out=$("$ZYGCTL" replace 2>&1 || true)
assert_contains "$out" "replace queued" "replace acks even when refused"

# Server should log "replace: blocked" since slow service is in STARTING
sleep 0.5
log=$(cat "$DAEMON_LOG")
assert_contains "$log" "replace: blocked" "server logs blocked refusal"
```

- [ ] **Step 4: Run the test suite**

Run: `tests/integration/run_tests.sh`
Expected: all existing tests pass (regression check) plus the new replace tests pass.

- [ ] **Step 5: Commit**

```bash
hg commit -m "tests: add Linux integration coverage for zygctl replace"
```

---

## Task 14: Hammerhead test plan + man page updates

**Files:**
- Create: `tests/integration/test_replace_hammerhead.md`
- Modify: `man/zygctl.8`
- Modify: `man/zyginit.8`

Hammerhead testing is manual — it requires `hh-prototest` and exercises the actual contract-driven supervision path. Document the test cases as a checklist the operator runs by hand.

- [ ] **Step 1: Create the Hammerhead test checklist**

Write `tests/integration/test_replace_hammerhead.md`:

````markdown
# Live Replace — Hammerhead Test Checklist

Manual checklist for validating `zygctl replace` on `hh-prototest`
(192.168.122.50). Companion to the spec at
`docs/superpowers/specs/2026-05-02-contract-readoption-design.md`.

## Build + deploy

```bash
# From dev host
scp src/replace.reef src/main.reef src/contract.reef src/socket.reef \
    src/contract_linux_stubs.c src/helpers.c \
    root@192.168.122.50:/root/zyginit/src/
scp tools/zygctl/src/zygctl.reef \
    root@192.168.122.50:/root/zyginit/tools/zygctl/src/

# On hh-prototest
ssh root@192.168.122.50 'cd /root/zyginit && \
    gcc -c src/helpers.c -o build/helpers.o && \
    reefc build -l contract --obj build/helpers.o && \
    cp build/zyginit /sbin/init.new && mv /sbin/init.new /sbin/init && \
    cd tools/zygctl && \
    gcc -c src/symlink_wrapper.c -o build/symlink_wrapper.o && \
    reefc build --obj build/symlink_wrapper.o && \
    cp build/zygctl /sbin/zygctl'
```

After the binary swap, the running zyginit is still the OLD one. The
first `zygctl replace` will swap to the new binary.

## Test cases

### 1. Smoke: replace into the same binary

- [ ] `zygctl status` — note all PIDs and ctids; capture uptimes
- [ ] `zygctl replace` — should print `replace queued (wait=0)` then close
- [ ] Wait ~1 second; `zygctl status` again
- [ ] **All services have the same PIDs** (kernel-level continuity)
- [ ] **All services have the same contract IDs**
- [ ] **Uptimes are continuous** (delta from last_start_time, not reset)
- [ ] **The SSH session driving this test is still alive**
- [ ] `who -b` and `who -r` still report the same boot_time / runlevel

### 2. Pre-condition refusal

- [ ] `zygctl restart sshd` (puts sshd briefly in STOPPING/STARTING)
- [ ] Immediately: `zygctl replace`
- [ ] Expect server log line: `zyginit: replace: blocked: sshd:stopping`
- [ ] Verify the running zyginit did NOT exec (check uptime via /proc/1/start)

### 3. `--wait=N` success

- [ ] `zygctl restart syslogd && zygctl replace --wait=10`
- [ ] Replace should succeed once syslogd stabilizes
- [ ] Confirm syslogd's restart_count incremented and is preserved post-replace

### 4. Dirty disable

- [ ] `rm /etc/zyginit/enabled.d/cron.toml`  (no `zygctl reload`)
- [ ] `zygctl replace`
- [ ] After replace, `zygctl status` does NOT list cron
- [ ] But: `pgrep cron` still shows the cron pid running (process not killed)
- [ ] Server log shows `zyginit: replace: orphaned cron`

### 5. Crash during exec gap

- [ ] Pick a service with `restart.on = "always"`, e.g. sshd
- [ ] `pkill -9 -f "sshd -D"` (or whatever the daemon's argv looks like)
- [ ] Within ~0.5 seconds: `zygctl replace`
- [ ] After replace, server log includes `zyginit: replace: sshd contract empty post-recovery, applying restart policy`
- [ ] sshd is restarted with restart_count incremented

### 6. Stale state.toml across real reboot

- [ ] `cp /var/run/zyginit/state.toml /tmp/saved-state.toml` while a replace
      hasn't been issued (file may not exist; create one by issuing replace
      first then copy quickly before the new zyginit deletes it)
- [ ] `zygctl reboot` (real reboot)
- [ ] After boot: `cp /tmp/saved-state.toml /var/run/zyginit/state.toml`
- [ ] Stop+start zyginit somehow OR observe that the next replace handles it
- [ ] Server log includes `replace: stale state file (boot_time ...)`
- [ ] state.toml is deleted after that read

### 7. Repeated replace

- [ ] Note sshd's restart_count and uptime
- [ ] `zygctl replace` × 5 in a row, ~5 seconds apart
- [ ] After all 5: restart_count is unchanged (replace doesn't bump it)
- [ ] Uptime is continuous from the original start (no resets)
- [ ] No leftover state.toml.tmp files in /var/run/zyginit/

## Reverting

If a replace puts the system in a bad state and SSH is still alive:

```bash
mv /sbin/init.bak /sbin/init  # if you saved the old binary
zygctl replace                # swap back to known-good
```

If SSH is dead but the VM is still running, `virsh reset hh-prototest`
will reboot. The new binary at /sbin/init persists (filesystem-level
swap), so a reboot uses the new binary cleanly — re-test from cold boot.
````

- [ ] **Step 2: Update `man/zygctl.8`**

Add a `replace` entry in the COMMANDS section. Example pattern (adapt to existing groff macros):

```groff
.TP
.B replace [--wait=N]
Replace the running zyginit process via in-place
.BR execve (2)
of
.IR /sbin/init .
Preserves all running services across the swap. With
.BR --wait=N ,
poll up to N seconds for any transient-state service to stabilize
before refusing. The operator must drop the new binary at
.I /sbin/init
before issuing this command. See
.BR zyginit (8)
for the live-replace mechanism.
```

- [ ] **Step 3: Update `man/zyginit.8`**

Add a "LIVE REPLACE" section describing:
- The state-file path (`/var/run/zyginit/state.toml`)
- The exec target is hard-wired to `/sbin/init`
- The boot-id sentinel (utmpx BOOT_TIME)
- That the design is operator-driven and does NOT cover crash recovery

Refer to the spec for full details.

- [ ] **Step 4: Verify man pages render**

Run:
```bash
man -l man/zygctl.8 | head -50
man -l man/zyginit.8 | head -50
```

Expected: no troff errors, the `replace` entry appears in zygctl.8.

- [ ] **Step 5: Commit**

```bash
hg add tests/integration/test_replace_hammerhead.md
hg commit -m "docs: Hammerhead replace test checklist + man page updates"
```

---

## Done — final verification

After all 14 tasks land:

- [ ] **Linux build clean:**
  ```bash
  clang -c src/helpers.c -o build/helpers.o
  clang -c src/contract_linux_stubs.c -o build/contract_linux_stubs.o
  reefc build --obj build/helpers.o --obj build/contract_linux_stubs.o
  cd tools/zygctl && reefc build --obj build/symlink_wrapper.o && cd ../..
  ```

- [ ] **All Linux integration tests pass:** `tests/integration/run_tests.sh`

- [ ] **Replace-probe tests pass:** `cd utils/replace_probe && ./build/replace_probe && cd ../..`

- [ ] **Spec coverage check** — re-read the spec, point at a task for each requirement:
  - §3 "no `adopt_contract`" → Task 9 (no adopt call in apply_recovered_state) ✓
  - §4 module split → Tasks 1-12 cover all five files ✓
  - §5 state file format → Tasks 4, 5 ✓
  - §5.2 boot-id staleness check → Tasks 1, 5 ✓
  - §5.3 atomic write → Task 4 ✓
  - §6.1 zygctl interface → Task 12 ✓
  - §6.2 --wait=N → Tasks 8, 10 ✓
  - §6.3 pre-conditions → Tasks 6, 10 ✓
  - §7.2 algorithm → Task 9 ✓
  - §7.3 config divergence → Task 9 (orphan abandon path) ✓
  - §8 error handling → Tasks 4, 5, 9, 10 (defensive throughout) ✓
  - §9.1 Linux tests → Task 13 ✓
  - §9.2 Hammerhead tests → Task 14 ✓
  - §11 decisions table → spec already mapped to design choices, reflected in implementation ✓

- [ ] **Hammerhead deploy + run the manual checklist** in `tests/integration/test_replace_hammerhead.md`. Capture results in a memory note (`pid1_replace_iter1.md` or similar).

---

## Non-trivial gotchas to watch for

- **Reef stdlib API names** in this plan (`toml.parse_ok`, `toml.get_int_or`, `fd.fd_fsync`, `str.starts_with`, `str.char_at`, `process.exit_now`) are best-effort. The implementer should grep the actual Reef stdlib (`~/repos/reef-lang/`) and use the real names, adjusting code locally. The shapes are right; only the spellings may differ.

- **`set_runtime_for_test` naming**. The proc is used by both the probe (legitimate test usage) and `apply_recovered_state` (production path). If the name bothers a reviewer, rename to `supervisor.adopt_runtime` and update both call sites. The function does what the production path needs; the test-y name is cosmetic.

- **`STATE_WAITING` ambiguity**. Conservative interpretation: refuse replace whenever any service is in WAITING. This includes the boot-time "not-yet-started" case, which can't actually happen at the point a replace is issued (since the table is fully started before the event loop entry that processes the replace flag). False positives only land in pathological timing. Document in commit message if concerns surface.

- **utmpx BOOT_TIME is read-once on Hammerhead** if the file is on a not-yet-mounted filesystem. The fallback to `time.time_now()` covers this gap, but means a panic-restart followed by a real reboot followed by another zyginit start will see different boot_times — that's CORRECT behavior (real reboot must invalidate state.toml).

- **`process.process_exec` argv handling**. Reef's `process_exec` may or may not require argv terminated by a sentinel. Match the pattern used elsewhere in the codebase (e.g., `start_service` in supervisor.reef). The existing pattern there uses `[string]` arrays — same pattern in `replace_self`.

- **socket.reef `cmd` parsing**. The existing dispatch only takes one string argument; for `--wait=N` we need to parse it from the `arg` string. Confirm by reading the existing handler shape; if `arg` is the rest of the line after the verb, the impl above is correct.