# Live Replace — Hammerhead Test Checklist

Manual checklist for validating `zygctl replace` on `hh-prototest`
(192.168.122.50). Companion to the spec at
`docs/superpowers/specs/2026-05-02-contract-readoption-design.md`.

## Build + deploy

```bash
# From dev host
scp src/replace.reef src/main.reef src/contract.reef src/socket.reef \
    src/contract_linux_stubs.c src/helpers.c \
    root@192.168.122.50:/root/zyginit/src/
scp tools/zygctl/src/zygctl.reef \
    root@192.168.122.50:/root/zyginit/tools/zygctl/src/

# On hh-prototest
ssh root@192.168.122.50 'cd /root/zyginit && \
    gcc -c src/helpers.c -o build/helpers.o && \
    reefc build -l contract --obj build/helpers.o && \
    cp build/zyginit /sbin/init.new && mv /sbin/init.new /sbin/init && \
    cd tools/zygctl && \
    gcc -c src/symlink_wrapper.c -o build/symlink_wrapper.o && \
    reefc build --obj build/symlink_wrapper.o && \
    cp build/zygctl /sbin/zygctl'
```

After the binary swap, the running zyginit is still the OLD one. The
first `zygctl replace` will swap to the new binary.

## Test cases

### 1. Smoke: replace into the same binary

- [ ] `zygctl status` — note all PIDs and ctids; capture uptimes
- [ ] `zygctl replace` — should print `replace queued (wait=0)` then close
- [ ] Wait ~1 second; `zygctl status` again
- [ ] **All services have the same PIDs** (kernel-level continuity)
- [ ] **All services have the same contract IDs**
- [ ] **Uptimes are continuous** (delta from last_start_time, not reset)
- [ ] **The SSH session driving this test is still alive**
- [ ] `who -b` and `who -r` still report the same boot_time / runlevel

### 2. Pre-condition refusal

- [ ] `zygctl restart sshd` (puts sshd briefly in STOPPING/STARTING)
- [ ] Immediately: `zygctl replace`
- [ ] Expect server log line: `zyginit: replace: blocked: sshd:stopping`
- [ ] Verify the running zyginit did NOT exec (check uptime via /proc/1/start)

### 3. `--wait=N` success

- [ ] `zygctl restart syslogd && zygctl replace --wait=10`
- [ ] Replace should succeed once syslogd stabilizes
- [ ] Confirm syslogd's restart_count incremented and is preserved post-replace

### 4. Dirty disable

- [ ] `rm /etc/zyginit/enabled.d/cron.toml`  (no `zygctl reload`)
- [ ] `zygctl replace`
- [ ] After replace, `zygctl status` does NOT list cron
- [ ] But: `pgrep cron` still shows the cron pid running (process not killed)
- [ ] Server log shows `zyginit: replace: orphaned cron`

### 5. Crash during exec gap

- [ ] Pick a service with `restart.on = "always"`, e.g. sshd
- [ ] `pkill -9 -f "sshd -D"` (or whatever the daemon's argv looks like)
- [ ] Within ~0.5 seconds: `zygctl replace`
- [ ] After replace, server log includes `zyginit: replace: sshd contract empty post-recovery, applying restart policy`
- [ ] sshd is restarted with restart_count incremented

### 6. Stale state.toml across real reboot

- [ ] `cp /var/run/zyginit/state.toml /tmp/saved-state.toml` while a replace
      hasn't been issued (file may not exist; create one by issuing replace
      first then copy quickly before the new zyginit deletes it)
- [ ] `zygctl reboot` (real reboot)
- [ ] After boot: `cp /tmp/saved-state.toml /var/run/zyginit/state.toml`
- [ ] Stop+start zyginit somehow OR observe that the next replace handles it
- [ ] Server log includes `replace: stale state file (boot_time ...)`
- [ ] state.toml is deleted after that read

### 7. Repeated replace

- [ ] Note sshd's restart_count and uptime
- [ ] `zygctl replace` × 5 in a row, ~5 seconds apart
- [ ] After all 5: restart_count is unchanged (replace doesn't bump it)
- [ ] Uptime is continuous from the original start (no resets)
- [ ] No leftover state.toml.tmp files in /var/run/zyginit/

## Reverting

If a replace puts the system in a bad state and SSH is still alive:

```bash
mv /sbin/init.bak /sbin/init  # if you saved the old binary
zygctl replace                # swap back to known-good
```

If SSH is dead but the VM is still running, `virsh reset hh-prototest`
will reboot. The new binary at /sbin/init persists (filesystem-level
swap), so a reboot uses the new binary cleanly — re-test from cold boot.