Live Replace — Hammerhead Test Checklist

Manual checklist for validating zygctl replace on hh-prototest
(192.168.122.50). Companion to the spec at
docs/superpowers/specs/2026-05-02-contract-readoption-design.md.

Build + deploy

# From dev host
scp src/replace.reef src/main.reef src/contract.reef src/socket.reef \
    src/contract_linux_stubs.c src/helpers.c \
    root@192.168.122.50:/root/zyginit/src/
scp tools/zygctl/src/zygctl.reef \
    root@192.168.122.50:/root/zyginit/tools/zygctl/src/

# On hh-prototest
ssh root@192.168.122.50 'cd /root/zyginit && \
    gcc -c src/helpers.c -o build/helpers.o && \
    reefc build -l contract --obj build/helpers.o && \
    cp build/zyginit /sbin/init.new && mv /sbin/init.new /sbin/init && \
    cd tools/zygctl && \
    gcc -c src/symlink_wrapper.c -o build/symlink_wrapper.o && \
    reefc build --obj build/symlink_wrapper.o && \
    cp build/zygctl /sbin/zygctl'

After the binary swap, the running zyginit is still the OLD one. The
first zygctl replace will swap to the new binary.

Test cases

1. Smoke: replace into the same binary

• zygctl status — note all PIDs and ctids; capture uptimes
• zygctl replace — should print replace queued (wait=0) then close
• Wait ~1 second; zygctl status again
• All services have the same PIDs (kernel-level continuity)
• All services have the same contract IDs
• Uptimes are continuous (delta from last_start_time, not reset)
• The SSH session driving this test is still alive
• who -b and who -r still report the same boot_time / runlevel

2. Pre-condition refusal

• zygctl restart sshd (puts sshd briefly in STOPPING/STARTING)
• Immediately: zygctl replace
• Expect server log line: zyginit: replace: blocked: sshd:stopping
• Verify the running zyginit did NOT exec (check uptime via /proc/1/start)

3. --wait=N success

• zygctl restart syslogd && zygctl replace --wait=10
• Replace should succeed once syslogd stabilizes
• Confirm syslogd's restart_count incremented and is preserved post-replace

4. Dirty disable

• rm /etc/zyginit/enabled.d/cron.toml (no zygctl reload)
• zygctl replace
• After replace, zygctl status does NOT list cron
• But: pgrep cron still shows the cron pid running (process not killed)
• Server log shows zyginit: replace: orphaned cron

5. Crash during exec gap

• Pick a service with restart.on = "always", e.g. sshd
• pkill -9 -f "sshd -D" (or whatever the daemon's argv looks like)
• Within ~0.5 seconds: zygctl replace
• After replace, server log includes zyginit: replace: sshd contract empty post-recovery, applying restart policy
• sshd is restarted with restart_count incremented

6. Stale state.toml across real reboot

• cp /var/run/zyginit/state.toml /tmp/saved-state.toml while a replace
  hasn't been issued (file may not exist; create one by issuing replace
  first then copy quickly before the new zyginit deletes it)
• zygctl reboot (real reboot)
• After boot: cp /tmp/saved-state.toml /var/run/zyginit/state.toml
• Stop+start zyginit somehow OR observe that the next replace handles it
• Server log includes replace: stale state file (boot_time ...)
• state.toml is deleted after that read

7. Repeated replace

• Note sshd's restart_count and uptime
• zygctl replace × 5 in a row, ~5 seconds apart
• After all 5: restart_count is unchanged (replace doesn't bump it)
• Uptime is continuous from the original start (no resets)
• No leftover state.toml.tmp files in /var/run/zyginit/

Reverting

If a replace puts the system in a bad state and SSH is still alive:

mv /sbin/init.bak /sbin/init  # if you saved the old binary
zygctl replace                # swap back to known-good

If SSH is dead but the VM is still running, virsh reset hh-prototest
will reboot. The new binary at /sbin/init persists (filesystem-level
swap), so a reboot uses the new binary cleanly — re-test from cold boot.