|

PID 1 burns ~1.6% CPU when idle: one LWP polling at ~800 syscalls/sec #8

New Chris Tusa opened this issue 29 days ago
Chris Tusa opened this issue 29 days ago

Summary

zyginit 0.6.0 running as PID 1 consumes ~1.6% of a CPU continuously while the
system is completely idle
. One LWP is making roughly 800 system calls and
800 voluntary context switches per second
— a poll loop waking about every
1.25 ms — with no services changing state and nothing else running.

Measurement

Host: hh-alpha12 (Hammerhead rev 510, 4 vCPU, 8 GiB, KVM guest). Idle, load
average 0.02, 27 processes, no user activity.

CPU accumulation over a 180-second window on an otherwise idle system:

t0    init=01:27:56
t180  init=01:27:59      ->  3 s / 180 s  =  ~1.6% of one CPU

Sustained, not bursty: over 3 d 23 h of idle uptime PID 1 had accumulated
1:27:56 of CPU time (~1.5% average, consistent with the instantaneous rate).
No other process comes close — moused accumulated 0 seconds across the same
180-second window.

Per-LWP microstate accounting, 5-second sample (prstat -mL -p 1 5 1):

   PID USERNAME USR SYS TRP TFL DFL LCK SLP LAT VCX ICX SCL SIG PROCESS/LWP
     1 root     0.2 1.3 0.0 0.0 0.0 0.0  98 0.2  4K   4  4K   0 zyginit/2
     1 root     0.0 0.0 0.0 0.0 0.0 0.0 100 0.0   5   0  60   0 zyginit/1
     1 root     0.0 0.0 0.0 0.0 0.0 100 0.0 0.0   0   0   0   0 zyginit/3

LWP 2 is the source: 4K syscalls and 4K voluntary context switches in 5
seconds (~800/s each), 1.3% SYS, 98% sleeping. The pattern — high VCX matching
high SCL, almost all time in SLP — is a short-timeout wait being re-armed
continuously rather than a blocking wait on an event.

LWP 1 is quiet (60 syscalls in 5 s). LWP 3 is parked on a lock (LCK 100), which
looks intentional.

Why it matters

  • It is a constant floor on idle power and on VM density: every Hammerhead guest
    burns ~1.6% of a core doing nothing, before any workload.
  • It is in PID 1, so it applies to every system, always, with no way to opt out.
  • Hammerhead is working toward shipping an optimized non-DEBUG default
    (hammerhead#15); a 1.6% idle floor in the supervisor is a meaningful share of
    what that effort is trying to recover.

What would help narrow it

We deliberately did not run pstack/truss against PID 1 here — the
machine was in use at the time and stopping init, even briefly, risks the host.
We can capture a stack or a truss -c on a disposable guest on request; say the
word and we will.

The likely shape from the numbers alone: a supervisor loop polling on a short
timeout (roughly 1 ms) instead of blocking on the event source until something
happens. If the loop is already event-driven, the ~800/s wakeups suggest the
timeout is being used as the primary tick rather than as a safety net.

Environment

  • zyginit 0.6.0 (zygctl --version = 0.6.0)
  • Hammerhead rev 510 (hammerhead-14046ed8b08d), DEBUG kernel
  • reefc 0.9.3, coral 0.6.0
  • 16 services running, 0 failed

Assignees

No assignees

Labels

No labels

Severity

None