Summary

Every kernel this tree builds is DEBUG (NIGHTLY_OPTIONS='-FntDAmr': D
builds DEBUG, F skips the non-DEBUG build), and the deployed kernel prints
DEBUG enabled. That is right for development and wrong for anything shipped.
Proposal: add a release profile that builds non-DEBUG and produces a
release boot environment, gated by its own boot matrices. Keep DEBUG as the
default for development.

Filed as a plan, not a bug: nothing is broken today.

What DEBUG costs, measured

alpha12 (8 GiB, 4 vCPU), A/B/A/B interleaved, two boots per arm, identical
userland, only /kernel /platform /usr/kernel differing. Each boot
verified by the DEBUG enabled console banner.

373 MB of kernel memory, 36% on fork/exec, 61% on a filesystem walk. On
alpha13's permanent 4 GiB that memory is ~9% of RAM. Code shrinks too: unix
.text -14%, genunix/zfs/mac -20%. Most of the memory is kmem debugging
(KMF_AUDIT|KMF_DEADBEEF|KMF_REDZONE|KMF_CONTENTS); the rest of the CPU win is
ASSERTs and other #ifdef DEBUG code.

Cost of the change: 2832 fewer DTrace probes (DEBUG-only functions), no
ASSERTs, and no kmem audit/redzone/deadbeef.

A non-DEBUG kernel builds clean

First time in this tree: uts in 29 min, rc=0, zero compiler errors.

bldenv +d ... 'cd $SRC/uts && gmake ROOT=/usr/src/hammerhead-build/root_amd64-nd install'

Three traps, all hit while measuring:

• +d is required. With -F in the env file, bldenv DEFAULTS to DEBUG
  even without -d; the first attempt silently built DEBUG into the -nd proto.
  Check the Build type banner, and obj64/ vs debug64/ objects.
• ROOT= on the command line is required. MULTI_PROTO=no, so ROOT takes
  no -nd suffix and the build would overwrite the validated proto.
• No -j. uts has two rules for some objects; parallel make ran both and
  ctfconvert then read a half-written file (Elf library failure).

uts alone does not build the kmdb dmods (46), misc/kmdbmod, or the 513
ucode data files; overlay those from the DEBUG proto. Clone a proto with
gtar, not cp -r (symlink cycles such as usr/lib/64 -> .).

Userland does not have to match

The kernel/userland boundary is the syscall ABI. Verified by running a hybrid
BE (existing userland + non-DEBUG kernel): clean boot, 0 failed services,
DTrace and mdb working -- that is where the numbers above come from.

A full release build would still flip userland: 99 makefiles outside uts add
-DDEBUG, 48 of them gated on the build type (zfs, zpool, zdb, smbd,
parts of mdb). Worth doing in a release profile; not required for the BE split.

Related latent bug: mdb/kmdb dmods compile kernel headers WITHOUT DEBUG and
read many structures by compiled layout (mdb_vread), so against today's DEBUG
kernel the structures with #ifdef DEBUG members (mac_impl_s, asycom,
squeue_s, threadtrans, cpr_storage_desc) can be misread silently. A
non-DEBUG default removes that mismatch rather than creating one.

Proposed shape

• A release profile that builds non-DEBUG kernel and userland.
• Two boot environments: hammerhead (non-DEBUG) as the pool's default, and
  hammerhead-debug (DEBUG kernel, plus the kmdb archive) as a second Limine
  entry. A ZFS clone makes the second cost only the kernel delta.
• hh-deploy needs a BE-name parameter and per-BE ESP payload names; it
  currently hardcodes one BE and points bootfs at it. The two-entry
  limine.conf pattern already exists from the kmdb work (rev 503), and an
  entry selects a BE with two knobs that must agree: path:/module_path: on
  the ESP, and zfs-bootfs= in the cmdline.
• Gate the release on matrices run against the kernel actually shipped.
  Today's matrices only cover DEBUG; shipping a non-DEBUG kernel without its
  own gate would mean validating something we do not ship.

Why not now

The cost falls on the machines where the work happens and the benefit on
machines that do not exist yet. ASSERTs and the kmem audit catch exactly the
bug class this project keeps hitting (pages simultaneously live and free,
memory nobody claims), the boot path is still changing weekly, and a second
kernel doubles both build time and the matrix surface. Natural trigger: when
zyginstall makes a release image real.

Note the big memory wins already landed unconditionally and are unrelated to
this: rev 504 strips DWARF from the boot archive and rev 505 frees it after
the root filesystem unmounts, so the default boot no longer pins 313.5 MiB.

If a single machine needs the memory before then, set kmem_flags = 0 in
/etc/system recovers 358 of the 373 MB with no build or tooling change, at
the cost of the audit checks.