Summary

Loading kmdb into a running system fails:

# mdb -F -K
mdb: failed to start kmdb: Resource temporarily unavailable

The nucleus-grow window that hb_fixup() maps above _end runs to the top of
the address space, so it covers SEGDEBUGBASE (0xffffffffff800000, 4 MiB),
the VA the kernel layout reserves for the debugger. kmdb refuses to activate
over an existing mapping. This is a side effect of the hammerhead#2 fix
(308:43bee0e84057), not of the linker work and not pre-existing illumos.

Boot-time -k kmdb is unaffected and works (revs 486/498/499): the boot path
allocates the debugger segment with BOP_ALLOC and never checks the VA.

Root cause

uts/i86pc/limine/limine_fixup.c:

navail = (size_t)((uintptr_t)0 - grow2m);
if (nsz > navail)
        nsz = P2ALIGN(navail, (size_t)TWO_MEG);
...
if (do_bsys_alloc(NULL, (caddr_t)grow2m, nsz, TWO_MEG) == NULL)

The clamp bounds the window by 2^64. The documented amd64 layout
(uts/i86pc/os/startup.c, "64-bit Kernel's Virtual memory layout") puts the
debugger segment above kernel data, and startup.c seg_attaches
kdebugseg there. So the window takes 4 MiB it does not own.

EAGAIN has exactly one source, kctl_dseg_alloc() in
cmd/mdb/common/kmdb/kctl/kctl_main.c:

/* make sure there isn't something there already (like kadb) */
if (hat_getpfnum(kas.a_hat, addr) != PFN_INVALID)
        return (EAGAIN);

Measured (alpha12, rev 499, fresh boot, before any mdb -K, read-only mdb -k)

The debugger segment is already mapped, contiguous, zero-filled:

0xffffffffff800000 -> 0x103e00000
0xffffffffff900000 -> 0x103f00000
0xffffffffffbff000 -> 0x1041ff000

kmdb has not touched it:

kctl_state = 0 (KCTL_ST_INACTIVE)   kctl_dseg = 0   kctl_dseg_size = 0
kctl_boot_loaded = 0                boothowto = 0x1000 (RB_DEBUG clear)

It is one linear run with a constant VA-PA offset, from the window base through
the top of the address space, ending at physical 0x104600000:

0xfffffffffc000000 -> 0x100600000
0xfffffffffd000000 -> 0x101600000
0xffffffffff800000 -> 0x103e00000   <- SEGDEBUGBASE
0xffffffffffe00000 -> 0x104400000

That matches the window hammerhead#2 measured at the time,
va=fffffffffbe00000 pa=100400000 high=4200000 (66 MiB).

Not a free-list hazard

The frames behind the window have no page_t: ::page_num2pp fails for pfns
0x102600, 0x103e00 and 0x104400, and they sit in the memseg hole
0x100400..0x104600. A positive control (the pfn of a heap-allocated
kmem_cache) does resolve, so the negative is real. Nothing is aliased.

Fix sketch

Bound the window by the debugger segment instead of 2^64:

navail = (size_t)((uintptr_t)SEGDEBUGBASE - grow2m);

With grow2m = 0xfffffffffbe00000 that is 58 MiB, above HH_NUCLEUS_MIN
(32 MiB). Also fix the "available above _end" wording in the bop_panic.

Gate (the property, not "it boots")

1. After boot, echo 0xffffffffff800000::vtop | mdb -k reports no mapping.
2. mdb -F -K from the console reaches the kmdb prompt, ::dmods lists
   modules, and :c resumes.
3. nucleus grow: line shows the new size; -k boot still loads kmdb.
4. alpha12 12/12 and alpha13 8/8 matrices, since this is the boot path.

Observation, not a claim

econtig = e_moddata = 0xfffffffffc0a0000, and the kernel kas segment ends
there. So most of the 66 MiB window is mapped but beyond anything the kernel
accounts for, and its frames are outside every memseg. Whether that is memory
legitimately held for kobj_segbrk growth or simply lost is worth a look,
especially on alpha13 at 4 GiB. Not measured further.