|

unix has no runtime CTF since rev 258: every D program fails to compile #8

Closed Chris Tusa opened this issue 8 days ago
Chris Tusa opened this issue 8 days ago

unix gets no CTF at runtime, so every D program fails to compile. DTrace is unusable on current Hammerhead.

Symptom

Any D program — not just -lv — fails on a current build:

# dtrace -n 'BEGIN { exit(0); }'
dtrace: invalid probe specifier BEGIN { exit(0); }: "/usr/lib/dtrace/procfs.d",
line 315: no symbolic type information is available for unix`kas:
Module does not contain any CTF data

procfs.d:315 is (T->t_procp->p_as == &kas). It is a **library** file, loaded for every compilation, so the failure is not specific to the probe requested. Plain dtrace -l` still works, which is why a probe-count check does not catch this.

Measurement

Two hosts, elfdump -c over every /system/object/*/object (the runtime image,
not the on-disk file):

alpha11, kernel of 2026-08-10 (pre-Limine) alpha12, rev 326
modules with runtime CTF 210 / 210 210 / 211
the one without unix
unix CTF on disk present present
any D program compiles yes no

The data is on disk and is not loaded. Nothing is being stripped.

Mechanism

uts/common/krtld/kobj.c:1332, added by rev 258 (4b1062bdb561, uts: Limine
unix boots to login
, 2026-08-21):

if (mp->flags & KOBJ_EXEC) {
#if defined(_DBOOT)
        /* Limine already loaded this unix. Re-reading symbols from the
         * archive's kernel/unix overwrites st_value with a different
         * binary's addresses (kobj_kdi_init then jumps into garbage). */
        mp->flags |= KOBJ_RELOCATED;
#else
        ... kobj_load_elfhdr / read shdrs / get_syms() / get_ctf() ...
#endif

kobj.o is always compiled with -D_DBOOT (uts/intel/Makefile.rules:308,
original, not new), so the #else arm is dead. That arm is the only caller of
get_ctf() for unix, so mp->ctfdata stays NULL and objfs exports no CTF.

The guard's stated reason concerns symbols (get_syms()), but it also removes
get_ctf(), which was not its target. A second consequence of the get_syms()
half is visible in probe counts: fbt unix is 5850 on the pre-Limine host vs 3687
at rev 326 (−2163), which accounts for essentially the entire dtrace -l delta
between the two hosts (60854 → 58709 fbt).

What this is not

It is not attributable to the GNU ld -r kmod work (REMOVE_SUNLD step A′):

  • unix is sunld-linked on both hosts (DT_SUNW_STRPAD + DT_SUNW_LDMACH
    present, no DT_SUNW_KMOD). Its link did not change.
  • All 210 GNU-linked kmods on alpha12 — e.g. zfs, whose elfdump -d shows
    SUNW_KMOD with STRPAD/LDMACH absent — have working runtime CTF.
  • 196 of 201 shared fbt modules have identical probe counts across the two
    hosts; the exceptions are unix and four console modules that went up.
  • The guard landed at rev 258; A′'s first commit is 272 and it became the
    default at 276.

Suggested approach

Keep skipping get_syms(), but still read the ELF header and section headers and
call get_ctf(). Hazard to design around first: CTF object/function sections
are positional against the module's symbol table. Under the guard the runtime
symbols are Limine's preloaded image while the CTF would come from
kobj_open_file("/kernel/unix") via the boot archive. If those two images can
differ — which is precisely what the guard's comment asserts — the indices will not
correspond and DTrace would report confidently wrong types, which is worse than
the current hard failure. Establish whether the archive copy and the loaded image
are guaranteed identical before wiring CTF back up.

Found while running the REMOVE_SUNLD step A′ validation gate, whose Task 7 Step 2
this was blocking. A′ is exonerated; see the plan docs.

Chris Tusa commented 8 days ago

Blast radius is wider than filed: mdb -k is down too, from the same guard

Confirmed at rev 340 on hh-alpha12. The guard skips three calls for unix,
not the two originally reported. kobj_load_elfhdr() is in the dead arm as well,
so unix's mp->hdr is never populated:

if (mp->flags & KOBJ_EXEC) {
#if defined(_DBOOT)
        mp->flags |= KOBJ_RELOCATED;          /* <- the whole #else is dead */
#else
        file = kobj_open_file(mp->filename);
        if (kobj_load_elfhdr(file, mp) < 0)   /* (1) never runs */
        ...
        if (get_syms(mp, file) < 0)           /* (2) never runs */
        if (get_ctf(mp, file) < 0)            /* (3) never runs */
#endif

ksyms_snapshot.c:148 builds /dev/ksyms by copying that header wholesale and
overriding only offsets and counts -- e_ident, and therefore the ELF magic,
comes straight from unix:

hdr.elf_hdr = ((struct module *)modules.mod_mp)->hdr;
hdr.elf_hdr.e_phoff = offsetof(ksyms_header_t, text_phdr);
hdr.elf_hdr.e_shoff = offsetof(ksyms_header_t, shdr);
...

modules is a bcopy of the head of krtld's list (kobj.c:525), which is unix
(KOBJ_EXEC|KOBJ_PRIM, kobj.c:865). So the header is a zeroed struct.

Measured:

# dd if=/dev/ksyms bs=16 count=1 | od -c
0000000  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0
# mdb -k -e '::status'
mdb: /dev/ksyms is not an ELF file

On hh-alpha11 (kernel of 2026-08-10, pre-guard) the same bytes are
177 E L F 002 001 001 006 and mdb -k reports "debugging live kernel (64-bit)
on hammerhead"
. Note this supersedes the standing belief that mdb is broken by
--export-dynamic
-- mdb itself works; it is /dev/ksyms that is malformed.

Full symptom set, one root cause

skipped call symptom
get_ctf() no runtime CTF for unix -> every D program fails to compile
get_syms() fbt unix probes 5850 -> 3687
kobj_load_elfhdr() /dev/ksyms has no ELF header -> mdb -k fails

Both kernel debuggers are therefore down on current Hammerhead.

Still not A′'s

Re-measured at rev 340 (full build of tip, GNU kmods default): 212 loaded
modules, exactly one without runtime CTF (unix); zfs shows DT_SUNW_KMOD
with STRPAD/LDMACH absent, i.e. GNU-linked, and has working CTF. ksyms takes
its header from unix alone, which is sunld-linked on both arms, so KMOD_LINKER
cannot reach any of this.

Raises the priority

Fixing the CTF half alone would leave mdb -k broken. Whoever takes this should
restore all three calls together, subject to the symbol-table correspondence
hazard already noted above -- which applies to get_syms() most of all, since
that is the one the guard was actually written to prevent.

Chris Tusa commented 8 days ago

Fixed in rev 342 (cda2fff331b8). Verified on a full hh-build of tip plus a marker-gated boot matrix.

Two changes were needed, not one

1. kobj.c — restore the three skipped calls, with a check.

Rather than reverting the guard outright, the block now verifies the on-disk image
is the one being run: DT_SYMTAB is read from the running .dynamic, so it must
equal the .dynsym sh_addr the file claims. On a mismatch it keeps the boot
dynsym, skips the file's symbols and CTF, and prints why — the kernel boots
degraded and loudly rather than jumping into garbage.

The guard's original premise was checked before removing it: deploy --target,
the path that refreshed the archive but never the ESP, is now hard-disabled, and
ESP == archive == BE was measured byte-identical on a deployed disk.

2. limine_xboot.c — rename a file-static that shadowed a kernel global.

Restoring the block initially broke the boot differently:

/kernel/fs/objfs: undefined symbol 'modules'
WARNING: mod_load: cannot load module 'objfs'
WARNING: Cannot mount /system/object

The adapter declared static struct boot_modules modules[HH_MAX_MODULES], which
collides with the kernel global struct modctl modules (conf/param.c). Statics
are absent from unix's .dynsym, so it was invisible while unix ran on the
dynsym; loading the full .symtab let the local shadow the global. Pre-Limine
unix has exactly one modules symbol, this build had two. Renamed to
hh_boot_modules.

The guard was concealing this second bug, which would surface for anything
reading unix's full symbol table.

Verification (rev 342, full build, hh-alpha12)

check before after
runtime CTF census 211 modules, 1 without (unix) 212, 0 without
dtrace -n 'BEGIN{exit(0)}' failed to compile rc=0, probe fires
dtrace -lvn 'fbt:genunix::entry' rc=1, 0 lines rc=0, 110928 lines (pre-Limine: 110928)
fbt unix probes 3687 5861 (pre-Limine 5850)
total probes 62346 64566 (pre-Limine 64497)
/dev/ksyms 16 zero bytes valid ELF, .symtab 0x123480, correct e_entry
modules in unix .symtab 2 1
console undefined symbol 'modules' clean

Build: build_ok=y, 0 errors after "Building OS-Net source".
Boot matrix, deploy-marker verified by sha256: 28/28 — 1 vCPU 4/4, 2 vCPU
12/12, 4 vCPU 12/12, every boot 17 daemons / 0 failed.

mdb -k is NOT fixed by this, and that is correct

The ksyms half of this issue is fixed — /dev/ksyms is now well-formed. mdb -k
still cannot attach, but for an unrelated defect this change merely stopped
masking: /dev/kmem returns EIO for every kernel-image address because those
physical pages are not in phys_install. Filed as
#9 with a reproducer
that does not involve mdb. Closing this one on the DTrace and ksyms criteria;
mdb -k tracks in #9.

Assignees

No assignees

Labels

No labels

Severity

High