|

Manifest inventory capped at 8192; OS-sized packages install incomplete #1

Closed Chris Tusa opened this issue 22 days ago
Chris Tusa opened this issue 22 days ago

Summary

coral build of an OS-sized package succeeds and the .pkg.tar.xz contains the full tree, but .MANIFEST is silently truncated at 8192 entries. coral install is manifest-driven, so it would install a subset and omit kernel/unix, libc, and Limine.

This is a Coral capacity bound (new [ManifestEntry](8192)), not a Reef compiler crash. Reef 0.8.6 walked the tree without a GC fault.

Measured (2026-08-17, hh-alpha11)

  • Coral 0.4.7 (/usr/bin/coral, 2026-08-10), reefc 0.8.6
  • Port: base/hammerhead-basesystem (build.sh stages a finished hh-build proto via hh-pkg --stage; Coral owns pack)
  • Command: PROTO_AREA=… HAMMERHEAD=… coral -y build base/hammerhead-basesystem → exit 0 in ~3 min
Count
Archive hammerhead-basesystem-0.0.0-1.pkg.tar.xz (240M)
Tar members 42 308 (includes ./kernel/unix, ./usr/lib/limine/BOOTX64.EFI; no opt/onbld, no usr/gcc)
.MANIFEST lines 8193 (#mtree + 8192 entries: 7813 file / 277 dir / 102 link)
Tar paths with no manifest line ~34 391

.MANIFEST includes ./sbin/zyginit (walked early). It does not include:

  • ./kernel/unix
  • ./lib/libc.so.1 / ./usr/lib/libc.so.1
  • ./usr/lib/limine/BOOTX64.EFI

generate_manifest collects until count == 8192, then sorts that prefix. Install uses only the manifest (package.install_filesread_manifest into another [8192]). A host installed from this archive would not get a kernel or libc.

Related: hh-pkg’s own xz (no .MANIFEST) fails install immediately (read_manifest → 0 → Failed to install files). Same root cause: install will not walk the archive.

Same 8192 appears in install.reef, remove.reef, verify.reef, database.reef, pkgdb.reef.

Where

  • src/util/mtree.reefgenerate_manifest / walk_directory(..., max=8192)
  • src/core/package.reefinstall_files / install_files_upgrade allocate [ManifestEntry](8192)
  • src/commands/build.reefgenerate_manifest_file (no check that walk covered the tree)

Still present in tree at Coral 0.4.8.

Expected

  • Inventory and install every staged path, or
  • Fail closed if the walk hits the cap (generate_manifest_file / install_files must not succeed with a silent prefix).

A zport-sized package is fine today. A single hammerhead-basesystem blob is ~42k members and is the intended OS package.

Suggested direction

  • Raise the bound far above 42k and treat “walk stopped at max” as an error, or
  • Stream the manifest (no fixed array): write .MANIFEST while walking; install streams the file instead of loading it all into [ManifestEntry](N).

Do not paper over this by installing from the tar listing while leaving .MANIFEST truncated — coral verify / remove / upgrade would still be wrong.

Chris Tusa commented 13 days ago

0.5.0 is the optparse / Reef 0.9.1 release (v0.5.0). This ticket is not in that drop.

Measured again on 2026-08-26 with host reefc 0.9.1 and the 0.5.0 coral: a 9000-file package still gets a silently truncated .MANIFEST (8192 entries). Reef 0.9.1 allocates new [int](50000) fine — the cap is Coral's new [ManifestEntry](8192) in mtree.generate_manifest, not a compiler limit.

Targeting 0.5.1: raise the bound above OS-sized trees (~42k for hammerhead-basesystem) and fail closed when the walk hits max. Do not install from the tar listing while leaving .MANIFEST truncated.

Chris Tusa commented 13 days ago

Fixed on @ as 90ce6b0b95f1 (PR coral!1, CI #14 green).

types.max_package_files() is 131072. Generate/parse fail closed on overflow; install/remove/verify/files/owner/db use the same cap.

Still on 0.5.0 until the 0.5.1 tag.

Assignees

No assignees

Labels

No labels

Severity

None