zygaena / coral
|
Files Changesets Issues Pull Requests Releases
root / docs / PACKAGE_FORMAT_V2_DESIGN.md
PACKAGE_FORMAT_V2_DESIGN.md markdown 409 lines 14.4 KB
Raw

Design: Package Format v2

Date: 2026-03-07
Status: Draft
Target: Coral v0.3
Authors: Zygaena Team

Overview

Redesign of the Coral package internal format to eliminate the rsync dependency,
add a rich file manifest for install/remove/verify operations, embed dependency
metadata in packages, fix config file handling, and clean up the scripting model.

These changes are possible now because Reef v0.5.0 provides native filesystem
operations (fs.stat, fs.perm, fs.link, fs.ops) that replace the need for
shell-outs to rsync, cp, chmod, chown, and ln.

No package has shipped to production yet, so there is no backward-compatibility
constraint.

1. Package Archive Structure

The archive format remains .pkg.tar.xz (gtar + xz). The internal layout changes:

name#version-release.pkg.tar.xz
  .PKGINFO              # TOML — package metadata + dependencies
  .MANIFEST             # mtree-format file inventory (replaces .FOOTPRINT)
  .CONFIG               # list of config-protected paths (unchanged)
  .SCRIPTS/             # optional hook scripts (zsh only)
      pre-install.sh
      post-install.sh
      pre-remove.sh
      post-remove.sh
      pre-upgrade.sh
      post-upgrade.sh
  usr/
  etc/
  ...                   # payload files at their install paths

Changes from v1

Item v1 v2
File manifest .FOOTPRINT (flat path list) .MANIFEST (mtree with metadata)
Dependencies not in package [dependencies] in .PKGINFO
Script hooks 4 (install/remove) 6 (+ upgrade pair)
Script runners .reef, .sh, bare .sh (zsh) and bare only
Install method rsync -a fs.ops driven by manifest

2. .PKGINFO — Package Metadata

TOML format. Adds a [dependencies] section with optional version constraints.

[package]
name = "openssl"
version = "3.2.1"
release = 1
description = "TLS/SSL and crypto library"
url = "https://www.openssl.org"
license = "Apache-2.0"
maintainer = "admin@zygaena.org"
arch = "x86_64"

[dependencies]
runtime = ["zlib", "libc"]
build = ["perl", "makedepend"]

Version constraints

Runtime and build dependencies support optional version operators:

runtime = ["zlib>=1.3", "libc"]

Supported operators: >=, <=, >, <, =. No operator means "any version".

Most dependencies will be unversioned. Version constraints are added only when a
specific ABI or feature is required (e.g., a soname bump, a new API entry point).
Exact version pinning (=) is discouraged — it creates rebuild cascades.

The port's package.toml remains the authoritative source. The [dependencies]
section in .PKGINFO is a copy embedded at build time so standalone packages are
self-describing without the ports tree.

Generation

generate_pkginfo() in build.reef writes the [dependencies] section by
reading ctx.port.deps.runtime and ctx.port.deps.build.

3. .MANIFEST — mtree File Inventory

Replaces .FOOTPRINT. Uses mtree-style format: one line per filesystem entry,
path first, then space-separated key=value attributes.

Format

#mtree
./usr/bin type=dir mode=0755 uname=root gname=root
./usr/bin/passwd type=file mode=4755 uname=root gname=sys sha256=a1b2c3d4e5f6...
./usr/bin/zlib-flate type=file mode=0755 uname=root gname=root sha256=d4e5f6a7...
./usr/include type=dir mode=0755 uname=root gname=root
./usr/include/zconf.h type=file mode=0644 uname=root gname=root sha256=e5f6a7b8...
./usr/lib type=dir mode=0755 uname=root gname=root
./usr/lib/libz.a type=file mode=0644 uname=root gname=root sha256=9c0d1e2f...
./usr/lib/libz.so type=link uname=root gname=root link=libz.so.1
./usr/lib/libz.so.1 type=link uname=root gname=root link=libz.so.1.3.1
./usr/lib/libz.so.1.3.1 type=file mode=0755 uname=root gname=root sha256=f0a1b2...

Fields

Key Required Values Notes
type always file, dir, link First attribute after path
mode files, dirs 4-digit octal Includes setuid/setgid/sticky
uname always user name String, not numeric uid
gname always group name String, not numeric gid
sha256 files only 64-char hex For install verification and coral verify
link symlinks only target path Relative or absolute link target

Sort order

Entries are sorted lexicographically by path. This ensures:

  • Directories appear before their contents (install order: top-down)
  • Removal in reverse order processes files before their parent directories

Extensibility

Unknown key=value pairs are ignored by the parser. Future attributes (e.g.,
xattr=..., acl=..., flags=...) can be added without breaking existing
Coral versions. However, ACLs and extended attributes are better handled in
post-install scripts where policy decisions belong.

Metadata exclusion

.PKGINFO, .MANIFEST, .CONFIG, and .SCRIPTS/ are not listed in the
manifest. They are package metadata, not payload files.

Generation

generate_manifest() in build.reef replaces generate_footprint(). It walks
the staging directory using fs.stat and fs.perm to read type, mode, owner,
and group for each entry. File checksums are computed with checksum.sha256_file().
Symlink targets are read with fs.link.readlink().

Parser module

New module: util/mtree.reef

export
    type ManifestEntry
    fn parse_manifest(content: string, entries: [ManifestEntry], max: int): int
    fn generate_manifest(pkg_dir: string): string
    fn entry_type(entry: ManifestEntry): string
    fn entry_path(entry: ManifestEntry): string
    fn entry_mode(entry: ManifestEntry): int
    fn entry_sha256(entry: ManifestEntry): string
    fn entry_link(entry: ManifestEntry): string
    fn entry_uname(entry: ManifestEntry): string
    fn entry_gname(entry: ManifestEntry): string
end export

4. Install Method — Replacing rsync

Current flow (v1)

extract .pkg.tar.xz to temp dir
  → find symlinks, remove conflicts in root
  → rsync -a --exclude=metadata temp_dir/ root/

New flow (v2)

extract .pkg.tar.xz to temp dir
  → parse .MANIFEST into entry array
  → for each entry (in manifest order):
      if type=dir:
          fs.ops.create_dir_all(root + path)        -- create if missing
          fs.perm.chmod(root + path, mode)           -- set permissions
          fs.perm.chown(root + path, uname, gname)   -- set ownership
      if type=file:
          check if config-protected (see section 6)
          if config file with user modifications → install as .pkg
          else:
              remove any existing conflicting path (file, symlink, or dir)
              fs.ops.copy_file_preserve(src, root + path)
              fs.perm.chmod(root + path, mode)
              fs.perm.chown(root + path, uname, gname)
              verify sha256 matches manifest
      if type=link:
          remove any existing conflicting path
          fs.link.symlink(link_target, root + path)

Advantages over rsync

  • Zero external dependencies — no rsync required during bootstrap
  • Manifest-driven — every file placed is accounted for, no silent extras
  • Per-file verification — sha256 checked on install, not just blindly copied
  • Config protection integrated — checked inline during install, not as a
    separate pass
  • Explicit permissions — set from manifest, not inherited from build host
    filesystem quirks

Error handling

If any file operation fails, installation stops and returns an error. Partial
installs are recorded so coral remove can clean up. This is safer than rsync's
exit code 23 (partial transfer accepted).

5. Scripting Model

Hooks

Six hooks, all executed with zsh:

Hook Arguments When Abort on failure
pre-install.sh $VERSION Before files installed (fresh install) Yes
post-install.sh $VERSION After files installed (fresh install) No*
pre-remove.sh $VERSION Before files removed (full remove) Yes
post-remove.sh $VERSION After files removed (full remove) No*
pre-upgrade.sh $OLD_VERSION $NEW_VERSION Before files replaced (upgrade) Yes
post-upgrade.sh $OLD_VERSION $NEW_VERSION After new files installed (upgrade) No*

*Post-scripts log failures as warnings but do not roll back, since the file
operation has already completed.

Execution

All scripts are invoked as:

zsh "/path/to/.SCRIPTS/hook-name.sh" [args...]

The .reef script type is removed. Requiring the Reef compiler at install time
creates a bootstrap dependency that cannot be satisfied when building the base OS.

Bare executables (no extension) are still supported — Coral executes them
directly, relying on a shebang line or ELF header.

Script search order

For each hook, Coral looks for (in order):

  1. hook-name.sh — executed with zsh
  2. hook-name — executed directly (must be executable with valid shebang or ELF)

If neither exists, the hook is silently skipped (not an error).

Upgrade flow

pre-upgrade.sh  $OLD_VERSION  $NEW_VERSION    # abort if non-zero
  → remove old files (driven by installed manifest)
  → install new files (driven by new manifest)
post-upgrade.sh  $OLD_VERSION  $NEW_VERSION   # warn if non-zero

If no pre-upgrade.sh or post-upgrade.sh exists, Coral performs the file
swap silently. It does not fall back to running pre-remove + post-install,
as those hooks may have side effects inappropriate for an upgrade (e.g.,
post-install creating a fresh database vs. preserving an existing one).

Stored scripts

Remove scripts (pre-remove.sh, post-remove.sh) must be available when a
package is removed, even if the original .pkg.tar.xz is gone. Coral stores
a copy of .SCRIPTS/ in the package database at install time
(/var/lib/coral/installed/<name>/scripts/). Upgrade scripts are stored there
as well, since they're needed when a future version replaces the current one.

6. Config File Protection — Checksum-Based

Current behavior (v1, broken)

Reads both files fully into memory and compares content with str.equals().
Both branches of the comparison (equal / not equal) execute the same code path —
always installs as .pkg. The "files are identical" case should overwrite in
place.

New behavior (v2)

Uses the sha256 checksum from the manifest to detect user modifications without
reading the full installed file into memory twice.

On fresh install:

  1. Install config file normally to dest_path
  2. Store the package-provided sha256 in the database for this file

On upgrade (file already exists):

  1. Compute sha256 of the currently installed file
  2. Compare against the stored checksum from the previously installed version
  3. Decision:
Installed matches old pkg checksum? Action
Yes (user has not modified) Overwrite with new version
No (user has modified) Install new version as dest_path.pkg-new, keep user's file
  1. Update stored checksum to the new package's sha256

On remove:

  • Config files listed in .CONFIG are not removed if they differ from the
    package-provided checksum (user has modified them)
  • Config files that match the package checksum are removed normally

This matches the proven dpkg conffile model.

Implementation

fn install_config_file(src: string, dest: string, manifest_sha256: string, stored_sha256: string): bool
    if not fs.stat.exists(dest)
        // Fresh install — copy file
        fs.ops.copy_file_preserve(src, dest)
        return true
    end if

    // File exists — check if user modified it
    let installed_sha256 = checksum.sha256_file(dest)

    if installed_sha256 == stored_sha256
        // User has not modified — safe to overwrite
        fs.ops.copy_file_preserve(src, dest)
        return true
    else
        // User has modified — preserve their version
        let new_path = dest + ".pkg-new"
        fs.ops.copy_file_preserve(src, new_path)
        // Log: "Config file preserved, new version at dest.pkg-new"
        return true
    end if
end install_config_file

7. Implementation Plan

Phase 1: mtree module and manifest generation

  • New module util/mtree.reef — parser and generator
  • Update build.reef: replace generate_footprint() with generate_manifest()
  • Use fs.stat, fs.perm, fs.link.readlink(), checksum.sha256_file()

Phase 2: fs.ops-based installer

  • Rewrite install_files() in package.reef to walk the manifest
  • Use fs.ops.copy_file_preserve, fs.perm.chmod, fs.perm.chown, fs.link.symlink
  • Remove rsync dependency from install path
  • Integrate config file checksum logic

Phase 3: Enhanced .PKGINFO

  • Update generate_pkginfo() to emit [dependencies] section
  • Update read_pkginfo() to parse dependencies
  • Add deps field to PackageInfo type or extend existing Dependencies

Phase 4: Script model cleanup

  • Add pre-upgrade.sh / post-upgrade.sh hook support
  • Remove .reef script type from run_script() and run_stored_script()
  • Change .sh invocation from sh to zsh
  • Pass version arguments to all hooks
  • Abort on non-zero exit from pre-* hooks
  • Update upgrade command to call upgrade hooks

Phase 5: Removal and verify

  • Rewrite remove_files() to use manifest (directories tracked explicitly)
  • Use fs.ops.remove_file, fs.ops.remove_tree, dir.remove_dir
  • Implement coral verify <pkg> — walk stored manifest, check sha256 and
    permissions against live filesystem

Dependencies

  • Reef >= 0.5.0 (for fs.stat, fs.perm, fs.link, fs.ops)
  • rsync is no longer required for package installation (remains used by
    coral repo mirror only)

8. Risks and Mitigations

Risk Mitigation
fs.ops.copy_file_preserve slower than rsync for large packages Benchmark during Phase 2; rsync does buffered I/O and copy_file_preserve uses 8KB buffer — comparable. Manifest-driven install skips metadata files without needing exclude flags.
mtree parser bugs cause install failures Comprehensive test coverage in Phase 1; mtree is a simple line-oriented format with no nesting or escaping edge cases beyond spaces in paths.
Packages with >8192 files overflow entry array Use manifest file_count pre-scan or dynamic sizing. Current .FOOTPRINT has same limit.
Upgrade hook ordering with dependency chains Out of scope for this design. Coral currently upgrades packages individually. Dependency-ordered upgrades (topological sort) are a future enhancement.