# Design: Package Format v2

**Date:** 2026-03-07
**Status:** Draft
**Target:** Coral v0.3
**Authors:** Zygaena Team

## Overview

Redesign of the Coral package internal format to eliminate the rsync dependency,
add a rich file manifest for install/remove/verify operations, embed dependency
metadata in packages, fix config file handling, and clean up the scripting model.

These changes are possible now because Reef v0.5.0 provides native filesystem
operations (`fs.stat`, `fs.perm`, `fs.link`, `fs.ops`) that replace the need for
shell-outs to rsync, cp, chmod, chown, and ln.

No package has shipped to production yet, so there is no backward-compatibility
constraint.

---

## 1. Package Archive Structure

The archive format remains `.pkg.tar.xz` (gtar + xz). The internal layout changes:

```
name#version-release.pkg.tar.xz
  .PKGINFO              # TOML — package metadata + dependencies
  .MANIFEST             # mtree-format file inventory (replaces .FOOTPRINT)
  .CONFIG               # list of config-protected paths (unchanged)
  .SCRIPTS/             # optional hook scripts (zsh only)
      pre-install.sh
      post-install.sh
      pre-remove.sh
      post-remove.sh
      pre-upgrade.sh
      post-upgrade.sh
  usr/
  etc/
  ...                   # payload files at their install paths
```

### Changes from v1

| Item | v1 | v2 |
|------|----|----|
| File manifest | `.FOOTPRINT` (flat path list) | `.MANIFEST` (mtree with metadata) |
| Dependencies | not in package | `[dependencies]` in `.PKGINFO` |
| Script hooks | 4 (install/remove) | 6 (+ upgrade pair) |
| Script runners | `.reef`, `.sh`, bare | `.sh` (zsh) and bare only |
| Install method | `rsync -a` | `fs.ops` driven by manifest |

---

## 2. .PKGINFO — Package Metadata

TOML format. Adds a `[dependencies]` section with optional version constraints.

```toml
[package]
name = "openssl"
version = "3.2.1"
release = 1
description = "TLS/SSL and crypto library"
url = "https://www.openssl.org"
license = "Apache-2.0"
maintainer = "admin@zygaena.org"
arch = "x86_64"

[dependencies]
runtime = ["zlib", "libc"]
build = ["perl", "makedepend"]
```

### Version constraints

Runtime and build dependencies support optional version operators:

```toml
runtime = ["zlib>=1.3", "libc"]
```

Supported operators: `>=`, `<=`, `>`, `<`, `=`. No operator means "any version".

Most dependencies will be unversioned. Version constraints are added only when a
specific ABI or feature is required (e.g., a soname bump, a new API entry point).
Exact version pinning (`=`) is discouraged — it creates rebuild cascades.

The port's `package.toml` remains the authoritative source. The `[dependencies]`
section in `.PKGINFO` is a copy embedded at build time so standalone packages are
self-describing without the ports tree.

### Generation

`generate_pkginfo()` in `build.reef` writes the `[dependencies]` section by
reading `ctx.port.deps.runtime` and `ctx.port.deps.build`.

---

## 3. .MANIFEST — mtree File Inventory

Replaces `.FOOTPRINT`. Uses mtree-style format: one line per filesystem entry,
path first, then space-separated `key=value` attributes.

### Format

```
#mtree
./usr/bin type=dir mode=0755 uname=root gname=root
./usr/bin/passwd type=file mode=4755 uname=root gname=sys sha256=a1b2c3d4e5f6...
./usr/bin/zlib-flate type=file mode=0755 uname=root gname=root sha256=d4e5f6a7...
./usr/include type=dir mode=0755 uname=root gname=root
./usr/include/zconf.h type=file mode=0644 uname=root gname=root sha256=e5f6a7b8...
./usr/lib type=dir mode=0755 uname=root gname=root
./usr/lib/libz.a type=file mode=0644 uname=root gname=root sha256=9c0d1e2f...
./usr/lib/libz.so type=link uname=root gname=root link=libz.so.1
./usr/lib/libz.so.1 type=link uname=root gname=root link=libz.so.1.3.1
./usr/lib/libz.so.1.3.1 type=file mode=0755 uname=root gname=root sha256=f0a1b2...
```

### Fields

| Key | Required | Values | Notes |
|-----|----------|--------|-------|
| `type` | always | `file`, `dir`, `link` | First attribute after path |
| `mode` | files, dirs | 4-digit octal | Includes setuid/setgid/sticky |
| `uname` | always | user name | String, not numeric uid |
| `gname` | always | group name | String, not numeric gid |
| `sha256` | files only | 64-char hex | For install verification and `coral verify` |
| `link` | symlinks only | target path | Relative or absolute link target |

### Sort order

Entries are sorted lexicographically by path. This ensures:
- Directories appear before their contents (install order: top-down)
- Removal in reverse order processes files before their parent directories

### Extensibility

Unknown `key=value` pairs are ignored by the parser. Future attributes (e.g.,
`xattr=...`, `acl=...`, `flags=...`) can be added without breaking existing
Coral versions. However, ACLs and extended attributes are better handled in
post-install scripts where policy decisions belong.

### Metadata exclusion

`.PKGINFO`, `.MANIFEST`, `.CONFIG`, and `.SCRIPTS/` are not listed in the
manifest. They are package metadata, not payload files.

### Generation

`generate_manifest()` in `build.reef` replaces `generate_footprint()`. It walks
the staging directory using `fs.stat` and `fs.perm` to read type, mode, owner,
and group for each entry. File checksums are computed with `checksum.sha256_file()`.
Symlink targets are read with `fs.link.readlink()`.

### Parser module

New module: `util/mtree.reef`

```
export
    type ManifestEntry
    fn parse_manifest(content: string, entries: [ManifestEntry], max: int): int
    fn generate_manifest(pkg_dir: string): string
    fn entry_type(entry: ManifestEntry): string
    fn entry_path(entry: ManifestEntry): string
    fn entry_mode(entry: ManifestEntry): int
    fn entry_sha256(entry: ManifestEntry): string
    fn entry_link(entry: ManifestEntry): string
    fn entry_uname(entry: ManifestEntry): string
    fn entry_gname(entry: ManifestEntry): string
end export
```

---

## 4. Install Method — Replacing rsync

### Current flow (v1)

```
extract .pkg.tar.xz to temp dir
  → find symlinks, remove conflicts in root
  → rsync -a --exclude=metadata temp_dir/ root/
```

### New flow (v2)

```
extract .pkg.tar.xz to temp dir
  → parse .MANIFEST into entry array
  → for each entry (in manifest order):
      if type=dir:
          fs.ops.create_dir_all(root + path)        -- create if missing
          fs.perm.chmod(root + path, mode)           -- set permissions
          fs.perm.chown(root + path, uname, gname)   -- set ownership
      if type=file:
          check if config-protected (see section 6)
          if config file with user modifications → install as .pkg
          else:
              remove any existing conflicting path (file, symlink, or dir)
              fs.ops.copy_file_preserve(src, root + path)
              fs.perm.chmod(root + path, mode)
              fs.perm.chown(root + path, uname, gname)
              verify sha256 matches manifest
      if type=link:
          remove any existing conflicting path
          fs.link.symlink(link_target, root + path)
```

### Advantages over rsync

- **Zero external dependencies** — no rsync required during bootstrap
- **Manifest-driven** — every file placed is accounted for, no silent extras
- **Per-file verification** — sha256 checked on install, not just blindly copied
- **Config protection integrated** — checked inline during install, not as a
  separate pass
- **Explicit permissions** — set from manifest, not inherited from build host
  filesystem quirks

### Error handling

If any file operation fails, installation stops and returns an error. Partial
installs are recorded so `coral remove` can clean up. This is safer than rsync's
`exit code 23` (partial transfer accepted).

---

## 5. Scripting Model

### Hooks

Six hooks, all executed with `zsh`:

| Hook | Arguments | When | Abort on failure |
|------|-----------|------|------------------|
| `pre-install.sh` | `$VERSION` | Before files installed (fresh install) | Yes |
| `post-install.sh` | `$VERSION` | After files installed (fresh install) | No* |
| `pre-remove.sh` | `$VERSION` | Before files removed (full remove) | Yes |
| `post-remove.sh` | `$VERSION` | After files removed (full remove) | No* |
| `pre-upgrade.sh` | `$OLD_VERSION $NEW_VERSION` | Before files replaced (upgrade) | Yes |
| `post-upgrade.sh` | `$OLD_VERSION $NEW_VERSION` | After new files installed (upgrade) | No* |

*Post-scripts log failures as warnings but do not roll back, since the file
operation has already completed.

### Execution

All scripts are invoked as:

```
zsh "/path/to/.SCRIPTS/hook-name.sh" [args...]
```

The `.reef` script type is removed. Requiring the Reef compiler at install time
creates a bootstrap dependency that cannot be satisfied when building the base OS.

Bare executables (no extension) are still supported — Coral executes them
directly, relying on a shebang line or ELF header.

### Script search order

For each hook, Coral looks for (in order):
1. `hook-name.sh` — executed with `zsh`
2. `hook-name` — executed directly (must be executable with valid shebang or ELF)

If neither exists, the hook is silently skipped (not an error).

### Upgrade flow

```
pre-upgrade.sh  $OLD_VERSION  $NEW_VERSION    # abort if non-zero
  → remove old files (driven by installed manifest)
  → install new files (driven by new manifest)
post-upgrade.sh  $OLD_VERSION  $NEW_VERSION   # warn if non-zero
```

If no `pre-upgrade.sh` or `post-upgrade.sh` exists, Coral performs the file
swap silently. It does **not** fall back to running pre-remove + post-install,
as those hooks may have side effects inappropriate for an upgrade (e.g.,
post-install creating a fresh database vs. preserving an existing one).

### Stored scripts

Remove scripts (`pre-remove.sh`, `post-remove.sh`) must be available when a
package is removed, even if the original `.pkg.tar.xz` is gone. Coral stores
a copy of `.SCRIPTS/` in the package database at install time
(`/var/lib/coral/installed/<name>/scripts/`). Upgrade scripts are stored there
as well, since they're needed when a future version replaces the current one.

---

## 6. Config File Protection — Checksum-Based

### Current behavior (v1, broken)

Reads both files fully into memory and compares content with `str.equals()`.
Both branches of the comparison (equal / not equal) execute the same code path —
always installs as `.pkg`. The "files are identical" case should overwrite in
place.

### New behavior (v2)

Uses the sha256 checksum from the manifest to detect user modifications without
reading the full installed file into memory twice.

**On fresh install:**
1. Install config file normally to `dest_path`
2. Store the package-provided sha256 in the database for this file

**On upgrade (file already exists):**
1. Compute sha256 of the currently installed file
2. Compare against the stored checksum from the *previously installed* version
3. Decision:

| Installed matches old pkg checksum? | Action |
|-------------------------------------|--------|
| Yes (user has not modified) | Overwrite with new version |
| No (user has modified) | Install new version as `dest_path.pkg-new`, keep user's file |

4. Update stored checksum to the new package's sha256

**On remove:**
- Config files listed in `.CONFIG` are **not** removed if they differ from the
  package-provided checksum (user has modified them)
- Config files that match the package checksum are removed normally

This matches the proven dpkg conffile model.

### Implementation

```
fn install_config_file(src: string, dest: string, manifest_sha256: string, stored_sha256: string): bool
    if not fs.stat.exists(dest)
        // Fresh install — copy file
        fs.ops.copy_file_preserve(src, dest)
        return true
    end if

    // File exists — check if user modified it
    let installed_sha256 = checksum.sha256_file(dest)

    if installed_sha256 == stored_sha256
        // User has not modified — safe to overwrite
        fs.ops.copy_file_preserve(src, dest)
        return true
    else
        // User has modified — preserve their version
        let new_path = dest + ".pkg-new"
        fs.ops.copy_file_preserve(src, new_path)
        // Log: "Config file preserved, new version at dest.pkg-new"
        return true
    end if
end install_config_file
```

---

## 7. Implementation Plan

### Phase 1: mtree module and manifest generation
- New module `util/mtree.reef` — parser and generator
- Update `build.reef`: replace `generate_footprint()` with `generate_manifest()`
- Use `fs.stat`, `fs.perm`, `fs.link.readlink()`, `checksum.sha256_file()`

### Phase 2: fs.ops-based installer
- Rewrite `install_files()` in `package.reef` to walk the manifest
- Use `fs.ops.copy_file_preserve`, `fs.perm.chmod`, `fs.perm.chown`, `fs.link.symlink`
- Remove rsync dependency from install path
- Integrate config file checksum logic

### Phase 3: Enhanced .PKGINFO
- Update `generate_pkginfo()` to emit `[dependencies]` section
- Update `read_pkginfo()` to parse dependencies
- Add `deps` field to `PackageInfo` type or extend existing `Dependencies`

### Phase 4: Script model cleanup
- Add `pre-upgrade.sh` / `post-upgrade.sh` hook support
- Remove `.reef` script type from `run_script()` and `run_stored_script()`
- Change `.sh` invocation from `sh` to `zsh`
- Pass version arguments to all hooks
- Abort on non-zero exit from pre-* hooks
- Update upgrade command to call upgrade hooks

### Phase 5: Removal and verify
- Rewrite `remove_files()` to use manifest (directories tracked explicitly)
- Use `fs.ops.remove_file`, `fs.ops.remove_tree`, `dir.remove_dir`
- Implement `coral verify <pkg>` — walk stored manifest, check sha256 and
  permissions against live filesystem

### Dependencies

- Reef >= 0.5.0 (for `fs.stat`, `fs.perm`, `fs.link`, `fs.ops`)
- rsync is no longer required for package installation (remains used by
  `coral repo mirror` only)

---

## 8. Risks and Mitigations

| Risk | Mitigation |
|------|------------|
| fs.ops.copy_file_preserve slower than rsync for large packages | Benchmark during Phase 2; rsync does buffered I/O and copy_file_preserve uses 8KB buffer — comparable. Manifest-driven install skips metadata files without needing exclude flags. |
| mtree parser bugs cause install failures | Comprehensive test coverage in Phase 1; mtree is a simple line-oriented format with no nesting or escaping edge cases beyond spaces in paths. |
| Packages with >8192 files overflow entry array | Use manifest file_count pre-scan or dynamic sizing. Current .FOOTPRINT has same limit. |
| Upgrade hook ordering with dependency chains | Out of scope for this design. Coral currently upgrades packages individually. Dependency-ordered upgrades (topological sort) are a future enhancement. |