Refactor: Source Mirrors Support

Status: Planning
Version: 0.1.3
Date: January 2026

Overview

Refactor the source download system to support multiple mirror URLs per source file, with automatic fallback when primary sources are unavailable.

Current Design

[sources]
urls = ["https://example.com/file.tar.gz"]
checksums = ["sha256:abc123..."]

Problems:

Multiple URLs treated as separate files, not mirrors
No fallback on download failure
Checksum tied to array index, fragile

New Design

Package.toml Format

[[source]]
file = "vim-9.1.0.tar.gz"
urls = [
    "https://github.com/vim/vim/archive/v9.1.0.tar.gz",
    "https://mirror.example.com/vim-9.1.0.tar.gz",
    "https://backup.example.org/vim-9.1.0.tar.gz"
]
checksum = "sha256:abc123..."

[[source]]
file = "illumos-compat.patch"
urls = ["https://patches.zygaena.org/vim/illumos.patch"]
checksum = "sha256:def456..."
extract = false

New Fields

Field	Required	Default	Description
`file`	Yes	-	Output filename (what to save as)
`urls`	Yes	-	Array of mirror URLs (tried in order)
`checksum`	No	"SKIP"	SHA256 checksum for verification
`extract`	No	true	Whether to extract archive

Implementation Plan

Phase 1: Type Definitions (types.reef)

New struct:

type Source = struct
    file: string        // Output filename
    urls: [string]      // Mirror URLs (tried in order)
    url_count: int      // Number of URLs
    checksum: string    // SHA256 checksum
    extract: bool       // Whether to extract (default: true)
end Source

Update SourceInfo:

type SourceInfo = struct
    sources: [Source]   // Array of Source structs
    count: int          // Number of sources
end SourceInfo

Add factory function:

fn new_source(): Source

Phase 2: TOML Parsing (port.reef)

Add support for parsing [[source]] table arrays:

Detect source. prefixed keys
Group by index (source.0.file, source.0.urls, etc.)
Parse each source into Source struct
Handle both old format (backward compat) and new format

Backward Compatibility:

If sources.urls exists (old format), convert to new format
Each URL becomes a separate Source with auto-detected filename
Checksums mapped by index

Phase 3: Download Logic (build.reef)

Update download_sources():

for each source in sources:
    if file exists in cache and checksum matches:
        skip download

    for each url in source.urls:
        try download to source.file
        if success:
            verify checksum
            if checksum ok:
                break to next source
            else:
                delete file, try next mirror
        else:
            try next mirror

    if all mirrors failed:
        return error

Phase 4: Extract Logic (build.reef)

Update extract_sources():

for each source in sources:
    if source.extract:
        extract archive
    else:
        copy file to work_dir (for patches, etc.)

Phase 5: Update Test Ports

Convert test ports to new format:

test/ports/base/vim/package.toml
test/ports/base/tree/package.toml
test/ports/core/coral/package.toml
etc.

File Changes Summary

File	Changes
`src/types.reef`	Add Source struct, update SourceInfo, add new_source()
`src/core/port.reef`	Parse [[source]] tables, backward compat for old format
`src/commands/build.reef`	Mirror fallback in download, respect extract flag
`src/util/http.reef`	Add download_with_mirrors() helper (optional)
`test/ports/*/package.toml`	Update to new format

Example Migration

Before (old format)

[sources]
urls = ["https://github.com/vim/vim/archive/v9.1.0.tar.gz"]
checksums = ["SKIP"]

After (new format)

[[source]]
file = "vim-9.1.0.tar.gz"
urls = [
    "https://github.com/vim/vim/archive/v9.1.0.tar.gz"
]
checksum = "SKIP"

With mirrors

[[source]]
file = "vim-9.1.0.tar.gz"
urls = [
    "https://github.com/vim/vim/archive/v9.1.0.tar.gz",
    "https://ftp.nluug.nl/pub/vim/unix/vim-9.1.tar.bz2",
    "https://mirror.freedif.org/vim/unix/vim-9.1.tar.bz2"
]
checksum = "sha256:1234567890abcdef..."

Testing Checklist

Single source, single URL (basic case)
Single source, multiple URLs (mirror fallback)
Multiple sources
Primary URL fails, mirror succeeds
All mirrors fail (error handling)
Checksum verification after download
Checksum mismatch triggers next mirror
extract = false for non-archive files
Backward compatibility with old format
Cached file with valid checksum skips download

Future Enhancements

Priority/weight for mirrors (prefer faster/closer)
Parallel downloads from multiple mirrors
Resume partial downloads
Mirror health tracking (remember failed mirrors)