Fedora 45 Release Process and Build Pipeline

Fedora 45 Release Process and Build Pipeline

Fedora 45 transforms source code into installable artifacts through a highly automated pipeline involving source control, clean-room builds, gated updates, and a multi-stage composition process. This system ensures that every ISO, cloud image, and container image is reproducible, auditable, and tested before reaching the end user.

Source Control and Package Definition

Fedora manages package definitions in individual Git repositories hosted at src.fedoraproject.org (migrating from Pagure to Forgejo). Each repository contains an RPM spec file, downstream patches, and a sources file pointing to upstream tarballs in a lookaside cache to keep binary files out of Git.

Packagers primarily use the fedpkg CLI to manage these repositories. When a packager runs fedpkg build, the tool generates a URL pointing to a specific Git commit and submits it to Koji, ensuring the build is fully reproducible from that commit hash.

The Build System: Koji

Koji serves as the central build system for Fedora. It employs a hub-and-spoke architecture where a passive XML-RPC server manages a PostgreSQL database, and builder daemons poll this hub for work.

To ensure reproducibility, Koji creates a fresh Mock chroot environment for every build, preventing "leakage" from previous builds. The system organizes builds using tags—named collections of builds. Build targets map requests to a build tag (defining the buildroot) and a destination tag (where the finished RPM lands). Koji also orchestrates non-RPM artifacts via plugins for Kiwi images, Image Builder, and OSTree composes.

Update Gating: Bodhi

For branched releases, RPMs do not move directly from Koji to users. Instead, they are gated by Bodhi, the update management system. Bodhi manages the transition of updates through three states: pending, testing, and stable.

Updates move toward stability based on "karma" (user and automated test feedback) and time spent in testing. Critical path packages—those essential for booting and basic functionality—face stricter requirements, including longer testing periods (14 days versus 7) and higher karma thresholds. Bodhi integrates with Greenwave and ResultsDB for CI gating, and once an update is marked stable, Bodhi invokes Pungi to compose the actual DNF repositories.

Release Composition: Pungi

Pungi is the orchestrator that coordinates various tools to turn individual RPMs into final products like ISOs and cloud images. The process follows a strict sequence to maintain consistency:

1. Package Set Freezing (Pkgset)

Pungi first snapshots a specific Koji tag. This Pkgset phase ensures that all subsequent steps use a frozen set of packages, making the entire compose auditable and preventing new Koji builds from sneaking into a release in progress.

2. Product Definition (Comps and Variants)

Pungi uses two XML inputs to determine product composition:

  • Comps (fedora-comps): Defines package groups (e.g., gnome-desktop) and the role of each package (mandatory, default, optional, or conditional).
  • Variants XML: Defines specific products (e.g., Workstation, Server, Silverblue) and which comps groups they include.

3. Boot Image and ISO Production

  • Buildinstall: Uses lorax to create boot.iso, the minimal image that boots the Anaconda installer. This is the foundation for all installer ISOs.
  • Createiso: Overlays packages and repository metadata onto boot.iso using xorriso to produce the final dvd.iso (primarily for the Server variant).

4. Specialized Image Building

Fedora uses three primary paths for non-traditional ISOs:

  • Kiwi: Builds cloud images (AWS, Azure, GCP), Vagrant boxes, container base images, and WSL images. It uses XML definitions and integrates with Koji via the kiwiBuild task type.
  • Image Builder: Based on osbuild, this handles OSTree-based and bootc-based artifacts, including Atomic Desktop ISOs, Fedora IoT, and Fedora Minimal. It uses a pipeline of sandboxed stages to process JSON manifests.
  • rpm-ostree: For Atomic Desktops (Silverblue, Kinoite, etc.), rpm-ostree compose tree creates a versioned, checksummed filesystem tree (an OSTree commit) based on YAML treefiles. A comps-sync.py script ensures these treefiles stay synchronized with the standard comps XML definitions.

Metadata and Validation

Every compose generates standardized metadata via the productmd library, producing files like composeinfo.json, images.json, and rpms.json. These files are critical for downstream tools: Anaconda uses them to find installation trees, and openQA uses them to locate images for testing.

Automated Testing with openQA

Once a compose is finished, openQA boots the images in virtual machines to run automated scenarios, including installation workflows, desktop functionality, and upgrade paths. Nightly Rawhide and milestone composes (Beta, Final) always trigger openQA runs, and blocking bugs found here can halt a release.

Governance: The Changes Process

Major modifications to the Fedora pipeline or system defaults are managed through the Changes process. Proposals are categorized as System-Wide (requiring FESCo approval and detailed contingency plans) or Self-Contained (scoped to specific packages). All changes must be implemented in Rawhide and meet specific deadlines (such as the Beta Freeze) to be included in a release.

Technical Insights from the Community

Community discussion highlights the importance of this end-to-end visibility for troubleshooting. One contributor noted that understanding the image production pipeline is essential for diagnosing filesystem permission issues that change between versions. However, some historical context was provided regarding Koji's clean-room builds, with one user noting that in the past, dependencies not listed in Build-Requires might have occasionally leaked into builds due to the environment state, though this is less common in modern iterations.

Sources