Epic Games Lore Version Control System

Epic Games has released Lore, an open-source version control system (VCS) specifically engineered to handle the scale of modern game development and entertainment projects. Unlike general-purpose VCS tools, Lore is optimized for repositories that combine source code with massive binary assets, such as 3D models, textures, and audio files.

Core Architecture and Technical Design

Lore is a centralized, content-addressed version control system that utilizes Merkle trees and an immutable revision chain to ensure data integrity and scalability.

Content-Addressed Storage and Merkle Trees

Lore stores repository data referenced by content hashes within a Merkle tree. This architecture enables rapid comparisons, efficient integrity checks, and the ability to reuse data across different branches and history without duplication.

Immutable Revision Chain

Every revision in Lore is cryptographically signed. The hash of a revision is derived from its state, including the hashes of its parent revisions and the contained data. This creates a tamper-evident, immutable chain of truth for the entire project history.

Binary Optimization via Chunking

To handle large files efficiently, Lore employs chunked storage. Files are broken into reusable chunks with indexed lookups, which minimizes duplication and optimizes the transfer and update process for large binary assets.

Sparse Workspaces and On-Demand Hydration

Lore reduces local storage requirements through on-demand hydration. Workspaces remain lightweight by fetching specific file data only when it is requested by the user, eliminating the need to download the entire repository history upfront.

Key Features and Integration

Lore provides a flexible interface designed for both developers and artists, focusing on high throughput for large teams.

  • Full-Surface API: Lore can be extended and integrated using C/C++, C#, Rust, Go, Python, or JavaScript. SDKs for JavaScript, Python, .NET (C#), and Go are available on GitHub.
  • Lightweight Branching: Branches are implemented as mutable references, allowing for fast creation and switching without duplicating underlying data.
  • Centralized Service with Caching: The system uses a service-backed architecture with caching layers in front of durable storage to maintain performance as team size and repository volume grow.
  • CLI-First Interface: Full functionality is accessible via a command-line interface (CLI), which users have noted provides more readable output than traditional Git logs.

Industry Context and Use Cases

Lore is positioned as a modern alternative to proprietary systems like Perforce (Helix Core), which has long been the industry standard for AAA game development due to its ability to handle large binaries and provide file locking.

Comparison with Git and Perforce

While Git is highly efficient for text-based code, it often struggles with large binary files, even with Git LFS. Community discussion highlights that game development requires specific features—such as exclusive file locking for artists—that are traditionally the domain of Perforce. Lore aims to bridge this gap by providing the scalability of a centralized system with the openness of an MIT-licensed project.

Provenance and Current Adoption

Lore was previously known as Unreal Revision Control and served as the built-in VCS for UEFN (Unreal Editor for Fortnite). It is currently being adopted by internal Epic teams and serves as the backing store for the UEFN cook pipeline to reduce redundant file transfers and accelerate the time between publishing and playability.

Community Insights and Critiques

Technical discussions surrounding the release have raised several points regarding the system's practical application:

  • Tooling and UI: Some users have expressed concern over the lack of a native GUI client for artists, noting that a CLI-only workflow may be a barrier for non-technical team members.
  • File Locking: There are questions regarding the current implementation of asset locking, a critical requirement for binary assets that cannot be merged.
  • Implementation Language: The system is implemented in Rust, a choice that has surprised some who expected Epic's internal C++ or Verse languages.
  • Comparison to Other Data Versioning: Some observers note that Lore shares conceptual similarities with other data-versioning tools like Pachyderm, LakeFS, and Oxen, suggesting a convergence between the needs of AI/ML data management and game development.

"Git is fine for text based files like code, but it's really bad at stuff like textures, 3D models, audio files, and other non-text files that game developers need to collaborate on... The SOTA in this area is Perforce... a proprietary system."

"Lore, formerly called Unreal Revision Control, is the built-in version control system for UEFN (Unreal Editor for Fortnite), where creators have been using it to version their islands."

Sources