JackHopkins/factorio-learning-environment

What it solves

This project provides a standardized framework for developing and testing Large Language Model (LLM) agents within the game Factorio. It aims to create challenging, open-ended evaluation benchmarks that remain difficult even for advanced frontier models, preventing the "saturation" of benchmarks where AI agents can easily solve all tasks.

How it works

Agents interact with the game environment using a code synthesis approach based on a REPL (Read-Eval-Print-Loop) pattern. The process follows three steps:

1. Observation: The agent reads the output streams (stdout/stderr) from its previous program to understand the state of the world.
2. Action: The agent writes a Python program to execute a specific action in the game.
3. Feedback: The environment runs the program, updates the game state and namespace, and returns the output stream to the agent.

Who it’s for

AI researchers and developers working on LLM agents, specifically those focused on long-term planning, resource management, and open-ended evaluation benchmarks.

Highlights

• Code Synthesis Interface: Agents control the game via Python programming rather than simple text commands.
• Evaluation Framework: Includes built-in tools for running evaluation trajectories and experiments.
• Extensible Architecture: Supports optional features like the Model Context Protocol (MCP) and PostgreSQL for data management.
• Integrated Cluster Management: Provides a CLI tool (fle cluster start) to manage the Factorio game clusters required for the environment.