rmalde/minecraft-agent

Astra planner and JEV controller for Minecraft, with native recording, tested routes, and run verification.

Astra & JEV Minecraft Agent

What it is – An open‑source bot that plays a full Minecraft Java 1.16.5 speedrun (the End‑dragon fight) by combining two language‑model services:

  • Astra (or the older Sol) acts as a planner: given the current world state it sets a high‑level objective, item targets and a waypoint.
  • JEV consumes the planner’s objective and the detailed game observation and picks a single concrete action from a bounded set (move, mine, craft, eat, combat, etc.).

The bot talks to a vanilla server via the standard player protocol (through the Mineflayer library). A hidden, read‑only Minecraft client mirrors the game state for video capture; no game rules or entities are altered by the bot.

How it works

Component Role
Server Official vanilla 1.16.5 server (provided in server/). Runs a custom Java sensor (DragonObserver) that exposes the dragon‑head position via a local HTTP endpoint.
Mineflayer Node‑JS library that implements the player connection, path‑finding and low‑level actions (dig, place, attack, open chest, etc.).
Planner (Astra / Sol) Called via OpenRouter (/api/v1/chat/completions). Receives the full structured observation (blocks, inventory, coordinates) and returns a goal and waypoint.
JEV Typesafe’s decision model (typesafe/jev-1.13) accessed through /api/alpha/decisions. It selects one legal action from the current observation.
Native client & recorder A hidden Minecraft client (built from source) renders the world and captures a 960 × 540 @ 20 fps video with an overlay showing model names, time, coordinates, objective and chosen action.
Glue scripts nether-agent.mjs orchestrates the loop: read sensor → send to planner → send planner output + observation to JEV → execute selected action via Mineflayer.

All communication is text‑based; the models never see screenshots. The bot can see block data in loaded chunks and is given the seed and known portal coordinates.

Getting started (quick‑start)

  1. Prerequisites
    • Node ≥ 18, Java 17, Python 3, a local OpenRouter API key stored in Google Secret Manager.
    • Minecraft Java 1.16.5 client (the repository does not ship it).
  2. Install
    npm install               # Node dependencies
cd native-client && python3 install.py && ./build.sh
  3. Start the server
    cd server && ./start-server.sh   # launches server.jar with the provided seed
  4. Run a fresh agent (replace NEW_RUN with a unique name)
    PLANNER_MODEL=openai/gpt-6-astra RUN_ID=NEW_RUN \
NATIVE_VIEW=1 WAIT_NATIVE=1 NATIVE_RECORD=1 \
DRAGON_SENSOR_URL=http://127.0.0.1:3093 \
node nether-agent.mjs NEW_RUN
    The script will wait for the hidden native client (python3 native-client/launch.py) before the first action.
  5. Verify
    node --test evidence.test.mjs native-mirror.test.mjs ...
    Tests check that the run produced the required victory evidence (victory.json) and that all policy checks passed.

Evidence of success

  • The README documents a full run (nether‑final‑08) that killed the Ender Dragon with six bed explosions, exited the portal with full health, and finished in 8 min 43 s – 40 % faster than the previous best.
  • Logs (runs/<run>/events.jsonl) contain every model request/response, selected action, and game outcome.
  • A locally stored video (full-playthrough-banner.mp4) shows the hidden client rendering, the overlay HUD, and the final victory event.
  • Automated tests (evidence.test.mjs, combat‑lab scripts) verify that the dragon‑death conditions and world state match the recorded evidence.

Scope & limitations

  • Speedrun‑specific – The route, seed and world configuration are fixed; the bot is not a general‑purpose Minecraft AI.
  • Read‑only sensor – The bot cannot grant items, modify health, or change dragon AI; it only issues normal player actions.
  • Model cost – Each run makes ~35 planner calls and ~130 JEV decisions; an OpenRouter credit is required.
  • Platform – Development scripts target macOS; other OSes need manual path adjustments.

Where to look next

  • optimization/nether/config.json – the fixed route and waypoints.
  • observer/DragonObserver.java – source for the dragon‑head sensor.
  • combat-lab/ – isolated test harness for bed‑explosion combat logic.
  • The linked TypeSafe and OpenAI documentation for deeper model‑API details.

Bottom line – This repository provides a reproducible, fully‑automated Minecraft speedrun bot that demonstrates structured‑state control using large language models (Astra) and a typed decision model (JEV). It is a concrete example of LLM‑driven embodied agents operating in a complex, real‑time virtual environment.

Related

  • Project
    nodetool-ai/nodetool

    NodeTool is an open‑source, agent‑first creative workspace that lets you build multimodal AI‑driven media (text, images, video, audio, 3‑D, games) via a node‑graph interface. Connect any LLM or generative model with your own API keys or run local open‑weight models, let the built‑in agent wire and run workflows, and edit the results in specialized editors (storyboard, timeline, sketch, 3‑D, game). Available as a free cross‑platform desktop/web app, extensible with custom TypeScript/Python nodes, and usable from a CLI or MCP server.

  • Project
    emiliaprotocol/emilia-protocol

    An open protocol and gateway for AI agents that ensures consequential actions are bound to specific human approvals and verified before execution to prevent unauthorized modifications.

  • Project
    sam70361/aora-bot

    A zero-dependency SVG and JavaScript expression engine that provides 32 emotional states for AI assistants, chatbots, and desktop pets.

  • Project
    24kchengYe/MemoMind

    MemoMind is a locally hosted, GPU‑accelerated memory system for AI coding agents. It stores extracted facts from chats, documents, and daily life events in PostgreSQL + pgvector, builds a knowledge graph, and offers fast 4‑way hybrid retrieval (semantic, BM25, graph, temporal). The agent can retain new information, recall relevant memories, and reflect across the whole store. All data stays on the user’s machine, works with any OpenAI‑compatible LLM, and includes a web dashboard for browsing and exporting memories.