WorldTUI 360M Model Trained to Simulate a Terminal

TL;DR

WorldTUI fine‑tunes a 360 M‑parameter Qwen2.5‑Coder model to act as a terminal emulator: each keystroke is fed to the model, which predicts screen‑grid updates. The resulting demo runs entirely in the browser (208 MB int4 model) and can create files, run vim, tmux, and even play snake, while remembering simple state across commands.

How the Model Represents a Terminal

Conclusion: The terminal is encoded as a grid of characters and the model predicts row‑wise delta operations rather than a linear token stream.

The input begins with a snapshot token block describing screen size, contents, and cursor location (e.g., BOS SCREEN_START 32 96 …).
Each keystroke is sent as raw bytes ('l', \r, ESC [ D, etc.) wrapped in INPUT_START/INPUT_END tokens.
The model outputs a set of operations such as ROW_0 17 'l' and CURSOR 0 18 1 inside OUTPUT_START/OUTPUT_END, indicating which cells to overwrite and the final cursor state.
A special WAIT token encodes elapsed time, enabling autonomous updates like the snake game.
The vocabulary is trimmed to 333 tokens: 256 byte‑value tokens, 64 row identifiers, and a handful of control tokens (e.g., COPY, WAIT). This compact vocabulary enables the model to fit into a browser‑friendly size.

Efficient Screen Updates with Row‑Copy Operations

Conclusion: Introducing a COPY operation reduces the token cost of scrolling and line‑manipulation from thousands to a few tokens per row.

When the terminal scrolls, instead of emitting every character of the new screen, the model emits ROW_i COPY j commands that copy a previous row to a new position.
This reduces a typical 3,000‑token full‑screen redraw to roughly 3 tokens per affected row.
The same mechanism handles clearing the screen, inserting/deleting lines in vim, and updating man pages.

Training Data Generation

Conclusion: The author generated 336 k synthetic terminal sessions (≈2 B tokens) by recording real Docker shells with randomized environments, ensuring perfect keystroke‑to‑screen alignment.

Random usernames, hostnames, prompts, and terminal dimensions.
Random file system layouts reflecting personas (developer, sysadmin, etc.).
Weighted activity selection (vim editing, python REPL, tmux, etc.).
State tracking to keep commands realistic (e.g., opening existing files).
Human‑like typing with a 1.2 % typo rate.
35 % of sessions start mid‑application to teach the model to continue from arbitrary states.

The final dataset spans 32×96, 24×80, and 40×120 screen sizes.

Fine‑Tuning and Self‑Correction

Conclusion: Adding DAgger‑style self‑correction examples dramatically improved memory consistency, raising the PresetBench score from 24/47 to 38/47 (int4).

After an initial supervised fine‑tune, the model was run on recorded sessions to collect its own mistakes.
For each mistake, a correction delta that would bring the model’s screen back to the ground‑truth was constructed and used as a training example.
Corrections are only allowed to use tokens that have already appeared on the real screen, preventing hallucinations.
Roughly one‑third of sampled sessions yielded useful corrections; three rounds of this process were performed.

Training (including the correction rounds) consumed ~75 GPU‑hours on an RTX 4090.

Benchmarks

Conclusion: WorldTUI outperforms prompted base models on both command output accuracy and simple file‑memory tasks.

Model	ShellBench (106 cmds)	File readbacks (13)
Qwen2.5‑Coder‑0.5B (prompted)	52	2
Qwen3‑0.6B (prompted)	60	3
WorldTUI	80	9

The biggest gain is in the memory test, where WorldTUI correctly reproduces 9/13 read‑back scenarios versus 2–3 for the base models.

Speed Optimizations

Conclusion: Parallel row decoding and speculative verification reduce large‑frame latency from 6.7 s to 0.87 s on an RTX 4090.

Parallel rows: Each row is decoded independently in a batch, sharing the KV cache, so the decoding cost equals the longest row (~~100 steps) rather than the sum of all rows (~~3,000 steps).
Speculative decoding: The parallel pass produces a draft for all rows; a second verification pass with the same model checks the draft and corrects the first disagreement, preserving consistency (e.g., correct vim window placement).
On a Mac M4, single‑character latency is ~60 ms; a full man page takes 2.5 s sequentially vs. 0.9 s with parallel drafting.

Model Size Reduction for the Browser

Conclusion: Trimming the embedding table to the 333‑token vocabulary and int4 quantization shrinks the model to 208 MB, enabling WebGPU inference.

Original Qwen2.5‑Coder‑0.5B: 494 M parameters, 990 MB.
After removing unused token embeddings: ~360 M parameters, 716 MB.
Int4 quantization: 208 MB download size.
Inference runs entirely client‑side via custom WGSL shaders; activations remain 32‑bit floats.

Distillation Experiments

Conclusion: Removing layers harms memory consistency even when next‑token accuracy remains high, indicating that memory relies on specific deeper representations.

Dropping layer 7 alone barely affected token accuracy (99.3 % → 98.9 %) but reduced exact memory readbacks from 64/64 to 0/64.
A 16‑layer student preserving layers 1, 7, 15, 22 matched the teacher on memory (64/64) but lost ShellBench performance (65 vs. 78 correct commands).
Further distillation attempts were abandoned due to this trade‑off.

Demonstrated Capabilities

Conclusion: WorldTUI can run interactive full‑screen applications (vim, tmux, snake) and retain simple state across sessions, though it still hallucinates on complex filesystem manipulations.

Memory: Files created, edited, and read back in separate tmux panes persist correctly most of the time.
Vim: Supports mode switching (Esc), insert, save, and quit.
Tmux: Handles pane creation, navigation, and environment variable expansion.
Python REPL: Accepts simple expressions; tolerates minor typos (pythoon launches Python).
Snake: Runs smoothly despite being compute‑intensive.

Limitations

Conclusion: The model lacks a real filesystem or external tools, leading to hallucinations and occasional path‑tracking errors.

No internet access inside Docker containers; tools like lynx are non‑functional.
Complex file operations (e.g., appending to an existing file) often produce incorrect output.
Nested directory navigation can lose the current path.
Firefox WebGPU performance is ~20× slower than Chrome; the cause is unknown.

Community Reactions (selected comments)

"WorldTUI is a very good way to explore the question: What's the difference between a model that predicts intelligent output, and intelligence itself?" – OtherShrezzing

"Very cool. Fun project. I feel like there’s a wide field to be explored once you understand fine‑tuning good existing models" – oilkillsbirds

"I tried it and it’s not working as close as the demo gif. It tells me a folder already exists when it doesn’t, then the folder is missing after ls." – donperignon

"The next step is training a model to pretend to be the internet. (You could run lynx in that terminal to train/test it without images)" – amelius

"I wonder how much a small model could do if it had a tool‑calling API for a filesystem, an embedded busybox, and a single output token for ‘emit what busybox would do here’" – Lerc

Takeaways

WorldTUI proves that a modest 360 M‑parameter model can learn a coherent, interactive terminal world when trained on perfectly aligned keystroke‑screen data.
Self‑correction via DAgger‑style examples is key to improving persistent memory.
Parallel row decoding and speculative verification are essential for achieving near‑real‑time latency on consumer GPUs and in the browser.
Future work could explore richer state representations (real filesystem APIs, tool calling, diffusion‑based screen updates) to reduce hallucinations and expand applicability.

Try it: The demo is available at https://simedw.com/projects/worldtui (208 MB download, Chrome/Edge recommended).