jevchat turns Jev into a (lousy) chatbot – features, modes, and community reactions

TL;DR

jevchat is an open‑source wrapper that turns the Jev API into a turn‑by‑turn chatbot by repeatedly asking "Given the user's question and the reply written so far, which symbol comes next?"; it supports several alphabets, sampling strategies, and beam search, but the approach is expensive and primarily a playful experiment.

---

What jevchat does

• At each generation step jevchat sends Jev a single‑choice question: which symbol should follow the current partial reply?.
• Jev returns a probability distribution over the chosen alphabet; a sampler draws the next symbol from the normalized distribution.
• The process repeats until the special STOP symbol is selected, producing the final answer.
• The tool displays live statistics (symbols/s, characters/s, latency per API call) and the top‑scoring symbols for each step, letting users watch the distribution in real time.

---

Installation and basic usage

# Install dependencies via Poetry
poetry install

# Store your Jev API key in .env (or use JEV_API_KEY / TYPESAFE_API_KEY env vars)
# Example .env content:
# api_key="YOUR_KEY"

# Interactive chat
poetry run jevchat

# One‑shot query
poetry run jevchat ask "do people need water?"

# List available alphabets
poetry run jevchat alphabets

# Benchmark all modes (hits the real API)
poetry run jevchat bench

Press Ctrl‑C once to stop after the current request finishes; press twice to abort immediately.

---

Sampling strategies

Strategy	How it works	Typical command
choice (default)	One question over the entire alphabet.	jevchat -s choice ask "how many eyes do people have?"
choice – no shuffle	Disables alphabet re‑ordering that mitigates Jev’s position bias (produces worst‑case quality).	jevchat -s choice --no-shuffle-criteria ask "…"
choice – ensemble	Sends N parallel questions with different alphabet orderings and averages the results.	jevchat -s choice --ensemble 4 ask "…"
bisect	Recursively splits the alphabet into halves, asking yes/no questions until a small group remains, then a final choice.	jevchat -s bisect ask "…"
bisect – larger cuts	Increases the group size before the final choice, reducing API calls.	jevchat -s bisect --bisect-cutoff 32 --no-bisect-swap ask "…"
buckets	Divides a large alphabet into many buckets, each with an OTHER escape; the only strategy that supports >255 symbols.	jevchat -a bpe5k -s buckets --bucket-size 127 ask "…"
refine	Uses buckets then a second‑stage choice over the winning bucket, optionally followed by a nucleus rescore.	jevchat -a words1k -s refine --refine-nucleus 6 --refine-rounds 2 ask "…"

---

Presentation modes

• hypothesis (default): each option is a complete text fragment; Jev scores the finished strings directly. This yields roughly a 3× top‑1 improvement on character alphabets.
• symbol: each option is a single symbol; Jev must imagine the concatenated result before scoring.

---

Beam search

poetry run jevchat -b 3 ask "what is the opposite of hot?"

Keeps N candidate replies alive simultaneously. Each beam incurs one score per step, and traditional temperature/top‑p/top‑k settings are ignored—beams are ranked purely by probability.

---

Alphabets

Alphabet	Description	When to use
lower26	a‑z plus space	Simple letter‑only demos
ascii	Full printable ASCII	Any text generation
tokens	Whole‑word tokens	Natural‑language queries
words1k	1 000 most common words (requires buckets)	Restricted vocab experiments
bpe2k, bpe5k	Byte‑pair‑encoding vocabularies of 2 k / 5 k tokens	Larger but still bounded vocab

---

Performance highlights (from jevchat bench)

• choice with the default alphabet is the slowest because it sends a separate request for every symbol.
• bisect reduces API calls roughly by half while keeping comparable quality.
• buckets and refine enable vocabularies >255 symbols with a modest increase in latency.
• beam width >1 multiplies cost linearly (one score per live beam per step) but can improve coherence.

---

Community reactions on Hacker News

• Eric PrUitt likened the experience to "Morty speaking with the death crystal"—the model iteratively refines its answer while watching its own demise.
• OtherShrezzing highlighted the humor: the model’s output often reads like a joke, producing a genuinely funny short story.
• boodleboodle connected the project to the older idea "BERT has a mouth and it must speak" (arXiv:1902.04094), noting the lineage of token‑by‑token generation.
• yipinwong compared the system to an "ADHD heavy friend" that constantly jumps between thoughts, reflecting Jev’s System 1 style.
• fen_wick wondered whether Jev’s structured data format helps or hinders rapid prototyping of chat‑style interfaces.
• K0IN asked why the approach can’t be reduced to a simple embedding similarity lookup, prompting discussion about the need for explicit decision‑making per symbol.
• petercooper suggested practical uses where a severely limited vocabulary is required (e.g., SQL generation) and reported that jevchat can produce reasonable queries, though traditional LLMs with linting still outperform it.
• IceDane criticized the use of Poetry for Python packaging, reflecting a common sentiment in the community.
• nowittyusername proposed an emoji‑only bot, noting that the architecture’s constraints might suit non‑textual output.

---

Limitations and cost considerations

• Each symbol requires a separate API call (or a small batch when using ensemble), making the approach expensive for longer replies.
• The quality varies dramatically with the chosen strategy and alphabet; the default choice mode can be noisy, while refine offers the best trade‑off.
• Real‑time interaction is limited by network latency and Jev’s per‑symbol scoring overhead.

---

Testing and reliability

• The repository includes 158 offline tests using a mocked HTTP client and a fake generation loop; no API key is needed for these tests.
• The jevchat bench command is the only part that contacts the live Jev service, allowing developers to gauge actual cost and latency.

---

Takeaway

jevchat demonstrates that a language model can be coerced into a deterministic, symbol‑level chatbot by repeatedly asking which symbol comes next? The project showcases a variety of sampling strategies, alphabet configurations, and beam search, turning Jev into an amusing, albeit costly, conversational toy. Community feedback underscores both the novelty and the practical constraints of this approach, hinting at niche applications where a tightly bounded vocabulary is advantageous.