xai-org/x-algorithm

X For You Feed Algorithm (x‑ai‑org/x‑algorithm)

What it is – This repository contains the production‑grade code that powers the For‑You timeline on X (the platform formerly known as Twitter). It stitches together two streams of content – posts from accounts a user follows (the in‑network path) and posts discovered from elsewhere (the out‑of‑network path) – then filters and ranks them with a transformer‑based model.

Why it matters – The feed is the core user‑experience on X. Understanding how it works is valuable for anyone studying large‑scale recommendation systems, real‑time ML pipelines, or the social‑impact of algorithmic curation.

Core concepts

Concept	Description
In‑network source	`thunder/` – an in‑memory store that serves recent tweets from accounts the viewer follows.
Out‑of‑network sources	`phoenix/` – a retrieval model that predicts relevance of tweets from accounts the viewer does not follow. `simclusters/` – a similarity‑based cluster that surfaces additional out‑of‑network tweets.
Scoring model	A transformer (named Phoenix) that consumes the viewer’s recent engagement history and predicts probabilities for actions such as Like, Retweet, Reply, Dwell‑time, Report, etc.
Weighting	Predicted action probabilities are multiplied by hand‑tuned weights (see `home-mixer/params/param.rs`) and summed to produce a single relevance score.
Visibility filtering	`visibility-filtering/` decides per‑post whether it is ALLOW, shown INTERSTITIAL (e.g., adult content warning), or DROP based on a rich set of signals (user blocks, mutes, account labels, country, subscription status, etc.).
Blending pipeline	After ranking, non‑post items (ads, “Who to Follow” suggestions, prompts, etc.) are interleaved via the Blender (`home-mixer/candidate_pipeline/for_you_candidate_pipeline.rs`).
Labeling path	Continuous background jobs label posts and accounts using a suite of classifiers (`agatha/`, `bdsm/`, `clip/`, `media‑model‑proxy/`) and rule engines (`scarecrow/`, `botmaker/`). These labels feed the visibility filter.

Repository layout (high‑level)

home-mixer/ – the request‑time pipeline that hydrates candidates, runs pre‑scoring filters, scores with Phoenix, applies post‑selection filters, and selects the top‑K posts.
visibility-filtering/ – logic that maps labels + viewer actions to ALLOW/INTERSTITIAL/DROP decisions.
phoenix/ – training code, inference wrapper, and synthetic data generation for the ranking transformer.
simclusters/ – cluster‑based retrieval of out‑of‑network tweets.
thunder/ – in‑memory cache of recent tweets from followed accounts.
botmaker/, botmaker-rules/, scarecrow/ – rule‑engine infrastructure that produces the labels consumed by the visibility filter.
agatha/, bdsm/, user-cred-v2/, clip/, media-model-proxy/ – ML models that score accounts, media, and text.
abuse-enforcement-service/ – enforcement logic that can suspend accounts or hide their content based on model scores.
under‑the‑hood‑label‑transparency‑tool/ – a UI that lets a user view aggregate statistics about the labels affecting their timeline.

How it works (request flow)

Query hydration – Pull the viewer’s recent actions, follow list, blocks/mutes, etc.
Candidate sources – Parallel fetch from thunder/ (in‑network) and from phoenix/ + simclusters/ (out‑of‑network).
Candidate hydration – Enrich each tweet with text, media embeddings, author labels, language, engagement counts, etc.
Pre‑scoring filters – Drop stale tweets, duplicates, self‑posts, subscriber‑only content, etc.
Scoring –
- PhoenixScorer predicts per‑action probabilities.
- RankingScorer applies the weighted sum, author‑decay, out‑of‑network discount, and new‑author boost.
- VMRanker optionally calls an external reranking service.
Selection – TopKScoreSelector keeps the highest‑scoring candidates.
Post‑selection filters – Visibility‑filtering checks each post against the label store; dropped posts are removed, interstitial posts stay but are flagged.
Blending – Non‑post items (ads, follow suggestions, prompts) are interleaved.
Side effects – Log impressions, refresh caches, record ad clicks, etc.

Notable recent updates (as of Aug 2026)

Weight documentation – Added comments in home-mixer/params/param.rs and home-mixer/scorers/ranking_scorer.rs clarifying that weights multiply predicted action probabilities, not raw engagement counts.
Brazil 2026 Election filter – New Brazil2026ElectionFilter removes posts from accounts flagged by Brazil’s electoral court unless the viewer follows them.
Synthetic training pipeline – phoenix/ now includes code to generate synthetic data and run a proof‑of‑concept training run, replacing the earlier placeholder “Phoenix demonstration model”.
Visibility‑filtering expansion – New modules (visibility-filtering/) and associated label sources added to make the filter more transparent and configurable.

What’s not in this repo

The repository deliberately excludes:

The live production data stores (e.g., tweet databases, user‑profile stores).
The full‑scale training infrastructure used at X (large‑scale Spark/TF pipelines).
Real‑time serving infrastructure outside the request‑path code (load balancers, API gateways, etc.).

License

The project is released under the Apache 2.0 License, permitting use, modification, and distribution with attribution.

TL;DR for a non‑expert

This code is the engine behind the personalized timeline you see on X.
It mixes posts you follow with algorithm‑chosen posts, scores them with a modern AI model, and then applies a set of safety and relevance rules before showing them.
The repo is a rare glimpse into a real‑world, production‑grade recommendation system that combines large‑scale ML, rule‑based moderation, and engineering tricks to keep the feed fast and safe.