xai-org/x-algorithm

X For You Feed Algorithm

What it is

• The open‑source core of the For You timeline on X (formerly Twitter). It decides which posts a viewer sees by mixing content from accounts they follow (in‑network) with content discovered from other accounts (out‑of‑network) and then ranking everything with a transformer‑based model.

Key components

• Home Mixer – the main request‑time pipeline that gathers candidates, hydrates them, applies pre‑scoring filters, scores them with the Phoenix model, re‑ranks, and finally selects the top‑K posts.
• Thunder – in‑memory cache of recent posts from followed accounts.
• Phoenix – retrieval model that predicts a viewer’s probability of taking actions (like, retweet, report, dwell time, etc.) on each candidate post.
• SimClusters – similarity‑based source that surfaces posts from accounts the viewer does not follow.
• Visibility‑filtering – a separate subsystem that decides per post whether it should be shown, shown with a notice, behind an interstitial, or dropped, based on labels, blocks, mutes, legal compliance, etc.
• Labeling path – continuous background pipelines that run classifiers (media‑model‑proxy, CLIP, Agatha, BDSM, etc.) and rule engines (Botmaker, Scarecrow, Abuse‑Enforcement) to attach safety and authenticity labels to posts and accounts.
• Blending pipeline – after ranking, non‑post items such as ads, “Who to Follow” recommendations, and prompts are interleaved with the organic timeline.

How it works (high‑level flow)

1. Query hydration – collect the viewer’s recent engagement history, follow list, blocks/mutes, etc.
2. Candidate sources – query Thunder (in‑network), Phoenix retrieval, SimClusters, and a shared popular‑posts list in parallel.
3. Candidate hydration – fetch post text, media, author details, existing labels, language, etc.
4. Pre‑scoring filters – drop duplicates, old posts, viewer’s own posts, blocked accounts, etc.
5. Scoring – Phoenix predicts action probabilities; a value model combines them with configurable weights; optional VMRanker can call an external reranker.
6. Selection – keep the top‑K scored candidates.
7. Post‑selection filters – visibility‑filtering decides the final fate of each post; deduplication of conversation threads follows.
8. Blending – ads, follow suggestions, and prompts are interleaved with the ranked posts.
9. Side effects – log serving events, refresh caches, record ad metrics.

Configuration & experimentation

• All tunable weights and feature flags live in home-mixer/params/param.rs. Experiments can enable/disable individual stages (e.g., the Brazil 2026 Election filter) without code changes.

What’s not in the repo

• Production data stores, real‑time model training pipelines, and the full set of private safety models. The repository contains the orchestration code and synthetic‑data generation for proof‑of‑concept training.

Transparency tools

• An Under the Hood label‑transparency UI reports to users why a post’s visibility was limited (e.g., legal takedown, compliance filters).

Typical use cases

• Researchers studying large‑scale recommendation pipelines.
• Engineers building similar feed systems who need a reference architecture for candidate generation, scoring, and safety filtering.
• Auditors or policy analysts examining how platform decisions (weights, legal filters) are implemented.

Getting started

1. Clone the repo.
2. Follow the README sections on synthetic data generation to run a small‑scale Phoenix training run.
3. Use the provided configuration files to spin up the Home Mixer service locally (requires Rust toolchain).
4. Explore the transparency tool by running the under_the_hood server and querying it with a test user ID.

---

License – see the LICENSE file in the repository.