ythx-101/x-tweet-fetcher

What it solves

This project provides a way to fetch data from X (Twitter) without requiring official API keys or user logins. It solves the problem of X's restrictive API and the fragility of standard web scraping, making it easier for AI agents and developers to retrieve tweets, timelines, and articles.

How it works

The tool uses a "smart backend routing" system that automatically switches between three different methods to get data:

1. FxTwitter: Used for fast, dependency-free retrieval of single tweets and user profiles.
2. Nitter: Uses Nitter instances (self-hosted or public) for timelines, search, and replies via direct HTTP requests.
3. Browser Driver: Employs Camofox or Playwright for complex tasks like fetching X Lists and full-text Articles, which require a real browser environment.

All data is normalized into a unified JSON schema, ensuring that regardless of the backend used, the output format remains consistent for downstream AI prompts.

Who it’s for

It is primarily designed for AI agents and developers who need reliable, login-free access to X data for monitoring, archiving, or content analysis.

Highlights

• Auto-Fallback: Automatically switches backends if one fails, with machine-readable error codes for agent branching.
• Local Archiving: Includes a "Ledger" feature that saves fetched tweets into a SQLite database for offline querying and statistics.
• Unified Schema: Normalizes different data sources into a single Tweet / Reply / Article format.
• Comprehensive Coverage: Supports single tweets, threaded replies, paginated timelines, search results, X Lists, and X Articles.
• Flexible Deployment: Supports both a CLI and a Python API.