liyedanpdx/reddit-ai-trends

Reddit AI Trend Reports

What it is – An open‑source pipeline that automatically pulls posts from a set of AI‑focused subreddits, enriches them with image analysis, YouTube transcript summarisation and web‑page scraping, then uses LLMs (via Groq or OpenRouter) to generate daily trend reports in English and Chinese.

Key capabilities

• Real‑time monitoring of dozens of AI sub‑communities (e.g., r/MachineLearning, r/LLMDevs, r/Rag, etc.).
• Multimodal enrichment: optional vision models (Qwen‑VL, Gemini) for images, YouTube transcript extraction, and Firecrawl‑based web‑page summarisation.
• Bilingual output – reports are produced in both English and Mandarin, with language selection controlled via REPORT_LANGUAGES.
• Historical storage – all raw posts, enrichments and generated reports are persisted in MongoDB, enabling week‑over‑week or month‑over‑month trend analysis.
• Cost‑aware design – smart caching of API results, configurable “smart” comment fetching, and the ability to toggle each enrichment step.
• Containerised deployment – Docker Compose spins up the app together with MongoDB; the repo includes a ready‑to‑run docker-compose.yml.

How it works (high‑level flow)

1. Collect – the Reddit fetcher pulls the newest POSTS_PER_SUBREDDIT posts from the subreddits listed in REDDIT_SUBREDDITS.
2. Enrich – optional modules run:
   • Vision model generates captions for any images.
   • YouTube URLs are turned into transcripts and summarised.
   • Linked web pages are scraped via Firecrawl and summarised.
   • Top comments are fetched (smart mode skips them when the post already has enough text).
3. Cache – every enrichment result is stored in MongoDB so repeated runs reuse the data and avoid extra API charges.
4. Prompt & generate – a JSON‑structured prompt is fed to the selected LLM (Groq or OpenRouter) which produces a markdown report containing:
   • Daily highlights
   • Weekly trend comparison
   • Monthly technology evolution
   • Selected deep‑dive sections
5. Persist – the report is written to reports/YYYY/MM/DD/ and a symlink (latest_report_en.md, latest_report_zh.md) is updated for quick access.

Getting started

# 1. Clone the repo
git clone https://github.com/liyedanpdx/reddit-ai-trends.git && cd reddit-ai-trends

# 2. Copy env template and fill in your keys
cp .env.example .env
# edit .env → add Reddit, Groq/OpenRouter, Firecrawl keys, etc.

# 3. Run with Docker Compose (recommended)
docker-compose up -d

# 4. Watch the logs – the app will generate a report each day at REPORT_GENERATION_TIME
docker-compose logs -f app

For a one‑off run without Docker:

pip install -r requirements.txt
python report_generation.py --languages en zh   # generate now

Typical use cases

• Researchers or product teams that need a concise, daily snapshot of what the AI community is discussing.
• Investors or analysts looking for early signals of emerging models, libraries, or research directions.
• Content creators who want a ready‑made briefing to base newsletters, podcasts or videos on.
• Anyone interested in bilingual AI news – the same pipeline produces both English and Chinese summaries.

Why it matters Reddit is a fast‑moving hub for AI breakthroughs, but the signal is buried in thousands of posts daily. This project automates the heavy lifting—data collection, multimodal enrichment, and summarisation—so users get a curated, historically‑trackable report without manual scraping or prompt‑engineering.

License – MIT (feel free to adapt, extend, or self‑host).