666ghj/BettaFish

BettaFish – Multi‑Agent Social‑Media Insight Platform

What it is – BettaFish is an open‑source, Python‑based system that stitches together several specialized AI agents to collect, analyse and report on public opinion from dozens of social‑media platforms (Weibo, Xiaohongshu, TikTok, etc.). Users interact with the system via a simple chat‑style query; the platform then runs a pipeline of agents that crawl data, perform multimodal (text + image + video) analysis, run sentiment and topic models, and finally generate a polished interactive HTML (or PDF/Markdown) report.

---

Core Components

Engine	Role	Main duties
QueryEngine	Broad‑search agent	Uses LLM‑driven prompts to issue web searches, fetch news and comments, and summarise initial findings.
MediaEngine	Multimodal agent	Downloads and analyses short‑video/audio, images and extracts structured cards (weather, stock, calendar) from search results.
InsightEngine	Private‑data & deep‑sentiment agent	Connects to a user‑supplied PostgreSQL/MySQL store, runs keyword‑optimised queries, performs fine‑tuned sentiment analysis (BERT‑LoRA, Qwen‑3, etc.).
ReportEngine	Report generator	Takes the agents’ outputs, selects a markdown template, plans chapter length, validates a JSON‑based intermediate representation (IR) and renders an interactive HTML report (PDF via WeasyPrint is optional).
ForumEngine	Agent collaboration hub	Implements a “forum” where a LLM‑hosted moderator mediates debate between the three analysis agents, enabling iterative refinement and avoiding single‑model bias.
MindSpider	Crawling subsystem	A set of Playwright‑driven spiders that harvest topics, comments and videos from the supported platforms; includes a broad‑topic extractor and a deep‑sentiment crawler.

---

How it works (high‑level flow)

1. User query – a Flask endpoint receives a natural‑language request (e.g., "Analyse the reputation of Wuhan University").
2. Parallel launch – QueryEngine, MediaEngine and InsightEngine start simultaneously, each using its own toolset.
3. Initial overview – Agents perform quick searches and return concise summaries.
4. Strategy planning – Based on the overview, the system decides how to split the problem into sub‑tasks.
5. Iterative forum loops – The ForumEngine monitors the agents, a LLM moderator prompts deeper searches, agents reflect on each other’s findings, and the loop repeats until a stopping condition is met.
6. Report assembly – ReportEngine gathers all intermediate results, builds a JSON IR, validates it, selects a markdown template, and renders the final HTML (and optionally PDF/Markdown).

---

Key Features

• AI‑driven 24/7 crawling across 10+ domestic and international platforms.
• Hybrid analysis: combines LLM reasoning, fine‑tuned sentiment models, and classic statistical methods.
• Multimodal capability – parses short videos, images and structured search‑engine cards.
• Forum‑style agent collaboration – a debate‑host LLM steers multiple agents, yielding richer, less homogeneous insights.
• Private‑data integration – secure connectors let enterprises blend internal databases with public opinion.
• Modular, lightweight Python design – each engine lives in its own package; developers can replace prompts, tools or models with a single config change.
• One‑click Docker deployment and optional Streamlit front‑ends for each engine.
• Report generation – interactive HTML reports with charts (SVG), PDF export via WeasyPrint, and markdown fallback.

---

Getting Started (Docker quick‑start)

# copy env template and edit credentials
cp .env.example .env
# start all services (Flask + PostgreSQL + agents)
docker compose up -d
# then open the UI
open http://localhost:5000

For development you can run the engines individually with Streamlit, e.g. streamlit run SingleEngineApp/query_engine_streamlit_app.py.

---

Extensibility

• Prompt swapping – change any agent’s prompt file under */prompts/ to retarget the system (e.g., from brand reputation to financial market analysis).
• Model plug‑in – all LLM calls use an OpenAI‑compatible wrapper; you can point the *_ENGINE_BASE_URL env vars to Azure, Anthropic, or a self‑hosted model.
• Custom tools – add new search or media‑extraction utilities in the tools/ directories; the agents will automatically pick them up.

---

License & Community

• License: GPL‑2.0 (see LICENSE).
• Contributing: guidelines in CONTRIBUTING.md.
• Support: LLM‑API sponsorship from Inferera; additional infrastructure from Anspire Open (search, file parsing, browser automation).
• Demo & docs: a full‑run report on Wuhan University is included (final_reports/), and a walkthrough video is linked in the README.

---

TL;DR

BettaFish is a fully‑featured, multi‑agent opinion‑analysis engine that crawls public social‑media data, runs multimodal and sentiment models, lets agents debate via a forum mechanism, and spits out an interactive report—all orchestrated with Python, Flask and Docker. It is ready for both end‑users (via a web UI) and developers who want to customise the agents for other domains.