SleepySoft/IntelligenceIntegrationSystem

Intelligence Integration System with AI and Workflow

What it solves

This project addresses the problem of information overload and poor quality of modern news media, where high volumes of "filler" content and clickbait titles often obscure actual intelligence value. It aims to transform raw, public news into structured, high-value intelligence by filtering out noise and summarizing key information.

How it works

The system follows a pipeline of scraping $\rightarrow$ submission $\rightarrow$ cleaning/AI analysis $\rightarrow$ filtering/re-publishing $\rightarrow$ archiving.

  1. Scraping: Uses RSS feeds and list-page scraping (via CrawlerServiceEngine.py) to collect news from a wide variety of international media outlets. It includes a "What You See Is What You Get" (WYSIWYG) tool for generating crawler configurations.
  2. AI Analysis: The IntelligenceAnalyzerProxy.py uses AI clients and specific prompts to analyze, score, and summarize the collected news. It utilizes json_repair to ensure high parsing rates of AI outputs.
  3. Storage & Retrieval: Raw and processed data are stored in MongoDB. A vector database (using the bge-m3 model) is used to enable similarity searches, related intelligence jumps, and associated intelligence deduction.
  4. Publication: A built-in web service (IntelligenceHubWebService.py) provides a management backend and a public-facing page to view the filtered intelligence.

Who it’s for

Users interested in Open Source Intelligence (OSINT) who want to automate the monitoring of global news sources and extract meaningful insights without reading through low-value articles.

Highlights

  • Automated OSINT Pipeline: Integrates scraping, AI-driven scoring, and AI-driven summarization into a single workflow.
  • Vector DB Integration: Supports advanced search and intelligence correlation/deduction using vector embeddings.
  • WYSIWYG Crawler Config: Simplifies the addition of new news sources via a visual interface.
  • Multi-source Support: Pre-configured for a wide range of global news agencies (e.g., BBC, Al Jazeera, TASS).
  • Modular Architecture: Separates the core hub, crawler engine, and vector database service for flexible deployment.

Related

  • Project
    taranis-ai/taranis-ai

    An AI-powered OSINT tool that automates the collection of unstructured news from the web and helps analysts transform it into structured intelligence reports.

  • Project
    KatrielMoses/voidaccess

    A self-hostable OSINT tool that automates dark-web research by collecting, enriching, and mapping threat intelligence into structured formats.

  • Project
    estebanpdl/osintgpt

    A document indexing and retrieval tool for analysts that allows querying private research collections with citations and support for fully local LLM pipelines.

  • Project
    VectifyAI/OpenKB

    An open-source system that compiles raw documents into a structured, interlinked wiki-style knowledge base using LLMs and vectorless, reasoning-based retrieval.

  • Project
    Thysrael/Horizon

    An AI-powered news radar that aggregates, filters, and deduplicates content from multiple sources to generate personalized daily briefings with context and community discussion.