SleepySoft/IntelligenceIntegrationSystem

Intelligence Integration System with AI and Workflow

What it solves

This project addresses the issue of information overload and low-quality journalism. It aims to filter out "watery" content and clickbait titles from massive amounts of daily news, ensuring that only high-value intelligence is surfaced for the reader.

How it works

The system follows a pipeline of scraping $\rightarrow$ submission $\rightarrow$ cleaning/AI analysis $\rightarrow$ filtering/re-publishing $\rightarrow$ archiving.

  1. Scraping: It uses RSS feeds and list-page scraping to collect news from various global media outlets. It includes a "What You See Is What You Get" (WYSIWYG) tool for generating crawler configurations.
  2. AI Analysis: Collected news is processed by an AI analyzer that uses specific prompts to score and analyze the intelligence value of the content. It utilizes json_repair to ensure high parsing rates of AI outputs.
  3. Vector Storage: It employs a vector database (using the bge-m3 model) to enable similarity searches, intelligence jumping, and correlation deduction.
  4. Management: A central hub (IntelligenceHub) manages the data flow, provides a web interface for administration, and publishes the filtered results as HTML pages.

Who it’s for

Users interested in Open Source Intelligence (OSINT) who want a streamlined, AI-powered way to monitor global news and extract meaningful insights without reading through low-value articles.

Highlights

  • AI-Driven Scoring: Automatically filters news based on intelligence value, aiming to separate high-value reports from noise.
  • Vector-Based Correlation: Uses a vector database to find similar intelligence and perform correlation deductions.
  • WYSIWYG Crawler Config: Simplifies the addition of new news sources through a visual configuration interface.
  • Global Media Integration: Pre-configured to scrape from a wide variety of international news agencies (e.g., BBC, Al Jazeera, TASS).
  • Comprehensive Storage: Uses MongoDB for structured intelligence archiving and a dedicated vector DB for semantic search.

Related

  • Project
    taranis-ai/taranis-ai

    An open-source intelligence (OSINT) tool that uses AI and NLP to collect unstructured news from diverse sources and transform it into structured intelligence reports.

  • Project
    KatrielMoses/voidaccess

    A self-hostable OSINT tool that automates dark-web research by collecting, enriching, and mapping threat intelligence into structured formats.

  • Project
    apurvsinghgautam/robin

    An AI-powered OSINT tool that leverages LLMs to refine queries, filter search results, and summarize findings from dark web search engines.

  • Project
    estebanpdl/osintgpt

    A Python package for OSINT research that leverages GPT models and Qdrant vector search to analyze text data and find similar documents.

  • Project
    Thysrael/Horizon

    An AI-powered news radar that fetches, deduplicates, and filters content from multiple sources to generate personalized daily briefings.