rahulnyk/knowledge_graph

What it solves

This project converts a corpus of text (such as a PDF) into a structured knowledge graph of concepts. It aims to move beyond simple entity extraction to capture meaningful concepts and their relationships, enabling a more profound way to analyze text or implement Graph Retrieval Augmented Generation (GRAG).

How it works

The system processes text through the following pipeline:

1. Chunking: The text is split into smaller segments.
2. Concept Extraction: A local LLM (Mistral 7B via Ollama) extracts concepts and their semantic relationships from each chunk.
3. Relationship Mapping: Edges are created based on both the explicit semantic relationships identified by the LLM and the contextual proximity (concepts appearing in the same chunk).
4. Weighting and Aggregation: Similar pairs are grouped, weights are summed, and relationships are concatenated to create a single edge between distinct concepts.
5. Analysis and Visualization: The tool calculates node degrees and communities for sizing and coloring, then uses NetworkX and Pyvis to generate an interactive web-based visualization.

Who it’s for

Users who want to transform unstructured text into a visual network of ideas, researchers looking to analyze the connectedness of concepts in a body of work, and developers interested in building Graph RAG systems using local, cost-effective LLMs.

Highlights

• Local Execution: Uses Ollama and Mistral 7B to ensure the process is free and runs entirely on a personal machine.
• Concept-Centric: Focuses on extracting "concepts" rather than just named entities for more meaningful graphs.
• Interactive Visualization: Generates JavaScript-based graphs that can be hosted on the web.
• Graph Analytics: Supports calculating centralities and communities to identify the most important concepts and clusters within the text.