nlweb-ai/NLWeb

Main reference implementation for NLWeb, implemented in Python.

What it solves

NLWeb simplifies the creation of conversational interfaces for websites, allowing them to be accessible to both human users and AI agents. It addresses the need for a standardized way for AI assistants to interact with web content using natural language APIs, leveraging existing semantic layers like Schema.org.

How it works

NLWeb provides a protocol and a set of tools to turn website content into a conversational endpoint. It uses Schema.org structured data to return responses in JSON format. Every NLWeb instance acts as an MCP (Model Context Protocol) server, supporting an ask method for natural language queries. The project includes several modules:

  • AskAgent: The core query agent that handles natural language queries against websites.
  • AgentFinder: A discovery service for routing to NLWeb agents across the web.
  • AgentFinder: A discovery service for routing to NLWeb agents and routing to NLWeb agents across the web.
  • DataFinder: A translator that converts natural language to SQL for enterprise data sources using Schema.org mappings.
  • ModelRouter: A tool for routing and scoring LLMs to select cost-effective models based on quality thresholds.
  • NLWebScorer: Neural scorer models for ranking and evaluating search results.

Who it’s for

Website owners and developers who want to implement natural language interfaces for their sites, as well as developers building AI agents that need a structured, standardized way to interact with the web.

Highlights

  • MCP Support: Natively supports the Model Context Protocol, making it easier for AI agents to interact with websites.
  • Broad Compatibility: Works across Windows, macOS, and Linux, and integrates with various vector stores (e.g., Qdrant, Milvus, Elasticsearch) and LLMs (OpenAI, DeepSeek, Gemini, Anthropic).
  • Semantic Web Integration: Leverages Schema.org and RSS to treat the web as a semantic layer for AI.
  • Platform Agnostic: Designed to be lightweight and scalable from laptops to data center clusters.

Related

  • Project
    mims-harvard/ToolUniverse

    An ecosystem for building AI scientist systems that provides a standardized protocol and access to over 1,000 scientific tools, ML models, and datasets.

  • Project
    microsoft/powerbi-modeling-mcp

    An MCP server that enables AI agents to build, modify, and manage Power BI semantic models using natural language and automated workflows.

  • Project
    browser-use/browser-use

    A library and framework that enables AI agents to control a web browser like a human to automate tasks such as filling forms and extracting data.

  • Project
    qingchencloud/clawpanel

    A visual management panel for AI Agent frameworks like OpenClaw, Hermes Agent, and DeepSeek Harness, featuring a built-in AI assistant for system diagnosis and configuration.

  • Project
    webmachinelearning/prompt-api

    A proposed web platform API that gives developers a direct, standardized way to access built-in browser or OS language models for local, private, and cost-effective AI integration.