tinyfish-io/bigset-oss

What it solves

BigSet allows users to create structured datasets from the live web using plain English descriptions. It eliminates the manual effort of stitching together search APIs, scraping frameworks, schema design, and scheduling tools to build and maintain current data tables.

How it works

1. Description: The user provides a natural language description of the desired dataset.
2. Schema Inference: AI automatically determines the necessary column names, data types, and primary keys.
3. Discovery: An orchestrator agent uses web search to find relevant entities.
4. Population: Parallel sub-agents research each entity, fetch real-time data, and verify it against sources.
5. Output: The resulting structured data is presented in a UI and can be exported as CSV or XLSX.
6. Maintenance: Users can set a refresh cadence (from 30 minutes to weekly) to keep the data current automatically.

Who it’s for

Researchers, lead generation specialists, and AI agent developers who need verified, structured, and regularly updated data from public web pages without writing custom scrapers.

Highlights

• Autonomous Research: Uses agents to search, fetch, and verify data rather than relying on static scraping.
• Automated Schema: Infers the table structure automatically from the user's prompt.
• Scheduled Refreshes: Built-in cron-like functionality to prevent datasets from going stale.
• CLI Support: Includes a command-line interface for creating and exporting datasets programmatically.
• Integration: Built on TinyFish APIs for web search and page fetching.