nshiab/simple-data-analysis

What it solves

SDA is a high-performance TypeScript library designed to bring powerful data analysis capabilities to the JavaScript/TypeScript ecosystem. It eliminates the need for developers to switch between Python (Pandas) or R (Tidyverse) for data crunching and JavaScript for visualization, allowing the entire data pipeline—from loading and cleaning to analysis and visualization—tobe completed in a single language.

How it works

SDA is built on top of DuckDB, an in-process analytical database, executing SQL queries via duckdb-node-neo. It uses a chainable, synchronous builder API inspired by Pandas and the Tidyverse. To optimize performance, SDA queues transformations and fuses compatible operations into single DuckDB statements to reduce database round trips. It also lazily loads extensions for geospatial computations, fuzzy matching, and vector search.

Who it’s for

It is primarily aimed at data journalists, researchers, and TypeScript developers who need to perform complex tabular and geospatial analysis on large datasets without leaving the JS/TS environment.

Highlights

• Multi-modal Data Support: Handles tabular, geospatial, and vector data.
• AI Integration: Includes methods for row-by-row LLM processing, embeddings, vector similarity, hybrid search, and RAG.
• Public Data Connectors: Built-in support for pulling data directly from Statistics Canada and OpenStreetMap.
• High Performance: Leverages DuckDB for efficient processing of millions of records, often outperforming traditional dataframe tools.
• Visualization: Integrated support for terminal-based charts and exporting to Observable Plot (PNG/SVG), Google Sheets, and Datawrapper.