nshiab/simple-data-analysis

What it is: A TypeScript library for tabular, geospatial, and AI-assisted data analysis that runs on Deno, Node.js, and Bun. It wraps DuckDB (an in-process analytical database) behind a Pandas/Tidyverse-inspired chainable API, and adds optional AI features like row-by-row LLM processing, embeddings, and RAG. It was created by a computational journalist at CBC News who wanted to do data analysis and visualization entirely in JavaScript instead of switching between Python, R, and JS.

Why it matters: JavaScript has historically lagged behind Python and R for serious data work (pandas, tidyverse, GeoPandas). SDA closes that gap by:

• Performance via DuckDB – Benchmarks in the README (22M temperature records, 335K geospatial points) show SDA runs close to raw DuckDB speed while beating pandas/tidyverse/GeoPandas, with lower memory.
• One language for the whole pipeline – Analysis, geospatial joins, charts, maps, and AI enrichment all happen in TypeScript without switching ecosystems.
• Lazy loading and query fusion – Core transformations queue up and get fused into single SQL statements, reducing round trips. AI/Google/charting dependencies only load when those specific methods are called, so core-only users don't pay the startup cost.
• Built-in AI features (in the extended package): aiRowByRow() sends column values to an LLM and stores structured responses (e.g., extracting country + continent from a city name), aiEmbeddings() converts text to vectors, and it supports clustering (HDBSCAN), hybrid search, RAG-style question answering, and natural-language queries. Google Gemini, Vertex AI, or local Ollama models are configurable via .env.
• Public data sources – Direct loaders for Statistics Canada tables and OpenStreetMap (local files or Overpass queries), with local caching for reproducibility.

How it works under the hood: SDA executes SQL in DuckDB via duckdb-node-neo, and uses DuckDB extensions (spatial for geospatial ops, plus fuzzy matching, full-text search, vector indexes) loaded lazily. The API is synchronous and chainable for transformations; you only await the final observer method (like .log(), .writeData(), or getting results).

Example (from README): Load a CSV of Canadian weather stations, rename columns, drop missing values, compute mean temperature per station, sort, print, and export:

const sdb = new SimpleDB();

const data = await sdb
  .newTable("temperatures")
  .loadData(".../dailyTemperatures.csv")
  .renameColumns({ t: "temperature", id: "station" })
  .removeMissing({ columns: "temperature" })
  .summarize({
    columns: "temperature", by: "station", stats: "mean", decimals: 2,
  })
  .sort({ mean: "desc" })
  .log();

await data.writeData("output/averageTemperatures.csv");
await sdb.close();

Who it's for: Data journalists, analysts, and developers who prefer JavaScript/TypeScript and need to handle large datasets, do geospatial joins, produce charts/maps, or enrich data with LLMs — without learning a second data language.

Caveats from the README: This is a real, substantial library, but it is not a full dataframe replacement for every Python use case; the AI features require API keys (Gemini/Vertex/Ollama) and the README notes that JSR documentation currently omits inherited methods (a known upstream limitation). Installation is via JSR or npm for Node/Bun/Deno, and there's a companion scaffolding tool (setup-data-project) to start new projects.