l: A New High-Performance Runtime for k, q, and qSQL

l: A New High-Performance Runtime for k, q, and qSQL

l is a new runtime designed for k4, q, and qSQL, the array programming languages and database tools widely used in high-frequency trading and financial services on Wall Street. By implementing transparent SIMD, automatic parallelism, and native operations on compressed vectors, l aims to provide a faster execution engine without requiring changes to existing k/q code.

Native Execution on Compressed Vectors

Performance in l is driven by the use of compressed vectors as the primary unit of execution. Rather than rebuilding a full array in memory before performing a calculation, l's primitives run directly on compressed structures. This approach reduces the amount of data moved between memory and the CPU, addressing the common hardware bottleneck where memory bandwidth is exhausted before compute capacity is reached.

Key characteristics of this compression model include:

  • Unified Format: Data maintains a single format across disk, memory, and Inter-Process Communication (IPC).
  • Reduced Memory Footprint: Vectors are automatically compressed, optimizing cache capacity.
  • Direct Computation: Primitives operate on compressed data at rest and in transit, eliminating the need for a separate decode pass.

Transparent Hardware Acceleration

L automates the selection of the most efficient execution path based on the hardware target and the size of the data being processed. Developers write standard k/q code (e.g., sum x), and the runtime determines the optimal execution strategy without requiring manual annotations.

Execution Paths

  • SIMD Vectorization: l utilizes NEON vectorization on Apple Silicon and AVX-512 support on x86 architectures.
  • Automatic Threading: The runtime intelligently applies threading based on vector length. Small vectors are processed via SIMD, while larger vectors utilize a combination of SIMD and multi-threaded fan-out.
  • Hardware Offloading: The runtime supports offloading computations to GPUs and NPUs.

Compatibility and Database Support

L provides full compatibility with k4 syntax, q, and qSQL, allowing users to run existing idioms and code without rewrites. This includes native support for:

  • Columnar tables
  • Dictionaries
  • Partitions
  • Splays

As a fully qSQL-compatible database, l supports all standard joins and partition logic, aiming to maintain the "legendary speed" associated with the kdb+ ecosystem while offering an independent implementation of the q interpreter.

Community Reception and Analysis

Industry observers and developers have noted that l enters a specialized design space of array languages (including APL, BQN, and J) where proprietary licensing is common.

Key Insights from the Community

  • Market Position: Some users view l as a potential alternative for those who find the commercial licensing of kdb+ (by Kx Systems) prohibitively expensive.
  • Production Goals: Unlike some other k-family implementations, l explicitly targets full production database compatibility and language compatibility.
  • Critiques: Discussion among developers has highlighted that the project is currently closed-source, which some consider a barrier to adoption for a language runtime. There has also been debate regarding the aesthetic and "vibe-coded" nature of the project's landing page.

"q is the programming language underpinning kdb+ by Kx Systems... L is an independent implementation of a q interpreter. To those who reluctantly commercially license kdb+ - this will be a welcome alternative."

"Unlike klong/ngn/bqn et al... this has the goal of full production database compatibility (and full language compatibility)."

Sources