webmachinelearning/prompt-api

What it solves

This project proposes a standardized JavaScript API that allows web developers to directly prompt language models built into the browser or operating system. It eliminates the need for developers to rely solely on cloud-based APIs or manually bundle and run large models via WASM and WebGPU, reducing costs, improving privacy through local processing, and enabling offline functionality.

How it works

The API provides a LanguageModel interface to create sessions. Developers can interact with the model using prompt() for single responses or promptStreaming() for real-time output. It supports advanced prompting techniques including:

• System Prompts: Setting the persona or context for the model (e.g., "Pretend to be an eloquent hamster").
• N-shot Prompting: Providing examples of user/assistant interactions to guide model behavior.
• Tool Use: Defining external capabilities (tools) via JSON schemas. The model can request a tool call, which the application executes and feeds back into the session as a tool response.
• Sampling Parameters: Adjusting temperature and topK to control the randomness and creativity of the output.

Who it’s for

Web developers who want to integrate AI capabilities like text classification, summarization, translation, and interactive assistants into their web applications without managing their own model infrastructure.

Highlights

• Local Execution: Leverages on-device models for faster results and lower latency.
• Model Agnostic: Provides a uniform API regardless of the specific model provided by the browser or OS.
• Tool Integration: Supports open-loop tool calling for extending model capabilities with external data or functions.
• Resource Efficient: Reduces user bandwidth and storage needs by using optimized, built-in models.