microsoft/TypeAgent

TypeAgent – A Sample Personal AI Agent

What it is

• A Microsoft‑maintained open‑source sample that shows how to build a single personal assistant that can be spoken to in natural language.
• The code demonstrates how to combine large language models (LLMs) with traditional software components (actions, memory, planning) using a structured‑prompting approach.

Core ideas

Principle	How it is applied
Distill models into logical structures	Actions are expressed as typed schemas (via TypeChat); memory is stored as ontologies; planning uses a “tree‑of‑thought” search where LLMs, quality models and humans collaborate.
Control information density with structure	Action contracts are discrete and densely described; memory is kept in tight semantic structures that fit within the LLM’s attention window; each search node focuses on a sub‑problem.
Enable collaboration through structure	Humans disambiguate ambiguous requests; simple parsers extract logical forms; multiple models (LLM, advantage/quality models) work together to expand the best‑first‑search tree.

Key components

• TypeAgent Shell – An Electron‑based UI that lets a user converse (voice‑enabled) with multiple registered agents through a single interface.
• Dispatcher – Routes a user’s natural‑language request to the agent whose typed contract best matches the intent, using structured prompting.
• KnowPro – Implements the Structured RAG memory layer: conversation history is indexed as logical entities, enabling precise queries like “what books did we discuss?”
• Cache – Uses LLM‑driven structured prompts to memoize action translations, reducing repeated calls to the model and lowering latency.
• Agent SDK – Defines the TypeScript interfaces that agents must implement to plug into the dispatcher and shell.

Sample agents included (all built on the same architecture)

• Music player, chat, web browser, VS Code, list manager, calendar, email, desktop automation, image editor, markdown editor, photo montage, etc.

How to try it

1. Follow the platform‑specific setup docs (Windows, WSL2, Linux, macOS) to install Node, Azure OpenAI keys, etc.
2. Run the TypeAgent Shell (ts/packages/shell) – you get a conversational window with voice support.
3. Interact with built‑in agents or add your own by implementing the SDK and registering the agent with the dispatcher (see the Echo tutorial).

Design highlights

• Structured RAG replaces classic vector‑only retrieval with a schema‑driven index, giving higher recall for conversational facts.
• Action‑memory coupling – Performing an action (e.g., “add a pickle‑ball game to my calendar”) automatically creates a memory entry that can be referenced by later actions.
• AMP architecture (Actions‑Memories‑Plans) ties the three flows together, allowing seamless information exchange.

Current status & limits

• Early‑stage sample code, not a production‑ready framework.
• Tested only with Azure OpenAI services on developer machines; English‑only evaluation.
• Relies on well‑crafted JSON schemas; if a schema does not capture user intent, the LLM may produce invalid responses.
• No built‑in telemetry; developers must supply their own API keys.

Who might find it useful

• Researchers or engineers exploring how to structure LLM‑driven agents.
• Developers who want a reference implementation for building personal assistants, action dispatchers, or memory‑augmented chatbots.
• Teams interested in the “structured prompting” pattern for reducing LLM cost and improving reliability.

Next steps planned by Microsoft

• Publish reusable libraries for the memory and dispatch layers.
• Provide installable packages and dynamic discovery of new agents.

---

All information above is taken directly from the repository’s README; no additional features are inferred.