Agents.js Release: Give Tools to LLMs using JavaScript

Hugging Face has introduced Agents.js, a JavaScript library designed to provide Large Language Models (LLMs) with tool access. This allows developers to build agents that can execute multi-modal tasks in either browser-based or server-side environments.

Core Functionality and Usage

The HfAgent object serves as the primary entry point for the library. Developers can instantiate an agent using a Hugging Face access token and interact with it using plain-text commands. The agent operates by generating JavaScript code to achieve a goal and then evaluating that code to produce a result.

Execution Workflow

There are two primary ways to execute tasks with Agents.js:

  1. Manual Code Generation and Evaluation: Using generateCode to create the JavaScript logic and evaluateCode to execute it. This is the recommended path for security reasons.
  2. Direct Execution: Using the run method to generate and execute code in a single step.

Data Output

Results returned by the agent are provided as Update objects, which contain a message (informational text) and data (which can be a string or a Blob). The use of Blobs allows the agent to return multi-modal outputs such as images or audio.

Security Considerations

Executing arbitrary code generated by an LLM poses a significant security risk. Hugging Face warns that this should not be performed in untrusted environments. To mitigate this risk, developers are encouraged to use the generateCode and evaluateCode sequence rather than the run method, allowing for a manual review of the generated code before it is executed.

Customization and Extensibility

Agents.js is designed for flexibility, allowing developers to swap the default language model and add specialized tools.

Custom LLMs

By default, the library uses the OpenAssistant/oasst-sft-4-pythia-12b-epoch-3.5 model via the Inference API. However, any asynchronous function that takes a string input and returns a promise for a string can be used as a custom LLM. This enables integration with other providers, such as OpenAI's text-davinci-003.

Custom Tools

Developers can extend the agent's capabilities by defining custom Tool objects. A tool definition requires:

  • A name and description to help the LLM understand when to use the tool.
  • Examples consisting of prompt-code pairs that guide the LLM on how to call the tool.
  • A call function that defines the actual logic to be executed.

These custom tools can be merged with the defaultTools provided by the library during agent initialization.

Multi-modal Input Support

Agents.js supports passing input files to the agent. By providing a FileList to generateCode or evaluateCode, the agent can incorporate these files into its workflow. For example, an agent can be instructed to caption an uploaded image and then convert that caption into speech using a combination of imageToText and evaluateCode.

Sources