modelscope/ms-agent

MS‑Agent – A lightweight, extensible framework for autonomous AI agents

What it is – MS‑Agent (formerly modelscope‑agent) is an open‑source Python library that lets developers build, run, and customise “agents” – LLM‑driven programs that can call tools, keep memory, and explore data on their own. It ships with a Model Calling Protocol (MCP) layer for talking to ModelScope’s inference service, a skill system that follows the Anthropic Agent‑Skills protocol, and a small React + FastAPI web UI for interactive chatting.

Key capabilities (as described in the README)

• Multi‑agent chat with tool‑calling – agents can invoke external utilities (search, code execution, video generation, etc.) via MCP.
• Deep Research / Agentic Insight – a specialised workflow for autonomous literature or data research, scoring highly on the DeepResearch benchmark.
• Code Generation & Video Generation – built‑in projects (Code Genesis, Singularity Cinema) for producing code artifacts and short (≈5 min) videos.
• Agent Skills – a knowledge‑driven skill system that retrieves relevant skills with hybrid FAISS + BM25 search, filters them with an LLM, builds a DAG of dependencies, and runs independent skills in parallel.
• Memory support – optional persistent memory via the mem0 library, enabling agents to remember user preferences across sessions.
• Multimodal inputs – images, video and other modalities can be fed to agents.
• Context compression – automatic pruning and summarisation of long interaction histories to stay within token limits.
• Web UI – a local React router + FastAPI server that streams agent responses via Server‑Sent Events.
• Agent Hub CLI – commands to sync agent workspaces with remote ModelScope repositories, watch for changes, and convert between different agent formats.

Typical workflow (quick‑start example)

1. Install the package from PyPI (basic or with the [research] extras).

   pip install ms-agent            # core
   pip install 'ms-agent[research]'  # deep‑research features
   

2. Export a ModelScope API key (required for the default LLM backend).

   export MODELSCOPE_API_KEY=YOUR_KEY
   

3. Write a short async script that creates an LLMAgent with an MCP configuration and calls run():

   import asyncio
   from ms_agent import LLMAgent
   
   mcp = {
       "mcpServers": {
           "fetch": {
               "type": "streamable_http",
               "url": "https://mcp.api-inference.modelscope.net/YOUR_UUID/mcp"
           }
       }
   }
   
   async def main():
       agent = LLMAgent(mcp_config=mcp)
       await agent.run('Introduce modelscope.cn')
   
   asyncio.run(main())
   

4. Run the script; the agent streams a response back, optionally via the provided Web UI.

Extending the framework

• Add new skills by placing Python modules under ms_agent/skill/ and describing them in the skill manifest; the system will automatically discover, retrieve, and compose them.
• Plug in custom tools (e.g., a financial‑data collector) and expose them through MCP so agents can call them during a conversation.
• Swap the LLM backend – while the default uses ModelScope’s inference API, the library can be pointed at any OpenAI‑compatible endpoint that respects the MCP format.
• Enable memory by installing mem0ai and providing a DashScope embedding key; agents will then store and retrieve context from a persistent vector store.

Who might use it

• Researchers building autonomous agents for literature review, data mining, or financial analysis.
• Developers who need a quick way to prototype multi‑tool LLM assistants (code generation, video creation, web search, etc.).
• Teams that want a self‑hosted UI for interacting with agents while keeping the underlying architecture modular.

Where to learn more

• Paper: https://arxiv.org/abs/2309.00986
• Docs (English): https://ms-agent-en.readthedocs.io
• Docs (Chinese): https://ms-agent.readthedocs.io/zh-cn
• MCP Playground: https://modelscope.cn/mcp/playground
• Discord community: https://discord.gg/qmTFPY9byM

---

TL;DR – MS‑Agent is a real, actively maintained Python framework for building autonomous, tool‑calling LLM agents. It bundles a protocol layer (MCP), a skill‑retrieval/execution engine, memory support, multimodal input, and a lightweight web UI, making it suitable for research prototypes and small‑scale production assistants.