mlflow/mlflow

MLflow – Open‑source AI‑engineering platform

What it is – MLflow is a mature, open‑source platform that helps teams build, track, evaluate, and deploy machine‑learning and generative‑AI (LLM/agent) applications. It started as a tool for classic ML experiment tracking and model registry, and has recently added a full GenAI stack (tracing, evaluation, prompt management, optimization, and an AI‑gateway). The project is maintained by the MLflow community and backed by Databricks.

Core capabilities

Area	What you get
Experiment tracking	Log parameters, metrics, artifacts, and source code for every run; view them in a web UI or query via the Python/Java SDK.
Model registry	Versioned model store with stage transitions (Staging → Production), lineage, and access‑control.
Deployment	One‑click serving to Docker, Kubernetes, Azure ML, SageMaker, and other runtimes.
LLM/Agent observability	Automatic OpenTelemetry‑based tracing for >60 LLM frameworks (LangChain, LangGraph, OpenAI Agent, etc.). Captures prompts, responses, token usage, and custom metadata.
Evaluation & monitoring	Built‑in suite of >50 quality metrics and LLM judges; tracks regressions over time.
Prompt registry & optimization	Versioned prompt store with lineage; optional automated prompt‑tuning using state‑of‑the‑art algorithms.
AI Gateway	Unified OpenAI‑compatible proxy that routes requests to any provider (OpenAI, Anthropic, Gemini, Bedrock, LiteLLM, …), enforces rate limits, cost caps, and guardrails.
Multi‑language support	SDKs for Python, TypeScript/JavaScript, Java, and a REST API, so you can instrument virtually any stack.

Typical workflow

1. Start a server – uvx mlflow server runs the tracking UI locally or in the cloud.
2. Instrument code – mlflow.openai.autolog() (or the equivalent for other providers) automatically logs prompts, responses, and metrics.
3. Run experiments – Execute your LLM or ML code; all runs appear in the UI with traces, evaluation scores, and cost information.
4. Iterate – Use the prompt registry to version new prompts, run systematic evaluations, and let the built‑in optimizer suggest improvements.
5. Deploy – Promote a vetted model or agent to the Model Registry and serve it via the AI Gateway or a containerized endpoint.

Ecosystem & integrations

• Agent frameworks: LangChain, LangGraph, OpenAI Agent, DSPy, CrewAI, LlamaIndex, AutoGen, Semantic Kernel, and many more (both Python and TypeScript/Java).
• LLM providers: OpenAI, Anthropic, Google Gemini, Amazon Bedrock, Databricks, LiteLLM, Mistral, etc.
• Observability: Native OpenTelemetry support; can forward traces to Jaeger, Zipkin, or any OTLP collector.
• Deployment targets: Docker, Kubernetes, Azure ML, AWS SageMaker, Databricks, and generic REST endpoints.

Getting started

# 1️⃣ Install the Python SDK
pip install mlflow

# 2️⃣ Spin up a local server (runs UI on http://localhost:5000)
uvx mlflow server

# 3️⃣ Enable automatic LLM logging
python - <<'PY'
import mlflow
mlflow.set_tracking_uri('http://localhost:5000')
mlflow.openai.autolog()

from openai import OpenAI
client = OpenAI()
client.responses.create(model='gpt-4o-mini', input='Hello!')
PY

Visit http://localhost:5000 to explore traces, metrics, and prompt versions.

Who should use it?

• Data‑science teams that need reproducible experiment tracking and model versioning.
• Engineering groups building production LLM applications or autonomous agents who want built‑in observability, cost‑control, and safety guardrails.
• Organizations looking for a single platform that covers both classic ML pipelines and modern GenAI workflows.

License – Apache 2.0 (per the LICENSE.txt file).

---

All details above are taken directly from the repository’s README; no additional features have been inferred.