replicate/cog

What it solves

Cog simplifies the process of packaging machine learning models into production-ready Docker containers. It removes the complexity of writing manual Dockerfiles, managing compatible CUDA/cuDNN/PyTorch/Tensorflow versions, and building custom HTTP inference servers, which often requires researchers to rely on engineers for deployment.

How it works

Users define their model's environment in a cog.yaml file (specifying GPU needs, system packages, and Python versions) and the model's execution logic in a run.py file using a standard Python class. Cog then automatically generates a Docker image following industry best practices and dynamically creates a high-performance RESTful HTTP API using a Rust/Axum server based on the defined Python input and output types.

Who it’s for

Machine learning researchers and developers who want to deploy their models to their own infrastructure or to Replicate without needing deep expertise in Docker or infrastructure engineering.

Highlights

• Automated Containerization: Generates Docker images with Nvidia base images and efficient caching without requiring a manual Dockerfile.
• Dependency Management: Automatically handles compatible combinations of CUDA, cuDNN, PyTorch, and TensorFlow.
• Automatic API Generation: Creates an OpenAPI schema and a high-performance HTTP inference server from Python type definitions.
• Flexible Deployment: Models can be run locally, built as images for any Docker-compatible environment, or deployed to Replicate.