How to Deploy and Fine‑Tune DeepSeek R1 Models on AWS

Hugging Face provides a step‑by‑step guide for deploying and fine‑tuning DeepSeek R1 models on AWS services.

What is DeepSeek‑R1?

DeepSeek‑R1 is an open‑source reasoning model released by DeepSeek AI, accompanied by six distilled versions based on Llama and Qwen architectures. The release followed OpenAI’s o1 model, which demonstrated that more compute at inference improves reasoning performance on tasks such as math, coding, and logic.

Deploying DeepSeek R1 Models on AWS

You can deploy DeepSeek R1 and its distilled models on AWS via Hugging Face Inference Endpoints, Amazon Bedrock Marketplace, Amazon SageMaker AI (GPU or Neuron), or EC2 Neuron with the Hugging Face Neuron Deep Learning AMI.

Deploy via Hugging Face Inference Endpoints

Hugging Face Inference Endpoints let you launch a managed endpoint for any DeepSeek R1 distilled model or the Unsloth GGUF quantized version, with autoscaling and scale‑to‑zero, at about $8.30 per hour for the base model. To deploy, open the model page on Hugging Face, click Deploy, then HF Inference Endpoints. The page prepopulates an optimized container and recommended hardware. After creation, you can send queries to the endpoint. The team is working on enabling DeepSeek models deployment on Inferentia instances.

Deploy via Amazon Bedrock Marketplace

The Bedrock Marketplace offers a one‑click way to launch a SageMaker‑hosted endpoint for DeepSeek distilled models, which deploys an endpoint in Amazon SageMaker AI under the hood. A short video demonstrates navigating the AWS console to complete the deployment.

Deploy via Amazon SageMaker AI with Hugging Face LLM DLCs (GPU)

For GPU‑based inference you can use the Hugging Face LLM DLCs on SageMaker Jumpstart or the Python SDK, with specific instance types recommended for each distilled variant.

Model	Instance Type	# of GPUs per replica
deepseek-ai/DeepSeek-R1-Distill-Llama-70B	ml.g6.48xlarge	8
deepseek-ai/DeepSeek-R1-Distill-Qwen-32B	ml.g6.12xlarge	4
deepseek-ai/DeepSeek-R1-Distill-Qwen-14B	ml.g6.12xlarge	4
deepseek-ai/DeepSeek-R1-Distill-Llama-8B	ml.g6.2xlarge	1
deepseek-ai/DeepSeek-R1-Distill-Qwen-7B	ml.g6.2xlarge	1
deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B	ml.g6.2xlarge	1

Before deploying, ensure a SageMaker Domain is configured, sufficient quota exists for the chosen instance type, and a JupyterLab space is available. For the 70B Llama distilled model, raise the default quota for ml.g6.48xlarge to 1.

Install the latest SageMaker SDK:

!pip install sagemaker --upgrade

Instantiate a session and retrieve the execution role:

import json
import sagemaker
import boto3
from sagemaker.huggingface import HuggingFaceModel, get_huggingface_llm_image_uri

try:
    role = sagemaker.get_execution_role()
except ValueError:
    iam = boto3.client("iam
    role = iam.get_role(RoleName="sagemaker_execution_role\["Role\]["Arn\)

Create the Hugging Face Model object:

model_id = "deepseek-ai/DeepSeek-R1-Distill-Llama-70B"
model_name = model_id.split("\/\["[-1].lower()

hub = {
    "HF_MODEL_ID": model_id,
    "SM_NUM_GPUS": json.dumps(8)
}

huggingface_model = HuggingFaceModel(
    image_uri=get_huggingface_llm_image_uri("huggingface", version="3.0.1\[
    env=hub,
    role=role,
)

Deploy to a SageMaker endpoint and test:

endpoint_name = f"{model_name}-ep"

predictor = huggingface_model.deploy(
    endpoint_name=endpoint_name,
    initial_instance_count=1,
    instance_type="ml.g6.48xlarge",
    container_startup_health_check_timeout=2400,
)

# send request
predictor.predict({"inputs": "What is the meaning of life?"})

After testing, delete the endpoint:

predictor.delete_model()
predictor.delete_endpoint()

The TGI v3 container automatically selects performant parameters for the hardware.

Deploy via Amazon SageMaker AI with Hugging Face LLM DLCs (Neuron)

For Neuron‑based inference (Trainium/Inferentia) you use the Hugging Face Neuron DLC, setting environment variables like HF_NUM_CORES and MAX_BATCH_SIZE, and deploy on ml.inf2.48xlarge.

Prerequisites are the same as for GPU deployment: a configured SageMaker Domain, sufficient quota for ml.inf2.48xlarge, and a JupyterLab space.

Instantiate a session and retrieve the execution role (same code as above).

Create the Hugging Face Model object with Neuron‑specific settings:

image_uri = get_huggingface_llm_image_uri("huggingface-neuronx", version="0.0.25\[
model_id = "deepseek-ai/DeepSeek-R1-Distill-Llama-70B"
model_name = model_id.split("\/\["[-1].lower()

hub = {
    "HF_MODEL_ID": model_id,
    "HF_NUM_CORES": "24",
    "HF_AUTO_CAST_TYPE": "bf16",
    "MAX_BATCH_SIZE": "4",
    "MAX_INPUT_TOKENS": "3686",
    "MAX_TOTAL_TOKENS": "4096",
}

huggingface_model = HuggingFaceModel(
    image_uri=image_uri,
    env=hub,
    role=role,
)

Deploy to a SageMaker endpoint and test:

endpoint_name = f"{model_name}-ep"

predictor = huggingface_model.deploy(
    endpoint_name=endpoint_name,
    initial_instance_count=1,
    instance_type="ml.inf2.48xlarge",
    container_startup_health_check_timeout=3600,
    volume_size=512,
)

# send request
predictor.predict(
    {
        "inputs": "What is is the capital of France?",
        "parameters": {
            "do_sample": True,
            "max_new_tokens": 128,
            "temperature": 0.7,
            "top_k": 50,
            "top_p": 0.95,
        }
    }
)

After testing, delete the endpoint as shown earlier.

Deploy on EC2 Neuron with Hugging Face Neuron Deep Learning AMI

You can run DeepSeek R1 distilled models on an EC2 inf2.48xlarge instance using the Hugging Face Neuron Deep Learning AMI and a Docker command that launches a TGI endpoint.

First, subscribe to the Hugging Face Neuron Deep Learning AMI on AWS Marketplace, launch an inf2.48xlarge instance with that AMI, and connect via SSH.

Then start the endpoint:

docker run -p 8080:80 \
    -v $(pwd)/data:/data \
    --device=/dev/neuron0 \
    --device=/dev/neuron1 \
    --device=/dev/neuron2 \
    --device=/dev/neuron3 \
    --device=/dev/neuron4 \
    --device=/dev/neuron5 \
    --device=/dev/neuron6 \
    --device=/dev/neuron7 \
    --device=/dev/neuron8 \
    --device=/dev/neuron9 \
    --device=/dev/neuron10 \
    --device=/dev/neuron11 \
    -e HF_BATCH_SIZE=4 \
    -e HF_SEQUENCE_LENGTH=4096 \
    -e HF_AUTO_CAST_TYPE="bf16" \
    -e HF_NUM_CORES=24 \
    ghcr.io/huggingface/neuronx-tgi:latest \
    --model-id deepseek-ai/DeepSeek-R1-Distill-Llama-70B \
    --max-batch-size 4 \
    --max-total-tokens 4096

Wait a few minutes for the compiled model to download and the TGI endpoint to start.

Test the endpoint:

curl localhost:8080/generate \
    -X POST \
    -d '{"inputs":"Why is the sky dark at night?"}' \
    -H 'Content-Type: application/json'

Pause the EC2 instance when finished testing. The team is working on enabling DeepSeek R1 deployment on Trainium & Inferentia with this AMI.

Fine‑tuning DeepSeek R1 Models on AWS

Fine‑tuning DeepSeek R1 models on AWS is currently being enabled; the guide points to upcoming support for Hugging Face Training DLCs on SageMaker and the Neuron Deep Learning AMI.

Fine‑tune on Amazon SageMaker AI with Hugging Face Training DLCs

The team is working to enable full fine‑tuning of DeepSeek R1 models with the Hugging Face Training DLCs on SageMaker; details will follow.

Fine‑tune on EC2 Neuron with Hugging Face Neuron Deep Learning AMI

Similarly, fine‑tuning on Neuron instances via the Hugging Face Neuron Deep Learning AMI is under development.