Fine-Tune Llama 3.x on SageMaker AI with JumpStart

Fine-tuning an 8B-parameter model used to be a research-team project. On SageMaker AI it is a JumpStart estimator, a JSON Lines file, and a single GPU instance, and with LoRA adapters the whole run usually costs less than a team lunch. This tutorial fine-tunes Llama 3.1 8B Instruct on your own instruction data, evaluates it, deploys it, and prices the run.

Prerequisites: SageMaker Python SDK 2.200 or newer, a SageMaker execution role, an S3 bucket, and acceptance of the Llama license (JumpStart prompts for this on first use).

1. Prepare the dataset

JumpStart's text-generation fine-tuning expects a train.jsonl file and, for instruction tuning, a template.json that tells the training script how to render each record. A minimal support-assistant dataset looks like this:

{"instruction": "How do I rotate an API key?", "context": "", "response": "Open Settings > API keys, choose Rotate next to the key, and update any clients within 24 hours; the old key keeps working until then."}
{"instruction": "Summarize the refund policy.", "context": "Refunds are issued within 14 days of purchase for annual plans only.", "response": "Annual plans can be refunded within 14 days of purchase; monthly plans cannot."}

And the template:

{
  "prompt": "Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.\n\n### Instruction:\n{instruction}\n\n### Input:\n{context}\n\n### Response:\n",
  "completion": "{response}"
}

Hold out 5 to 10 percent of records as validation.jsonl before uploading; you will want a loss number that is not on the training data.

import sagemaker

sess = sagemaker.Session()
bucket = sess.default_bucket()
train_uri = sess.upload_data("data/", bucket=bucket, key_prefix="llama-ft/train")

Upload train.jsonl and template.json into the same prefix (the trainer looks for both), and put validation.jsonl in a sibling prefix.

Dataset quality dominates everything else in this tutorial. A few hundred carefully written, deduplicated examples in the exact style you want beats ten thousand scraped ones.

2. Configure the estimator

from sagemaker.jumpstart.estimator import JumpStartEstimator

model_id = "meta-textgeneration-llama-3-1-8b-instruct"

estimator = JumpStartEstimator(
    model_id=model_id,
    instance_type="ml.g5.12xlarge",
    environment={"accept_eula": "true"},
)

# Inspect the defaults before overriding anything.
print(estimator.hyperparameters())

Print the hyperparameters rather than copying them from a blog post (including this one); JumpStart versions change the exact names and defaults. The ones that matter for a LoRA run:

estimator.set_hyperparameters(
    instruction_tuned="True",
    chat_dataset="False",
    epoch="3",
    learning_rate="0.0001",
    per_device_train_batch_size="4",
    max_input_length="2048",
    peft_type="lora",
    lora_r="8",
    lora_alpha="32",
    lora_dropout="0.05",
    int8_quantization="True",   # quantize the frozen base: the QLoRA configuration
    enable_fp16="True",
)

Setting peft_type="lora" with int8_quantization="True" gives you the QLoRA recipe: the base weights are loaded quantized and frozen, and only the low-rank adapter matrices train. That is what lets an 8B model fine-tune on a single ml.g5.12xlarge (four A10G GPUs, 96 GB total) without tensor-parallel gymnastics. If you have a chat-format dataset (lists of role / content messages), set chat_dataset="True" and drop the template file.

3. Launch the training job

estimator.fit(
    {
        "training": train_uri,
        "validation": f"s3://{bucket}/llama-ft/validation",
    },
    job_name="llama-3-1-8b-support-lora",
)

The job pulls the base weights from JumpStart's bucket (no Hugging Face token needed), trains, merges or saves the adapter depending on the JumpStart version, and writes model.tar.gz to S3. Training and validation loss stream to CloudWatch under /aws/sagemaker/TrainingJobs; a validation loss that stops falling after the first epoch is the usual signal to reduce epoch and save the money.

Expect a three-epoch run on a few thousand examples to take 30 to 90 minutes on this instance.

4. Evaluate before you deploy

Loss is a proxy. Before spending on an endpoint, run the fine-tuned model against a held-out task set and compare it with the base model on the same prompts. The cheapest way is a batch transform job or a temporary endpoint driving a small script:

import json

eval_prompts = [json.loads(l) for l in open("data/eval.jsonl")]

def render(rec):
    return template["prompt"].format(**rec)

results = []
for rec in eval_prompts:
    out = predictor.predict({
        "inputs": render(rec),
        "parameters": {"max_new_tokens": 256, "temperature": 0.0},
    })
    results.append({"prompt": rec["instruction"], "expected": rec["response"], "got": out})

Score the results with whatever matches your task: exact match or ROUGE for extractive answers, an LLM-as-judge rubric (run through Amazon Bedrock) for open-ended ones, and always a human read of twenty random samples. If the fine-tuned model does not beat the base model plus a good system prompt, ship the system prompt.

5. Deploy to an endpoint

predictor = estimator.deploy(
    instance_type="ml.g5.2xlarge",
    initial_instance_count=1,
    endpoint_name="llama-3-1-8b-support",
    accept_eula=True,
)

print(predictor.predict({
    "inputs": render({"instruction": "How do I rotate an API key?", "context": ""}),
    "parameters": {"max_new_tokens": 128, "temperature": 0.1},
}))

JumpStart deploys with the Large Model Inference container and sensible defaults. For production traffic, add autoscaling and streaming as described in our real-time endpoint tutorial, and capture request / response data for later evaluation.

Delete the endpoint when you are done experimenting:

predictor.delete_endpoint(delete_endpoint_config=True)
predictor.delete_model()

6. Price out the run

Everything in this tutorial is on-demand SageMaker AI pricing, so the cost is easy to estimate from the pricing page:

• Training: ml.g5.12xlarge hours x the hourly rate. A one-hour LoRA run is a single-digit to low-double-digit dollar figure in most regions. Spot training (use_spot_instances=True with max_wait) typically cuts that by more than half and is safe here because JumpStart checkpoints.
• Evaluation: an hour or two of ml.g5.2xlarge, a few dollars.
• Serving: the endpoint is the recurring cost. One ml.g5.2xlarge running 24/7 is a four-figure monthly number; this is why the evaluation step exists.
• Storage: model.tar.gz for an 8B model in FP16 is about 16 GB in S3, which is pennies.

Log the job's billable seconds from describe_training_job and keep it with the evaluation results, so the next person can see what a rerun will cost.

Going further

• Larger models (Llama 3.3 70B, Qwen, Mistral) follow the same pattern with a bigger instance or a multi-node job; for multi-day runs, SageMaker HyperPod with a flexible training plan removes the capacity gamble.
• Keep training data, template, hyperparameters, and evaluation scores together in the Model Registry so fine-tunes are reproducible.
• If you are deciding between fine-tuning on SageMaker AI and calling a managed model on Bedrock, read our 2026 decision guide.

Want a fine-tuned model in production with the evaluation to back it? Contact NeuralArmada.