+1 (726) 227-3241
NeuralArmada

Serve Many Fine-Tunes from One GPU: Multi-Adapter LoRA Inference on SageMaker AI

Fine-tuning a small open-weight model per customer, per language, or per document type is now cheap. Serving thirty of those fine-tunes is not — unless you stop treating each one as a separate model. If every tenant gets its own ml.g5.2xlarge endpoint, you are paying for thirty GPUs to do the work of one.

LoRA (Low-Rank Adaptation) fine-tunes produce small adapter weights — often 20–200 MB — that sit on top of an unchanged base model. SageMaker AI can host one copy of the base model on one GPU and load many adapters against it, routing each request to the adapter named in the payload. This tutorial shows how to build that setup end to end: train adapters, lay them out in S3, deploy a multi-adapter endpoint, invoke a specific adapter, add and remove adapters without redeploying, and check whether the economics actually work for your traffic.

It assumes you already know how to deploy a normal real-time endpoint. If you do not, start with our tutorial on deploying a Hugging Face model to a real-time endpoint, then come back.

When multi-adapter serving is the right answer

Use it when all of the following hold:

  • One base model, many behaviours. Every variant is a LoRA/QLoRA fine-tune of the same base checkpoint at the same precision. Different base models cannot share a GPU this way.
  • Per-variant traffic is low and bursty. Ten requests a minute per tenant does not justify a dedicated GPU, but the aggregate keeps one GPU usefully busy.
  • You need per-request isolation, not per-request throughput records. Multi-adapter inference costs a little latency versus a single merged model — usually a few percent, more if adapters thrash in and out of GPU memory.

Do not use it when a tenant needs guaranteed isolation for compliance reasons (noisy-neighbour and blast-radius arguments still apply — a bad deploy takes everyone down), when adapters have wildly different ranks and target modules that break the server's batching, or when one tenant's traffic alone saturates the instance. In that last case, give them their own endpoint and keep the shared endpoint for the long tail.

The alternative for many-model-on-one-endpoint situations is inference components, which pack different models onto shared instances. Adapters are the finer-grained tool: same base weights, many behaviours, no duplicated base model in memory.

Step 1: train adapters that are actually compatible

The single most common failure in this pattern is adapters that were trained against subtly different base weights. Pin the base model revision and record it.

from peft import LoraConfig

peft_config = LoraConfig(
    r=16,
    lora_alpha=32,
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
)

Two rules that save you a week later:

  1. Keep target_modules identical across every adapter you intend to co-host. Servers batch requests across adapters far more efficiently when the adapted layers line up.
  2. Do not merge the adapter into the base weights at the end of training. model.save_pretrained(...) on the PEFT model is what you want — it writes adapter_model.safetensors and adapter_config.json, and nothing else. If your output directory is 15 GB, you merged by accident.

Sanity-check the artifact before it ever reaches an endpoint:

tar tzf adapter.tar.gz
# adapter_config.json
# adapter_model.safetensors

Step 2: lay out adapters in S3

Each adapter gets its own prefix. The base model lives separately.

s3://neuralarmada-models/base/llama-3.1-8b-instruct/
s3://neuralarmada-models/adapters/
    tenant-acme/adapter_config.json
    tenant-acme/adapter_model.safetensors
    tenant-globex/adapter_config.json
    tenant-globex/adapter_model.safetensors
    support-triage-v3/...

Flat, one directory per adapter, no nesting. The adapter name your callers use will be the directory name, so treat these as a stable public interface: version in the name (support-triage-v3), never mutate an existing prefix in place. Mutating a prefix means two callers get two different models under one name, and you will not be able to reproduce a complaint.

Step 3: deploy the base model with adapter support

Deploy the base model on a Large Model Inference (LMI) container with dynamic adapter loading enabled. The OPTION_ENABLE_LORA flag turns on the adapter path; OPTION_MAX_LORAS caps how many adapters stay resident in GPU memory at once.

import sagemaker
from sagemaker.djl_inference import DJLModel

role = sagemaker.get_execution_role()

model = DJLModel(
    model_id="s3://neuralarmada-models/base/llama-3.1-8b-instruct/",
    role=role,
    env={
        "OPTION_ROLLING_BATCH": "vllm",
        "OPTION_ENABLE_LORA": "true",
        "OPTION_MAX_LORAS": "8",
        "OPTION_MAX_LORA_RANK": "16",
        "OPTION_MAX_CPU_LORAS": "32",
        "OPTION_TENSOR_PARALLEL_DEGREE": "1",
    },
)

predictor = model.deploy(
    initial_instance_count=1,
    instance_type="ml.g6.2xlarge",
    endpoint_name="shared-llama-adapters",
    container_startup_health_check_timeout=900,
)

The three numbers that matter:

  • OPTION_MAX_LORAS — adapters kept in GPU memory. Requests for an adapter outside this set trigger a swap from CPU memory, which adds tens of milliseconds. Set it to cover your hot set, not your whole catalogue.
  • OPTION_MAX_CPU_LORAS — adapters cached in host memory. Make this comfortably larger than MAX_LORAS; a swap from CPU is cheap, a fetch from S3 is not.
  • OPTION_MAX_LORA_RANK — must be at least the largest r you trained with. A rank-64 adapter against a rank-16 ceiling fails at load time, not at deploy time, which is a nasty way to find out.

Give the container a generous health-check timeout. An 8B model plus adapter warm-up regularly takes six to ten minutes to become healthy, and the default will fail the deploy while the container is still perfectly fine.

Step 4: register adapters as SageMaker components

SageMaker AI models adapters as inference components of kind adapter, attached to the base model's component. That is what lets you add and remove adapters without touching the endpoint.

import boto3

sm = boto3.client("sagemaker")

sm.create_inference_component(
    InferenceComponentName="adapter-tenant-acme",
    EndpointName="shared-llama-adapters",
    Specification={
        "BaseInferenceComponentName": "base-llama-31-8b",
        "Container": {
            "ArtifactUrl": "s3://neuralarmada-models/adapters/tenant-acme/"
        },
    },
)

Adapter components inherit the base component's compute; they do not request GPUs of their own. Repeat per adapter, or loop over a manifest so that onboarding a tenant is one row of config rather than one deployment.

Step 5: invoke a specific adapter

The caller names the adapter in the request body. Everything else looks like a normal invocation.

import json, boto3

rt = boto3.client("sagemaker-runtime")

resp = rt.invoke_endpoint(
    EndpointName="shared-llama-adapters",
    ContentType="application/json",
    InferenceComponentName="adapter-tenant-acme",
    Body=json.dumps({
        "inputs": "Summarise this ticket in one sentence: ...",
        "parameters": {"max_new_tokens": 128, "temperature": 0.2},
    }),
)

print(json.loads(resp["Body"].read())["generated_text"])

Omit InferenceComponentName and you get the raw base model. That is a useful fallback — and a useful bug. Make the adapter name a required argument in your internal client library so a dropped header never silently returns un-fine-tuned output to a customer.

Step 6: add, update, and remove adapters at runtime

This is the operational payoff. Onboarding a new tenant is a create_inference_component call — no redeploy, no downtime for anyone else:

sm.create_inference_component(
    InferenceComponentName="adapter-tenant-initech",
    EndpointName="shared-llama-adapters",
    Specification={
        "BaseInferenceComponentName": "base-llama-31-8b",
        "Container": {
            "ArtifactUrl": "s3://neuralarmada-models/adapters/tenant-initech/"
        },
    },
)

Shipping a new adapter version means creating a new component (adapter-support-triage-v4), shifting callers, then deleting the old one:

sm.delete_inference_component(InferenceComponentName="adapter-support-triage-v3")

Because adapters are cheap to add and remove, resist the urge to update in place. Create-new-then-cut-over gives you an instant rollback: point traffic back at v3. Pair it with the deployment guardrails you already use for base-model updates — updating the base model on a shared endpoint affects every tenant at once and deserves canary shifting.

Step 7: does the money work?

Take twelve tenants, each averaging 8 requests per minute at ~600 output tokens.

Dedicated endpoints: 12 × ml.g6.2xlarge, on-demand, 730 hours/month. At roughly $1.20/hour that is about $10,500/month, with every GPU idling most of the time.

One shared endpoint: aggregate load is ~96 requests/minute. Two ml.g6.2xlarge instances behind one endpoint handle that with headroom for adapter swaps and traffic spikes: about $1,750/month — an 83% reduction, before you add autoscaling.

The break-even is straightforward: shared serving wins whenever aggregate utilisation across variants would leave individual GPUs below roughly 40% busy. Above that, per-tenant endpoints start to look reasonable again on latency grounds alone.

Two costs the arithmetic hides. First, cold adapters: a request for an adapter that is neither on GPU nor in CPU cache pays an S3 fetch — hundreds of milliseconds. Keep MAX_CPU_LORAS above your active adapter count and this stays rare. Second, shared fate: a base-model upgrade or a bad instance affects every tenant. Budget for a canary endpoint carrying a copy of your two or three highest-value adapters, and validate base upgrades there first. That extra instance is still a rounding error against twelve dedicated GPUs.

Monitoring that tells you which adapter is hurting

Endpoint-level metrics average over all tenants and will hide a single pathological adapter. Two things to add on day one:

  1. Per-component metrics. CloudWatch publishes invocation and latency metrics per inference component. Build one dashboard row per adapter; a tenant whose p99 doubled is invisible in the endpoint aggregate.
  2. A swap-rate signal. Log adapter name and time-to-first-token per request. A rising share of slow first tokens for a given adapter usually means it has fallen out of the resident set — the fix is raising MAX_LORAS, adding an instance, or moving that tenant to their own endpoint.

Then layer normal drift monitoring on top, keyed by adapter, so quality regressions are attributable to a variant rather than to "the endpoint".

The short version

One base model, many LoRA adapters, one GPU pool. Train adapters with matching target modules against a pinned base revision; store one adapter per S3 prefix with the version in the name; deploy the base on an LMI container with LoRA enabled; register each adapter as an inference component; require the adapter name in every call; and monitor per component so a single bad variant does not hide in the average. For fleets of low-traffic fine-tunes, this is usually the single largest inference-cost reduction available on SageMaker AI.

Running a fleet of per-customer fine-tunes and unsure whether to consolidate them? Get in touch — our SageMaker consultants can size the shared endpoint, model the break-even against your traffic, and plan the migration.