NeuralArmada | Transcribe Hours of Audio Without a Timeout: Whisper on SageMaker AI Asynchronous Inference

Speech-to-text is the workload that most often breaks a team's first SageMaker AI deployment. The model works fine in a notebook, then the endpoint starts returning ModelError and 504s as soon as somebody uploads a ninety-minute earnings call. Nothing is wrong with the model. The problem is the hosting option.

This tutorial deploys Whisper for long-form transcription on an asynchronous inference endpoint: payloads arrive by S3 URI, results are written back to S3, callers get an SNS notification, and the endpoint scales to zero instances when the queue is empty. Code below assumes sagemaker>=2.200, a SageMaker execution role, and a Studio space or any machine with AWS credentials.

Why not a real-time endpoint

Real-time SageMaker AI endpoints are bounded in three ways that matter for audio:

Invocation timeout. InvokeEndpoint is capped at 60 seconds of model response time. Transcribing an hour of audio does not fit, even on a GPU.
Payload size. The request body limit is 6 MB. A single 45-minute WAV file is larger than that before you start thinking about batches.
Cost shape. Transcription traffic is bursty — nothing for six hours, then 400 files after a conference call ends. A provisioned real-time endpoint bills through the quiet hours.

Asynchronous inference removes all three. You pass an S3 object reference instead of bytes, so payloads go up to 1 GB; the processing budget is up to one hour per request; and the endpoint's autoscaling minimum can legitimately be 0, because the internal queue holds requests while instances come up.

Batch Transform is the other candidate, and it is the right answer when you have 10,000 files sitting in a bucket and no latency expectation at all. Choose async when requests arrive one at a time from an application and someone is waiting for a callback. (We compared all four hosting modes in our inference-types guide; this post is the implementation of one of them.)

Step 1: an inference handler that streams from disk

Use a custom handler rather than a stock container. faster-whisper (CTranslate2 backend) is roughly 4x faster than the reference PyTorch implementation at the same accuracy and, importantly, accepts a file path so you never hold the whole decoded waveform in memory.

code/
  inference.py
  requirements.txt

code/requirements.txt:

faster-whisper==1.1.1
ctranslate2>=4.5

code/inference.py:

import json, os, tempfile
from faster_whisper import WhisperModel

MODEL_SIZE = os.environ.get("WHISPER_MODEL_SIZE", "large-v3")


def model_fn(model_dir):
    # compute_type=float16 on GPU; use int8 on CPU instances.
    return WhisperModel(MODEL_SIZE, device="cuda", compute_type="float16",
                        download_root=os.path.join(model_dir, "hf"))


def input_fn(request_body, content_type):
    if content_type in ("audio/wav", "audio/x-wav", "audio/mpeg", "application/octet-stream"):
        suffix = ".mp3" if content_type == "audio/mpeg" else ".wav"
        fd, path = tempfile.mkstemp(suffix=suffix)
        with os.fdopen(fd, "wb") as f:
            f.write(request_body)
        return {"path": path, "opts": {}}
    if content_type == "application/json":
        payload = json.loads(request_body)
        return {"path": payload["audio_path"], "opts": payload.get("options", {})}
    raise ValueError(f"Unsupported content type: {content_type}")


def predict_fn(data, model):
    segments, info = model.transcribe(
        data["path"],
        beam_size=data["opts"].get("beam_size", 5),
        vad_filter=True,                      # drop silence; big win on call recordings
        word_timestamps=data["opts"].get("word_timestamps", False),
        language=data["opts"].get("language"),  # None = autodetect
    )
    out = [
        {"start": round(s.start, 3), "end": round(s.end, 3), "text": s.text.strip()}
        for s in segments
    ]
    try:
        os.remove(data["path"])
    except OSError:
        pass
    return {
        "language": info.language,
        "language_probability": round(info.language_probability, 3),
        "duration": round(info.duration, 2),
        "segments": out,
        "text": " ".join(s["text"] for s in out),
    }


def output_fn(prediction, accept):
    return json.dumps(prediction), "application/json"

Two details that save a debugging afternoon:

model.transcribe is a generator. If you return segments without materialising the list, the handler returns instantly and the response serialises to nothing. Always consume it inside predict_fn.
SageMaker AI downloads the async payload and hands your input_fn the raw bytes. You do not call S3 yourself for the input, only for anything extra (diarisation reference files, glossaries) you want to fetch.

If you want fully air-gapped, VPC-only inference, bake the model weights into model.tar.gz instead of letting faster-whisper download them at container start — a cold start that has to pull 3 GB from the internet is both slow and impossible inside a private subnet.

Step 2: deploy the async endpoint

import sagemaker
from sagemaker.pytorch import PyTorchModel
from sagemaker.async_inference import AsyncInferenceConfig

sess = sagemaker.Session()
role = sagemaker.get_execution_role()
bucket = sess.default_bucket()

model = PyTorchModel(
    model_data=f"s3://{bucket}/whisper/model.tar.gz",   # weights + empty code dir placeholder
    role=role,
    entry_point="inference.py",
    source_dir="code",
    framework_version="2.6",
    py_version="py312",
    env={"WHISPER_MODEL_SIZE": "large-v3",
         "TS_DEFAULT_RESPONSE_TIMEOUT": "3600"},
)

async_config = AsyncInferenceConfig(
    output_path=f"s3://{bucket}/whisper/output",
    failure_path=f"s3://{bucket}/whisper/failure",
    max_concurrent_invocations_per_instance=2,
    notification_config={
        "SuccessTopic": "arn:aws:sns:us-east-1:111122223333:whisper-success",
        "ErrorTopic": "arn:aws:sns:us-east-1:111122223333:whisper-error",
    },
)

predictor = model.deploy(
    initial_instance_count=1,
    instance_type="ml.g5.xlarge",
    endpoint_name="whisper-async",
    async_inference_config=async_config,
)

Set failure_path explicitly. Without it, failed invocations produce an SNS error message pointing at nothing useful, and you end up re-running jobs to find out what broke.

max_concurrent_invocations_per_instance is the knob that decides whether your GPU is saturated or thrashing. Whisper large-v3 in float16 wants roughly 5 GB of VRAM per concurrent stream; on a 24 GB ml.g5.xlarge, 2 is safe, 4 is optimistic, and 6 will OOM on long files with word timestamps enabled.

Step 3: invoke and collect results

import boto3, json

sm = boto3.client("sagemaker-runtime")

resp = sm.invoke_endpoint_async(
    EndpointName="whisper-async",
    InputLocation=f"s3://{bucket}/whisper/input/board-meeting-2026-01-14.wav",
    ContentType="audio/wav",
    InvocationTimeoutSeconds=3600,
    Accept="application/json",
)

print(resp["OutputLocation"])   # where the transcript will appear
print(resp["FailureLocation"])  # where the stack trace will appear

The call returns immediately with an InferenceId and the two S3 locations. Production consumers should subscribe a Lambda to the success topic rather than polling:

def handler(event, context):
    for record in event["Records"]:
        msg = json.loads(record["Sns"]["Message"])
        if msg["invocationStatus"] != "Completed":
            # msg["failureReason"] plus the object at failureLocation
            raise RuntimeError(msg.get("failureReason", "unknown"))
        transcript_uri = msg["responseParameters"]["outputLocation"]
        # index it, diff it against the previous version, hand it to an LLM, ...

To pass options rather than raw audio, send a small JSON object instead — put the JSON in S3, use ContentType="application/json", and have input_fn read audio_path. In that shape your handler is responsible for the S3 GetObject, so grant the execution role read access to the audio prefix.

Step 4: scale to zero (the whole point)

Async endpoints are the one hosting mode where a minimum capacity of zero has always been allowed, because queued requests survive the cold start.

import boto3

aas = boto3.client("application-autoscaling")
resource_id = "endpoint/whisper-async/variant/AllTraffic"

aas.register_scalable_target(
    ServiceNamespace="sagemaker",
    ResourceId=resource_id,
    ScalableDimension="sagemaker:variant:DesiredInstanceCount",
    MinCapacity=0,
    MaxCapacity=4,
)

aas.put_scaling_policy(
    PolicyName="whisper-backlog",
    ServiceNamespace="sagemaker",
    ResourceId=resource_id,
    ScalableDimension="sagemaker:variant:DesiredInstanceCount",
    PolicyType="TargetTrackingScaling",
    TargetTrackingScalingPolicyConfiguration={
        "TargetValue": 2.0,
        "CustomizedMetricSpecification": {
            "MetricName": "ApproximateBacklogSizePerInstance",
            "Namespace": "AWS/SageMaker",
            "Dimensions": [{"Name": "EndpointName", "Value": "whisper-async"}],
            "Statistic": "Average",
        },
        "ScaleInCooldown": 600,
        "ScaleOutCooldown": 300,
    },
)

Target-tracking on ApproximateBacklogSizePerInstance will not lift you off zero on its own, because with zero instances the per-instance metric is undefined. Add a step-scaling alarm on HasBacklogWithoutCapacity >= 1 (period 60s, one datapoint) that sets desired capacity to 1. That pair — step out of zero, target-track above it — is the standard configuration and it is the part people forget.

Keep ScaleInCooldown generous. A GPU container that has to reload 3 GB of weights takes minutes to come back, and flapping between 0 and 1 every ten minutes costs more in latency than it saves in instance-seconds.

What to watch in CloudWatch

Metric	Why it matters
`ApproximateBacklogSize`	Total queue depth. A backlog that never drains means your concurrency setting or instance count is wrong.
`HasBacklogWithoutCapacity`	Requests waiting with zero instances. Drives the scale-from-zero alarm.
`TimeInBacklog`	End-to-end wait users actually feel. Alarm on p99, not average.
`ModelLatency`	Per-file GPU time. Divide by audio duration to get a real-time factor; large-v3 on g5 should land near 0.05–0.15x with VAD on.
`Invocation4XXErrors`	Usually a bad `ContentType` or an S3 object the role cannot read.

Failure modes we see in client accounts

Silent truncation. A handler that returns segments lazily, or a container response timeout left at the 60-second default, produces a partial transcript with no error. Set TS_DEFAULT_RESPONSE_TIMEOUT and always compare info.duration against the source file length.
Autodetect drifting on short clips. Whisper's language detection reads only the first 30 seconds. For multilingual call centres, pin language per tenant rather than letting a hold-music intro decide.
Hallucinated text over silence. Whisper fills long silences with invented sentences. vad_filter=True removes most of it; also drop segments whose no_speech_prob is high if you enable that output.
One giant file per invocation. The 1 GB payload and one-hour processing caps are real. For multi-hour recordings, chunk on silence boundaries upstream and fan out several invocations, then stitch with an offset per chunk.
Unbounded S3 output. Transcripts accumulate forever. Put a lifecycle rule on the output and failure prefixes on day one.

Where this fits

An async transcription endpoint is rarely the deliverable on its own. In most engagements it is stage one of a pipeline: transcribe, then redact PII, then summarise or extract structure with an LLM behind a guardrail, then index for search. Each stage is its own endpoint or job with its own scaling profile, wired together with Step Functions or SageMaker Pipelines so a failure at stage three does not re-run the GPU work at stage one.

If you are standing up speech workloads on SageMaker AI — or trying to work out why an existing endpoint times out on long audio — get in touch and we will walk your architecture with you.