Tabular ML That Still Pays the Bills: AutoMLV2 Baselines and XGBoost Tuning on SageMaker AI

Almost every generative-AI roadmap we are handed has a churn model, a fraud score, or a demand forecast quietly holding up the revenue behind it. Those models are tables, not tokens, and they are still where SageMaker AI earns its keep. This tutorial builds a defensible tabular pipeline end to end: an automated baseline you can produce in an afternoon, a tuned gradient-boosting model that has to beat it, and a deployment shape that does not cost money while it idles.

Everything here uses the SageMaker Python SDK v2 (pip install "sagemaker>=2.200") and boto3 from a Studio space or any machine with a SageMaker execution role.

Step 0: split the data before you look at it

The most expensive tabular bug is not a bad hyperparameter, it is leakage. Two rules cover most of it.

Split on time if the prediction happens in time. Churn, fraud, demand, and conversion are all forecasts. A random split lets the model learn from the future and inflates validation AUC by five to fifteen points, which you then discover in production.

import pandas as pd

df = pd.read_parquet("events.parquet").sort_values("event_ts")
cut_train = df["event_ts"].quantile(0.70)
cut_valid = df["event_ts"].quantile(0.85)

train = df[df["event_ts"] <= cut_train]
valid = df[(df["event_ts"] > cut_train) & (df["event_ts"] <= cut_valid)]
test  = df[df["event_ts"] > cut_valid]

Split on the entity, not the row. If one customer contributes 400 rows, those rows must all land in the same split. GroupShuffleSplit on the customer ID does this; a naive train_test_split does not.

Write the three frames to S3 as CSV with a header (Autopilot expects the target as a named column):

for name, frame in [("train", train), ("valid", valid), ("test", test)]:
    frame.to_csv(f"s3://my-bucket/churn/{name}/{name}.csv", index=False)

Step 1: an AutoMLV2 baseline in one call

Autopilot's second-generation API (AutoMLV2) runs feature preprocessing, model selection, and tuning as a managed job and hands back a leaderboard plus a deployable candidate. Treat it as a baseline generator, not as the final answer: its value is telling you what score is achievable with zero cleverness, so you know whether your hand-built model is actually earning its maintenance cost.

import sagemaker
from sagemaker.automl.automlv2 import AutoMLV2, AutoMLDataChannel, AutoMLTabularConfig

session = sagemaker.Session()
role = sagemaker.get_execution_role()

automl = AutoMLV2(
    problem_config=AutoMLTabularConfig(
        target_attribute_name="churned",
        problem_type="BinaryClassification",
        max_candidates=20,
        max_runtime_per_training_job_in_seconds=1800,
    ),
    base_job_name="churn-baseline",
    role=role,
    sagemaker_session=session,
    output_path="s3://my-bucket/churn/automl/",
    job_objective={"MetricName": "AUC"},
)

automl.fit(
    inputs=[
        AutoMLDataChannel(s3_data_type="S3Prefix",
                          s3_uri="s3://my-bucket/churn/train/",
                          channel_type="training"),
        AutoMLDataChannel(s3_data_type="S3Prefix",
                          s3_uri="s3://my-bucket/churn/valid/",
                          channel_type="validation"),
    ],
    wait=True,
)

best = automl.best_candidate()
print(best["CandidateName"], best["FinalAutoMLJobObjectiveMetric"])

Passing your own validation channel matters. If you let Autopilot split for you it splits randomly, and on time-series-shaped data your baseline becomes optimistic for exactly the reason described above.

Two things to collect before moving on:

• The objective metric of the best candidate. That is the number to beat.
• The candidate definition notebook and data exploration notebook that Autopilot writes to S3. The exploration notebook flags high-cardinality columns, missing-value rates, and target imbalance; it is free EDA you would otherwise pay an engineer a week to produce.

Cost control: max_candidates and max_runtime_per_training_job_in_seconds are the two knobs that stop a baseline job from quietly running for a day. Twenty candidates on a mid-sized tabular set is usually enough to establish the ceiling.

Step 2: beat the baseline with tuned XGBoost

If the baseline is good enough and nobody needs to explain the pipeline, ship the Autopilot candidate. Usually somebody does need to explain it, and you want a single artifact you control. Use the managed XGBoost container with Automatic Model Tuning (AMT).

Use Hyperband as the strategy for anything iterative. It early-stops unpromising configurations against validation:auc at intermediate rounds, which typically finds a comparable optimum for a third to a half of the compute of Bayesian search.

from sagemaker.estimator import Estimator
from sagemaker.image_uris import retrieve
from sagemaker.inputs import TrainingInput
from sagemaker.tuner import (HyperparameterTuner, ContinuousParameter,
                             IntegerParameter, HyperbandStrategyConfig,
                             StrategyConfig)

image = retrieve(framework="xgboost", region=session.boto_region_name, version="1.7-1")

xgb = Estimator(
    image_uri=image,
    role=role,
    instance_count=1,
    instance_type="ml.m6i.2xlarge",
    output_path="s3://my-bucket/churn/xgb/",
    sagemaker_session=session,
    use_spot_instances=True,
    max_run=3600,
    max_wait=7200,
)

xgb.set_hyperparameters(
    objective="binary:logistic",
    eval_metric="auc",
    num_round=2000,
    early_stopping_rounds=50,
    scale_pos_weight=9,   # ~10% positives
)

ranges = {
    "eta": ContinuousParameter(0.01, 0.3, scaling_type="Logarithmic"),
    "max_depth": IntegerParameter(3, 10),
    "min_child_weight": ContinuousParameter(1, 60, scaling_type="Logarithmic"),
    "subsample": ContinuousParameter(0.5, 1.0),
    "colsample_bytree": ContinuousParameter(0.4, 1.0),
    "lambda": ContinuousParameter(0.1, 100, scaling_type="Logarithmic"),
    "alpha": ContinuousParameter(0.0001, 10, scaling_type="Logarithmic"),
}

tuner = HyperparameterTuner(
    estimator=xgb,
    objective_metric_name="validation:auc",
    objective_type="Maximize",
    hyperparameter_ranges=ranges,
    strategy="Hyperband",
    strategy_config=StrategyConfig(
        hyperband_strategy_config=HyperbandStrategyConfig(min_resource=1, max_resource=2000)
    ),
    max_jobs=60,
    max_parallel_jobs=6,
)

tuner.fit({
    "train": TrainingInput("s3://my-bucket/churn/train/", content_type="text/csv"),
    "validation": TrainingInput("s3://my-bucket/churn/valid/", content_type="text/csv"),
})

For the built-in XGBoost container, CSV input must have the target in the first column and no header row, which is a different layout from the Autopilot channel. Write a second copy of the split rather than trying to make one file serve both; the storage is pennies and the alternative is an hour of silent label-shift debugging.

A few tuning habits that matter more than the search space:

• Keep max_parallel_jobs low relative to max_jobs. Bayesian and Hyperband search both learn from completed trials. Running 30 of 60 jobs in parallel turns the search into a random grid.
• Use managed spot (use_spot_instances=True with max_wait). Tuning trials are short, restartable, and the typical saving is 60-70%.
• Set early_stopping_rounds so individual trials stop on their own even before Hyperband intervenes.
• Log-scale the regularization parameters. Linear sampling over lambda in [0.1, 100] spends almost all its budget on large values.

Step 3: judge the winner on the test split, with the right metric

AUC is a fine tuning objective and a poor business metric. For an imbalanced target, compare candidates on the metric tied to the decision you will actually make.

from sklearn.metrics import average_precision_score, roc_auc_score
import numpy as np

p = predictor.predict(X_test)      # probabilities
print("AUC   ", roc_auc_score(y_test, p))
print("PR-AUC", average_precision_score(y_test, p))

# precision/recall at the capacity you actually have
k = int(0.05 * len(p))             # top 5% get a retention call
idx = np.argsort(-p)[:k]
print("precision@5%", y_test.values[idx].mean())

precision@k is the number the business owner understands: of the customers we can afford to call, how many were really going to churn. Run it on the held-out test split, which neither Autopilot nor AMT ever saw. If the tuned XGBoost beats the Autopilot candidate by less than the noise of that split, ship Autopilot and spend the time elsewhere.

Also check calibration if the score feeds a threshold or an expected-value calculation. Gradient boosting with scale_pos_weight produces confident, badly calibrated probabilities; a CalibratedClassifierCV isotonic wrapper fit on the validation split usually fixes it.

Step 4: deploy where idle costs nothing

Tabular scoring traffic is almost never smooth. Batch scoring nightly, or serverless inference for spiky request traffic, both beat a provisioned endpoint that sits at 3% utilization.

from sagemaker.serverless import ServerlessInferenceConfig

predictor = tuner.best_estimator().deploy(
    serverless_inference_config=ServerlessInferenceConfig(
        memory_size_in_mb=2048,
        max_concurrency=20,
    ),
    endpoint_name="churn-scorer",
)

Serverless charges per millisecond of compute plus data processed, and nothing while idle. The trade is a cold start of roughly a second or two on first invocation after quiet periods, which a nightly or interactive scoring workload can absorb. If a whole population is scored on a schedule, skip endpoints entirely and use a Batch Transform job against the model artifact.

Register the winner in the SageMaker Model Registry with its test metrics attached so the approval step has evidence, and put the whole flow (split, baseline, tune, evaluate, register) into a SageMaker Pipeline once it stops changing daily. That is what turns a notebook result into something a team can retrain in six months without archaeology.

The shortlist

• Split on time and on entity before anything else; leakage beats every hyperparameter.
• Use AutoMLV2 as a baseline generator and take its data-exploration notebook for free.
• Give Autopilot and AMT your own validation channel so their splits match reality.
• Hyperband plus managed spot plus a modest max_parallel_jobs is the cheap, effective tuning default.
• Decide with precision@k or PR-AUC on an untouched test split, not with the tuning objective.
• Serverless or Batch Transform for bursty tabular traffic; provisioned endpoints are for steady load.

If you have a tabular model that has been "nearly ready" for two quarters, or an Autopilot result nobody trusts enough to deploy, NeuralArmada's SageMaker consultants do this work as a fixed-scope engagement. Get in touch with the dataset shape and the decision it needs to support, and we will tell you what the realistic ceiling looks like.