Models and inference

Deploy a model on a cloud machine

Serve a catalog model, or a trained adapter, on a dedicated GPU machine. You choose the machine and a spending limit; GFT provisions it. To serve on your own Kubernetes cluster, prepare a Helm bundle instead.

You need the CLI signed in with the deploy scope.

Choose a machine#

bash

gft deploy machines Qwen/Qwen3.8-27B

Deploy#

bash

gft deploy cloud Qwen/Qwen3.8-27B --rate <tariff-id> --budget 20 \
  --dp 2 --tp 2 --speculation mtp4 --prefix-cache --request-id my-deploy-001

GFT checks GPU capacity, model compatibility, budget and permissions before it allocates the machine. The whole machine is billed. Reuse the same --request-id when you retry, so a retry never creates a second deployment.

Option	Meaning
`--rate`	Tariff ID from `gft deploy machines`.
`--budget`	Maximum reserve in USD.
`--runtime`	Maximum runtime in seconds. Default `3600`.
`--artifact`	Optional trained adapter ID.
`--dp`	Number of independent model replicas.
`--tp`	GPUs per replica: 1, 2, 4 or 8.
`--speculation`	`none`, `mtp4` or `suffix4`.
`--prefix-cache`, `--no-prefix-cache`	Turn prefix caching on or off.

With no advanced options, the server runs one replica across all GPUs of the machine. To choose another layout, set both --dp and --tp. Continuous batching is automatic, up to 64 active requests per replica within your workspace and API limits.

Manage these endpoints on the Models page. In Python, use client.deployments.cloud(..., inference_options={...}).

Prepare a BYOC bundle#

To serve on your own cluster, save an immutable snapshot plan as plan.json:

json

{
  "snapshot_claim": "model-snapshots",
  "snapshot_subpath": "snapshots/my-model-v1",
  "snapshot_digest": "sha256:<64 hex characters>",
  "full_model_snapshot": true
}

Then build the bundle:

bash

gft deploy bundle --config plan.json --output qwen-dp2-tp2.zip \
  --dp 2 --tp 2 --speculation suffix4 --prefix-cache

The bundle is prepared by the local Studio operator API at http://127.0.0.1:8322 (change it with --api-url), using GFT_STUDIO_API_TOKEN if that API requires one. It never sends your cloud API key or starts GPUs. Options override the JSON plan, and an existing output file is never overwritten.

Before you install the Helm bundle, build and configure the included suffix image and finish the platform settings. In Python, use client.deployments.bundle(plan).