Models and inference

Chat with a model and measure speed

Send a prompt to a model from the terminal, and measure its latency and throughput.

You need a key with the inference scope. Copy the model ID from Playground.

Chat#

bash

gft chat <model-id> "Explain this circuit" --stream --max-tokens 512

--temperature, --top-p, --system and --max-tokens control generation. With --json --stream, the CLI prints one JSON event per line. Inference requests are never retried automatically.

In Python, use client.chat.create(...) or client.chat.stream(...).

Benchmark#

bash

gft bench <model-id> --requests 10 --concurrency 1,4,8 --max-tokens 128 --output bench.json

bench asks you to confirm charges before it sends requests; add --yes in scripts. Each concurrency level sends up to --requests calls in small batches, and the first failed batch stops further requests.

The report shows time to first token (TTFT), p50 and p95 latency, and output tokens per second, all measured by the client. They include network and request overhead, not only GPU time. Buffered cloud replies have no TTFT.

Use a local Studio#

With a local Studio installation, add --local to use its provider endpoints. The provider account is billed:

bash

gft chat fireworks:endpoint_id "Hello" --local --api-url http://127.0.0.1:8322 --stream

If the local API requires a token, set GFT_STUDIO_API_TOKEN. It is separate from the cloud GFT_API_KEY. --reasoning none works with supported local models.