# TrainingClient

## *class* [**tinker.TrainingClient**](https://github.com/thinking-machines-lab/tinker/blob/main/src/tinker/lib/public_interfaces/training_client.py#L90)(*holder*, *model_seq_id*, *model_id*)

Client for training ML models with forward/backward passes and optimization.

The TrainingClient corresponds to a fine-tuned model that you can train and sample from. You typically get one by calling `service_client.create_lora_training_client()`. **Key methods:**

- forward_backward() - compute gradients for training
- optim_step() - update model parameters with Adam optimizer
- save_weights_and_get_sampling_client() - export trained model for inference

```python
training_client = service_client.create_lora_training_client(base_model="Qwen/Qwen3-8B")
fwdbwd_future = training_client.forward_backward(training_data, "cross_entropy")
optim_future = training_client.optim_step(types.AdamParams(learning_rate=1e-4))
fwdbwd_result = fwdbwd_future.result()  # Wait for gradients
optim_result = optim_future.result()    # Wait for parameter update
sampling_client = training_client.save_weights_and_get_sampling_client("my-model")
```

**Parameters:**

- [**holder**](https://github.com/thinking-machines-lab/tinker/blob/main/src/tinker/lib/public_interfaces/training_client.py#L115) (*[InternalClientHolder](https://github.com/thinking-machines-lab/tinker/blob/main/src/tinker/lib/internal_client_holder.py#L185)*) – Internal client managing HTTP connections and async operations
- [**model_seq_id**](https://github.com/thinking-machines-lab/tinker/blob/main/src/tinker/lib/public_interfaces/training_client.py#L115) (*int*)
- [**model_id**](https://github.com/thinking-machines-lab/tinker/blob/main/src/tinker/lib/public_interfaces/training_client.py#L115) (*types.[ModelID](https://github.com/thinking-machines-lab/tinker/blob/main/src/tinker/types/model_id.py#L5)*) – Unique identifier for the model to train. Required for training operations.

### [**forward**](https://github.com/thinking-machines-lab/tinker/blob/main/src/tinker/lib/public_interfaces/training_client.py#L212)(*data*, *loss_fn*, *loss_fn_config=None*)

Compute forward pass without gradients.

**Parameters:**

- [**data**](https://github.com/thinking-machines-lab/tinker/blob/main/src/tinker/lib/public_interfaces/training_client.py#L214) (*List\[types.[Datum](https://tinker-docs.thinkingmachines.ai/tinker/api-reference/types/datum/index.md)\]*) – List of training data samples
- [**loss_fn**](https://github.com/thinking-machines-lab/tinker/blob/main/src/tinker/lib/public_interfaces/training_client.py#L215) (*types.[LossFnType](https://github.com/thinking-machines-lab/tinker/blob/main/src/tinker/types/loss_fn_type.py#L5)*) – Loss function type (e.g., "cross_entropy")
- [**loss_fn_config**](https://github.com/thinking-machines-lab/tinker/blob/main/src/tinker/lib/public_interfaces/training_client.py#L216) (*Dict[str, float] | None*, default: `None`) – Optional configuration for the loss function

**Returns:** [`APIFuture`](https://tinker-docs.thinkingmachines.ai/tinker/api-reference/apifuture/index.md) containing the forward pass outputs and loss

```python
data = [types.Datum(
model_input=types.ModelInput.from_ints(tokenizer.encode("Hello")),
loss_fn_inputs={"target_tokens": types.ModelInput.from_ints(tokenizer.encode("world"))}
)]
future = training_client.forward(data, "cross_entropy")
result = await future
print(f"Loss: {result.loss}")
```

*Async variant:* `forward_async()`

### [**forward_backward**](https://github.com/thinking-machines-lab/tinker/blob/main/src/tinker/lib/public_interfaces/training_client.py#L250)(*data*, *loss_fn*, *loss_fn_config=None*)

Compute forward pass and backward pass to calculate gradients.

**Parameters:**

- [**data**](https://github.com/thinking-machines-lab/tinker/blob/main/src/tinker/lib/public_interfaces/training_client.py#L252) (*List\[types.[Datum](https://tinker-docs.thinkingmachines.ai/tinker/api-reference/types/datum/index.md)\]*) – List of training data samples
- [**loss_fn**](https://github.com/thinking-machines-lab/tinker/blob/main/src/tinker/lib/public_interfaces/training_client.py#L253) (*types.[LossFnType](https://github.com/thinking-machines-lab/tinker/blob/main/src/tinker/types/loss_fn_type.py#L5)*) – Loss function type (e.g., "cross_entropy")
- [**loss_fn_config**](https://github.com/thinking-machines-lab/tinker/blob/main/src/tinker/lib/public_interfaces/training_client.py#L254) (*Dict[str, float] | None*, default: `None`) – Optional configuration for the loss function

**Returns:** [`APIFuture`](https://tinker-docs.thinkingmachines.ai/tinker/api-reference/apifuture/index.md) containing the forward/backward outputs, loss, and gradients

```python
data = [types.Datum(
model_input=types.ModelInput.from_ints(tokenizer.encode("Hello")),
loss_fn_inputs={"target_tokens": types.ModelInput.from_ints(tokenizer.encode("world"))}
)]

# Compute gradients
fwdbwd_future = training_client.forward_backward(data, "cross_entropy")

# Update parameters
optim_future = training_client.optim_step(
types.AdamParams(learning_rate=1e-4)
)

fwdbwd_result = await fwdbwd_future
print(f"Loss: {fwdbwd_result.loss}")
```

*Async variant:* `forward_backward_async()`

### [**forward_backward_custom**](https://github.com/thinking-machines-lab/tinker/blob/main/src/tinker/lib/public_interfaces/training_client.py#L383)(*data*, *loss_fn*, *loss_type_input='logprobs'*)

Compute forward/backward with a custom loss function.

Allows you to define custom loss functions that operate on log probabilities. The custom function receives logprobs and computes loss and gradients.

**Parameters:**

- [**data**](https://github.com/thinking-machines-lab/tinker/blob/main/src/tinker/lib/public_interfaces/training_client.py#L385) (*List\[types.[Datum](https://tinker-docs.thinkingmachines.ai/tinker/api-reference/types/datum/index.md)\]*) – List of training data samples
- [**loss_fn**](https://github.com/thinking-machines-lab/tinker/blob/main/src/tinker/lib/public_interfaces/training_client.py#L386) (*[CustomLossFnV1](https://github.com/thinking-machines-lab/tinker/blob/main/src/tinker/lib/public_interfaces/training_client.py#L82)*) – Custom loss function that takes (data, logprobs) and returns (loss, metrics)
- [**loss_type_input**](https://github.com/thinking-machines-lab/tinker/blob/main/src/tinker/lib/public_interfaces/training_client.py#L388) (*Literal['logprobs']*, default: `'logprobs'`) – Input space for `loss_fn`. Currently the only supported value is `"logprobs"`.

**Returns:** [`APIFuture`](https://tinker-docs.thinkingmachines.ai/tinker/api-reference/apifuture/index.md) containing the forward/backward outputs with custom loss

```python
def custom_loss(data, logprobs_list):
# Custom loss computation
loss = torch.mean(torch.stack([torch.mean(lp) for lp in logprobs_list]))
metrics = {"custom_metric": loss.item()}
return loss, metrics

future = training_client.forward_backward_custom(data, custom_loss)
result = future.result()
print(f"Custom loss: {result.loss}")
print(f"Metrics: {result.metrics}")
```

*Async variant:* `forward_backward_custom_async()`

### [**optim_step**](https://github.com/thinking-machines-lab/tinker/blob/main/src/tinker/lib/public_interfaces/training_client.py#L547)(*adam_params*)

Update model parameters using Adam optimizer.

The Adam optimizer used by tinker is identical to [torch.optim.AdamW](https://docs.pytorch.org/docs/stable/generated/torch.optim.AdamW.html). Note that unlike PyTorch, Tinker's default weight decay value is 0.0 (no weight decay).

**Parameters:**

- [**adam_params**](https://github.com/thinking-machines-lab/tinker/blob/main/src/tinker/lib/public_interfaces/training_client.py#L547) (*types.[AdamParams](https://tinker-docs.thinkingmachines.ai/tinker/api-reference/types/adamparams/index.md)*) – Adam optimizer parameters (learning_rate, betas, eps, weight_decay)

**Returns:** [`APIFuture`](https://tinker-docs.thinkingmachines.ai/tinker/api-reference/apifuture/index.md) containing optimizer step response

```python
# First compute gradients
fwdbwd_future = training_client.forward_backward(data, "cross_entropy")

# Then update parameters
optim_future = training_client.optim_step(
types.AdamParams(
learning_rate=1e-4,
weight_decay=0.01
)
)

# Wait for both to complete
fwdbwd_result = await fwdbwd_future
optim_result = await optim_future
```

*Async variant:* `optim_step_async()`

### [**save_state**](https://github.com/thinking-machines-lab/tinker/blob/main/src/tinker/lib/public_interfaces/training_client.py#L615)(*name*, *ttl_seconds=None*, *overwrite=False*)

Save model weights to persistent storage.

**Parameters:**

- [**name**](https://github.com/thinking-machines-lab/tinker/blob/main/src/tinker/lib/public_interfaces/training_client.py#L616) (*str*) – Name for the saved checkpoint
- [**ttl_seconds**](https://github.com/thinking-machines-lab/tinker/blob/main/src/tinker/lib/public_interfaces/training_client.py#L616) (*int | None*, default: `None`) – Optional TTL in seconds for the checkpoint (None = never expires)
- [**overwrite**](https://github.com/thinking-machines-lab/tinker/blob/main/src/tinker/lib/public_interfaces/training_client.py#L616) (*bool*, default: `False`) – If True, overwrite any existing checkpoint with the same name

**Returns:** [`APIFuture`](https://tinker-docs.thinkingmachines.ai/tinker/api-reference/apifuture/index.md) containing the save response with checkpoint path

```python
# Save after training
save_future = training_client.save_state("checkpoint-001")
result = await save_future
print(f"Saved to: {result.path}")
```

*Async variant:* `save_state_async()`

### [**load_state**](https://github.com/thinking-machines-lab/tinker/blob/main/src/tinker/lib/public_interfaces/training_client.py#L711)(*path*, *weights_access_token=None*)

Load model weights from a saved checkpoint.

This loads only the model weights, not optimizer state (e.g., Adam momentum). To also restore optimizer state, use load_state_with_optimizer.

**Parameters:**

- [**path**](https://github.com/thinking-machines-lab/tinker/blob/main/src/tinker/lib/public_interfaces/training_client.py#L712) (*str*) – Tinker path to saved weights (e.g., "tinker://run-id/weights/checkpoint-001")
- [**weights_access_token**](https://github.com/thinking-machines-lab/tinker/blob/main/src/tinker/lib/public_interfaces/training_client.py#L712) (*str | None*, default: `None`) – Optional access token for loading checkpoints under a different account.

**Returns:** [`APIFuture`](https://tinker-docs.thinkingmachines.ai/tinker/api-reference/apifuture/index.md) containing the load response

```python
# Load checkpoint to continue training (weights only, optimizer resets)
load_future = training_client.load_state("tinker://run-id/weights/checkpoint-001")
await load_future
# Continue training from loaded state
```

*Async variant:* `load_state_async()`

### [**load_state_with_optimizer**](https://github.com/thinking-machines-lab/tinker/blob/main/src/tinker/lib/public_interfaces/training_client.py#L744)(*path*, *weights_access_token=None*)

Load model weights and optimizer state from a checkpoint.

**Parameters:**

- [**path**](https://github.com/thinking-machines-lab/tinker/blob/main/src/tinker/lib/public_interfaces/training_client.py#L745) (*str*) – Tinker path to saved weights (e.g., "tinker://run-id/weights/checkpoint-001")
- [**weights_access_token**](https://github.com/thinking-machines-lab/tinker/blob/main/src/tinker/lib/public_interfaces/training_client.py#L745) (*str | None*, default: `None`) – Optional access token for loading checkpoints under a different account.

**Returns:** [`APIFuture`](https://tinker-docs.thinkingmachines.ai/tinker/api-reference/apifuture/index.md) containing the load response

```python
# Resume training with optimizer state
load_future = training_client.load_state_with_optimizer(
"tinker://run-id/weights/checkpoint-001"
)
await load_future
# Continue training with restored optimizer momentum
```

*Async variant:* `load_state_with_optimizer_async()`

### [**save_weights_for_sampler**](https://github.com/thinking-machines-lab/tinker/blob/main/src/tinker/lib/public_interfaces/training_client.py#L830)(*name*, *ttl_seconds=None*)

Save model weights for use with a SamplingClient.

**Parameters:**

- [**name**](https://github.com/thinking-machines-lab/tinker/blob/main/src/tinker/lib/public_interfaces/training_client.py#L831) (*str*) – Name for the saved sampler weights
- [**ttl_seconds**](https://github.com/thinking-machines-lab/tinker/blob/main/src/tinker/lib/public_interfaces/training_client.py#L831) (*int | None*, default: `None`) – Optional TTL in seconds for the checkpoint (None = never expires)

**Returns:** [`APIFuture`](https://tinker-docs.thinkingmachines.ai/tinker/api-reference/apifuture/index.md) containing the save response with sampler path

```python
# Save weights for inference
save_future = training_client.save_weights_for_sampler("sampler-001")
result = await save_future
print(f"Sampler weights saved to: {result.path}")

# Use the path to create a sampling client
sampling_client = service_client.create_sampling_client(
model_path=result.path
)
```

*Async variant:* `save_weights_for_sampler_async()`

### [**get_info**](https://github.com/thinking-machines-lab/tinker/blob/main/src/tinker/lib/public_interfaces/training_client.py#L884)()

Get information about the current model.

**Returns:** [`GetInfoResponse`](https://tinker-docs.thinkingmachines.ai/tinker/api-reference/types/getinforesponse/index.md) with model configuration and metadata

```python
info = training_client.get_info()
print(f"Model ID: {info.model_data.model_id}")
print(f"Base model: {info.model_data.model_name}")
print(f"LoRA rank: {info.model_data.lora_rank}")
```

*Async variant:* `get_info_async()`

### [**get_tokenizer**](https://github.com/thinking-machines-lab/tinker/blob/main/src/tinker/lib/public_interfaces/training_client.py#L904)()

Get the tokenizer for the current model.

**Returns:** `PreTrainedTokenizer` compatible with the model

```python
tokenizer = training_client.get_tokenizer()
tokens = tokenizer.encode("Hello world")
text = tokenizer.decode(tokens)
```

### [**create_sampling_client**](https://github.com/thinking-machines-lab/tinker/blob/main/src/tinker/lib/public_interfaces/training_client.py#L919)(*model_path*, *retry_config=None*)

Create a SamplingClient from saved weights.

**Parameters:**

- [**model_path**](https://github.com/thinking-machines-lab/tinker/blob/main/src/tinker/lib/public_interfaces/training_client.py#L920) (*str*) – Tinker path to saved weights
- [**retry_config**](https://github.com/thinking-machines-lab/tinker/blob/main/src/tinker/lib/public_interfaces/training_client.py#L920) (*[RetryConfig](https://github.com/thinking-machines-lab/tinker/blob/main/src/tinker/lib/retry_handler.py#L39) | None*, default: `None`) – Optional configuration for retrying failed requests

**Returns:** [`SamplingClient`](https://tinker-docs.thinkingmachines.ai/tinker/api-reference/samplingclient/index.md) configured with the specified weights

```python
sampling_client = training_client.create_sampling_client(
"tinker://run-id/weights/checkpoint-001"
)
# Use sampling_client for inference
```

*Async variant:* `create_sampling_client_async()`

### [**save_weights_and_get_sampling_client**](https://github.com/thinking-machines-lab/tinker/blob/main/src/tinker/lib/public_interfaces/training_client.py#L951)(*name=None*, *retry_config=None*)

Save current weights and create a SamplingClient for inference.

**Parameters:**

- [**name**](https://github.com/thinking-machines-lab/tinker/blob/main/src/tinker/lib/public_interfaces/training_client.py#L952) (*str | None*, default: `None`) – Deprecated, has no effect. Will be removed in a future release.
- [**retry_config**](https://github.com/thinking-machines-lab/tinker/blob/main/src/tinker/lib/public_interfaces/training_client.py#L952) (*[RetryConfig](https://github.com/thinking-machines-lab/tinker/blob/main/src/tinker/lib/retry_handler.py#L39) | None*, default: `None`) – Optional configuration for retrying failed requests

**Returns:** [`SamplingClient`](https://tinker-docs.thinkingmachines.ai/tinker/api-reference/samplingclient/index.md) configured with the current model weights

```python
# After training, create a sampling client directly
sampling_client = training_client.save_weights_and_get_sampling_client()

# Now use it for inference
prompt = types.ModelInput.from_ints(tokenizer.encode("Hello"))
params = types.SamplingParams(max_tokens=20)
result = sampling_client.sample(prompt, 1, params).result()
```

*Async variant:* `save_weights_and_get_sampling_client_async()`

## Referenced by

- [ServiceClient.create_lora_training_client](https://tinker-docs.thinkingmachines.ai/tinker/api-reference/serviceclient/#create_lora_training_client)
- [ServiceClient.create_training_client_from_state](https://tinker-docs.thinkingmachines.ai/tinker/api-reference/serviceclient/#create_training_client_from_state)
- [ServiceClient.create_training_client_from_state_with_optimizer](https://tinker-docs.thinkingmachines.ai/tinker/api-reference/serviceclient/#create_training_client_from_state_with_optimizer)
- [tinker_cookbook.checkpoint_utils.save_checkpoint](https://tinker-docs.thinkingmachines.ai/cookbook/api-reference/checkpoint_utils/save_checkpoint/index.md)
- [tinker_cookbook.eval.TrainingClientEvaluator.__call__](https://tinker-docs.thinkingmachines.ai/cookbook/api-reference/eval/trainingclientevaluator/#trainingclientevaluator-call)