# tinker_cookbook.tokenizer_utils.TmlTokenizer

## *class* [**tinker_cookbook.tokenizer_utils.TmlTokenizer**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/tokenizer_utils.py#L64)(*Protocol*)

Structural type for the underlying `tml_renderers` tokenizer.

Cookbook never constructs this object (`tml_renderers.tokenizers` does), and `tml_renderers` is lazily imported, so its concrete type is not referenceable here. This protocol pins the small surface the cookbook adapter and renderer actually rely on, making the contract explicit instead of `Any` duck-typing.

**Fields:**

- [**bos_token**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/tokenizer_utils.py#L74) (*str*)
- [**eos_token**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/tokenizer_utils.py#L75) (*str*)

## Referenced by

- [tinker_cookbook.tokenizer_utils.SupportsTmlTokenizer](https://tinker-docs.thinkingmachines.ai/cookbook/api-reference/tokenizer_utils/supportstmltokenizer/index.md)
- [tinker_cookbook.tokenizer_utils.TmlRenderersTokenizerAdapter](https://tinker-docs.thinkingmachines.ai/cookbook/api-reference/tokenizer_utils/tmlrendererstokenizeradapter/index.md)