tiktoken-rs

Rust library for tokenizing text with OpenAI models using tiktoken.

This library provides a set of ready-made tokenizer libraries for working with GPT, tiktoken and related OpenAI models. Use cases cover tokenizing and counting tokens in text inputs.

This library is built on top of the tiktoken library and includes some additional features and enhancements for ease of use with Rust code.

Supports all current OpenAI models including GPT-5.4, GPT-5, GPT-4.1, GPT-4o, o1, o3, o4-mini, and gpt-oss models.

Scope: This crate is focused on OpenAI tokenizers (tiktoken). For non-OpenAI models
(Llama, Gemini, Mistral, etc.), use the HuggingFace tokenizers crate.

Examples

For full working examples for all supported features, see the examples directory in the repository.

Usage

1. Install this tool locally with cargo

shCopy
cargo add tiktoken-rs

Then in your rust code, call the API

Counting token length

rustCopy
use tiktoken_rs::o200k_base;

let bpe = o200k_base().unwrap();
let tokens = bpe.encode_with_special_tokens(
  "This is a sentence   with spaces"
);
println!("Token count: {}", tokens.len());

For repeated calls, use the singleton to avoid re-initializing the tokenizer:

rustCopy
use tiktoken_rs::o200k_base_singleton;

let bpe = o200k_base_singleton();
let tokens = bpe.encode_with_special_tokens(
  "This is a sentence   with spaces"
);
println!("Token count: {}", tokens.len());

Upgrading encode calls

CoreBPE::encode mirrors upstream tiktoken and returns a Result.
Propagate or unwrap the result before using the tokens:

rustCopy
use tiktoken_rs::o200k_base;

let bpe = o200k_base().unwrap();
let allowed = bpe.special_tokens();
let (tokens, last_piece_token_len) = bpe.encode("hello <|endoftext|>", &allowed).unwrap();

The generic encode_as and count helpers also return Result.
This release also follows upstream's Rust 2024 core, so projects need
Rust 1.85 or newer. That floor comes from the vendored upstream code,
not from a direct dependency MSRV.

Counting max_tokens parameter for a chat completion request

rustCopy
use tiktoken_rs::{get_chat_completion_max_tokens, ChatCompletionRequestMessage};

let messages = vec![
    ChatCompletionRequestMessage {
        content: Some("You are a helpful assistant that only speaks French.".to_string()),
        role: "system".to_string(),
        ..Default::default()
    },
    ChatCompletionRequestMessage {
        content: Some("Hello, how are you?".to_string()),
        role: "user".to_string(),
        ..Default::default()
    },
    ChatCompletionRequestMessage {
        content: Some("Parlez-vous francais?".to_string()),
        role: "system".to_string(),
        ..Default::default()
    },
];
let max_tokens = get_chat_completion_max_tokens("o1-mini", &messages).unwrap();
println!("max_tokens: {}", max_tokens);

Counting max_tokens parameter for a chat completion request with async-openai

Need to enable the async-openai feature in your Cargo.toml file.

rustCopy
use tiktoken_rs::async_openai::get_chat_completion_max_tokens;
use async_openai::types::chat::{
    ChatCompletionRequestMessage, ChatCompletionRequestSystemMessage,
    ChatCompletionRequestSystemMessageContent, ChatCompletionRequestUserMessage,
    ChatCompletionRequestUserMessageContent,
};

let messages = vec![
    ChatCompletionRequestMessage::System(ChatCompletionRequestSystemMessage {
        content: ChatCompletionRequestSystemMessageContent::Text(
            "You are a helpful assistant that only speaks French.".to_string(),
        ),
        name: None,
    }),
    ChatCompletionRequestMessage::User(ChatCompletionRequestUserMessage {
        content: ChatCompletionRequestUserMessageContent::Text(
            "Hello, how are you?".to_string(),
        ),
        name: None,
    }),
];
let max_tokens = get_chat_completion_max_tokens("o1-mini", &messages).unwrap();
println!("max_tokens: {}", max_tokens);

tiktoken supports these encodings used by OpenAI models:

Encoding name	OpenAI models
o200k_harmony	gpt-oss-20b, gpt-oss-120b
o200k_base	GPT-5 series, o1/o3/o4 series, gpt-4o, gpt-4.5, gpt-4.1, codex-*
cl100k_base	gpt-4, gpt-3.5-turbo, text-embedding-ada-002, text-embedding-3-*
p50k_base	Code models, text-davinci-002, text-davinci-003
p50k_edit	Edit models like text-davinci-edit-001, code-davinci-edit-001
r50k_base (or gpt2)	GPT-3 models like davinci

Context sizes

Model	Context window
gpt-5.4, gpt-5.4-pro	1,050,000
gpt-4.1, gpt-4.1-mini, gpt-4.1-nano	1,047,576
gpt-5, gpt-5-mini, gpt-5-nano, gpt-5.4-mini, gpt-5.4-nano	400,000
o1, o3, o3-mini, o3-pro, o4-mini	200,000
codex-mini	200,000
gpt-oss	131,072
gpt-4o, gpt-4o-mini	128,000
o1-mini, gpt-5.3-codex-spark	128,000
gpt-3.5-turbo	16,385
gpt-4	8,192

See the examples in the repo for use cases. For more context on the different tokenizers, see the OpenAI Cookbook

Encountered any bugs?

If you encounter any bugs or have any suggestions for improvements, please open an issue on the repository.

Acknowledgements

Thanks @spolu for the original code, and .tiktoken files.

License

This project is licensed under the MIT License.