Introducing tiktoken-php: A BPE tokeniser to use with OpenAI’s models

Are you a developer working with the OpenAi API in PHP and need to manage token usage efficiently? Look no further! Introducing tiktoken-php, a handy PHP library that allows you to count tokens in text strings, making it easier to stay within the usage of your OpenAI or ChatGPT account.

As an developer who’s building an Ai project on top of OpenAi, I wanted to create a PHP version of the popular tiktoken Python library developed by OpenAI. The tiktoken library in Python is a powerful tool for counting tokens in a text string, which is essential for managing token usage when working with the OpenAi models. Inspired by its functionality, I set out to create a similar library in PHP to make it accessible to PHP developers as well.

What are tokens and why do they matter?

The GPT family of models process text using tokens, which are common sequences of characters found in text.

I highly recommend you to read this link for explicit information about tokens and use this link count and test the token counts of your sentences.

How to use tiktoken-php

Using tiktoken-php is straightforward. First, you need to install the library using Composer, the popular PHP dependency management tool. Simply run the following command in your project directory:

composer require mehrab-wj/tiktoken-php

Once tiktoken-php is installed, you can use it in your PHP code by including the autoloader and creating an instance of the TikToken class, as shown in the following example:

require_once 'vendor/autoload.php';

use TikToken\Encoder;
$prompt = "Ai is cool";
$encoder = new Encoder();

$tokens = $encoder->encode($prompt); // [32, 72, 318, 3608]

// Get tokens count:
echo count($tokens); // 4

The encode() method takes a text string as input and returns the tokens as an array. So you can use the output of encode() method to manage your token usage and stay within the limits.