Hidden costs in AI deployment: Why Claude models may be 20-30% more expensive than GPT in enterprise settings

The various model families are a well-known fact that families can use different tbehenizers. However, there is limited analysis of how the process will be “tohinalization“ It varies in these tochenizers. Do all the tokenizers result in the same number of verses for a specific entry text? If not, how different are the verses created? How important are the differences?

In this article, we are investigating these questions and examine the practical effects of the change in ticalization. We present a comparative story of two border model families: OpenChatgpt vs AnthropicalClaude. Although the advertised “valuable Token” figures are highly competitive, experiments reveal that anthropical models can be 20-30% more expensive than GPT models.

API Assessment – Claude 3.5 Sonnet vs GPT-4O

By June 2024, the price structure for these two advanced border models is highly competitive. Both Anthropic Clone Clone 3.5 Sonnet and Openai’s GPT-4O, the same costs for output tokens, Claude 3.5 offers a 40% lower price for Sonnet signs.

Source: Vantage

Hidden “inefficiency of Tokenizer”

Despite the decrease in the lower drop of an anthropical model, it is cheaper than the total cost of experience working with GPT-4O (with a set of fixed tips), Claude Sonnet-3.5.

Why?

Anthropic Tokenizer tends to break the same entries to more signs compared to Openai’s Tokenizer. This means that anthropical models for the same requests are more verses than Openai colleagues. As a result, if the significant value for the entrance of 3.5 Sonnet can replace these deposits, these deposits can be formed, and leading to higher total costs in practical use.

This hidden expense is caused by the path of the anthropy of the Tokenizer, provides information using more verses to represent the same content. The number of Token has a serious impact on inflation costs and the use of the context window.

Domain dependent Tanhenization inefficiency

Domain content Domain content is toxenized in a different way by the anthropy Tokenizer, causing a different level of increasing token compared to Openai models. AI research community recorded similar tochenization differences here. In three popular domains of our findings, ie: English articles, code (python) and math.

Domain	Model Login	GPT Tokens	Klost Tokens	Token surface
English articles		77	89	~ 16%
Code (Python)		60	78	~ 30%
Mathematics		114	138	~ 21%

Clod 3 3.5 Sonet Tokenizer (GPT-4O) Source of Token (GPT-4O) Source: Lavaya Gupta

Clode 3.5 When comparing Sonet to GPT-4O, Tokenizer rate changes significantly in the inhabitant of inhabit. For English articles, the Claude toxinizer produces about 16% more than 16% more than GPT-4O for the same login text. This surface increases sharply with more structured or technical content.

These variability arises, because some types of content and code such as the code contain patterns and symbols that have patterns and symbols, which are higher than a small number of cases. In contrast, the more natural language content is tended to show a place in a low sign.

Other practical results of Tokenizer

Outside the direct impact on the cost, there is an indirect impact on the use of the context window. Anthropic models, unlike 128k verses of Openai, can be even smaller for an effective use of Token Space, Anthrop models. Thus, the “advertised” context against the “effective” context window can be a small or large difference with the window sizes of the “advertised” context.

Application of Tokenizers

Uses GPT models Byte Double Coding (BPE)(Combining frequent character pairs to create token. In particular, the latest GPT models use open source O200K_Base Tokenizer. The actual verses used by GPT-4O (in Tiktoken Tokenizer) can be viewed here.

JSON
 
{
    #reasoning
    "o1-xxx": "o200k_base",
    "o3-xxx": "o200k_base",

    # chat
    "chatgpt-4o-": "o200k_base",
    "gpt-4o-xxx": "o200k_base",  # e.g., gpt-4o-2024-05-13
    "gpt-4-xxx": "cl100k_base",  # e.g., gpt-4-0314, etc., plus gpt-4-32k
    "gpt-3.5-turbo-xxx": "cl100k_base",  # e.g, gpt-3.5-turbo-0301, -0401, etc.
}

Unfortunately, their tchinizer is not directly and easily as GPT because it is not much about anthropic tochinizers. Anthropical In December 2024, Token was released. However, it was deleted soon in the 2025 versions.

Late red Reporting that “Anthropic, Openai uses a unique Tokenizer with only 65,000 token changes compared to 100.261 Token changes for GPT-4.” This Colab notebook Contains Python code to analyze the differences in Tokenization between GPT and Claude models. Another instrument Some common, confirm our findings that allow you to communicate with open toheads.

Tokens is an active assessment capability (without inviting the actual model API) and budget expenditures are very important for AI businesses.

Key Takeaways

• The competitive prices of anthropy are coming with hidden costs:
  The anthropic Claode 3.5 Sonneti, when Openai’s GPT-4o offers 40% lower access signs, this visible maya can be wrong due to the differences in how to enter the input text.
  
• Hidden “inefficiency of Tokenizer”:
  Anthropic models are more verbose. This inconsistency is comprehensible when evaluating the true cost of understanding this inconsistency for large-scale text preparations.
  
• Domain dependent Tokenizer Inefficiency:
  When choosing between Openai and Anthropic Models, Evaluate the character of your login text. The cost difference for natural language tasks may be minimal, but technical or structured domains can cause significant high costs with anthropic models.
  
• An effective context window:
  According to an anthropy’s tanizer version, the 200K Context Window, which has been advertised, can offer a lesser effective place for up to 128K, a potential Space between advertisement and true context window.

Anthropic, Venturebeat’s press time did not meet their requests for comment. We will update the story if answered.