Helicone

Log inLog into community

Helicone Community Page

Updated 2 months ago

Tokens calculation is wrong

Tokens calculation is wrong

·

Hey guys, found the issue with tokens calculation which affects cost projection. Instead of 2-4k we get ~22 tokens 😀. Noticed that for all OpenAI models almost a month ago (streaming is on). Have you met the same before 😀 ? We're using proxy integration (https://docs.helicone.ai/integrations/openai/python) with the following headers

headers = {
"Helicone-Auth": f"Bearer AUTHKEY",
"Helicone-Cache-Enabled": "true",
"Helicone-Property-CUSTOMPROPERTY": "custom_prop_value",
"Helicone-User-Id": user_id
}

Attachments

2

N

J

O

39 comments

Tested with disabled cache - got the same problem. Request IDs:
f8f66937-ba2c-4f58-8a9f-303171437e39
dbd193e8-9757-4773-9d07-b5a6ee15136d
c90556fc-e0c2-46f1-9b91-92692a5f980a
b34e6482-d751-4eef-8d35-a51ae396cb86

Attachment

Hi are you using streaming? If you are you might need to add the following options
https://docs.helicone.ai/use-cases/enable-stream-usage

Hey , yep we're using streaming. It helped, thanks a lot! 😄

Hi , we've been noticing the same issues with cost estimation of OpenAI's models since yesterday (streaming is on). Have smth changed in your API? 🙂

Request ID: 77051de5-4200-406f-8039-5942f378029c

Attachment

Hi Thanks for flagging I am taking a look now

Thanks a lot!

Am I on your org?

Nope 🙂

can you add me please? justin@helicone.ai?

and send me the org name?

Oh, turned out I don't have rights to add members :D. Jsonify

can you send me the org id?

ohh

okay

I see

25d70dfb-342e-45fb-8632-56b3ab0f13ac

Hi Jsonify

❤️ love your product btw

🤩

We love helicone as well ^_^

ahh I see the issue :/

Thanks you for flagging!

We fixed this and we are rolling out a fix. We are going to try to restore the previous logs..

Awesome, ty! ❤️

Hi we were able to restore these logs! thanks again for flagging

Hey , glad to hear it! 😀

OOlivier Balais

Hi there, we also have a problem with token computation on Helicone with latest OpenAI models. Is there something we can do?

Attachment

Hi is this still an issue?

If it is - can you please share your organization ID and I can take a look?

OOlivier Balais

Hello Justin, yes, it is still a problem. Here is our organization ID: b4f3ca3a-9749-4231-82f7-4ac53003e4ba. Thanks for your help.

Hi @Olivier Balais you will need to include this flag:

Plain Text

    stream_options={
        "include_usage": True # set to true
    }

https://docs.helicone.ai/use-cases/enable-stream-usage

@Justin - If I'm using the Vercel AI library, do you know if there is any way to pass in that stream_options value?

Hi @Justin we just started using Helicone (after moving away from Portkey).

We are right now on the free tier testing things out, first thing I hit is the cost calculation for OpenAI streaming is not being calculated, I'm using Vercel AI specifically the streamUI which as far as I know do not have any support for stream_options.

Is there already a solution or workaround for this? If not, in the streamUI we can capture the usage on the onFinish event, is it somehow possible to pass that to calculate the cost?

We really love the product but this is quite crucial for us. Please guide me here, thanks!

Hi Adel - looks like @assaf also asked this and we missed it

Let me see if there is a way

Hi @Adel and @assaf - I just enabled a header that will force this flag to be there. Can you please try it out?

https://docs.helicone.ai/faq/enable-stream-usage#method-2-helicone-header

Attachment

Wow 🥹 it is working now! thanks alot for the prompt response Justin

no problem! It was a small fix 😉

https://github.com/Helicone/helicone/commit/a94112d4a8674b47110960d64514159c4f6b9d67

Add a reply

Sign up and join the conversation on Discord