Hacker News
new
|
past
|
comments
|
ask
|
show
|
jobs
|
submit
login
contravariant
on May 17, 2023
|
parent
|
context
|
favorite
| on:
Numbers every LLM developer should know
How come the token to word ratio is smaller than 1 if tokens are either words or part of words? Shouldn't you expect
more
tokens than words?
yonixw
on May 17, 2023
|
next
[–]
That is how I understood it, a token is on average a 3/4 of a word. "Token to word". So if you want to buy 1000 tokens you would get effectively 750 words.
waleedk
on May 17, 2023
|
prev
|
next
[–]
[Author] Fair point -- I clarified the language and gave a concrete example. Hope that helps!
renewiltord
on May 17, 2023
|
prev
|
next
[–]
It's the token to word multiplier, yeah. i.e. x tokens = 0.75x words.
furyofantares
on May 17, 2023
|
prev
[–]
I think all the ratios given are x:1 and they tell you x.
qeternity
on May 17, 2023
|
parent
|
next
[–]
It’s the other way around.
1 GPT4 token is equivalent to 50 GPT3.5 tokens.
1 token is equivalent to 0.75 words.
contravariant
on May 17, 2023
|
parent
|
prev
[–]
That would make it 0.75 tokens to 1 word right?
furyofantares
on May 17, 2023
|
root
|
parent
[–]
lol, yes, I'm glad they clarified because I understood it correctly then made the mistake GP did when I replied to them.
Guidelines
|
FAQ
|
Lists
|
API
|
Security
|
Legal
|
Apply to YC
|
Contact
Search: