Hacker Newsnew | past | comments | ask | show | jobs | submit | 1aurent29's commentslogin

Common enough words like `function` and `class` are generally encoded as a single token by the tokenizer and may provide a slightly better context to the LLM. For openai you can test this stuff at https://platform.openai.com/tokenizer


Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: