Assistant
I can help with reading practice using the files in this workspace. What would you like to work on?
What is a token?Understand · 1 of 3
Learning lab
— tokens— charactersLoading tokenizer…
What is a token?
A token is a small piece of text represented by a number. It can be a word, part of a word, punctuation, or whitespace. Colors show the boundaries; they do not indicate meaning.
This view uses the o200k_base encoding. Different encodings can split the same text differently. OpenAI tokenizer ↗
Select a token to inspect its ID.
Some characters span multiple tokens. Those tokens share a color in Text view to preserve the character; Token IDs shows every ID.
Previewing does not load a file into contextEsc to close