🤖 AI & ML
Text as tokens
LLMs see text as tokens (pieces: words or subwords). Longer text = more tokens. We can simulate by splitting a string into words and counting.
3 min 10 XP Lesson 18 of 20
LLMs see text as tokens (pieces: words or subwords). Longer text = more tokens. We can simulate by splitting a string into words and counting.