Through the Token Lens: How AI Truly Understands Us

Through the Eyes of a Machine: Reflections on How LLMs Understand Us

I’ve been reflecting lately on how large language models (LLMs)—the very same technology powering most of our AI tools—actually “see” the world we share with them. It’s fascinating, and honestly, the more I learn about it, the more I realize how much of what we call “intelligence” is built on something as deceptively simple as breaking language into little puzzle pieces called tokens.

Think of it like this: when I write “Hello world!”, I see two words and some punctuation. An LLM, on the other hand, doesn’t really see it that way. It splits the phrase into tokens, which might look like: ["Hello", " world", "!", "\n"]. Each of those parts becomes the model’s raw material for understanding. Tokens are the currency of AI conversations. They determine how much we pay for API calls, how much an AI can “remember” at one time, and how smooth or clunky our interactions feel.

When I first learned this, I had that “a-ha” moment: no wonder models sometimes fumble with typos, non-English words, or even simple math. They’re not looking at whole ideas the way we do—they’re juggling these tokens in patterns they’ve learned from enormous datasets. And sometimes those patterns don’t quite line up with the way numbers or words naturally work for us.

Get Featured by Publishers Driving Holiday Amazon Sales

This holiday season, top publishers are hand-selecting Amazon brands for gift guides, newsletters, reviews, and more — sending high-intent shoppers directly to storefronts.

Levanta is partnering with these publishers to connect them with a select group of 7–9 figure brands ready to scale.

Check if your brand qualifies

Breaking Words, Building Meaning

What makes this even more interesting is the way tokenization strategies differ. Some models use Byte Pair Encoding (BPE), others use WordPiece, SentencePiece, or even Unigram. Each method shapes the way text is split up before it even gets fed into the neural net. For example, “unhappy” might become ["un", "##happy"] in WordPiece, or ["▁un", "happy"] in SentencePiece. These tiny differences in how words get chopped up actually ripple all the way up into how well the model understands context, meaning, or even cultural nuance.

That explains why some models feel “smarter” in one language than another. It’s not always about training data—it can come down to whether the tokenizer respects the natural structure of the language. In English, most words tokenize fairly cleanly. In languages like Japanese or Chinese, though, it’s a much harder puzzle. A poor tokenizer can triple the number of tokens needed just to express a simple idea, leaving the model with less “room” in its context window to keep track of the conversation.

Here’s one tip I’ve picked up: when writing prompts, simpler is often better. Avoid piling on obscure formatting or mixing multiple languages unless necessary, because every quirky character can eat up more tokens than you expect. Even emojis can balloon into several tokens, which is a strange thought when we’re so used to thinking of them as one neat symbol.

Former Zillow exec targets $1.3T

The top companies target big markets. Like Nvidia growing ~200% in 2024 on AI’s $214B tailwind. That’s why the same VCs behind Uber and Venmo also backed Pacaso. Created by a former Zillow exec, Pacaso’s co-ownership tech transforms a $1.3 trillion market. With $110M+ in gross profit to date, Pacaso just reserved the Nasdaq ticker PCSO.

Invest for $2.90/Share

_{Paid advertisement for Pacaso’s Regulation A offering. Read the offering circular at invest.pacaso.com. Reserving a ticker symbol is not a guarantee that the company will go public. Listing on the NASDAQ is subject to approvals.}

Numbers, Code, and Confusion

Now, here’s where things get even trickier: numbers and code. You might have heard of the infamous “3.11 vs 3.9” problem. For us, it’s obvious that 3.11 is greater than 3.9. But to a language model, “3.11” might tokenize as ["3", ".", "11"] and “3.9” as ["3", ".", "9"]. Suddenly, instead of two comparable values, it’s juggling separate symbols. No wonder the AI sometimes gets it wrong.

This is one reason LLMs often need external tools to really shine. Whether it’s a calculator, a code interpreter, or a search engine, these add-ons help the model bridge the gap between statistical pattern-matching and actual reasoning. Without them, it’s like asking someone to compare recipes while blindfolded—they might guess right sometimes, but they’re not actually tasting the dish.

What strikes me is how much of what we call “intelligence” in AI is actually collaborative. The model itself is one part, tokenization is another, and then we layer in tools, monitoring systems, and human feedback. It’s not a single brain at work; it’s an ecosystem.

If you’re working with AI tools, consider integrating monitoring systems like Sentry or similar. These give you insight not just into “what broke,” but into how your agent is behaving under the hood. Sometimes the issue isn’t the model at all, but a slow API call or a malformed token sequence.

The Context Window Challenge

The idea of a “context window” is another fascinating limitation. Models can only keep track of a certain number of tokens at once, and once you overflow that window, the oldest parts of the conversation slip away. Early models like GPT-2 had room for about 1,000 tokens—barely a page of text. Today, some cutting-edge models can handle a million or more.

But even with these huge upgrades, context windows shape how we need to think when prompting. If you overload your input with long documents or unnecessary repetition, you’re wasting precious space. If, instead, you’re precise—summarizing where needed, trimming filler—you give the model more breathing room to focus on the parts that actually matter.

This is where I think prompt writing becomes less of a technical skill and more of an art form. We’re not just feeding information to a machine; we’re negotiating with the boundaries of tokenization, context, and computation. It’s almost like writing a letter where every word costs a cent, and you have to decide: what’s worth saying?

Outrank Your Competitors on Amazon—Fast!

Stack Influence automates micro-influencer collaborations to significantly boost external traffic and organic rankings on Amazon. Trusted by top brands like Magic Spoon and Unilever, it's your secret to dominating page one.

Get 10% OFF this month!

Seeing Ourselves in the Mirror

To bring this full circle, I find myself looking at language differently now. Tokens aren’t just abstract pieces of AI math—they’re a reminder that meaning is always constructed from fragments. Whether it’s the syllables in our speech, the characters on a page, or the tokens inside a machine, we build understanding step by step.

That’s why AI sometimes stumbles, and why it sometimes surprises us with brilliance. It’s both constrained and creative, operating within the rules of tokenization but discovering patterns we might not notice ourselves.

For me, the takeaway is simple: if we want better AI experiences, we need to understand the lens it uses to see us. That means respecting the quirks of tokenization, designing prompts with awareness of cost and context, and remembering that behind every polished output, there’s a lot of invisible processing.

As a final tip: always check how many tokens your text consumes. A rule of thumb for English is that one token equals about four characters or three-quarters of a word. So if you’re drafting prompts, keep in mind that 100 tokens is about a paragraph. This helps you control costs and manage expectations.

So the next time you type a sentence into ChatGPT—or any AI assistant—pause for a second. Remember that it’s not “reading” the way you are. It’s seeing you in tokens, turning them into vectors, and then weaving them back into something human-readable. In a way, it’s like holding up a mirror that doesn’t quite reflect you perfectly, but still tells you something true about how we construct meaning.

Until next time, let’s keep exploring the world—token by token.

What’s your next spark? A new platform engineering skill? A bold pitch? A team ready to rise? Share your ideas or challenges at Tiny Big Spark. Let’s build your pyramid—together.

That’s it!

Keep innovating and stay inspired!

If you think your colleagues and friends would find this content valuable, we’d love it if you shared our newsletter with them!

PROMO CONTENT

Can email newsletters make money?

With the world becoming increasingly digital, this question will be on the minds of millions of people looking for new income streams in 2025.

The answer is—Absolutely!

That’s it for this episode!

Thank you for taking the time to read today’s email! Your support allows me to send out this newsletter for free every day. 

 What do you think for today’s episode? Please provide your feedback in the poll below.

Share the newsletter with your friends and colleagues if you find it valuable.

Disclaimer: The "Tiny Big Spark" newsletter is for informational and educational purposes only, not a substitute for professional advice, including financial, legal, medical, or technical. We strive for accuracy but make no guarantees about the completeness or reliability of the information provided. Any reliance on this information is at your own risk. The views expressed are those of the authors and do not reflect any organization's official position. This newsletter may link to external sites we don't control; we do not endorse their content. We are not liable for any losses or damages from using this information.