Intro

ChatGPT is to LLM, as Kleenex is to tissue

What are LLMs?

Tokens

Demo - Manual LLM inference 

LLM generate text

What LLMs predict

Vectors, intuitively

Word embeddings and nearest neighbors

Demo - Semantic meaning of word embeddings 

The architecture for a Large Language Model

How LLMs predict

Self-attention adds context

Demo - Adding "context" to a vector

MLP transforms

Demo - The necessity of non-linearities

How Transformers predict

Absolute positional encoding

Demo Cons of absolute positional bias

Demo - skip connections

Batch norm

RMS norm

How LLMs use position

Workshop-feedback-qa

How LLMs attend

Modern day transformer architectures

Modern LLM connection to papers

You can take the workshop from anywhere in the world, as long as you
have a computer and an internet connection. You also have the opportunity to ask the instructors questions live.

Live and remote

Learn at your own pace, whenever it's convenient for you. With no
rigid schedule to worry about, you can take the course on your own
terms.

Recorded

Learn by building while you learn the concepts.

Build

Join a [vibrant community](/discord) of other students who
are also learning with Fundamentals of transformers. Ask questions, get feedback and
collaborate with others to take your skills to the next level.

Community

In this workshop, we dive deep into Large Language Models (LLMs) to help you understand, build, and optimize their architecture for real-world applications. LLMs are transforming industries—from customer support to content creation—but understanding how these models work, let alone optimizing them, can be challenging. 

In this comprehensive 9-module series, we cover:

The technical essentials of LLMs, including autoregressive decoding, positional encoding, and multi-head attention
The entire LLM lifecycle, from pretraining on massive datasets to fine-tuning and instruction tuning for specialized tasks
Best practices for evaluating LLMs, identifying bottlenecks, and leveraging state-of-the-art architectures for efficiency and scalability

this workshop includes hours of in-depth instruction, hands-on coding exercises, and access to a community forum for support and discussions. You'll also gain exclusive access to source code templates, an expansive reference library, and downloadable materials for continued learning.

It's taught by [Alvin Wan](https://www.linkedin.com/in/alvinwan/), a Senior Research Scientist at Apple and a PhD student at UC Berkeley with international recognition for his impactful contributions in efficient AI and design. With his practical industry experience and research insights, you’ll be guided from fundamentals to advanced concepts with clarity and precision.

By the end of this workshop, you’ll not only understand how to create and optimize LLMs but also how to apply this knowledge across various applications in tech and business.

Everyone knows chatgpt, but how do modern large language models fully work? The fundamentals start at the transformer. This workshop is a workshop to dymstify the transformer and be able to run through concept to code on how the transformer work. This workshops combines concept at an intutive level, to code, to math all with the intent at providing an end to end understanding at the fundamentals of large language models.

Fundamentals of transformers - Live Workshop

Miniaturized version of Huggingface’s LLM inference utility

Interactive demos to understand word embeddings and model representation capacity

Visualization utilities for real-world LLMs, to understand self-attention

Intro to LLM Basics: Learn foundational concepts, including terminology like models, data, algorithms, and optimization.

Autoregressive Decoding: Grasp how LLMs predict words through conditional generation, supported by manual inference demos.

LLM Prediction Mechanism: Explore LLM architecture, with an intuitive look at vectors and word embeddings.

Semantic Meaning in Embeddings: See how word embeddings represent semantic meaning through nearest neighbors and vector demos.

Transformer Core Mechanics: Unpack the inner workings of a transformer layer, including self-attention and context addition.

Non-Linear Transformations: Discover why non-linearities are essential, supported by hands-on matrix multiplication and MLP demos.

Positional Encoding: Learn absolute and relative positional encoding techniques, plus RMS Norm for positional bias management.

Differences between absolute and relative positional encoding

Attention Mechanisms: Delve into “forward-facing” and multi-head attention to understand attention values.

Advanced Attention: Study grouped-query attention and its importance in handling large data.

Current Transformer Models: Analyze academic and modern transformer diagrams, identifying bottlenecks in today’s LLMs.

Build a Mini LLM Inference Tool: Create a simplified version of Huggingface’s LLM utility to understand LLM operation.

Understand Word Embeddings: Develop interactive demos exploring word embeddings and how models represent words

Visualize Self-Attention: Use visualization tools to understand the role of self-attention in language models.

---
title: Tokens
privateVideoUrl: https://stream.mux.com/HEVvNcVs00rcmrYvFPcJ900UqlVF1Qvh9NlfYzMVBQCvc.m3u8
isPublicLesson: true
---


Alvin Wan

We've seen now a bit of code that actually runs inference for us.

Now I'm going to jump right back to the slides.

And again, if you didn't catch all this, this is in the notebook that I sent

Okay. Oh, sorry. One more thing. One more thing here.

Okay, so I didn't explain what tokens are, yes.

Let me explain what that is in second here.

And let's just say that we want to understand how many words there are.

Okay, I'll give back to that question, Ken. That's a good question.

So here it says that there are seven tokens, right?

The tokenizer here, the main goal is to actually break up our input text into

And how it does that is up to the model designers.

So in this case, we've broken up the input text into seven chunks.

Throughout this presentation, I'm just going to say instead of tokens,

So let's just say the input is in theory, seven words.

Now let's look at how many outputs there are.

And so if I run this input IDs, so notice that I actually did something

LLM.generate gives you a bunch of text like we saw here, right?

Now here, what I'm doing is I'm doing something a little bit different.

So I'm feeding it a bunch of inputs and it's giving me outputs.

I'm looking directly at the outputs of the model.

Well, we'll talk about what that looks like later for now.

What we want to do is understand how many tokens the model, the LLM output,

And let's look at how many tokens there are.

We, and I'll explain why that's weird in a second.

But anyways, the point is here we have seven input tokens,

The number of tokens that we input is the number of tokens that we output.

So let me get, let me start looking at the questions.

I've always wondered how punctuation formatting works in LLM output.

Maybe it's time that I introduce what a token is more formal.

So as we talked about, a token is a bunch of chunks, right?

But most importantly, this tokenizing call actually gives us a set of IDs.

So first thing that it does is it chunks up the input into a bunch of blocks or

The second is each chunk now has an ID, right?

So you could roughly imagine that this input right here,

but it looks like something like one, two, 500, and so on, so forth, right?

So then the LLM takes in that list of numbers and it gives us back a list of

And those numbers may encode punctuation, right?

So in fact, what we can do is look at what these tokens look like right here.

And then let me actually just output these input IDs.

So you can see that there are some IDs here, two to 58, nine, nine.

And I think as it turns out, we can actually get the ID for comma if you wanted

But actually we're going to do that in the next demo.

So in the next demo, we'll actually see what the input at what the ID is for

can feel free to remind me if I don't do that.

So the next question is, are tokens related to features, feature extraction,

In its current form, what we've shown here, tokens are just basically,

they're basically just IDs that correspond to some text.

And later on, okay, there's some other stuff I need to explain first.