GPT in 500 Lines of SQL

I love this, something that starts off as some kind of sorcery a year ago is now being explained so well and almost in a childish way.

I think, I think, GPT creating GPT... creating GPT will be a thing soon. GPTception.

This is great. In a similar vein, I implemented GPT entirely in spreadsheet functions with accompanying video tutorials

https://spreadsheets-are-all-you-need.ai/

These marvels need to be preserved. Just posting the archive link here in case the blog is not maintained in future.

https://archive.is/VAGzF

I can feel the SQL-Force in this young jedi, Midichlorian level is on another level

I’ve completely avoided GPT and LLMs. This looks like it would generate some level of fluidity in text output, but not be able to parse and answer a question.

Is there any simplistic blog posts / training courses which go through how they work, or expose a toy engine in python or similar that? All the training I’ve seen so far seems oriented at how to use the platforms rather than how they actually work.

In the tokenization example for '"Mississippilessly", why is 'si' not part of the combined column? It appears twice in the text. My speculation is that it got collapsed out with 'iss' (a longer token). Is that right?

This should be illegal

Great write up, I enjoyed the reading the explanations for each piece and found them to be clear and quite thorough.

I did make the mistake though of clicking "+ expand source", and after seeing the (remarkable) abomination I can sympathize with ChatGPT's "SQL is not suitable for implementing large language model..." :)

Serious question, how do I get this smart?

Is this an accurate representation of the GPT driver loop?

    def generate(prompt: str) -> str:
      # Transforms a string into a list of tokens.
      tokens = tokenize(prompt) # tokenize(prompt: str) -> list[int]
    
      while True:
     
        # Runs the algorithm.
        # Returns tokens' probabilities: a list of 50257 floats, adding up to 1.
        candidates = gpt2(tokens) # gpt2(tokens: list[int]) -> list[float]
     
        # Selects the next token from the list of candidates
        next_token = select_next_token(candidates)
        # select_next_token(candidates: list[float]) -> int
     
        # Append it to the list of tokens
        tokens.append(next_token)
     
        # Decide if we want to stop generating.
        # It can be token counter, timeout, stopword or something else.
        if should_stop_generating():
          break
 
      # Transform the list of tokens into a string
      completion = detokenize(tokens) # detokenize(tokens: list[int]) -> str
      return completion

I keep reading that GPT is a "smarter" "complex" Markov, in the end, just a function spitting out next word with some probability.

But from my experience that cannot be true - it has to learn somehow. There is an easy example to make. Tell it something that happened today and contradicts the past (I used to test this with the Qatar World Cup), and then ask questions that are affected by that event, and it replied correctly. How is that possible? How a simple sentence (the information I provide) changes the probabilites for next token by that far?

It's a nice demo. Unfortunately the article mixes up things in the explanation for causal masking, because it seems the author conflates aspects from training and inference. First, causal masking exists to prevent the model from "peeking" at future tokens during training, and second (at least for GPT-like architectures) for enforcing the autoregressive aspect during inference. During inference we only use the last token anyways, so it will attend the entire input sequence. So this token is definitely not decided only from the last token's embedding.

Tại xỉu

This is a great read, I didn't expect to scroll right back to the top as soon as I finished it the first time.

Related:

A GPT in 60 Lines of NumPy - https://news.ycombinator.com/item?id=34726115 - February 2023 (146 comments)

This is a very good article and introduction.

Fantastic article. It kept my eyes on the screen for 2 hours, without interruption The author is a genius.

Unexpectedly insightful and answers some of the same questions I had early on: not just “how” questions, but “why” as well. You see the pattern with softmax quite often. I wish it was taught as “differentiable argmax” rather than by giving people a formula straight away. That’s not all it is, but that’s how it’s often used.

This is beautiful. I'd actually been going down this same rabbit hole with sqlite, I hadn't gotten far enough to bring a neural net into it.

I'd been inspired by the makemore lecture series[0]. At the 1hr mark or so, he switches from counting to using a nn, which is about as far as I've gotten. Breaking it down into a relational model is actually a really great exercise.

[0] https://www.youtube.com/watch?v=PaCmpygFfXo

What is this sorcery?

Plain Unicode, however, doesn't really work well with neural networks.

That is not true. See ByT5, for example.

As an illustration, let's take the word "PostgreSQL". If we were to encode it (convert to an array of numbers) using Unicode, we would get 10 numbers that could potentially be from 1 to 149186. It means that our neural network would need to store a matrix with 149186 rows in it and perform a number of calculations on 10 rows from this matrix.

What the author calls alphabet here, is typically called vocabulary. And you can just use UTF-8 bytes as your vocabulary, so you end up with 256 tokens, not 149186. That is what ByT5 does.