The Engineer’s Guide to Deep Learning: Understanding the Transformer Model

There are many others that are better.

1/ The Annotated Transformer Attention is All You Need http://nlp.seas.harvard.edu/annotated-transformer/

2/ Transformers from Scratch https://e2eml.school/transformers.html

3/ Andrej Karpathy has really good series of intros: https://karpathy.ai/zero-to-hero.html Let's build GPT: from scratch, in code, spelled out. https://www.youtube.com/watch?v=kCc8FmEb1nY GPT with Andrej Karpathy: Part 1 https://medium.com/@kdwa2404/gpt-with-andrej-karpathy-part-1...

4/ 3Blue1Brown: But what is a GPT? Visual intro to transformers | Chapter 5, Deep Learning https://www.youtube.com/watch?v=wjZofJX0v4M Attention in transformers, visually explained | Chapter 6, Deep Learning https://www.youtube.com/watch?v=eMlx5fFNoYc Full 3Blue1Brown Neural Networks playlist https://www.youtube.com/playlist?list=PLZHQObOWTQDNU6R1_6700...

question to experts of HN in ML/AI. Could you please share the beginner resources you think would worth for a person who wants to switch their domain from CRUD/backend APIs to ML/AI. There seems to be many branches of this domain, not sure where to start.

Is my understanding correct?

    * ML engineer -> engineer who builds ML models with pytorch (or similar frameworks)
    * AI engineer -> engineer who builds applications on top of AI solutions (prompt engineering, OpenAI, Claude APIs,....)
    * ML ops -> people who help with deploying, serving models

It uses keras, which is obsolete. Nobody uses that thing anymore.

Stay away from this.

No content besides a few paragraphs of intro. Actual content has 404 not found errors.

When you send me an email, please provide at least two SNS [social networking service] addresses (e.g. LinkedIn, Twitter) for verification purposes. ... I no longer accept contact from anonymous individuals.

It's pretty sad to see that social networking is being adopted as an identification and trust mechanism even by technical people. It was bad enough when some governments began demanding social networking usernames for visa/immigrant screening, but we can't even send an email without social proof to other technical people now?

This is a very compressed work-through from perceptron to transformer.

When he is working through the gradients of an LSTM, for example, it is to help understanding, not help you implement it in your favourite framework.

When he is showing solutions in various frameworks, the purpose is to help create connections between what the math looks like and what code can look like.

One of the most frustrating things about all the documentation on Transformers is the sole emphasis on NLP.

In particular, one of the most interesting parts of the Transformer architecture to me is the attention mechanism which is permutation invariant (if not for the positional embeddings people use to counteract this inherent quality of attention layers). Also the ability to arbitrarily mask this or that node in the graph -- or even individual edges -- gives the whole thing so much flexibility for encoding domain knowledge into your architecture.

Positional embeddings may still be required in many cases but you can be clever about them beyond the overly restrictive perspective of attention layer inputs purely as one-dimensional sequences.

Very good resource on building from the basics in a concise manner.

Transformer tutorial is like the new "Monad tutorial".

In contrast, the AI technology of the current golden age, which began in the mid-2010s, has consistently exceeded our expectations.

Well, until recently, that is. It looks like we hit the wall as for what LLMs can do - some might call it a plateau of productivity. Namely, as far as coding is concerned, LLMs can successfully create chunk of code of limited length and tiny programs, can also review small pieces of code and suggest improvements that are not related to the context of the whole program (unless it can fit in the context window). In spite of huge effort put in creating a system where LLM agents could work together to create software such as AutoGPT, no non-trivial program has been created in this way so far.