The Google employees who created transformers

I'm not 100% sure I agree...

Google is an ad company at the end of the day. Google is still making obscene amount of money with ads. Currently Gen AI is a massive expense in training and running and is only looking like it may harm future ad revenue (yet to be seen). Meanwhile OpenAI has not surpassed that critical threshold where they dominate the market with a moat and become a trillion dollar company. It is typically very hard for a company to change what they actually do, so much so that in the vast majority of cases its much more effective (money wise) to switch from developing products to becoming a rent seeking utility by means of lobbying and regulation.

Simply put, Google itself could not have succeeded with GenAI without an outside competitor as it would have become its own competition and the CEO would have been removed for such a blunder.

Their wildest success at Google would have meant the company's stock price doubling or tripling and their stock grants being worth a couple million dollars at most. Meanwhile all of them had blank checks from top VCs in the valley and the freedom to do whatever they wanted with zero oversight. What exactly could Sundar have done to retain them?

I think the issue is that there is no future for a trustworthy AI that doesn't completely cannibalize their ad revenue cash cow.

Like, who wants to use an AI that says things like, "... and that's why you should wear sunscreen outside. Speaking of skin protection, you should try Banana Boat's new Ultra 95 SPF sunscreen."

He was too distracted with all the other things going on under his watch. He spent too much time trying to convince rank and file that it's OK to do business with customers they don't like, trying to ensure arbitrary hiring goals were made, and appeasing the internal activists that led protests and pressured to exit people that didn't fall in line.

This is why the CEO of Coinbase sent the memo a few years back stating that these types of people (not mission focused, self absorbed, and distracting) should leave the company. The CEO of Google should be fired.

Google doesn't have to try to be the best at AI now, it can just buy whoever is the winner later - a lesson learned from google video.

I mean if you had the pedigree these people have, the right move is to leave a big company and go start your own thing, there is simply a lot more upside there. You’d be foolish to stay with a big company, not sure what Google could have done here. Not everyone wants to be Jeff Dean.

Just like Xerox and Bell. Innovation isn’t necessarily their most important job, at least when innovations are being made.

Well, ads are doing well.

Among other things (like cloud for instance). I always think Sundar is the Steve Ballmer of Google.

Why would that be of note, especially in America? I think it would be an interesting observation in China or Japan, or some other country which is generally less welcoming to immigrants than the US

The more interesting thing to me is that only one of them went to an elite American undergrad (Duke). Rest went to undergrads in India, Ukraine, Germany, Canada (UToronto has a 43% acceptance rate).

Yes, this is one of the actually admirable qualities of the US and California in particular. CA has one of the world’s largest economies because it attracts and embraces people from just about every part of the world.

The problem is that chatting with an LLM is extremely disruptive to their business model and it's difficult for them to productize without killing the golden goose.

What is OpenAI today? Can you elaborate? Google is a varied $trillion company. OpenAI sells access to large generative models.

Well at the time before Microsoft got involved it was sort of an unspoken rule among the AI community to be open and not release certain models to the public.

They'd have to fall pretty far to be OpenAI.

I think it’s evidence that timing is everything. In the 2010s deep learning was still figuring out how to leverage GPUs. The scale of compute required for everything after GPT-2 would have been nearly impossible in 2017/2018–our courses at Udacity used a few hours of time on K80 GPUs. By 2020 it was becoming possible to get unbelievable amounts of compute to throw at these models to test the scale hypothesis. The rise of LLMs is stark proof of the Bitter Lesson because it’s at least as much the story of GPU advancement as it is about algorithms.

Do you think this current AI hype machine came from something other than transformers?

It's 2024; do we really have to point out the fact that publications take liberties with headlines? It's been that way forever.

Besides that, are you denying that transformers are the fundamental piece behind all the AI hype today? That's the point of the article. I think the fact that a mainstream publication is writing about the Transformers paper is awesome.

AI is in the eye of the beholder.

Transformer is an evolution of large scale training of seq2seq. Diffusion models is an evolution of even older approach - training denoising autoencoders.

It had been a relatively gradual and accelerating progress, with number of people working in the field increasing exponentially, since 2010 or so, when Deep Learning on GPUs was popularized at NIPS by Theano.

Tens of thousands people working together on Deep Learning. Many more on GPUs.

Modern AI != Machine Learning. Artificial intelligence has always been about models being able to accomplish a broad swath of tasks that they weren't explicitly trained on. It seems hard to argue we had accomplished this before massive transformers came along.

A better example of that behaviour would be Kodak which invented the first digital camera as soon as 1975 then killed the project, because it was a threat to their chemical business.

I was privy to the details of this. Sundar blocked the NEMA team from presenting at I/O with its gpt3 level product in the form of a character.

Sundar was afraid of the technology and how it would be received and tried to ice it.

On the other hand, Alphabet's inability to deploy GPT-3 or GPT-3.5 has led to the possibility of its disruption, so anti-trust treatment may not be necessary.

Honestly, this is part of the reason why I don't think Google will be a dominant business in 10 years. Searching the web for information helped us service a lot of useful 'jobs to be done' ... but most of those are now all done better by ChatGPT, Claude, etc.

Sure we have Gemini but can Google take a loss in revenue in search advertising in their existing product to maybe one day make money from Gemini search? Advertising in the LLM interface hasn't been figured out yet.

Google (kind of) feels like an old school newspaper in the age of the internet. Advertising models for the web took a while to shake out.

I’m studying neuroscience but very interested in how ai works. I’ve read up on the old school but phrases like memory graph and energy minimization are new to me. What modern papers/articles would you recommend for folks who want to learn more?

This is an uncharitable and oddly dismissive take (i.e. perfect for HN, I suppose).

Today's incredible state-of-the-art does not exist without the transformer architecture. Transformers aren't merely some lucky passengers riding the coattails of compute scale. If they were, then the ChatGPT app which set the world ablaze would've instead been called ChatMLP, or ChatCNN. But it's not. And in 2024 we still have no competing NLP architecture. Because the transformer is a genuinely profound, remarkable idea with remarkable properties (e.g. training parallelism). It's easy to downplay GPTs as a mostly derivative idea with the benefit of hindsight. I'm sure we'll perform the same revisionist history with state-space models, or whatever architecture eventually supplants transformers. Do GPTs build on prior work? Do other approaches and ideas deserve recognition? Yeah, obviously. Like...welcome to science. But the transformer's architects earned their praise -- including via this article -- which isn't some slight against everyone else, as if accolades were a zero-sum game. These 8 people changed our world and genuinely deserve the love!

but we didn't see what another group found: that scale in compute with simple algorithms were just better

The bitter lesson [0] strikes again.

[0] http://www.incompleteideas.net/IncIdeas/BitterLesson.html

They didn’t invent it, they advanced it. I agree.

However, some advances can have huge consequences to the field compared to others, even if at the technical level they appear comparable.

One example that comes to mind is CRISPR.

This is a classic... This isn't the first time this piece has been posted before either. Here's one from FT last year[0] or Bloomberg[1]. You can find more. Google certainly played a major role, but it is too far to say they invented it or that they are the ones that created modern AI. Like Einstein said, shoulder of giants. And realistically, those giants are just a bunch of people in trench coats. Millions of researchers being unrecognized. I don't want to undermine the work of these researchers, but that doesn't mean we should also undermine the work of the many others (and thank god for the mathematicians who get no recognition and lay all the foundation for us).

And of course, a triggered Yann[2] (who is absolutely right).

But it is odd since it is actually a highly discussed topic, the history of attention. It's been discussed on HN many times before. And of course there's Lilian Weng's very famous blog post[3] that covers this in detail.

The word attention goes back well over a decade and even before Schmidhuber's usage. He has a reasonable claim but these things are always fuzzy and not exactly clear.

At least the article is more correct specifying Transformer rather than Attention, but even this is vague at best. FFormer (FFT-Transformer) was a early iteration and there were many variants. Do we call a transformer a residual attention mechanism with a residual feed forward? Can it be a convolution? There is no definitive definition but generally people mean DPMHA w/ skip layer + a processing network w/ skip layer. But this can be reflective of many architectures since every network can be decomposed into subnetworks. This even includes a 3 layer FFN (1 hidden layer).

Stories are nice, but I think it is bad to forget all the people who are contribution in less obvious ways. If a butterfly can cause a typhoon, then even a poor paper can contribute to a revolution.

[0] https://www.ft.com/content/37bb01af-ee46-4483-982f-ef3921436...

[1] https://www.bloomberg.com/opinion/features/2023-07-13/ex-goo...

[2] https://twitter.com/ylecun/status/1770471957617836138

[3] https://lilianweng.github.io/posts/2018-06-24-attention/

In fairness, an article that’s about “the Google engineers who incrementally advanced AI” wouldn’t sell many advertisements.

I argue they definitely 'changed AI' but definitely agree they didn't 'invent modern AI'.

Personally, I think both compute and NN architecture is probably needed to get closer to AGI.

I guess the same thing happened with special relativity and Poincaré was forgotten.

Tax-free ? This was a regular Google team.

That would make sense, if companies and executives paid their damn taxes in the first place.

I mean this is how the US operates. Other countries continually try to refine their existing industries and societal structures. In the US, you let everything rot and hope that a hail Mary innovation comes out of somewhere and creates a breath of fresh air. It creates quite a wild society, but it seems to have worked for now!

Many inventions of the modern age were largely R&D bloat as you'd describe it - in this age R&D is done in university, spun out into a startup, and dies shortly after the IPO - but there's a flicker of it occasionally lasting on if it can hide under the guise of a tax deduction.

Weren't transistors specifically invented to improve telephone networks?

Nice story, but Google's heyday was probably 10 years before that. By 2014 the decline had already started.

Hahaha...

I worked at Borg

The quota system can kick in at whatever time the limits are reached.

And GPUs are scattered across borg cells, limiting the ceiling. That's why XBorg was created so that a global search among all Borg cells for researchers.

And data center Capex is around 5 billion each year.

Google makes hundres of billions of revenue each year.

You are asking what people would do in impossible situation. Like "what you do after you are dead", literally I could do nothing after I am dead.

I cannot even understand what I do stands for in the context of your question. The above is my direct reaction in the line that he assumes he had unlimited budget.

I shared an office with Uszkoreit when I was a phd intern and I always admired him for him having dropped out of his phd program.

Those were fun times! (& great to see you again after all these years). It's astonishing to me how far the tech has come given what we were working on at the time.

Why do you think transformers won't be key in making a level 4+ self-driving AI. It seems to me that Vision-capable multi-modal transformers could be the missing part: they can understand what is happening in the world in a deductive way.

The Vision transformer is capable of predicting that a running child is likely to follow that rolling soccer ball. It is capable of deducting that a particular situation looks dangerous or unusual and it should slow down, or stay away from danger in ways that previous crop of AI could not.

Imo, the only thing currently preventing transformers to change everything is the large amount of compute power required to run them. It's not currently possible to imagine GPT4-V running on a embedded computer inside a car. Maybe AI asic type of chip will solve that issue, maybe Edge computing and 5G will find it's use-case... Let's wait and see, but I would bet that transformer will find it's way in many places and change the world in many more ways than bringing us chatbots.

I think LLMs provide more utility than further dependence on cars.

All modern AI is based on this, including self driving. Check out Waymo's research papers. They even have something called "Wayformer".

with periods of solo uninterrupted deep focus

Which is usually impossible in the office. So more like a mix, which is what all reasonable people are saying.

Not so subtle!

+1. I worked remotely the past 2 years. I just RTO'd (new team) and it's life-changing when you actually want to get stuff done quickly.

“Office” does not have to mean open office. Academics all have offices with doors for a reason. I can’t stand open office, but private office in a building with other people is great.

Same thing is happening with the Figure One robot. There isn't much new in the demo Figure did recently, all the work has been published in the palm saycan paper and other deepmind papers. Google is good at laying golden eggs and not realizing that they've laid them. Then employees get fed up, leave and start their own companies so they can productize their work without management getting in the way.

This is a story that could be written over and over again. Companies with big R&D budgets but not nimble enough any more to take advantage.

Xerox 2: Electric Boogoogle