Nvidia CEO Jensen Huang announces new AI chips: ‘We need bigger GPUs’

They are priced as if they are the only ones who are capable of creating chips that can crunch LLM algos. But AMD, Google, Intel, and even Apple are also capable.

Apple is in talks with Google to bring Gemini to the iPhone, and it will obviously also be on android phones. So almost every phone on earth is poised to be using Gemini in the near future, and Gemini runs entirely on Google's own custom hardware (which is at parity or better than nVidia's offerings anyway).

Because their stock value is highly coupled with crypto mining and AI craze.

The move from PoW to PoS for most crypto networks in combination with bust of ‘22. NVDA slid down in value.

OpenAI debuts ChatGPT in late 2022 and now it’s suddenly bumping in price as the hype and rush for GPUs from companies of all types buys up their stock of GPUs. Demand is far outpacing the supply. Nvda can’t keep up.

Thus, share price is brittle. Competition in the GPU market is dominantly owned by Nvidia. That can change, but so far openai loves using nvidia for some reason.

No moat.

Yes, CUDA, but CUDA is maaaaaybe a few tens of billion USD deep and a few (more) years wide. When the rest of the industry saw compute as a vanity market, that was sufficient. Now, it's a matter of time before margins go to, uhhh, less than 90%.

Does that make shorting a good idea? I wouldn't count on it. The market can always remain irrational longer than you can remain solvent.

72 P/E ratio while they have a mere monopoly on one the most valuable resource in the world.

Competition WILL come. Maybe it's Groq, maybe AMD, maybe Cerebras. Maybe there's a stealth startup out there. Point is, they're going to be challenged soon.

Their market cap is 2.2T $.

In the past year, they had a revenue of 60B $ and net income of 30B $. Absolutely amazing numbers, I agree. The year before they had a revenue of 30B $ and a net income of 4.5B $ - and it was a rather good year. What happens next of course depend of how you judge the situation - was it a peak hype demand ? Will it stabilize now ? Grow at current extraordinary rates ?

Scenario 1 - margins get back to normal due to hype going down, competition improving etc - in this case the company is worth at best ~200B $ - or 1/10 of what it is now.

Scenario 2 - they maintain current revenue and the exceptional margins - the company would be worth ~1T - or 1/2 of what it is now.

Scenario 3 - they current growth rate (based on past 12 months) continue for ~5 years. This is the case the company is worth ~2T $.

But they are in a business where most money come from a handful of customers, all of which are working on similar chips - and given the sums in play now, the incentives are *very* strong.

My opinion, is that the company is already priced for perfection - basically the current price reflects the perfect scenario. I struggle to see any upside, unless we have AGI in the next 5 years and it decides it can only run on Nvidia chips.

All of this is akin to Tesla in the past years. They grew from a small startup to a medium car maker - the % growth rate was huge of course - an amazing achievement in itself. But people projected that the % growth rate would continue - and the stock was priced accordingly. Reality is catching up on Tesla, even if some projections are still absolutely crazy.

Because he missed the train. My guess.

That's exactly what I'm saying below - PE is still very high, hence projecting the past growth into the future. But the scale changed a bit. A 37 PE ratio is extremely high by historical standards - this was reserved for very promising, small startups. Not for 2T companies. I know this got distorted in the past 15 years by abnormally low interest rates, but sooner or later it will come back to something that makes sense.

Buying long-term put options on Nvidia now is extremely expensive - the stock was so volatile that the price you pay for those options almost annihilate any gains you could expect, even if the stock losses 50% in 12 months.

You got me curious about those 5-7 trillions. Where these numbers come from ?

We are at a very unique time. The stock market has basically been in a bull market for 15 years with some very short-lived sell-offs along the way. During that time we've had some incredible innovations such as the iPhone, FANG stock dominance and unprecedented profitability for years.

You've also had three or four bona fide bubbles in that span, starting around 2017. First was Bitcoin along with the stock market as a whole (with Nvidia being one of the leading stocks of that bull market advance).

Then you had Tesla go parabolic and lots of people become rich. Then you had the whole post-COVID speculative mania.

The result of this has been extreme credulity by the average person. Today's keynote is the perfect summation of this phenomenon. I saw multiple people who almost certainly couldn't explain in any level of detail how Nvidia GPUs are used for training and inference, but rather rely on the secondhand talking points like CUDA that they've learned by watching Jim Cramer, watching this keynote with excitement and anticipating how much it would pump their shares or call options.

Contrast this with Steve Jobs keynotes from 15 years ago when Apple's best days were well ahead of them. Most keynotes were questioned, in some cases even mocked. When Tesla stock broke out, many people couldn't make sense of it. Ditto for cryptocurrencies. But now, taking their cues from those cycles, the average person wants to ride the next bubble to riches and is trying to catch the wave and so now believes every story attached to a rising asset price.

CEO's aren't blind to this and are using every opportunity to create favorable storylines. The leadup to a keynote like this carries with it an enormous amount of pressure to deliver. Hence a company like Nvidia leaning into generative artwork and straight up made up storylines like robot development.

At the end of the day, I'm afraid that there likely isn't all that much substance and the evidence is beginning to pile up that the megacap tech stocks have run out of ideas which is why they are laying off people en masse and appealing to the AI hype cycle to carry their stocks higher.

Consider that Nvidia has gone up 8x- 800%!- in just over a year. The cycles are moving faster and faster. I remember just a year ago when lots of people said Nvidia at $250 was insane. Now here we are with the stock at more than three times that level and most people are calling it cheap. The stock market seems to have in certain areas like semis, completely disconnected from the fundamentals and taken flight. Yes, Nvidia earnings have grown. But understand that this is all part of a positive feedback loop where tech CEO's are pressured by their competitors and shareholders to show that they are investing in AI. Thus they all talk about it on their earnings calls and spend massively. All of their stocks rise in unison as you have a market that increasingly looks like its chasing momentum stock trends up. Nvidia's moves of late have almost nothing to do with any fundamental developments in the company. It has been routinely trading upwards of $45 billion a day. The Friday before last that number was over $100 billion. These are absolutely insane figures. Compare that to Microsoft, the largest company by market cap in the world, which trades on average around $8 billion per day.

I think this is generally how bull markets end and I think we may be actually forming the top of the great bull market for the megacaps that began around 2010 but really hit its stride starting in 2017.

I don't understand AMD in this. Isn't it insanity that they're not throwing all they've got at their software stack?

It changes alot. It is not legal to make a 'CUDA' driver for an AMD GPU as Nvidia own cuda. You can see there was a open implementation of this that AMD sponsored until they got threatened with a lawsuit by Nvidia

Getting this working might be worth a trillion $ to AMD - they should be doing more than just waiting for a bootstrapped startup to debug their drivers for them.

CUDA is not open. See what happened with ZLUDA.

They did force Microsoft to make Office work on Mac though... (Office for Mac already existed but I think MS agreed to not cancel it.)

By "Opening Up" they cement their leadership position. AI frameworks are already targeting CL, SPIR-V, etc. The low level details will fade and so will Nvidias api dominance.

The MI300 smokes the H100 yet here we are.

I get it, the service model always shines the brightest in the eye of the revenue calculator. But I have immediate skepticism they’ll be able to execute at a competitive level. Their core competency has always been manufacture and production, not service-based things. It’s a big rock to push up a tall hill.

- Training isn’t done at 4-bits, to date this small size has only been for inference.

Wasn't there a paper from Microsoft two weeks ago or so where they trained on log₂(3) bits?

Edit: https://arxiv.org/pdf/2402.17764.pdf

Intuitively, there is a ton of redundancy and we still have a long way we can still compress things.

Maybe the rounding errors are noise that is somewhat useful in a big enough neutral net. Image generators also generate noise to work on

Thanks. First source i see for what fp4 is. Gotta say I'm surprised: I would have chosen to lose one value, but have a zero. (though I have no doubt those people are much more clever and knowledgeable than I am)

Thanks, I was thinking that zero, negative zero, inf, negative inf, and the NaN's were included like in IEEE 754

1 bit (not floating point)

I like how you specified that it's not floating point.

I see, so the idea is that enough of the quantization errors are sort of averaged out across the dimensions of the vector space to still be useful?

Half the reason why they move back up to 8/16 bit is that current hardware doesn't properly support 4 bit floats, and you get better performance from the conversion. I think once this hardware hits, most of the computation will shift to native 4 bit just for efficency's sake.

...assuming the recent 1.58b paper doesn't render the entire float quantization approach obsolete by then.

Llama.cpp and many others support 4 bit weights and lower

From how I understood it, it means they optimised the entire stack from CUDA to the networking interconnects specifically for data centers, meaning you get 30x more inference per dollar for a datacenter. This is probably not fluff, but it's only relevant for a very very specific use-case, ie enterprises with the money to buy a stack to serve thousands of users with LLMs.

It doesn't matter for anyone who's not microsoft, aws or openai or similar.

Isn't sparsity almost always a win at this point? Making everything fully connected is a major waste.

'narrow scenario,' perhaps, but one that also happens to closely match rumors for GPT4's size

Is that how it works? Why don't we just put many chips in one computer?

two fused chips

Jensen's comment about being first was such a dig to Emerald Rapids.

Because it involved scaling in chip area needed for FP8. AI community realized that FP8 training is possible few years back so the transistors given for FP8 was scaled. Overall I think transistors grow just by ~50% per generation so most of the gains comes from removing FP32/FP64 share which were dominant 10 years back, but there is only some point it could go to.

As someone who has used Ubuntu in the past and has now moved onto greener pastures, I appreciate everything Canonical and Ubuntu have done for the Linux community but there are many better options today and Canonical is already far from the company it once used to be.

Tbh, Ubuntu’s only pull is the support and breadth of users. As a desktop, it’s let down by Unity, which IMHO is basically a port of Windows 8 tablet UI.

If they defaulted back to a menu and taskbar-based WM, it might actually be more approachable to users who are more familiar with macOS and Windows.

the desktop Linux ecosystem can survive w/o Ubuntu. Silverblue / Universal Blue for instance is quite compelling.

What happened to Red Hat? As far as I see they’re continuing to invest in linux. Im glad they are keeping CoreOs around as FCOS.

That's not culture, that's Shuttleworth.

> Apple shows that owning the entire stack is immensely profitable.

Apple shows no such thing. Apple, sells pretty, reliable and safe. A car is a car, but apple is a sports car, or a saloon. Vertical integration is the way they chose to deliver that, and pretty and reliable are all normal people care about.

Nvidia is gonna have to think long and hard about the "whole stack". 20 years ago they might have been able to pull a next, but right now anything that isnt LINUX is a rounding error, and they dont want to turn into SUN (no one at nividia is smart enough to make them the next sun).

Nvidia architecture + Nvidia os is not something that I see them pulling off for the datacenter.

This is sort of already a reality. Their vGPU functionality (partitioning a single physical GPU into multiple virtual GPUs) is already separately licensed - https://www.nvidia.com/en-us/data-center/buy-grid/

And that's once you've bought an expensive Tesla/Quadro GPU too.

Cerebras: -Hold my beer

Matthew Bryant would disagree.

https://findthatmeme.com/blog/2023/01/08/image-stacks-and-ip...

Currently AI folks create bigger models and feed larger training data sets, because we still don't know the efficiency limits. IOW, currently we can't accelerate the learning beyond a certain point with less training data. This is esp. true on LLM/GenAI space.

On the other areas where NNs are used, what I can see is training models with less data is not only plausible, but very possible, esp. in image processing, probably an image carries more information than a single sentence.

I think AI falls on the spectrum with a slight bias to the latter group. Because if you can shrink a model 10% and lose a month, you'd rather have a 10% bigger model now, and reap the money^H^H^H^H^H fame.

I don't know anything about a typical web app, because I'm not a typical developer developing web apps. :)

Just because you're squeezing out every bit of performance from a processor doesn't mean all the work you're doing is meaningful or can't be optimized.

I work on material simulations. I make processors hit their TDPs, saturate their pipelines and make them go as fast as they can. However, sometimes we come up with a formula optimization which does things 1-2% faster, which means we can save hours on a bigger computation. Utilization doesn't change, but speed does.

I think it is more of an information problem.

It's an interesting point of view, and partially true. However, we're still wasting "space" by just adding bits to the network to make it contain more data.

There's a long way to go. "Wasteful development" is a phase and always be part of software development. The important part is not forgetting that optimization exists. Otherwise we can't sustain ourselves much with all that energy use.

It doesn't imply a full 25x reduction in power consumption though, that might "only" go down by 10x.