Intel CEO: 'The entire industry is motivated to eliminate the CUDA market'

I wonder if the next generation chips are going to just have a dedicated hardware RNG per-core if that's an issue?

Generating random numbers is a bit complicated!

I know! I just wrote a whole paper and published a library on this!

But really, perhaps not as much as many from outside might think. The core of a Philox implementation can be around 50 lines of C++ [1], with all the bells and whistles maybe around 300-400. That implementation's performance equals CuRAND's , sometimes even surpasses it! (the API is designed to avoid maintaining any rng states on device memory, something curand forces you to do).

running the same PRNG with the same seed on all your cores will produce the same result

You're right. Solution here is to utilize multiple generator objects, one per thread, ensuring each produces statistically independent random streams. Some good algorithms (Philox for example), allow you to use any set of unique values as seeds for your threads (e.g. thread id).

[1] https://github.com/msu-sparta/OpenRAND/blob/main/include/ope...

for GPGPU, the better approach is CBRNG like random123.

https://github.com/DEShawResearch/random123

if you accept the principles of encryption, then the bits of the output of crypt(key, message) should be totally uncorrelated to the output of crypt(key, message+1). and this requires no state other than knowing the key and the position in the sequence.

the direct-port analogy is that you have an array of CuRand generators, generator index G is equivalent to key G, and you have a fixed start offset for the particular simulation.

moreover, you can then define the key in relation to your actual data. the mental shift from what you're talking about is that in this model, a PRNG isn't something that belongs to the executing thread. every element can get its own PRNG and keystream. And if you use a contextually-meaningful value for the element key, then you already "know" the key from your existing data. And this significantly improves determinism of the simulation etc because PRNG output is tied to the simulation state, not which thread it happens to be scheduled on.

(note that the property of cryptographic non-correlation is NOT guaranteed across keystreams - (key, counter) is NOT guaranteed to be uncorrelated to (key+1, counter), because that's not how encryption usually is used. with a decent crypto, it should still be very good, but, it's not guaranteed to be attack-resistant/etc. so notionally if you use a different key index for every element, element N isn't guaranteed to be uncorrelated to element N+1 at the same place in the keystream. If this is really important then maybe you want to pass your array indexes through a key-spreading function etc.)

there are several benefits to doing it like this. first off obviously you get a keystream for each element of interest. but also there is no real state per-thread either - the key can be determined by looking at the element, but generating a new value doesn't change the key/keystream. so there is nothing to store and update, and you can have arbitrary numbers of generators used at any given time. Also, since this computation is purely mathematical/"pure function", it doesn't really consume any memory-bandwidth to speak of, and since computation time is usually not the limiting element in GPGPU simulations this effectively makes RNG usage "free". my experience is that this increases performance vs CuRand, even while using less VRAM, even just directly porting the "1 execution thread = 1 generator" idiom.

Also, by storing "epoch numbers" (each iteration of the sim, etc), or calculating this based on predictions of PRNG consumption ("each iteration uses at most 16 random numbers"), you can fast-forward or rewind the PRNG to arbitrary times, and you can use this to lookahead or lookback on previous events from the keystream, meaning it serves as a massively potent form of compression as well. Why store data in memory and use up your precious VRAM, when you could simply recompute it on-demand from the original part of the original keystream used to generate it in the first place? (assuming proper "object ownership" of events ofc!) And this actually is pretty much free in performance terms, since it's a "pure function" based on the function parameters, and the GPGPU almost certainly has an excess of computation available.

--

In the extreme case, you should be able to theoretically "walk" huge parts of the keystream and find specific events you need, even if there is no other reference to what happened at that particular time in the past. Like why not just walk through parts of the keystream until you find the event that matches your target criteria? Remember since this is basically pure math, it's generated on-demand by mathing it out, it's pretty much free, and computation is cheap compared to cache/memory or notarizing.

(ie this is a weird form of "inverted-index searching", analogous to Elastic/Solr's transformers and how this allows a large number of individual transformers (which do their own searching/indexing for each query, which will be generally unindexable operations like fulltext etc) to listen to a single IO stream as blocks are broadcast from the disk in big sequential streaming batches. Instead of SSD batch reads you'd be aiming for computation batch reads from a long range within a keystream. (And this is supposition but I think you can also trade back and forth between generator space and index hitrate by pinning certain bits in the output right?)

--

Anyway I don't know how much that maps to your particular use-case but that's the best advice I can give. Procedural generation using a rewindable, element-specific keystream is a very potent form of compression, and very cheap. But, even if all you are doing is just avoiding having to store a bunch of CuRand instances in VRAM... that's still an enormous win even if you directly port your existing application to simply use the globalThreadIdx like it was a CuRand stateful instance being loaded/saved back to VRAM. Like I said, my experience is that because you're changing mutation to computation, this runs faster and also uses less VRAM, it is both smaller and better and probably also statistically better randomness (especially if you choose the "hard" algorithms instead of the "optimized" versions like threefish instead of threefry etc). The bit distribution patterns of cryptographic algorithms is something that a lot of people pay very very close attention to, you are turning a science toy implementation into a gatling gun there simply by modeling your task and the RNG slightly differently.

That is the reason why you shouldn't do the "just download random numbers", as a sibling comment mentions (probably a joke) - that consumes VRAM, or at least system memory (and pcie bandwidth). and you know what's usually way more available as a resource in most GPGPU applications than VRAM or PCIe bandwidth? pure ALU/FPU computation time.

buddy, everyone has random numbers, they come with the fucking xbox. ;)

How does that work? Why not pick up where the previous team left off instead of everyone starting new ones? Or are they all targeting different backends and hardware?

Yeah. In this instance, I was talking about the same algorithm (phillox), the difference is purely in implementation.

I have the fastest random number generator in the world. And it works in parallel too!

https://i.stack.imgur.com/gFZCK.jpg

Appreciate it, will do.

Or you could just re-use the same number; no one can prove it is not random.

https://xkcd.com/221/

For a given set kernels, and a limited set of architectures, the problem is relatively easy.

But covering all the important kernels acros all the crazy architecture out there and with relatively good performance and numerical accuracy ... Much harder

The amount of VRAM is the absolute killer USP for the current large AI model hobbyist segment. Something that had just as much VRAM as a 3090 but at half the speed and half the price would sell like hot cakes.

This is pretty disappointing to hear. I’m really surprised they can’t even get a clean build script for users, let alone integrate into the regular Pytorch releases.

oh no, I just bought a refurbed A770 16GB for tinkering with GPGPU lol. It was $220, return?

It wasn’t always like this. Nvidia did the initial heavy lifting to get cuda off the ground to a point where other people could use it.

That's because CUDA is a clear, well-functioning library and Intel has no equivalent. It makes any "you just have to get Pytorch working" a little less plausible.

You know how it works. The same busybodies who are putting out this useless noise releases are the ones who squandered Intel's lead, and now are patting themselves on the back for figuring out that with this they'll again be on top for sure!

There was a post on HN a few months ago about how Nvidia's CEO still has meetings with engineers in the trenches. Contrast that with what we know of Intel, which is not much good, and a lot of bad. (That they are notoriously not-well-paying, because they were riding on their name recognition.)

Its purpose was to create compatibility between Unix vendors so developers could write software compatible with different flavors. The primary market for Unix vendors is servers, which to this day are still about CLI and daemons, and POSIX systems continue to have dominant market share in that market.

The point of the open standard is to untether your prospective customers from the incumbent. That means nobody is going to monopolize that technology anymore, but that works out fine when it's not the thing you're trying to sell -- AMD and Intel aren't trying to sell software libraries, they're trying to sell GPUs.

And this strategy regularly works out for companies. It's Commoditize Your Complement.

If you're Intel you support Linux and other open source software so you can sell hardware that competes with vertically integrated vendors like DEC. This has gone very well for Intel -- proprietary RISC server architectures are basically dead, and Linux dominates much of the server market in which case they don't have to share their margins with Microsoft. The main survivor is IBM, which is another company that has embraced open standards. It might have also worked out for Sun but they failed to make competitive hardware, which is not optional.

We see this all over the place. Google's most successful "messaging service" is Gmail, using standard SMTP. It's rare to the point of notability for a modern internet service to use all proprietary networking protocols instead of standard HTTP and TCP and DNS, but many of them are extremely successful.

And some others are barely scraping by, but they exist, which they wouldn't if there wasn't a standard they could use instead of a proprietary system they were locked out of.

That doesn't really make sense. Pretty much all LLMs are trained in pytorch, which is open source. LLMs only reached the state it is now because many academic conferences insisted that paper submissions have open source code attached to it. So much of the ML/AI ecosystem is open source. Pretty much only CUDA is not open source.

In a business-class network, even one running Windows, if DNS breaks "everything" is going to break.

What _7il4 removed were these two comments:

"AMD is not serious, and neither is Intel for that matter. Their software are piles of proprietary garbage fires. They may say they are serious but literally nothing indicates they are."

"Yes, and ROCm also doesn't work on anything non-AMD. In fact it doesn't even work on all recent AMD gpus. T"

How’s SYCL proprietary exactly?

ROCm is open source.

and: https://www.youtube.com/watch?v=pVl25BbczLI

It is way, way better in the last year or so. Perfectly reasonable cards for inference if you actually understand the stack and know how to use it. Is nVidia faster? Sure, but at twice the price for 20-30% gains. If that makes sense for you, keep paying the tax.

Agreed. Time will tell.

MI300x and ROCm 6 and their support of projects like Pytorch, are all good steps in the right direction. HuggingFace now supports ROCm.

but it's all inference, and most of Nvidias moat is in training afaik.

That’s my point though - they’re catching up, not leading, which rather implies that they absolutely missed a beat, and don’t therefore understand where the market is going before it goes there.

Krzanich started out as an engineer

Otellini also made the historical decision to pass on the iPhone chip...

owned the top500 with xeon phi.

This is interesting - because what I heard (within Intel at the time, circa 2015) was Xeon Phi was a disaster. The programming model was bad and they couldn't sell them.

as technical and strategic

It seems that this is an AND not an OR.

It still doesn't change mistake in your original message.

Precisely. The problem I found on HN is that It is hard to have any meaningful discussion on anything Hardware. Especially when it is mixed with business or economics models.

I was greedy and was hoping Intel could fall closer to $20 in early 2023 before I load up more of their stock. Otherwise I would put more money where my mouth is.

it's not about revenue, it's about investment. It is closely related to future profit. Not so much to current revenue ...

AMD could use testers(cough clients i mean) like you. Jokes aside, please report bugs to rocm github..

Not only that, for a “first” (not sure how much of Larrabee was salvaged) discrete GPU attempt, Intel Arc is fantastic. Look at the first GPUs Nvidia and ATI launched.

It’s only when you put them up against Nvidia and AMDs comes-with-decades-of-experience offerings that Intel’s GPUs seem less than stellar.

Vulkan 1.3 has pointers, thanks to buffer device address[1]. It took a while to get there, and earlier pointer support was flawed. I also don't know of any major applications that use this.

Modern Vulkan is looking pretty good now. Cooperative matrix multiplication has also landed (as a widely supported extension), and I think it's fair to say it's gone past OpenCL.

Whether we get significant adoption of all this I think is too early to say, but I think it's a plausible foundation for real stuff. It's no longer just a toy.

[1] https://community.arm.com/arm-community-blogs/b/graphics-gam...

I understand why you've picked D3D11, but people have to understand that comes with serious limitations. There are no subgroups, which also means no cooperative matrix multiplication ("tensor cores"). For throughput in machine learning inference in particular, there's no way D3D11 can compete with either CUDA or a more modern compute shader stack, such as one based on Vulkan 1.3.

They seemed exceedingly hard to use well but interestingly capable & full of promise. And they were made in a much more primitive software age.

I'd love to hear about what didn't work. OpenMP support seemed ok maybe but OpenMP is just a platform, figuring out software architectures that's mechanistically sympathetic to the system is hard. It would be so interesting to see what Xeon Phi might have been if we had Calcite or Velox or OpenXLA or other execution engine/optimizers that can orchestrate usage. The possibility of something like Phi seems so much higher now.

There's such a consensus around Phi tanking, and yes, some people came and tried and failed. But most of those lessons, of why it wasn't working (or was!) never survived the era, never were turned into stories & research that illuminates what Phi really was. My feeling is that most people were staying the course on GPU stuff, and that there weren't that many people trying Phi. I'd like more than the heresay heaped at Phi's feed to judge by.

Hence why " they botched it".

Intel is releasing a 288-core x86

This made me wonder a couple of things-

What kind of workloads and problems is that best suited for? It’s a lot of cores for a CPU, but for pure math/compute, like with AI training and inference and with graphics, 288 cores is like ~1.5% of the number of threads of a modern GPU, right? Doesn’t it take particular kinds of problems to make a 288 core CPU attractive?

I also wondered if the ratio of the highest core count CPU to GPU has been relatively flat for a while? Which way is it trending- which of CPUs or GPUs are getting more cores faster?

To be clear, you can probably make a 1000 core x86 machine, and those 1000 cores can probably even be pretty powerful. I don't doubt that. I think Azure even has crazy 8-socket multi-sled systems doing hundreds of cores, today. But this thread is about CUDA. Sierra Forest will get absolutely obliterated by a single A100 in basically any workload where you could reasonably choose between the two as options. I'm not saying they can't exist. Just that they will be (very) bad in this specific competition. I made an edit to my comment to reflect that.

But what you mention is important, and also a reason for the ultimate demise of e.g. Xeon Phi. Intel surely realized they could just scale their existing Xeon designs up-and-out further than expected. Like from a product/SKU standpoint, what is the point of having a 300 core Phi where every core is slow as shit, when you have a 100 core 4-socket Xeon design on the horizon, using an existing battle-tested design that you ship billions of dollars worth every year? Especially when the 300 core Xeon fails completely against the competition. By the time Phi died, they were already doing 100-cores-per-socket systems. They essentially realized any market they could have had would be served better by the existing Xeon line and by playing to their existing strengths.

288 Cores or Threads? Cuz to my knowledge AMD already has a 128 Core, 256 Thread Processor with the Epyc 9754

People argue for ROCm to support older cards because that is all they have accessible to them.

What they really need is to support the less expensive cards, of which the older cards are a large subset. There are a lot of people who will make contributions and fix bugs if they can actually use the thing. Some CS student at the university has to pay tuition and therefore only has an old RX570, and that isn't going to change in the next couple years, but that kind of student could fix some of the software bugs currently preventing the company from selling more expensive GPUs to large institutions. If the stack supported their hardware.

The way I see it, the whole point of ROCm support is being able to service the many users who have pre-existing AMD cards and nothing else available. If someone is going to rent a GPU, I don't need to bother with adding extra features for them, because they can just rent a CUDA-capable GPU instead.

I'm considering adding ROCm support for some ML-enabled tool - no matter if it's a commercial product or an open source library - the thing I need from AMD is to ensure that the ROCm support I make will work without hassle for these end-users with random old AMD gaming cards (because these are the only users who need the tool to have ROCm support), and if ROCm upstream explicitly drops support for some cards because AMD no longer regularly test it, well, the ML tool developers aren't going to do that testing for them either; that's simply AMD intentionally refusing to do even the bare minimum (do a lot of testing for a wide variety of hardware to fix compatibility issues) that I'd expect to be table stakes for saying that "AMD is doubling down on ROCm".

Well yeah. Before I go renting a super GPU in the cloud, I'd like to get my feet wet with the 5 year old but reasonably well specced AMD GPU (Vega 48) in my iMac...but I can't. It's more rational for me to get an fancy 2021 GPU or a Jetson and stick it in an enclosure or build a Linux box around it. At least I know CUDA is a mature ecosystem and is going to be around for a while, so whatever time I invest in it is likely to pay for itself.

I get your point about AMD not wanting to spend money on supporting old hardware, but how do they expect to build a market without a fan base?

CUDA was redesigned to follow C++'s memory model introduced in C++11, yet another compatibility pain point with other hardware vendors.

It's pretty much based on the same underlying featureset. Which is why trying to target Vulkan Compute from it is messy enough, whereas OpenCL is a natural target.

Now, but that isn't how it started after the OpenCL 3.0 reboot, where 3.0 == 1.0.

Also some of that work, we have to thank Codeplay for, before their acquistion from Intel.

No, further, return to sea. Exist as peaceful algae.

No, it's the other direction (away from monke): go toward open source :)

That's the way we do SaaS over here in Australia. Kicks tyres. She'll be right mate.

And on the way down, there was an unholy congress between SharePoint and OneDrive to create a new eldritch horror.

In my opinion the main issues are a layer up from that, it's more of a customer experience and brand building exercise to solve by actually improving the tools people use. Today NVIDIA is the default if you want to get work done and not have a lot of tooling issues. Nobody is going to forego that experience to work with flaky slow products from a late number three entrant but that's more or less the perception today. What's weird is they actually have the high ground in an important area: They ship more GPUs than just about everyone and have for a long time, their integrated graphics drivers are pretty good (virtualization!), and they have a deep bench of performance engineering talent. NVIDIA structurally has no way to reach the same volume. Intel with some investment of time could be the most productive platform for AI developers but it would take creativity that is hard in a big company and acknowledging that in that effort they're starting as a distant #3.

If it's easy why don't they support it instead of watch billions of dollars fly away I wonder. If whatever cuda program you build would run the same on intel or amd, do those offshoot projects really matter?