63 replies
2h54m
“The Heavy Press Program was a Cold War-era program of the United States Air Force to build the largest forging presses and extrusion presses in the world.” This ”program began in 1944 and concluded in 1957 after construction of four forging presses and six extruders, at an overall cost of $279 million. Six of them are still in operation today, manufacturing structural parts for military and commercial aircraft” [1].
$279mm in 1957 dollars is about $3.2bn today [2]. A public cluster of GPUs provided for free to American universities, companies and non-profits might not be a bad idea.
[1] https://en.m.wikipedia.org/wiki/Heavy_Press_Program
[2] https://data.bls.gov/cgi-bin/cpicalc.pl?cost1=279&year1=1957...
Doubtful that GPUs purchased today would be in use for a similar time scale. Govt investment would also drive the cost of GPUs up a great deal.
Not sure why a publicly accessible GPU cluster would be a better solution than the current system of research grants.
Totally agree. That doesn't mean it can't generate massive ROI.
Difficult to say this ex ante. On its own, yes. But it would displace some demand. And it could help boost chip production in the long run.
Those receiving the grants have to pay a private owner of the GPUs. That gatekeeping might be both problematic, if there is a conflict of interests, and inefficient. (Consider why the government runs its own supercomputers versus contracting everything to Oracle and IBM.)
It would be better that the government removes IP on such technology for public use, like drugs got generics.
This way the government pays 2'500 USD per card, not 40'000 USD or whatever absurd.
You want to punish NVIDIA for calling its shots correctly? You don't see the many ways that backfires?
No. But I do want to limit the amount we reward NVIDIA for calling the shots correctly to maximize the benefit to society. For instance by reducing the duration of the government granted monopolies on chip technology that is obsolete well before the default duration of 20 years is over.
That said, it strikes me that the actual limiting factor is fab capacity not nvidia's designs and we probably need to lift the monopolies preventing competition there if we want to reduce prices.
Why do you think these private entities are willing to invest the massive capital it takes to keep the frontier advancing at that rate?
Why wouldn't NVIDIA be a solid steward of that capital given their track record?
Because whether they make 100x or 200x they make a shitload of money.
The problem isn't who is the steward of the capital. The problem is that economically efficient thing to do for a single company is (given sufficient fab capacity, and a monopoly) to raise prices to extract a greater share of the pie at the expense of shrinking the size of the pie. I'm not worried about who takes the profit, I'm worried about the size of the pie.
It's not a certainty that they 'make a shitload of money'. Reducing the right tail payoffs absolutely reduces the capital allocated to solve problems - many of which are risky bets.
Your solution absolutely decreases capital investment at the margin, this is indisputable and basic economics. Even worse when the taking is not due to some pre-existing law, so companies have to deal with the additional uncertainty of whether & when future people will decide in retrospect that they got too large a payoff and arbitrarily decide to take it from them.
You can't just look at the costs to an action, you also have to look at the benefits.
Of course I agree I'm going to stop marginal investments from occurring into research into patent-able technologies by reducing the expect profit. But I'm going to do so very slightly because I'm not shifting the expected value by very much. Meanwhile I'm going to greatly increase the investment into the existing technology we already have, and allow many more people to try to improve upon it, and I'm going to argue the benefits greatly outweigh the costs.
Whether I'm right or wrong about the net benefit, the basic economics here is that there are both costs and benefits to my proposed action.
And yes I'm going to marginally reduce future investments because the same might happen in the future and that reduces expected value. In fact if I was in charge the same would happen in the future. And the trade-off I get for this is that society gets the benefit of the same actually happening in the future and us not being hamstrung by unbreachable monopolies.
I think you're shifting it by a lot. If the government can post-hoc decide to invalidate patents because the holder is getting too successful, you are introducing a substantial impact on expectations and uncertainty. Your action is not taken in a vacuum.
I think this is a much more speculative impact. Why will people even fund the improvements if the government might just decide they've gotten too large a slice of the pie later on down the road?
No the trade-off is that materially less is produced. These incentive effects are not small. Take for instance, drug price controls - a similar post-facto taking because we feel that the profits from R&D are too high. Introducing proposed price controls leads to hundreds of fewer drugs over the next decade [0] - and likely millions of premature deaths downstream of these incentive effects. And that's with a policy with a clear path towards short-term upside (cheaper drug prices). Discounted GPUs by invalidating nvidia's patents has a much more tenuous upside and clear downside.
[0]: https://bpb-us-w2.wpmucdn.com/voices.uchicago.edu/dist/d/312...
You have proposed state ownership of all successful IP. That is a massive change and yet you have demonstrated zero understanding of the possible costs.
Your claim that removing a profit motivation will increase investment is flat out wrong. Everything else crumbles from there.
You're massively increasing uncertainty.
Why would you expect it would ever happen again? What you want is an unrealized capital gains tax. Not to nuke our semiconductor industry.
there is no such thing as a lump-sum transfer, this will shift expectations and incentives going forward and make future large capital projects an increasingly uphill battle
So, if a private company is successful, you will nationalize its IP under some guise of maximizing the benefit to society? That form of government was tried once. It failed miserably.
Under your idea, we’ll try a badly broken economic philosophy again. And while we’re at it, we will completely stifle investment in innovation.
There was a post[0] on here recently about how the US went from producing woefully insufficient numbers of aircraft to producing 300k by the end of world war 2.
One of the things that the post mentioned was the meager profit margin that the companies made during this time.
But the thing is that this set the America auto and aviation industry up to rule the world for decades.
A government going to a company and saying 'we need you to produce this product for us at a lower margin thab you'd like to' isn't the end of the world.
I don't know if this is one of those scenarios but they exist.
[0] https://www.construction-physics.com/p/how-to-build-300000-a...
In the case of NVIDIA it's even more sneaky.
They are an intellectual property company holding the rights on plans to make graphic cards, not even a company actually making graphic cards.
The government could launch an initiative "OpenGPU" or "OpenAI Accelerator", where the government orders GPUs from TSMC directly, without the middleman.
It may require some tweaking in the law to allow exception to intellectual property for "public interest".
y'all really don't understand how these actions would seriously harm capital markets and make it difficult for private capital formation to produce innovations going forward.
If we have public capital formation, we don’t necessarily need private capital. Private innovation in weather modelling isn’t outpacing government work by leaps and bounds, for instance.
because it is extremely challenging to capture the additional value that is being produced by better weather forecasts and generally the forecasts we have right now are pretty good.
private capital is absolutely the driving force for the vast majority of innovations since the beginning of the 20th century. public capital may be involved, but it is dwarfed by private capital markets.
It’s challenging to capture the additional value and the forecasts are pretty good because of continual large-scale government investment into weather forecasting. NOAA is launching satellites! it’s a big deal!
Private nuclear research is heavily dependent on governmental contracts to function. Solar was subsidized to heck and back for years. Public investment does work, and does make a didference.
I would even say governmental involvement is sometimes even the deciding factor, to determine if research is worth pursuing. Some major capital investors have decided AI models cannot possibly gain enough money to pay for their training costs. So what do we do when we believe something is a net good for society, but isn’t going to be profitable?
To the extent these are incremental units that wouldn't have been sold absent the government program, it's difficult to see how NVIDIA is "harmed".
20-25 year old drugs are a lot more useful than 20-25 year old GPUs, and the manufacturing supply chain is not a bottleneck.
There's no generics for the latest and greatest drugs, and a fancy gene therapy might run a lot more than $40k.
Of course they won't. The investment in the Heavy Press Program was the initial build, and just citing one example, the Alcoa 50,000 ton forging press was built in 1955, operated until 2008, and needed ~$100M to get it operational again in 2012.
The investment was made to build the press, which created significant jobs and capital investment. The press, and others like it, were subsequently operated by and then sold to a private operator, which in turn enabled the massive expansion of both military manufacturing, and commercial aviation and other manufacturing.
The Heavy Press Program was a strategic investment that paid dividends by both advancing the state of the art in manufacturing at the time it was built, and improving manufacturing capacity.
A GPU cluster might not be the correct investment, but a strategic investment in increasing, for example, the availability of training data, or interoperability of tools, or ease of use for building, training, and distributing models would probably pay big dividends.
I don't think there's a shortage of capital for AI... probably the opposite
Of all the things to expand the scope of government spending why would they choose AI, or more specifically GPUs?
Look at it from the perspective of an elected official:
If it succeeds, you were ahead of the curve. If it fails, you were prudent enough to fund an investigation early. Either way, bleeding edge tech gives you a W.
There may however, be a shortage of capital for open source AI, which is the subject under consideration.
As for the why... because there's no shortage of capital for AI. It sounds like the government would like to encourage redirecting that capital to something that's good for the economy at large, rather than good for the investors of a handful of Silicon Valley firms interested only in their own short term gains.
Would you mind expanding on these options? Universal training data sounds intriguing.
Sure, just on the training front, building and maintaining a broad corpus of properly managed training data with metadata that provides attribution (for example, content that is known to be human generated instead of model generated, what the source of data is for datasets such as weather data, census data, etc), and that also captures any licensing encumbrance so that consumers of the training data can be confident in their ability to use it without risk of legal challenge.
Much of this is already available to private sector entities, but having a publicly funded organization responsible for curating and publishing this would enable new entrants to quickly and easily get a foundation without having to scrape the internet again, especially given how rapidly model generated content is being published.
there are many things i think are more capital constrained, if the government is trying to subsidize things.
A much better investment would be to (somehow) revolutionize production of chips for AI so that it's all cheaper, more reliable, and faster to stand up new generations of software and hardware codesign. This is probably much closer to the program mentioned in the top level comment: It wasn't to produce one type of thing, but to allow better production of any large thing from lighter alloys.
How about using some of that money to develop CUDA alternatives so everyone is not paying the Nvidia tax?
It seems like rocm is already fully ready for transformer inference, so you are just referring to training?
ROCm is buggy and largely undocumented. That’s why we don’t use it.
Either you port Tensorflow (Apple)[1] or PyTorch to your platform or you allow CUDA to run on your hardware (AMD) [2]. Companies are incentives to not have NVIDIA having a monopoly but the thing is that CUDA is a huge moat due to compatibility of all frameworks and everyone knows it. Also, all of the cloud or on premises providers use NVIDIA regardless.
[1] https://developer.apple.com/metal/tensorflow-plugin/ [2] https://www.xda-developers.com/nvidia-cuda-amd-zluda/
It would be probably cheaper to negate some IP. There are quite some projects and initiatives to make CUDA code run on AMD for example, but as far as I know, they all stopped at some point, probably because of fear of being sued into oblivion.
That's the kind of work that can come out of academia and open source communities when societies provide the resources required.
Please start with the Windows Tax first for Linux users buying hardware...and the Apple Tax for Android users...
The National Science Foundation has been doing this for decades, starting with the supercomputing centers in the 80s. Long before anyone talked about cloud credits, NSF has had a bunch of different programs to allocate time on supercomputers to researchers at no cost, these days mostly run out of the Office of Advanced Cyberinfrastruture. (The office name is from the early 00s) - https://new.nsf.gov/cise/oac
(To connect universities to the different supercomputing centers, the NSF funded the NSFnet network in the 80s, which was basically the backbone of the Internet in the 80s and early 90s. The supercomputing funding has really, really paid off for the USA)
This would be the logical place to put such a programme.
The DoE has also been a fairly active purchaser of GPUs for almost two decades now thanks to the Exascale Computing Project [0] and other predecessor projects.
The DoE helped subsidize development of Kepler, Maxwell, Pascal, etc along with the underlying stack like NVLink, NGC, CUDA, etc either via purchases or allowing grants to be commercialized by Nvidia. They also played matchmaker by helping connect private sector research partners with Nvidia.
The DoE also did the same thing for AMD and Intel.
[0] - https://www.exascaleproject.org/
As you've rightly pointed out, we have the mechanism, now let's fund it properly!
I'm in Canada, and our science funding has likewise fallen year after year as a proportion of our GDP. I'm still benefiting from A100 clusters funded by tax payer dollars, but think of the advantage we'd have over industry if we didn't have to fight over resources.
Yeah, the specific AI/ML-focused program is NAIRR.
https://nairrpilot.org/
Terrible name unless they low-key plan to make AI researchers' hair fall out.
Don't these public clusters exist today, and have been around for decades at this point, with varying architectures? In the sense that you submit a proposal, it gets approved, and then you get access for your research?
Not--to my knowledge--for the GPUs necessary to train cutting-edge LLMs.
All of the major cloud providers offer grants for public research https://www.amazon.science/research-awards https://edu.google.com/intl/ALL_us/programs/credits/research https://www.microsoft.com/en-us/azure-academic-research/
NVIDIA offers discounts https://developer.nvidia.com/education-pricing
eg. for Australia, the National Computing Infrastructure allows researchers to reserve time on:
- 160 nodes each containing four Nvidia V100 GPUs and two 24-core Intel Xeon Scalable 'Cascade Lake' processors.
- 2 nodes of the NVIDIA DGX A100 system, with 8 A100 GPUs per node.
https://nci.org.au/our-systems/hpc-systems
This is the most recent iteration of a national platform. They have tons of GPUs (and CPUs, and flash storage) hooked up as a Kubernetes cluster, available for teaching and research.
https://nationalresearchplatform.org/
The problem is that any public cluster would be outdated in 2 years. At the same time, GPUs are massively overpriced. Nvidia's profit margins on the H100 are crazy.
Until we get cheaper cards that stand the test of time, building a public cluster is just a waste of money. There are far better ways to spend $1b in research dollars.
What about dollar cost averaging your purchases of GPUs? So that you're always buying a bit of the newest stuff every year rather than just a single fixed investment in hardware that will become outdated? Say 100 million a year every year for 20 years instead of 2 billion in a single year?
The private companies buying hundreds of billions of dollars of GPUs aren't writing them off in 2 years. They won't be cutting edge for long. But that's not the point--they'll still be available.
I don't see how the current practice of giving a researcher a grant so they can rent time on a Google cluster that runs H100s is more efficient. It's just a question of capex or opex. As a state, the U.S. has a structual advantage in the former.
One assumes the U.S. government wouldn't be paying list price. In any case, the purpose isn't purely research ROI. Like the heavy presses, it's in making a prohibitively-expensive capital asset generally available.
Overall government doing anything is a bad idea. There are cases however where government is the only entity that can do certain things. These are things that involve military, law enforcement etc. Outside of this we should rely on private industry and for-profit industry as much as possible.
The American healthcare industry demonstrates the tremendous benefits of rigidly applying this mindset.
Why couldn’t law enforcement be private too? You call 911, several private security squads rush to solve your immediate crime issue, and the ones who manage to shoot the suspect send you a $20k bill. Seems efficient. If you don’t like the size of the bill, you can always get private crime insurance.
That’s not correct. The American health care system is an extreme example of where private organisations fail overall society.
I'd like to see big programs to increase the amount of cheap, clean energy we have. AI compute would be one of many beneficiaries of super cheap energy, especially since you wouldn't need to chase newer, more efficient hardware just to keep costs down.
Yeah this would be the real equivalent of the program people are talking about above. That an investing in core networking infrastructure (like cables) instead of just giving huge handouts to certain corporations that then pocket the money.....
What about distributed training on volunteer hardware? Is that feasible?
It is an exciting concept, there's a huge wealth of gaming hardware deployed that is inactive at most hours of the day. And I'm sure people are willing to pay well above the electricity cost for it.
Unfortunately, the dominant LLM architecture makes it relatively infeasible right now.
- Gaming hardware has too limited VRAM for training any kind of near-state-of-the-art model. Nvidia is being annoyingly smart about this to sell enterprise GPUs at exorbitant markups.
- Right now communication between machines seems to be the bottleneck, and this is way worse with limited VRAM. Even with data-centre-grade interconnect (mostly Infiniband, which is also Nvidia, smart-asses), any failed links tend to cause big delays in training.
Nevertheless, it is a good direction to push towards, and the government could indeed help, but it will take time. We need both a more healthy competitive landscape in hardware, and research towards model architectures that are easy to train in a distributed manner (this was also the key to the success of Transformers, but we need to go further).
So we'll have the government bypass markets and force the working class to buy toys for the owning class?
If anything, allocate compute to citizens.
If something like this were to become a reality, I could see something like "CitizenCloud" where once you prove that you are a US Citizen (or green card holder or some other requirement), you can then be allocated a number of credits every month for running workloads on the "CitizenCloud". Everyone would get a baseline amount, from there if you can prove you are a researcher or own a business related to AI then you can get more credits.
I just watched this 1950s DoD video on the heavy press program and highly recommend it: https://www.youtube.com/watch?v=iZ50nZU3oG8
It makes much more sense to invest in a next generation fab for GPUs than to buy GPUs and more closely matches this kind of project.
Great idea, too bad the DOE and NSF were there first.
The size of the cluster would have to be massive or else your job will be on the queue for a year. And even then what are you going to do downsize the resources requested so you can get in earlier? After a certain point it starts to make more sense to just buy your own xeons and run your own cluster.
Imagine if they made a data center with 1957 electronics that cost $279 million.
They probably won't be using it now because the phone in your pocket is likely more powerful. Moore law did end but data center stuff are still evolving order of magnitudes faster than forging presses.