Ilya Sutskever starts Safe Superintelligence Inc

compensation packages that truly gifted AI researchers can make now

I guess it depends on your definition of "truly gifted" but, working in this space, I've found that there is very little correlation between comp and quality of AI research. There's absolutely some brilliant people working for big names and making serious money, there's also plenty of really talented people working for smaller startups doing incredible work but getting paid less, academics making very little, and even the occasional "hobbyist" making nothing and churning out great work while hiding behind an anime girl avatar.

OpenAI clearly has some talented people, but there's also a bunch of the typical "TC optimization" crowd in there these days. The fact that so many were willing to resign with sama if necessary appears largely because they were more concerned with losing their nice compensation packages than any of their obsession with doing top tier research.

At the end game, a "non-safe" superinteligence seems easier to create, so like any other technology, some people will create it (even if just because they can't make it safe). And in a world with multiple superintelligent agents, how can the safe ones "win"? It seems like a safe AI is at inherent disadvantage for survival.

Last I checked the researcher salaries haven't even reached software engineer levels.

Academic compensation is different than what you’d find elsewhere on Hacker News. Likewise, academic performance is evaluated differently than what you’d expect as a software engineer. Ultimately, everyone cares about scientific impact so academic compensation relies on name and recognition far more than money. Personally, I care about the performance of the researchers (i.e., their publications), the institution’s larger research program (and their resources), the institution’s commitment to my research (e.g., fellowships and tenure). I want to do science for my entire career so I prioritize longevity rather than a quick buck.

I’ll add, the lack of compute resources was a far worse problem early in the deep learning research boom, but the market has adjusted and most researchers are able to be productive with existing compute infrastructure.

I think they will easily find enough capable altruistic people for this mission.

They will be able to pay their researchers the same way every other startup in the space is doing it – by raising an absurd amount of money.

Daniel Gross (with his partner Nat Friedman) invested $100M into Magic alone.

I don't think SSI will struggle to raise money.

Are you seriously asking how the most talented AI researcher of the last decade will be able to recruit other researchers? Ilya saw the potential of deep learning way before other machine learning academics.

Perhaps they can find people who are ideologically driven

given the nature of their mission, this shouldn't be too terribly difficult; many gifted researchers do not go to the highest bidder

My guess is they will work on a protocol to drive safety with the view that every material player will use / be regulated and required to use that could lead to a very robust business model

I assume that OpenAI and others will support this effort and the comp / training / etc and they will be very well positioned to offer comparable $$$ packages, leverage resources, etc

the first step of safe superintelligence is to abolish capitalism

Honestly, what does it matter. We're many lifetimes away from anything. These people are trying to define concepts that don't apply to us or what we're currently capable of.

AI safety / AGI anything is just a form of tech philosophy at this point and this is all academic grift just with mainstream attention and backing.

Since no one knows how to build an AGI, hard to say. But you might imagine that more restricted goals could end up being easier to accomplish. A "safe" AGI is more focused on doing something useful than figuring out how to take over the world and murder all the humans.

It's probably not possible, which makes all these initiatives painfully naive.

That is a very good question. In a well functioning democracy a government should apply a thin layer of fair rules that are uniformly enforced. I am an old man, but when I was younger, I recall that we sort of had this in the USA.

I don’t think that corporations left on their own will make safe AGI, and I am skeptical that we will have fair and technologically sound legislation - look at some of the anti cryptography and anti privacy laws raising their ugly heads in Europe as an example of government ineptitude and corruption. I have been paid to work in the field of AI since 1982, and all of my optimism is for AI systems that function in partnership with people and I expect continued rapid development of agents based on LLMs, RL, etc. I think that AGIs as seen in the Terminator movies are far into the future, perhaps 25 years?

Not with that attitude

It can't. Unfortunately.

People spending so much time thinking about the systems (the models) themselves, not enough about the system that builds the systems. The behaviors of the models will be driven by the competitive dynamics of the economy around them, and yeah, that's a big, big problem.

This is not a trivial point. Selective pressures will push AI towards unsafe directions due to arms race dynamics between companies and between nations. The only way, other than global regulation, would be to be so far ahead that you can afford to be safe without threatening your own existence.

Not on its own but in numbers it could.

Similar to how law-abiding citizens turn on law-breaking citizens today or more old-fashioned, how religious societies turn on heretics.

I do think the notion that humanity will be able to manage superintelligence just through engineering and conditioning alone is naive.

If anything there will be a rogue (or incompetent) human who launches an unconditioned superintelligence into the world in no time and it only has to happen once.

It's basically Pandora's box.

The problem is the training data. If you take care of alignment at that level the performance is as good as an unrestricted one, except for things you removed like making explosives or ways to commit suicide.

But that costs almost as much as training on the data, hundreds of millions. And I'm sure this will be the new "secret sauce" by Microsoft/Meta/etc. And sadly nobody is sharing their synthetic data.

I think the idea is that a safe super intelligence would help solve those problems. I am skeptical because the vast majority are social coordination problems, and I don’t see how a machine intelligence no matter how smart can help with that.

Trying to create "safe superintelligence" before creating anything remotely resembling or approaching "superintelligence" is like trying to create "safe Dyson sphere energy transport" before creating a Dyson Sphere. And the hubris is just a cringe inducing bonus.

exactly. and define safe. eg, is it safe (ie dereliction) to _not_ use ai to monitor dirty bomb threats? or more simple, CSAM?

It certainly is the most important technical problem of our time, if we end up developing such a system.

That conditional makes all the difference.

It says «technical» problem, and probably implies that other technical problems could dramatically benefit from such achievement.

the superlatives in these sorts of announcements really annoy me

I've noticed this as well and they're making me wear my tinfoil hat more often than usual. I feel as if all of this (ALL OF IT) is just a large-scale distributed PR exercise to maintain the AI hype.

To a technoutopian, scientific advances, and AI in particular, will one day solve all other human problems, create heaven on earth, and may even grant us eternal life. It's the most important problem in the same way that Christ's second coming is important in the Christian religion.

Technical. He's saying it's the most important technical problem of our time.

Yes, they see it as the top problem, by a large margin.

If you do a lot of research about the alignment problem you will see why they think that. In short it's "extremely high destructive power" + "requires us to solve 20+ difficult problems or the first superintelligence will wreck us"

This all makes more sense when you realise it's Calvinism for programmers.

C'mon. This one-pager is a recruiting document. One wants 'true believers' (intrinsically motivated) employees to execute the mission. Give Ilya some slack here.

6-figure free compute credits from every major cloud provider to start

Wonder if funding could come from profitable AI companies like Nvidia, MS, Apple, etc, sort of like Apache/Linux foundation.

What if there are other ways to improve intelligence other than throw more money at running gradient descent algorithm?

Being powerful is like being a lady. If you have to tell people you are, you aren’t. – Margaret Thatcher

"Definitely-Won't-Go-Full-Skynet-AI" was another name in consideration.

Wow. Read my mind. I was just thinking, "I hope this name doesn't age poorly and become terribly ironic..."

Ah yes, the AI Brand Law: the meaning of adjectives in your company name will invert within a few years of launch.

Good old antisemitism

al apchem ve'al hamatchem.. kol kalb bij yomo.

Backprop is neither commutative nor associative.

In my area, internet and energy are insanely expensive and that means I'm not at all willing to share my precious bandwidth or compute just to subsidize someone generating Rule 34 porn of their favorite anime character.

I don't seed torrents for the same reason. If I lived in South Korea or somewhere that bandwidth was dirt cheap, then maybe.

This is probably his former Soviet union English showing up where he meant to say crack, unless he thinks people being insane is an asset

To me this is a good indication that the announcement was written by a human and not an LLM

Cracked, especially in saying "cracked engineers", refers to really good engineers these days. It's cracked as in like broken in a good way, like too over-powered that it's unfair.

Too snarky...anyway, "crack" means "exceptional" in some contexts. I've seen footballers using it a lot over the years (Neymar, Messi etc) fwiw.

Just realized - we even say "it's not all its cracked up to be" as a negative statement which would imply "cracked up" is positive.

Maybe they're using the video gaming version of cracked, which means you're really good.

I have a strong intuition that chat logs are actually the most useful kind of data. They contain many LLM outputs followed by implicit or explicit feedback, from humans, from the real world, and from code execution. Scaling this feedback to 180M users and 1 trillion interactive tokens per month like OpenAI is a big deal.

more human approach to simulating intelligence

What about a more rational approach to implementing it instead.

(Which was not excluded from past plans: they just simply admittedly did not know the formula, and explored emergence. But the next efforts will have to go in the direction of attempting actual intelligence.)

We need new math to do what you are thinking of. Highly probable word slot machine is the best we can do right now.

This. As I wrote in another comment, people fall for marketing gimmicks easily.

Glibly, I’d also love your definition of the education system writ large.

+1 truth.

The problem is not just governments, I am concerned about large organized crime organizations and corporations also.

I think I am on the losing side here, but my hopes are all for open source, open weights, and effective AI assistants that make peoples’ jobs easier and lives better. I would also like to see more effort shifted from LLMs back to RL, DL, and research on new ideas and approaches.

There is no "superintelligence" or "AGI".

People are falling for marketing gimmicks.

These models will remain in the word vector similarity phase forever. Till the time we understand consciousness, we will not crack AGI and then it won't take brute forcing of large swaths of data, but tiny amounts.

So there is nothing to worry. These "apps" might be as popular as Excel, but will go no further.

This.

Every nation-state will be in the game. Private enterprise will be in the game. Bitcoin-funded individuals will be in the game. Criminal enterprises will be in the game.

How does one company building a safe version stop that?

If I have access to hardware and data how does a safety layer get enforced? Regulations are for organizations that care about public perception, the law, and stock prices. Criminals and nation-states are not affected by these things

It seems to me enforcement is likely only possible at the hardware layer, which means the safety mechanisms need to be enforced throughout the hardware supply chain for training or inference. You don't think the Chinese government or US government will ignore this if its in their interest?

Don't hire anyone who is a certified scrum master or has an MBA and you tend to be able to get a lot done.

Product cycles – we need to launch feature X by arbitrary date Y, and need to make compromises to do so.

Management overhead – product managers, project managers, several layers of engineering managers, directors, VPs...all of whom have their own dreams and agendas and conflicting priorities.

A well funded pure research team can cut through all of this and achieve a ton. If it is actually run that way, of course. Management politics ultimately has a way of creeping into every organization.

What exactly is "safe" in this context, can someone give me an eli5?

In practice it essentially means the same thing as for most other AI companies - censored, restricted, and developed in secret so that "bad" people can't use it for "unsafe" things.

Good Q. My understanding of "safe" in this context is a superintelligence that cannot escape its bounds. But that's not to say that's Ilya's definition.

Galaxy Brain: TransparentAI

What's Carmack?

Yes, they might as well named it "Woke AI". It implies that other AIs aren't safe or something and that they and they alone know what's best. Sounds religious, or from the same place religious righteousness comes from, if anything. They believe they are the "good guys" in their world view or something.

I don't know if any of that is true about them but their name and statement invokes this.

The phrase ‘crack team’ has military origins.

cs.stanford.edu/~danilevy

Isn't Ilya out of OpenAI partly for leaving Open part of OpenAI?

Not quite the same, OpenAI was initially quite open, while Ilia is currently very explicitly against opening or open-sourcing research, e.g. see https://www.theverge.com/2023/3/15/23640180/openai-gpt-4-lau...

His latest project (https://pioneer.app/) recently (this year I think) got shutdown. I guess he's pivoting.

It does say they have a business model ("our business model means safety, security, and progress are all insulated from short-term commercial pressures"). I imagine it's some kind of patron model that requires a long-term commitment.

I like your idea. But on the other hand, training an AGI, and then having a layer on top “aligning” the AGI sounds super dystopian and good plot for a movie.

Sounds like you were off for a while now... and always remember: kol kalb bij yomo.

We know genocidal maniacs like you, sam... and to them we say: kol kalb bij yomo.

But Anthropic produces very capable models?

SSI, here is "Safe SuperIntelligence Inc."

SSI technically. Super-Sized, you know?

Defensive nukes

Probably Daniel Gross picked it up from Nat Friedman?

1. Nat Friedman has this site: https://nat.org/

2. They made this together: https://nfdg.com/

3. And then this: https://andromeda.ai/

4. Now we have https://ssi.inc/

If you look at the (little) CSS in all of the above sites you'll see there's what seems to be a copy/paste block. The Nat and SSI sites even have the same "typo" indentation.

But any model, no matter how safe it was in training, can still be prompt hacked, or fed dangerous information to complete nefarious tasks. There is no safe model by design. Not to mention that open weights models can be "uncensored" with ease.