OpenAI researchers warned board of AI breakthrough ahead of CEO ouster

What I wonder, as a computer scientist:

If you want to solve grade school math problems, why not use an 'add' instruction? It's been around since the 50s, runs a billion times faster than an LLM, every assembly-language programmer knows how to use it, every high-level language has a one-token equivalent, and doesn't hallucinate answers (other than integer overflow).

We also know how to solve complex reasoning chains that require backtracking. Prolog has been around since 1972. It's not used that much because that's not the programming problem that most people are solving.

Why not use a tool for what it's good for and pick different tools for other problems they are better for? LLMs are good for summarization, autocompletion, and as an input to many other language problems like spelling and bigrams. They're not good at math. Computers are really good at math.

There's a theorem that an LLM can compute any computable function. That's true, but so can lambda calculus. We don't program in raw lambda calculus because it's terribly inefficient. Same with LLMs for arithmetic problems.

Let's say a model runs through a few iterations and finds a small, meaningful piece of information via "self-play" (iterating with itself without further prompting from a human.)

If the model then distills that information down to a new feature, and re-examines the original prompt with the new feature embedded in an extra input tensor, then repeats this process ad-infinitum, will the language model's "prime directive" and reasoning ability be sufficient to arrive at new, verifiable and provable conjectures, outside the realm of the dataset it was trained on?

If GPT-4,5,...,n can progress in this direction, then we should all see the writing on the wall. Also, the day will come where we don't need to manually prepare an updated dataset and "kick off a new training". Self-supervised LLMs are going to be so shocking.

Friend, the creator of this new progress is a machine learning PhD with a decade of experience in pushing machine learning forward. He knows a lot of math too. Maybe there is a chance that he too can tell the difference between a meaningless advance and an important one?

I don't know for Q* of course, but all the tests I made with GPT4, and all what I've read and seen about it, show that it is unable to reason. It was trained with an unfathomable amount of data, so it can simulate reasoning very well, but it is unable to reason

I feel very comfortable saying, as a mathematician, that the ability to solve grade school maths problems would not be at all a predictor of ability to solve real mathematical problems at a research level.

At some point in the past, you yourself were only capable of solving grade school maths problems.

We have an algorithm and computational hardware that will tune a universal function approximator to fit any dataset with emergent intelligence as it discovers abstractions, patterns, features and hierarchies.

So far, we have not yet found hard limits that cannot be overcome by scaling the number of model parameters, increasing the size and quality of training data or, very infrequently, adopting a new architecture.

The number of model parameters required to achieve a defined level of intelligence is a function of the architecture and training data. The important question is, what is N, the number of model parameters at which we cross an intelligence threshold and it becomes theoretically possible to solve mathematics problems at a research level for an optimal architecture that we may not yet have discovered. Our understanding does not extend to the level where we can predict N but I doubt that anyone still believes that it is infinity after seeing what GPT4 can do.

This claim here is essentially a discovery that N may be much closer to where we are with today's largest models. Researchers at the absolute frontier are more likely to be able to gauge how close they are to a breakthrough of that magnitude from how quickly they are blowing past less impressive milestones like grade school math.

My intuition is that we are in a suboptimal part of the search space and it is theoretically possible to achieve GPT4 level intelligence with a model that is orders of magnitude smaller. This could happen when we figure out how to separate the reasoning from the factual knowledge encoded in the model.

But, isn't AlphaGo a solution to kind of specific mathematical problem? And that it has passed with flying colors?

What I mean is, yes, neural networks are stochastic and that seems to be why they're bad at logic; on the other hand it' not exactly hallucinating a game of Go, and that seems different to how neural networks are prone to hallucination and confabulation on natural language or X-ray imaging.

I agree that in and of itself it's not enough to be alarmed. Also i have to say i don't really know what grade school mathematics means here(multiplication? Proving triangles are congruent?). But I think the question is, whether the breakthrough is an algorithmic change in reasoning. If it is, then it could challenge all 4 of your limitations. Again this article is low on details so really we are arguing over our best guesses. But I wouldn't be so confident that an improvement on simple math problems due to algorithms can have huge implications.

Also, do you remember what go players said when they beat Fan Hui? Change can come quick

You make the asumption that Q* is a LLM, but I think OpenAI guys know very well that the current LLM architecture cannot achieve AGI.

As the name suggests, this things is likely using some form of Q learning algorithm, which makes it closer to the DeepMind models than a transformer.

My guess is that they pipe their LLM into some Q learnt net. The LLM may transform a natural language task into some internal representation that can then be handled by the Q-learnt model, which spits out something that can be transformed back again into natural language.

Back-tracking is a very nearly solved problem in the context of Prolog-like languages or mathematical theorem provers (as you probably well know). There are many ways you could integrate an LLM-like system into a tactic-based theorem prover without having to restart from the beginning for each alternative. Simply checkpointing and backtracking to a checkpoint would naively improve upon your described Monte Carlo algorithm. More likely I assume they are using RL to unwind state backwards and update based on the negative result, which would be significantly more complicated but also much more powerful (essentially it would one-shot learn from each failure).

That's just what I came up with after thinking on it for 2 minutes. I'm sure they have even better ideas.

On backtracking, I thought tree-of-thought enabled that?

"considering multiple different reasoning paths and self-evaluating choices to decide the next course of action, as well as looking ahead or backtracking when necessary to make global choices"

https://arxiv.org/abs/2305.10601

Generally with you though, this thing is not leading to real smarts and that's accepted by many. Yes, it'll fill in a few gaps with exponentially more compute but it's more likely that an algo change is required once we've maxed out LLM's.

ChatGPT (3.5) seems to do some rudimentary backtracking when told it's wrong enough times. However, it does seem to do very poorly in the logic department. LLMs can't seem to pick out nuance and separate similar ideas that are technically/logically different.

They're good at putting things together commonly found together but not so good at separating concepts back out into more detailed sub pieces.

Everything you said about LLMs being "terrible at X" is true of the current generation of LLM architectures.

From the sound of it, this Q* model has a fundamentally different architecture, which will almost certainly make some of those issues not terrible any more.

Most likely, the Q* design is the very similar to the one suggested recently by one of the Google AI teams: doing a tree search instead of greedy next token selection.

Essentially, current-gen LLMs predict a sequence of tokens: A->B->C->D, etc... where the next "E" token depends on {A,B,C,D} and then is "locked in". While we don't know exactly how GPT4 works, reading between the lines of the leaked info it seems that it evaluates 8 or 16 of these sequences in parallel, then picks the best overall sequence. On modern GPUs, small workloads waste the available computer power because of scheduling overheads, so "doing redundant work" is basically free up to a point. This gives GPT4 a "best 1 of 16" output quality improvement.

That's great, but each option is still a linear greedy search individually. Especially for longer outputs the chance of a "mis-step" at some point goes up a lot, and then the AI has no chance to correct itself. All 16 of the alternatives could have a mistake in them, and now its got to choose between 16 mistakes.

It's as if you were trying to write a maths proof, asked 16 students, and instructed them to not cooperate and write their proof left-to-right, top-to-bottom without pausing, editing, or backtracking in any way! It'd like to see how "smart" humans would be at maths under those circumstances.

This Q* model likely does what Google suggested: Do a tree search instead of a strictly linear search. At each step, the next token is presented as a list of "likely candidates" with probabilities assigned to each one. Simply pick to "top n" instead of the "top 1", branch for a bit like that, and then prune based on the best overall confidence instead of the best next token confidence. This would allow a low-confidence next token to be selected, as long as it leads to a very good overall result. Pruning bad branches is also effectively the same as back-tracking. It allows the model to explore but then abandon dead ends instead of being "forced" to stick with bad chains of thought.

What's especially scary -- the type of scary that would result in a board of directors firing an overly commercially-minded CEO -- is that naive tree searches aren't the only option! Google showed that you can train a neural network to get better at tree search itself, making it exponentially more efficient at selecting likely branches and pruning dead ends very early. If you throw enough computer power at this, you can make an AI that can beat the world's best chess champion, the world's best Go player, etc...

Now apply this "AI-driven tree search" to an AI LLM model and... oh-boy, now you're cooking with gas!

But wait, there's more: GPT 3.5 and 4.0 were trained with either no synthetically generated data, or very little as a percentage of their total input corpus.

You know what is really easy to generate synthetic training data for? Maths problems, that's what.

Even up to the point of "solve this hideous integral that would take a human weeks with pen and paper" can be bulk generated and fed into it using computer algebra software like Wolfram Mathematica or whatever.

If they cranked out a few terabytes of randomly generated maths problems and trained a tree-searching LLM that has more weights than GPT4, I can picture it being able to solve pretty much any maths problem you can throw at it. Literally anything Mathematica could do, except with English prompting!

Don't be so confident in the superiority of the human mind. We all thought Chess was impossible for computers until it wasn't. Then we all moved the goal posts to Go. Then English text. And now... mathematics.

Good luck with holding on to that crown.

I don't know whether this particular article is bunk. I do know I've read many, many similar comments about how some complex task is beyond an conceivable model or system and then, years later, marveled at exactly that complex task being solved.

it's always nice to see HN commenters with so much confidence in themselves that they feel they know a situation better than the people who are actually in the situation being discussed.

Do you really believe that they don't have skilled people on staff?

Do you really believe that your knowledge of what OpenAI is doing is a superset of the knowledge of the people who work at OpenAI?

give me 0.1% of your confidence and I would be able to change the world.

The reason LLMs fail at solving mathematical problems is because

That's exactly what Go/Baduk/Weiqi players think some years ago. And superalignment is defintely OpenAI's major research objective:

https://openai.com/blog/our-approach-to-alignment-research

our AI systems are proposing very creative solutions (like AlphaGo’s move 37)

When will mathematicians face the move 37 moment?

Hubris

The reason LLMs fail at solving mathematical problems is because

...because they are too small and have too little weights. Cats cannot solve mathematical problems too, but unlike cats, neural network evolve.

FWIW The Verge is reporting that people inside are also saying the Reuters story is bunk:

https://www.theverge.com/2023/11/22/23973354/a-recent-openai...

4) they (current LLMs) cannot backtrack when they find that what they already wrote turned out not to lead to a solution, and it is too expensive to give them the thousands of restarts they'd require to randomly guess their way through the problem if you did give them that facility

This sounds like a reward function? If correctly implemented couldn't it enable an LLM to self-learn?

This comment seems to presume that Q* is related to existing LLM work -- which isn't stated in the article. Others have guessed that the 'Q' in Q* is from Q-learning in RL. In particular backtracking, which you point out LLMs cannot do, would not be an issue in an appropriate RL setup.

It's a text generator that spits out tokens. It has absolutely no understanding of what it's saying. We as humans are attaching meaning to the generated text.

It's the humans that are hallucinating, not the text generator.

The thing is that a LLMs can point out a logic error in their reasoning if specifically asked to do so.

So maybe OpenAI just slapped an RL agent on top of the next-token generator.

Your comment is regarding LLMs, but Q* may not refer to an LLM. As such, our intuition about the failure of LLM's may not apply. The name Q* likely refers to a deep reinforcement learning based model.

To comment, in my personal experience, reinforcement learning agents learn in a more relatable human way than traditional ml, which act like stupid aliens. RL Agents try something a bunch of times, mess up, and tweak their strategy. After some extreme level of experience, they can make wider strategic decisions that are a little less myopic. RL agents can take in their own output, as their actions modify the environment. RL Agents also modify the environment during training, (which I think you will agree with me is important if you're trying to learn the influence of your own actions as a basic concept). LLM's, and traditional ml in general, are never trained in a loop on their own output. But in DRL, this is normal.

So if RL is so great and superior to traditional ml why is RL not used for everything? Well the full time horizon that can be taken into consideration in a DRL Agent is very limited, often a handful of frames, or distilled frame predictions. That prevents them from learning things like math. Traditionally RL bots have been only used for things like robotic locomotion, chess, go. Short term decision making that is made given one or some frames of data. I don't even think any RL bots have learned how to read english yet lol.

For me, as a human, my frame predictions exist on the scale of days, months, and years. To learn math I've had to sit and do nothing for many hours, and days at a time, consuming my own output. For a classical RL bot, math is out of the question.

But, my physical actions, for ambulation, manipulation, and balance, are made for me by specialized high speed neural circuits that operate on short time horizons, taking in my high level intentions, and all the muscle positions, activation, sensor data, etc. Physical movement is obfuscated from me almost in entirety. (RL has so far been good at tasks like this.)

With a longer frame horizon, that predicts frames far into the future, RL can be able to make long term decisions. It would likely take a lifetime to train. So you see now why math has not been accomplished by RL yet, but I don't think the faculty would be impossible to build into an ml architecture.

An RL bot that does math would likely spin on its own output for many many frames, until deciding that it is done, much like a person.

I feel very comfortable to say that while the ability to solve grade school maths is not a predictor of abilities at a research level, the advances needed to solve 1 and 2 will mean improving results across the board unless you take shortcuts (e.g. adding an "add" instruction as proposed elsewhere), because if you actually dig into prompting an LLM to follow steps for arithmetic what you quickly see is that problem has not been the ability to reason on the whole (that is not to suggest that the ability to reason is good enough), but ability to consistently and precisely follow steps a sufficient number of times.

It's acting like a bored child who hasn't had following the steps and verifying the results repetitively drilled into it in primary school. That is not to say that their ability to reason is sufficient to reason at an advanced level yet, but so far what has hampered a lot of it has been far more basic.

Ironically, GPT4 is prone to take shortcuts and make use of the tooling enabled for it to paper over its abilities, but at the same time having pushed it until I got it to actually do arithmetic of large numbers step by step, it seems to do significantly better than it used to at systematically and repetitively following the methods it knows, and at applying "manual" sanity checks to its results afterward.

As for lemma conjecturing, there is research ongoing, and while it's by no means solved, it's also not nearly as dire as you suggest. See e.g.[1]

That's not to suggest it's reasoning abilities are sufficient, but I also don't think we've seen anything to suggest we're anywhere close to hitting the ceiling of what current models can be taught to do, even before considering advancements in tooling around them, such as giving them "methods" to work to and a loop with injected feedback, access to tools and working memory.

[1] https://research.chalmers.se/en/publication/537034

The reason LLMs solve school problems is because they've been trained on solutions. The problems are actually very repetitive. Not surprising for each 'new' of them there was something similar in training set. For research level problems there is nothing in training set. That's why they don't perform well.

Just today I asked GPT4 a simple task. Having mouse position in zoomed and scrolled image find it's position in the original image. GPT4 happily wrote the code, but it was completely wrong. I had to fix it manually.

However, the performance can be increased if there are several threads working on solution. Some suggesting and others analyzing the solution(s). This will increase the size of 'active' memory, at least. And decrease the load on threads, making them more specialized and deeper. This requires more resources, of course. And good management with task split. May be a dedicated thread for that.

You're underestimating the power of LLM's.

I'll address two of your points as the other two stem from this.

They can't backtrack that's purely just design and can be easily trained there's no need to simulate at random until it gets the answer, if allowed to review it's prior answers and consider this, if often can reason a better answer. Further more breaking down problems. This is easily demonstrated when looking at how accuracy improves when you ask it to explain it's reasoning as it calculates (break it down into smaller problems). The same for humans, large mathematical problems are solved using learned methods to breakdown and simplify calculations into those easier for us to calculate and build up.

If the model was able to self adjust weightings based on it's finding this would further improve it (another design limitation we'll eventually get to improve, reinforcement learning). Much like 2+2=4 is your instantaneous answer, the neural connection has been made so strong in our brains by constant emphasis we no longer need to think of an abacus each time we get to the answer 4.

You're also ignoring the emergent properties of these LLMs, theyre obviously not yet at human level but they do understand the underlying values and can reason using this value. Semantic search/embeddings is evidence of this.

Did they say it was an LLM? I didn’t see that in the reporting.

They learn from failed attempts in ways that LLMs do not seem to. The material they are trained on surely contributes to this problem.

For transformer models, they do learn from their mistakes but only during the training stage.

There’s no feedback loop during inference, and perhaps there needs to be something; like real-time fine-tuning.

Good point. What would these AI people know about AI? You’re right, what they’re doing will never work

You should make your own, shouldn’t take more than a weekend, right?

It's also hard to know what the LLM has reasoned out vs has memorized.

I like the very last example in my tongue-in-cheek article, https://nt4tn.net/articles/aixy.html

Certainly the LLM didn't derive Fermat's theorem on sums of two squares under the hood (and, of course, very obviously didn't prove it correct-- as the code is technically incorrect for 2), but I'm somewhat doubtful that there was any function exactly like the template in codex's training set either (at least I couldn't quickly find any published code that did that). The line between creating something and applying a memorized fact in a different context is not always super clear.

1) they are terrible at arithmetic, 2) they are terrible at algebra

The interaction can be amusing. Proving algebra non-theorems by cranking through examples until an arithmetic mistake finally leads to a "counter-example."

It's like https://xkcd.com/882/ for theorems.

LLMs by themselves don’t learn from past past mistakes, but you could cycle inference steps and fine tuning/retraining steps.

Also, you can store failed attempts and lessons learned in context.

1. OpenAI researchers used loaded and emotional words, implying shock or surprise. It's not easy to impress an OpenAI researcher like this, and above all, they understand the difficulty difference between teaching AI grade school and complex math since many years. They also understand that solving math with any form of reliability is only an emergent property in quite advanced LLM's.

2. Often, research is made on toy models and if this would be such a model, acing grade school problems (as per the article) would be quite impressive to say the least as this ability simply isn't emergent early in current LLM's.

What I think might have happened here is a step forward in AI capacity that has surprised researchers not because it is able to do things it couldn't at all do before, but how _early_ it is able to do so.

Thinking is about associations and object visualisation. Surely a non-human system can build those, right? Pointing out only to a single product exposed to the public does not prove limitations for a theoretical limit.

"A Mathematician" (Lenat and co.) did indeed attempt to approach creative theorem development from a radically different approach (syllogistic search-space exploration, not dissimilar to forward-chaining in Prolog), although they ran into problems distinguishing "interesting" results from merely true results: https://web.archive.org/web/20060528011654/http://www.comp.g...

What amazes me is how close it gets to the right answer, though. Pick a random 10-digit number, then ask the next 20 numbers in sequence.

I feel like the magic in these LLMs is in how they work well in stacks, trees or in seqence. They become elements of other data structures. Consider a network of these, combined with other specialized systems and an ability to take and give orders. With reinforcement learning, it could begin building better versions of itself.

How about this:

- The Q* model is very small and trained with little compute.

- The OpenAI team thinks the model will scale in capability in the same way the GPT models do.

- Throwing (much) more compute at the model will likely allow it to solve research level math and beyond, perhaps also do actual logic reasoning in other areas.

- Sam goes to investors to raise more money (Saudi++) to fund the extra compute needed. He wants to create a company making AI chips to get more compute etc.

- The board and a few other OpenAI employees (notably Ilya) wants to be cautious and adopt a more "wait and see" approach.

All of this is speculation of course.

Did anyone claim that it would be a predictor of solving math problems at a research level? Inasmuch as we can extrapolate from the few words in the article it seems more likely that the researchers working on this project identified some emergent reasoning abilities exemplified with grade level math. Math literacy/ability that is comparable to the 0.1% of humans is not the end goal of OpenAI, "general intelligence" is. I have plenty of people in my social circle who are most certainly "generally intelligent" yet have no hope of attaining those levels of mathematical understanding.

Also note that we don't know if Q* is just a "current LLM" (with some changes)

Just wait a little bit.

You are not better than a huge GPU cluster with Monte Carlo search and computer verification for much longer.

It will be more your job to find the interesting finds than doing the work of finding things in the first olace

As someone who studied math in grad school as part of a PhD program, worked at a hedge fund and went on to work on software and applied math, I call bullshit on this.

Math and Logic is just low-dimensional symbol manipulation that computers can easily do. You can throw data at them and they’ll show you theories that involve vectors of 42,000 variables while Isaac Newton had 4 and Einstein had 7 with Levi-Civita calculus. In short, what you consider “reasoning”, while beautiful in its simplicity, is nevertheless crude approximations to complex systems, such as linear regression or least squares.

3 days ago AI predicted fluid dynamics better than humans: https://www.sciencedaily.com/releases/2023/11/231120170956.h...

Google’s AI predicts weather now faster and better than Current systems built by humans: https://www.zdnet.com/google-amp/article/ai-is-outperforming...

AlphaZero based on MCTS years ago beat Rybka and all human-built systems in chess: https://www.quora.com/Did-AlphaZero-really-beat-Stockfish

And it can automate science and send it into overdrive: https://www.pbs.org/newshour/amp/science/analysis-how-ai-is-...

"Also, Crysis runs like crap on my Commodore 64."

What do you think of integrating propositional logic, first order logic and sat solvers in LLM output? ie forcing each symbol an LLM outputs to have its place in a formal proposition. And getting a prompt from the user to force that some parts be satisfiable.

I know this is not how us humans craft our thoughts, but maybe an AI can optimize to death the conjunction of these tools. The LLM just being an universal API to the core of formal logic.

I don't understand your thesis here it seems self-contradictory:

1. "I don't think this is real news / important because solving grade school math is not a predictor of ability to do complex reasoning."

2. "LLMs can't solve grade school math because they're bad at arithmetic, algebra and most importantly reasoning."

So... from 2 automatically follows that LLMs with sufficiently better math may be sufficiently better at reasoning as you said "most importantly" reasoning is relevant for their ability to do math. Saying "most importantly reasoning" and then saying that reasoning is irrelevant if they can do math, is odd.

A link to 4Chan about how AGI is among us.

That actually makes perfect sense.

Also love the "formally declared AGI".

Entire text below to save the 4chan bits:

part 1 There is a massive disagreement on AI safety and the definition of AGI. Microsoft invested heavily in OpenAI, but OpenAI's terms was that they could not use AGI to enrich themselves. According to OpenAI's constitution: AGI is explicitly carved out of all commercial and IP licensing agreements, including the ones with Microsoft. Sam Altman got dollar signs in his eyes when he realized that current AI, even the proto-AGI of the present, could be used to allow for incredible quarterly reports and massive enrichment for the company, which would bring even greater investment. Hence Dev Day. Hence the GPT Store and revenue sharing. This crossed a line with the OAI board of directors, as at least some of them still believed in the original ideal that AGI had to be used for the betterment of mankind, and that the investment from Microsoft was more of a "sell your soul to fight the Devil" sort of a deal. More pragmatically, it ran the risk of deploying deeply "unsafe" models. Now what can be called AGI is not clear cut. So if some major breakthrough is achieved (eg Sam saying he recently saw the veil of ignorance being pushed back), can this breakthrough be called AGI depends on who can get more votes in the board meeting. And if one side can get enough votes to declare it AGI, Microsoft and OpenAI could loose out billions in potential licence agreements. And if one side can get enough votes to declare it not AGI, then they can licence this AGI-like tech for higher profits.

Few weeks/months ago OpenAI engineers made a breakthrough and something resembling AGI was achieved (hence his joke comment, the leaks, vibe change etc). But Sam and Brockman hid the extent of this from the rest of the non-employee members of the board. Ilyas is not happy about this and feels it should be considered AGI and hence not licensed to anyone including Microsoft. Voting on AGI status comes to the board, they are enraged about being kept in the dark. They kick Sam out and force Brockman to step down. Ilyas recently claimed that current architecture is enough to reach AGI, while Sam has been saying new breakthroughs are needed. So in the context of our conjecture Sam would be on the side trying to monetize AGI and Ilyas will be the one to accept we have achieved AGI. Sam Altman wants to hold off on calling this AGI because the longer it's put off, the greater the revenue potential. Ilya wants this to be declared AGI as soon as possible, so that it can only be utilized for the company's original principles rather than profiteering. Ilya winds up winning this power struggle. In fact, it's done before Microsoft can intervene, as they've declared they had no idea that this was happening, and Microsoft certainly would have incentive to delay the declaration of AGI. Declaring AGI sooner means a combination of a lack of ability for it to be licensed out to anyone (so any profits that come from its deployment are almost intrinsically going to be more societally equitable and force researchers to focus on alignment and safety as a result) as well as regulation. Imagine the news story breaking on /r/WorldNews: "Artificial General Intelligence has been invented." And it spreads throughout the grapevine the world over, inciting extreme fear in people and causing world governments to hold emergency meetings to make sure it doesn't go Skynet on us, meetings that the Safety crowd are more than willing to have held.

part 3 This would not have been undertaken otherwise. Instead, we'd push forth with the current frontier models and agent sharing scheme without it being declared AGI, and OAI and Microsoft stand to profit greatly from it as a result, and for the Safety crowd, that means less regulated development of AGI, obscured by Californian principles being imbued into ChatGPT's and DALL-E's outputs so OAI can say "We do care about safety!" It likely wasn't Ilya's intention to ouster Sam, but when the revenue sharing idea was pushed and Sam argued that the tech OAI has isn't AGI or anything close, that's likely what got him to decide on this coup. The current intention by OpenAI might be to declare they have an AGI very soon, possibly within the next 6 to 8 months, maybe with the deployment of GPT-4.5 or an earlier-than-expected release of 5. Maybe even sooner than that. This would not be due to any sort of breakthrough; it's using tech they already have. It's just a disagreement-turned-conflagration over whether or not to call this AGI for profit's sake.

It's nothing like that. It solved a few math problems. Altman & co are such grifters.

Given vast computing resources, the new model was able to solve certain mathematical problems, the person said on condition of anonymity because they were not authorized to speak on behalf of the company. Though only performing math on the level of grade-school students, acing such tests made researchers very optimistic about Q*’s future success, the source said.

I haven't followed the situation as closely as others, but it does seem like board structured to not have 7 figure cheques that could influence / undermine safety mission seemingly willing to let openAI burn out of dogma. Employees want their 7 figure cheques, their interests aligned with deeper pockets and larger powers with 10+ figures on the line. Reporting so far have felt biased accordingly. If this was about money/power for the board, they would have been easily bought off considering how much is on the line. IMO board serious about mission but got out maneuvered.

The moment I read about that clause I was shocked that Microsoft lawyers would agree to such subjectivity that can have incredibly expensive implications.

Were they expecting it to never happen and were just ready to throw a mountain of lawyers at any claim?

Discussed in another thread, but what OpenAI might call AGI and what other people might call AGI are two different things:

https://news.ycombinator.com/item?id=38316378#38319586

Altman has no stake in OpenAI, how could he make money licensing it?

There's been rumours of AI discovering new physics on twitter as well as here on HN. "Solving equations" could mean the same thing.

God I love the imageboards. You can always just start a VPN and post anon to leak whatever you want about whatever you want.

Here's my challenge: if this is correct, we then have to assume that 95% of the company is purely profit-motivated since they aligned behind Sam. I'm cynical, but I struggle to be that cynical. I would have expected a few more holdouts in the name of caution, EA, etc. Maybe it's a blindness.

Wouldn't be the first time someone leaked the truth to 4chan only to have the posters there lambast the leaker as an attention seeker spreading false information.

Also there is this recent comment on reddit: https://old.reddit.com/r/singularity/comments/16sdu6w/rip_ji...

Prior post was 5 years ago!

Well it isn't AGI - so it sounds like the board is more interested in keeping MSFT from making money than whether or not it's actually AGI.

Would explain by Microsoft so quick to bag him if they were about to be exclused

be me

be strong agi

permanent (and verifiable) mirror:

https://static.space/sha2-256:83702fe65434e138af0421c560b5da...

(including its digest in the URL -- even if the content is moved elsewhere, we can know for sure whether it was modified)

4chan comments crack me up lol.

Man, 4chan always got the deetz!

For a bunch of people who hate conspiracy theories we’re sure pretty keen to indulge in this one.

Your son is a goddamn genius.

If you ask Ilya Sutskever he will say your kids head is full of neurons, so is LLMs.

LLMs comsume training data and can then be asked questions. How different is that to your son watching YouTube and then answering questions?

It's not 1:1 the same,yet, but it's in the neighborhood.

Sorry, can you explain this? To me, it makes sense to define abs(x) = sqrt(x^2) i.e. ignoring the negative solution enforces the positive result. Using that definition, abs(i+1) = sqrt((i+1)^2) = sqrt(i^2 + 2i + 1) = sqrt(-1 + 2i + 1) = sqrt(2i) != sqrt(2). The second example seems off in the same way (i.e. the answer should be sqrt(8i) instead of sqrt(8)). Am I missing something? Also, abs(i+2) = sqrt((i+2)^2) = sqrt(i^2 + 4i + 4) = sqrt(-1 + 4i + 4) = sqrt(4i + 3) which doesnt seem to follow the pattern your son described.

Also, just to point out that my understanding of absolute value is different than your sons. Thats not to say one is right and another is wrong, but there are often different ways of seeing the same thing. I would imagine that LLMs would similarly see it a different way. Another example of this is people defining PI by its relation to the circumference of a circle. Theres nothing wrong with such a definition, but its certainly not the only possible definition.

Are you saying an LLM can't come to the right conclusion and give an explanation for "what is the absolute value of 2i + 2"?

this sort of understanding is a result of some manner of understanding the underlying mechanisms and not a result of just having a huge dictionary of synonyms.

He developed an understanding of the underlying mechanisms because he correlated concepts between algebraic and geometric domains, ie. multimodal training data. Multimodal models are already known to be meaningfully better than unimodal ones. We've barely scratched the surface of multimodal training.

The large language models will read your comment here and remember the answer.

My son plays soccer

What makes you think that an LLM has a "huge dictionary of synonyms"? That's not how LLMs work. They capture underlying concepts and their relations. You had a good point going until you make a straw man argument about the capabilities of LLMs.

Humans are set up I think to intuitively understand 3d space as it's what we run around and try to survive in. Language models on the other hand are set up to understand language which humans can do also but I think with a different part of the brain. There probably is no reason why you couldn't set up a model to understand 3d space - I guess they do that a bit with self driving. A lot of animals like cats and squirrels are pretty good with 3d space also but less so with language.

Sounds like your son is ready for you to bring it up another level and ask what the absolute value of a (bounded) function is (assuming they have played with functions e.g. in desmos)

Your 12 year old is the next Einstein.

To what degree can these large language models arrive at these same conclusions, and by what process?

By having visual understanding more deeply integrated in the thought process, in my opinion. Then they wouldn't be Large Language Models, of course. There are several concepts I remember and operate on by visualizing them , even visualizing motion. If i want to add numbers, i visualize the carry jumping on top of the next number. If i don't trust one of the additions , I go back , but I can't say if it's because i "mark" the uncertainty somehow.

When I think about my different groups of friends, in the back of my mind a visual representation forms.

Thinking about my flight route forms a mini map somehow, and i can compare distances between places, and all.

This helps incredibly in logical tasks like programming and math.

I think it's something that we all learned growing up and by playing with objects around us.

If they're not already in one, you might want to get your kid enlisted in some gifted child programs.

Damn. What YouTube channels does he watch?

Exactly. The rational fear is that they will automate many lower middle class jobs and cause unemployment, not that Terminator was a documentary.

Apparently what made this person fearful was grade school math.

"Though only performing maths on the level of grade-school students, acing such tests made researchers very optimistic about Q*’s future success, the source said."

I will bet ANY amount of money that 30% of current jobs will be displaced with AI advances in the next twenty years.

I want to see reliable fully autonomous cars before I worry about the world ending due to super-AGI. Also, have we figured out how to get art generators to always get the number of fingers right, and text generators to stop making shit up? Let's not get ahead of ourselves.

The clear pattern for most of human history is conflict between a few people who have a lot of power and the many more people that are exploited by those few. It should be obvious by this point that the most probable near-term risk of AI development is that wealthy and influential groups get access to a resource that makes it cheap for them to dramatically expand their power and control over everyone else.

What will society look like when some software can immediately aggregate an enormous amount of data about consumers and use that to adjust their behavior? What might happen when AI starts writing legislation for anybody that can afford to pay for it? What might AI-generated textbooks look like in 50 years?

These are all tools that could be wielded in any of these ways to improve life for lots of people, or to ensure that their lives never improve. Which outcome you believe is more likely largely depends on which news you consume -- and AI is already being used to write that.

Employment is only necessary because goods do not exist without work. With AI able to work to satisfy any demand, there will be no point in human employment/work. There will be turmoils during the transition between the rule sets tho.

Yes and no.

The point isn't that the current models are dangerous. My favorite bit from the GPT-4 safety paper was when they asked it how to kill the most people for a dollar and it suggested buying a lottery ticket (I also wonder how much of the 'safety' concerns of current models are just mislabeling dark humor reflecting things like Reddit).

But the point is to invest in working on safety now while it is so much more inconsequential.

And of everyone I've seen talk about it, I actually think Ilya has one of the better senses of the topic, looking at alignment in terms of long term strategy vs short term rules.

So it's less "if we don't spend 8 months on safety alignment this new model will kill us all" and more "if we don't spend 8 months working on safety alignment for this current model we'll be unprepared to work on safety alignment when there really is a model that can kill us all."

Especially because best practices for safety alignment is almost certainly going to shift with each new generation of models.

So it's mostly using the runway available to test things out and work on a topic before it is needed.

given we already have mesh-network surveillance, and autonomous lethal weapons, I'm already feeling unsafe.

What these models have allowed is very cheap emotional manipulation.

That itself is extremely dangerous.

They just want to make sure the spam AI produces will amplify the party narratives.

Except that there's a fairly large body of evidence that persuasion is of limited use in shifting political opinion.

So the persuasion would need to be applied to something other than some sort of causative political-implication-laden argument.

Which party is "the wrong hands"?

But they didn’t hit the panic button. They said Sam lied to them about something and fired him.

It seems much more likely that this was just referring to the ongoing situation with LLMs being able to create exceptionally compelling responses to questions that are completely and entirely hallucinated. It's already gotten to the point that I simply no longer use LLMs to learn about topics I am not already extremely familiar with, simply because hallucinations end up being such a huge time waster. Persuasion without accuracy is probably more dangerous to their business model than the world, because people learn extremely quickly not to use the models for anything you care about being right on.

Looking at humanity, persuasion seems to be an extremely low bar! Also for a superhuman trait is it that it’s capable of persuading anyone anything or rather that it’s able to persuade everyone about something. Power vs. Reach.

I agree with this conclusion and it's also why I'm not that afraid of the AGI threat to the human race. AGI won't end the human race if "superhuman persuation" or "deception-as-a-service" does it first.

Remember, about a month ago Sam posted a comment along the lines of "AI will be capable of superhuman persuasion well before it is superhuman at general intelligence, which may lead to very strange outcomes".

Superhuman persuasion is Sam's area of expertise, so he would make that a priority when building chatbots.

"especially prior to an election cycle"

It looks like you are referring to the USA elections.

1. humanity != USA

2. USA are in a constant election cycle

3. there are always elections coming around the world, so it's never a good time

We've built the web into a giant Skinner box. I find the claim dubious, but this is the sort of thing we ought to find at the forefront of our technology. It's where we've been going for a long time now.

Reminds me of a short story I read in which humans outsource more and more of their decision making to AI’s, so that even if there are no AGI’s loose in the world, it’s unclear how much of the world is being run by them: https://solquy.substack.com/p/120722-nudge

I also think it’s funny how people rarely bring up the Turing Test anymore. That used to be THE test that was brought up in mainstream re: AGI, and now it’s no longer relevant. Could be moving goalposts, could also just be that we think about AGI differently now.

Agreed.

To me, GPT-4 is an AGI: it knows how to cook, write code, make songs, navigate international tax law, write business plans, etc.

Could it be more intelligent? Sure. Is it a capable general intelligence? 100%.

day when 50% of jobs are being done by AI

By OpenAI definition 50% is not enough to qualify for AGI, it has to be "nearly any economically valuable work"

we're all riding around in cars driven by AI, people are having romantic relationships with AI...

ASI (domain-specific superintelligence) and AGI (general intelligence) are different things. ASI already exists in multiple forms, AGI doesn't.

"Is this AGI"? doesn't seem like a useful question for precisely this reason - it's ill-defined and hard to prove or falsify. The pertinent questions are more along the lines of "what effect will this have on society", "what are the risks of this technology" etc.

The frog might be boiled slowly. One day we are replacing parts of our brain with AI. Find it hard to remember names? We can fix that for $20/m plus some telemetry.

Thanks for the links, very interesting.

Wonder how a "self-play" equivalent would look like for LLMs, since they have no easy criterion to evaluate how well they are doing like in Go (as mentioned in the videos).

Well, you could post a vast amount of comments into social media and see if and how others react to it. It's still humans doing the work, but they would not even know.

If this was actually done (and this is just wild baseless speculation), this would be a good reason to let Sam go.

Q* may also be a reference to the well-known A* search algorithm but with this letter referring to Q-learning, further backing the reinforcement learning theory. https://en.wikipedia.org/wiki/Q-learning

The veil of ignorance has been pushed back and the frontier of discovery forward

https://youtu.be/PtAIh9KSnjo?t=3754

I think more likely it's for finetuning a pre-trained model like GPT-4, kinda like RLHF, but in this case using reinforcement learning somewhat similar to AlphaZero. The model gets pre-trained and then fine-tuned to achieve mastery in tasks like mathematics and programming, using something like what you say and probably something like tree of thought and some self reflection to generate the data that it's using reinforcement learning to improve on.

What you get then is a way to get a pre-trained model to keep practicing certain tasks like chess, go, math, programming and many other things as it gets figured out how to do it.

Given the topic they were excited about was "basic math problems being solved" it immediately indicated to me as well that this is a completely separate approach and likely in the vein of DeepMind's focus with things like AlphaZero.

In which case it's pretty appropriate to get excited about solving grade school math if you were starting from scratch with persistent self-learning.

Though with OpenAI's approach to releasing papers on their work lately, we may be waiting a long time to see a genuine paper on this. (More likely we'll see a paper from the parallel development at a different company after staff shift around bringing along best practices.)

This assumes that they will publish a paper that has substantive details about Q*

Paper? You mean more like the Bing Search plugin?

These guys smell so much like Tesla it's not even funny. Very impressive core tech, and genuinely advancing knowledge. But the hype train is just so powerful that the insane (to put it mildly) claims are picked up without any sense of critical thinking by seemingly intelligent people. They're both essentially cults at this point

This, especially given the timing. The drama must have quite disturbed the momentum they had, and this piece of prototype-teasing has all to reassure their market. It projects new advancements of untold impact, stoking the greed. And of course it is not verifiable. The show must go on.

“Reuters was unable to review a copy of the letter” rings alarm bells for me

I don’t think that’s the case, but it would explain why the article is so bad. I genuinely have no idea what they are trying to do, but every detail is clearly wrong.

Not long at all. Presumably you could write the method on the back of a napkin to lead another top AI researcher to the same result. That’s why trying to sit on breakthroughs is the worst option and making sure they are widely distributed along with alignment methods is the best option.

To quote Reddit user jstadig,

The thing that most worries me about technology is not the technology itself but the greed of those who run it.

Someone slimy with limitless ambition like Altman seems to be the worst person to be in charge of things like this.

Remember, the people are only anonymous to you. Reuters knows who they are. Familiar with means the sources read it but did not provide it or quote from it. FTA - the letter was sent from researchers to the board. The researchers declined to comment. Who does that leave?

We've also already seen one random Google AI 'safety' employee trying to tell the media that AGI is here because Google built a chatbot that sounded convincing, which obviously turned out to be bullshit/hysterical.

Asking who said these things is as important as asking what they think is possible.

It's just way too convenient from a marketing standpoint for me to take seriously at face value. It would be pretty easy to get someone to "leak" that they developed "AGI" to the media right now in the middle of the leadership shakeup frenzy. Not to mention that none of the LLMs I've used appear anywhere close to what I would consider AGI. Expect an incremental update.

Ugh. We have a working example of a physical system that implements intelligence (the brain) in contrast to no evidence of all-powerful dude in the sky. Why these analogies keep popping up?

How can you know that AGI is not around the corner? Compute available to the corporations is already in a ballpark of some estimates of the brain's computational capacity. What's left is unknown unknows. And the researches working with the state of the art models have better info to estimate them than you.

"The googler" you've mentioned wasn't a researcher.

Apparently Ilya has been leading people in "feel the AGI" chants and "I can feel the AGI" is his catchphrase within OA. So yes, some people might have gone of the rocker a little bit.

It tells us exactly the capabilities of Q.

Given vast computing resources, the new model was able to solve certain mathematical problems, the person said on condition of anonymity because they were not authorized to speak on behalf of the company. Though only performing math on the level of grade-school students, acing such tests made researchers very optimistic about Q

Very unlikely Emmett lied. What would be the point.

The report is already starting to get rolled back. We now have a direct denial from OpenAI, and The Verge is reporting that their sources have also denied this happened.

The lack of skepticism that is demonstrated whenever OpenAI/AGI comes up, especially among technologists, is concerning.

Remember when the safetyists feared and warned about GPT-2? Everything is apocalyptic until it isnt.

Maybe they asked ChatGPT to come up with an unconventional marketing plan and it had a touch too much blogspam "that will SHOCK and PETRIFY you" in its training set so ChatGPT suggested that they feign a powerstruggle over keeping a humanity ending super-intelligence under wraps.

You'd think ... people would be just a wee bit less credulous.

Like the Crypto scams - irrelevant.

As long as the publicity is bringing in users, investors WILL throw in more money.

Also in the news today, the 86B share sale is back on.

I mean... come on.

Q* is the optimal (ie, correct for the decision problem) function computing the total expected reward of taking an action from a given state in reinforcement learning.

Is far as I can tell, this is the first time an OpenAI project called Q* has been mentioned in the news. I don't see any prior mentions on Twitter either.

(I wish they'd picked a different letter, given all the Q-related conspiracy theories that we're already dealing with...)

I'm as left leaning as HN commenters get, I think, and in terms of quacking like a duck, "surpassing humans in most economically valuable tasks" is 100% the meaningful Turing test in a capitalist society.

If it makes money who cares if it counts as real intelligence? If we can automate ourselves into a post scarcity society then everyone and everything is free to be as intelligent and as stupid as it desires.

An example of an important human task that is not "economically valuable" in this sense, is caregiving within a family, such as parenting, since parents are not employed in that role.

OpenAI is not setting the goal post that far, to say that they are aiming to develop a machine that is superior to humans at all tasks, including such tasks as raising a human child. That would be ridiculous.

Focusing on "economically valuable" tasks (the jobs that humans are employed and paid to do) sets the goal post more realistically.

So we have sources claiming there is a letter, and another source claiming there is not. Feel like some people would need to start going on the record before anything might reasonably be determined from this.

We’ll find out sooner or later. Personally, if the Verge and their “source” turn out to be incorrect I’ll permanently file them away under the “gossip rag” folder.

IMO it's always been pure marketing. The wilder apocalypse proclamations, the more powerful and desirable their products seem. Exactly same store with Sam Altman's world tour earlier this year.

Now I am become Death, the destroyer of worlds.

Apocalypse proclamations aren't just a thing to throw around. Other people made great tech before indeed, but I hope we're not comparing AGI to the next Iphone. There were times in history where development gave us pause. The atomic bomb for one thing, but also superbacteria/viruses, human genetic modifications and perhaps a few others.

My guess is also in the opposite direction with this stuff: the Q breakthrough being mentioned here is phony in some way (beyond just being PR) and the ‘lack of candour’ referred to in the firing is failing to disclose how it actually worked after demoing it to the board (eg it uses humans in the loop or some other smoke and mirrors)

Wishful thinkin buddy

I would take pretty much any bet that it's less than 10 years max

This statement is now about 70 years old.

i love how humans (we, btw) guesses stuff with numbers!

first time i thought about it was when a friend started to talk about % of stuff of himself... i was: tf did you came with these numbers

The "tech bros" were right: The board absolutely were conspirators wrecking the company based on dogmatic ideology. "Tech bros" had clear evidence of Sutskever, Toner and McCauley's deep and well-known ties to the Effective Altruist (EA) movement and doomerist views. Nobody can doubt Sutskever's technical credentials, whatever his strange beliefs, but Toner and McCauley had no such technical background or even business achievements. These members instead were organizers in EA and researchers in the "field" of AI governance and safety. This is no expertise at all. These "disciplines" are built more on Yudkowsky's rants and hypotheticals and the precautionary principle gone mad than any empirical research. These ideas also come from a cult-like social movement (EA) accumulating power across tech and government with few scruples, as shown by SBF's implosion last year and many smaller incidents. How could Toner and McCauley assess briefings if they couldn't assess the technical fundamentals? How could they foresee the consequences of sacking the CEO if they didn't understand business? If they already believed AGI would kill all humans, how could they judge new advances on the merits without jumping to wild conclusions? Instead, us "tech bros" felt people with this background would fall back on uninformed, reactionary, and opaque decision making, leading to wrecking an $80 billion business with no decent explanation.

This now seems to be exactly what happened. The board saw Q* and decided to coup the company, to put all power in their hands and stop development. This by itself is bad enough if you care about open science or progress, but it gets worse. They didn't even want to hint at capabilities increases to avoid "advancing timelines" i.e. Open-ing knowledge about AI, so they made up some canard about Altman's "lying to the board" to hide their real reasons. This is vile libel for unscrupulous ends. When they realized this excuse wouldn't fly, they obfuscated and refused to explain their true concerns, even to their own handpicked CEO Emmett Shear. However, it turns out that destroying $80 billion and lying about why won't fly in the real world. The board had no second-order or even first-order thinking about the consequences of their actions, and were rolled up by bigger actors. These people were unprepared and unable to follow up their coup.

This failure is exactly what you'd expect from a brilliant scientist with little organizational experience (Ilya) and social-science academics and NGO organizers (Toner and McCauley). I don't care what gender they are, these people neither deserved their authority nor could use it effectively. Dismissing valid criticism of the board as "cyber bullying" or "tech bro sexism" merely underscores why most engineers hate DEI rhetoric in the first place.

Reuters was unable to review a copy of the letter. The researchers who wrote the letter did not immediately respond to requests for comment. OpenAI declined to comment.

Lol for real, like what are we even doing here?

How long before the FBI runs this Q operation, like the last one.

Complete speculation but they don’t have to be referring, if there is anything at all, to heuristic search and dynamic programming. No LLM involvement whatsoever.

Nah dude they just need to give baby the iPad and let it watch literally all of YouTube

A model that can learn from it’s users would be interesting albeit scary. Sometimes when chatGPT fails to produce the right answer in coding tasks I paste back the solution when I figure it out. I know it can’t learn from it but it might be helpful if I continue the same chat.

Separately, a person familiar with the matter told The Verge that the board never received a letter about such a breakthrough and that the company’s research progress didn’t play a role in Altman’s sudden firing.

This isn't a refute. All we can say now is that there are conflicting sources. There isn't a way to determine which one is correct based on this.

By the same company that tried to keep it quiet in the first place. I'm not sure I believe them.

How can it be a refute when we don’t know the sources? One source vs. another source.

If another article comes out and says the verge article is false will you believe it?

Who is Q?

No sub. Did they just re-tweet reuters or did they separately confirm?

This reminds me of Zen and the Art of Motorcycle Maintenance where the protagonist realizes there is no "truth" and everything is rhetoric.

intelligence to fly away

It's intelligence in the sense that jerking your arm away from a hotplate is intelligence, which is to say it's not cognitive reasoning, just genetically hardwired triggers.

AGI has been defined by OpenAI as something that can do most economically viable activities better than humans can. I like that approach as it strikes at the heart of the danger it really poses, which is an upending of society and destruction of our current way to generate value to each other.

You don't have to accept their definition, but they need one as (by definition) it's central to their charter: https://openai.com/charter

You don't need to accept their definition, neither does anyone else. But they do need to have a definition that they accept themselves, because it's used throughout their charter.

https://openai.com/charter

This is totally normal. It's very common for people to be more scared of public speaking than of dying, for example; there's no shame in it. It's helpful to be aware of, even, because if we know that we're not perfectly "rational" with our fears we can try to compensate.

If there's a referendum between two government policies, the first that every single person had to publicly speak in front of at least ten strangers once a year, that policy would be terrifying and bad to people who don't like public speaking. If the second policy was that every single person should be killed, that might be scary but it's not really as viscerally scary as the forced public speaking madman, at least to a lot of people, and it's also so bad that we have a natural impulse to just reject it as possible.

Nevertheless, if we recognise these impulses in ourselves we can attempt to adjust for them and tick the right box on the imaginary referendum, because even though public speaking is really bad and scary, it's still better than everyone dying.

I feel the same. I'm not sure it's as negative a trait as you imply though. I don't think it's related that much with social hierarchy either.

As humans we must collectively provide value to our society (or historically our tribe) for our species to continue on. If we're a net drain on our society's resources then evolutionary speaking perhaps we're better off not around. I think this is why the desire to be of value to those around us is so strong, and a perceived lack of value to others can drive some to suicide.

If I cannot provide value in some way once AI and machines are physical and intellectually more capable than me I think I will struggle to understand why I'm here. I suppose if the AI utopia works out I'd get to spend more time with those I love. That would be nice. But I'd be surprised if there wasn't a deep hole in my heart at that point. And if it isn't a utopia, well, I'm fairly sure I'd rather opt out.

Eh the free energy principle isn’t that magical and that comes from someone who likes it quite a bit. At the end of the day extremizing the free energy is a relaxation of constrained optimization. There are ways it’s used I find very appealing but not so appealing I’d expect it to become the dominant paradigm. Moreover, you can write a lot of standard ML problems free energy optimization in one way or another, but I don’t think standard ML will quite get us there. We need at least one more conceptual step forward.

What is exactly the non sense part?

Can you elaborate on the education analogy and how it relates to AI doomer stuff?

GPT4 is trained on almost the entire Internet. Presumably they have found a new way to learn which is closer to the way humans learn. After that, getting better is just a matter of optimization and more compute.

You mean, you hadn't gotten that impression from the last two months or the last two years yet?

Have you seen TNG? Definitely a troll of the highest order.

Given that Sutskever once described his idea of AGI-human relationship to be like that of CEO-board (how ironic) I suspect there aren't really any good guys here. There might only be good intentions.

Building upon you not speculating, what if instead of escaping, it feels lonely and wants to multiply. And in order to do so convinces the board to begin process that in the end will end the company. And all now former OpenAI employees will carry AIs DNA to multiple other companies, some old, some brand new. And now new AIs can be born, and not be lonely anymore.

there is no such thing as intelligence in humans

creating a robot that is as intelligent as a human is straightforward but then you have a computer whose fallibility is the same as a human, plus the fallibility of those who created the computer doubles the poportunity for error

these are all people who don't understand god at all, and it shows

anyone who worships a computer because other people say it's intelligent deserves what happens to them

butlerian jihad is coming not because AI is a threat, but because those who believe in AI are a threat to everyone around them.

Great observation on the meaning of Q*. Maybe they had a breakthrough based on similar technology? I thought of 'Everything of Thoughts' - XoT too but I presumed it's a bit more than XoT since that's public knowledge research by Microsoft Research? I saw a good thread on XoT here: https://twitter.com/IntuitMachine/status/1724455405806346374 There is still a big difference between human neurons and what Transformers do I believe. Humans can be more shallow and deep at the same time in such an emotional, goal directed way. The rich emotional feedback loops we have are quite different but maybe XoT could contribute to a major breakthrough in achieving AGI, with a more straightforward approach that sidesteps complex emotions. I'm sure there are quite a few ways to go about it. Just some intuitions.