The pace that ML seems to be advancing right now is amazing. I don’t believe in the singularity but it’s changing software and then society in ways no one can predict.
The pace that ML seems to be advancing right now is amazing. I don’t believe in the singularity but it’s changing software and then society in ways no one can predict.
How’s Gemini looking?
It is interesting how persistently dominant GPT-4 is: https://twitter.com/lmsysorg/status/1735729398672716114
Off the top of my head, I can think for at least five foundation models (Llama, Claude, Gemini, Falcon, Mistral) that are all trading blows, but GPT is still a head above them and has been for a year now. Transformer LLMs are simple enough that, demonstrably, anyone with a million bucks of GPU time can make one, but they can't quite catch up with OpenAI. What's their special sauce?
Their only special sauce is the first-mover advantage. Then it attracted users (data), brand recognition, talent and became a positive feedback cycle.
GPT4 was created before most feedback cycle. They had GPT4 ready before ChatGPT launch.
If I recall right, GPT4 got done in October. After that, it was RLHF and safety work (Bing starts using GPT4 publicly in February, a month earlier than official launch)
If I recall right, before ChatGPT launched Google already had LaMDA which an employee believed to be sentient and was subsequently fired. The foundation model was definitely done, but to launch Bard, Google needed a kick in the ass in additional RLHF, safety and groundedness work.
Ultimately though, it's futile to argue which model got done first, as long as the models were behind closed doors. But ChatGPT launched before Bard did and that's the pertinent part that gave OpenAI the first-mover advantage.
The LaMDA is sentient guy gave me the impression of being a bit nuts. I'm sure google would show their weight and out-compete openai if they could. We all know all this "AI safety" is for show, right?
No, it's for brand safety and reputation. In 2016 Microsoft released Tay [1] without or lacking guards and it ended up being a failure and hurter the Microsoft brand.
We all know all this "AI safety" is for show, right?
No. A lot of people think it really matters
A lot of other people pretend to care about it because it also enables stifling the competition and attempting regulatory capture. But it's not all of them.
I'm personally devoting my career to AI safety, on a volunteer basis, because I think it's is legitimately of high importance. (See my blog, e.g. https://amistrongeryet.substack.com/p/implications-of-agi, if you want to understand where I'm coming from.)
What makes you think it is for show?
LaMDA is really far from being sentient.
It's outputs non-sensical (aka highly hallucinating) or relatively useless but coherent text.
It really needs further refinement.
This is one big reason why GPT-4 is still the most popular.
GPT-4 was done training August 2022
Thanks!
The RHLF is probably quite important even on top of a good base model.
That’s not it. It’s not just hype. The underlying model is better.
Their special sauce is most probably the quality of data and the amount of data cleaning effort they put in.
I’m speculating here but I think Google always refrains from getting into the manual side of things. With LLMs, it became obvious so fast that data is what matters. Seeing Microsoft’s phi-2 play, I’m convinced more about this.
DeepMind understood the properties, came up with Chinchilla but DeepMind couldn’t integrate well with Google, in terms of understanding what kind of data Google should supply to increase model quality.
OpenAI put annotation/cleaning work almost right from the start. Not too familiar with this but human labor was heavily utilized to increase training data quality after ChatGPT started.
Indeed, making poor people in 3rd world countries rate the worst sludge of the internet for 8+h a day might backfire on your marketing... OpenAI could risk it, Google maybe doesn't want to...
This is a naive take. How do you think Google collects or collected data for their safe-search classifiers? Now that's a sludge.
Or how do you think Google evaluates search-ranking changes (or gather data for training various ad-ranking & search-ranking models).
I don't know. How do they?
I was going to make a joke about all those CAPTCHAs we've solved, but I don't have an answer here.
Their instructions for human raters is public info.
Overview: https://blog.google/products/search/overview-our-rater-guide...
Full PDF: https://static.googleusercontent.com/media/guidelines.raterh...
Given that many western companies hire poor people to do all sorts of horrible work I doubt it’s that. More likely it’s to avoid suggestions of bias across their product range.
I kinda wonder if maybe it's at least partially due to openai hitting a kind of hyperparameter lottery. When each experiment costs millions it might be that (aside from good/ unique data) they just have a good set of hyperparameters used in training and it's too expensive for a competitor to find equal or better settings
Sorry for my ignorance: why does each experiment cost millions?
It’s the cost of compute hardware required to train a model of that size
Because training a model costs millions, so each time you experiment with trying to create a new kind of model it costs millions.
I would be surprised if this is the case. Neural scaling laws are well known and are used by all big industry players to extrapolate experiments.
Are they really "laws" my impression is its all just a bunch of empirical trends.
We cannot know truly how these parameters interact at large scale and also how they interact with each other.
Is it really the case that openai has data that Google doesn't?
Beside the fact that Gemini pro is more comparable to GPT-3.5, one more interesting observation is that even OpenAI themselves was not able (or didn't intend) to deliver a significantly better model than GPT-4 almost over a year. And OpenAI does not seem to hide their own magical "AGI" behind the scene as they've been more focused on efficiency and engineering works reportedly, primarily driven by Sam, rather than developing a new model. I'm reasonably sure that the current transformer itself as an architecture is at its peak and most improvements will be mostly incremental.
You cannot compare GPT 4 to Gemini Pro. They are different classes of models.
Note, Gemini Ultra, which they claim is competitive with or possibly even better than GPT-4, isn’t out yet. They have released a weaker model, Gemini Pro.
It will be interesting to see how capable Gemini Ultra actually is. For now we wait.
These ML-compilers are being overhyped. It's all the same trade-off as a traditional compiler: you get a lot more throughput than hiring a specialist performance programmer, but the latter will typically outperform, possibly by orders of magnitude.
These things are inferior at many levels: - Algorithmic: These things aren't feeding back to their human masters tips and tricks on how to modify the network to go faster beyond some very basic signals. - Loss of intent: ML network designers are specifying architecture in python, and by the time it's gone through many layers of lowering, you can get some complete garbage. Highly efficient garbage, but still garbage. (recent example, we caught one of these compilers doing a slice update operation by first forming the range of all possible indices to the array, slicing that to get indices to update, and then doing a scatter; we replaced it with a single memcpy call). - Inefficient kernels. Every time we see the output of these compilers go head-to-head with an expert assembly programmer, the compiler loses, often by 30%+. This always seems like the sort of thing that should be easy to solve, but given no-one seems to have cracked it in the past 50 years, it's obviously not as simple as it sounds.
Take a look at the chess engine Stockfish: they tossed out years and years of human written heuristics in board evaluation, to a small neural net that does the same but better.
Now consider all the heuristics for inlining, loop unrolling, vectorization etc in compilers, certainly a neural net can be beneficial and possibly easier to maintain than tons of human written heuristics.
We'll have to see. I could definitely see someone spending a lot of time training for a specific algorithmic kernel and microarchitecture and beating the best human results (by a few percent).
I'd be very surprised if that can be extended to a large complex algorithmic system that is amenable to mathematical reformulations (at least within the next 10 years).
Another bitter lesson? http://www.incompleteideas.net/IncIdeas/BitterLesson.html
My understanding is that stockfish retains and uses its classical evaluation model in addition to the NNUE model
No, they removed the code in Stockfish 16, released June 30, 2023 but it wasn't used for much or made much difference before then, after they introduced the neural net.
https://github.com/official-stockfish/Stockfish/commit/af110...
Humans designing algorithms by hand will go the way of the dodo bird
big +1 - IMHO the future of optimizers (and probably compilers...) are almost certainly ML-based.
These ML-compilers are being overhyped. It's all the same trade-off as a traditional compiler
Funny you should say that. Because traditional compilers have been incredibly useful.
Right, but we still tend to sidestep the compiler and/or spend hours of human time tuning the input to get the right output for core kernels.
It's all the same trade-off as a traditional compiler: you get a lot more throughput than hiring a specialist performance programmer, but the latter will typically outperform, possibly by orders of magnitude.
That throughput is the point though? You cannot have performance specialists on every single ML workload. It's still significantly better than not having these kinds of optimization.
Exactly! Why would anyone use gcc/clang when you can just hire someone to hand-write assembly instead?
Hardware (and performance) can always be improved without involvement of users so this is actually pretty useful.
Comment seem extremely dismissive and close minded.
Can anyone bring this down to earth for me?
What's the actual state of these "ML compilers" currently, and what is rhe near term promise?
One of the easiest approache is torch.compile, it's the latest iteration of pytorch compiler (previous methods were : TorchScript and FX Tracing.)
You simply write model = torch.compile(model)
"Across these 163 open-source models torch.compile works 93% of time, and the model runs 43% faster in training on an NVIDIA A100 GPU. At Float32 precision, it runs 21% faster on average and at AMP Precision it runs 51% faster on average."[1]
What google is trying to do, is to involve more people in the R&D of these kind of methods.
Thanks for this summary
Excellent. Read that entire article and still was not sure what Google was pitching.
It actually sounds very useful and cool, I just completely did not get that from the article.
Check out torch.compile
The near term promise is that you can use AMD, CUDA, TPUs, CPUs etc without explicit vendor support for the framework on which the model was developed.
Disclaimer: I will be very handwavey, reality is complex.
This is achieved by compiling the graph into some intermediate representation. And then implementing the right backend. For projects here, look at stableHLO, IREE, openXLA.
You can argue that Jax's jit compiler is a form of such compiler, mapping the traced operations down to XLA, which then does its own bit of magic to make it work on your backend.
It's transformations and abstractions all the way down.
Can anyone explain how conv works in that graph. You have a tensor of shape [2,4,16] and you convolve with a kernel of shape [4,16,8] and that gives you a [2,8] tensor? How's that possible?
Does this help?
*1. Input:*
* Tensor shape: [2,4,16] * `2`: This represents the *batch size*, meaning there are two independent data samples being processed. * `4`: This is the *input feature dimension*, indicating each sample has 4 features. * `16`: This is the *input channel dimension*, suggesting each feature has 16 channels of information.
*2. Kernel:*
* Shape: [4,16,8] * `4`: This is the *kernel size*, meaning the filter window used to convolve has a width of 4. * `16`: This matches the *input channel dimension*, ensuring the filter operates on the same number of channels as the input. * `8`: This is the *output channel dimension*, indicating the convolution produces 8 new channels of information per sample.
*3. Output:*
* Shape: [2,8] * `2`: This remains the *batch size* as the operation is applied to each sample independently. * `8`: This matches the *output channel dimension* of the kernel, signifying the final tensor has 8 new features extracted from the input.
*4. How is it possible?*
Despite the seemingly mismatched dimensions in the input and output, convolution on graphs works by leveraging the *neighborhood structure* of the graph. Here's a simplified explanation:
* The kernel slides across the graph, applying its weights to the features of the current node and its neighbors within a specific radius. * This weighted sum is then aggregated to form a new feature for the current node in each output channel. * As the kernel moves across the graph, it extracts information from the local neighborhood of each node, creating new features that capture relationships and patterns within the graph.
*Additional considerations:*
* The graph structure and edge weights likely play a role in how information propagates during the convolution process. * Specific details of the convolution implementation, including padding and stride, might also influence the output shape.
What about transformer itself, any indication that it is optimal in some way?
Feels like they bury the lede with the first paragraph, but otherwise cool stuff!
summary: improve prediction of run-time performance of a computation graph using GNN, they use an embedding dictionary for each node's opcode along with some other node features (eg shape, bits, window size, see [1]), they released a big dataset of these graphs in [2] with varying XLA compilation configurations and their resulting perf on TPUs, they did some stuff to improve prediction on bigger graphs than before in [3] by partitioning the graph (METIS graph partition, new to me) and other training things
This is only about predicting performance of a given graph and not about improving/suggesting/editing a new equivalent graph. As in FunSearch, models which have decent predictive power could be used with evolutionary search.
[1] https://github.com/google-research-datasets/tpu_graphs#featu...
[2] TpuGraphs: A Performance Prediction Dataset on Large Tensor Computational Graphs https://arxiv.org/abs/2308.13490
[3] Learning Large Graph Property Prediction via Graph Segment Training https://arxiv.org/abs/2305.12322
I really don’t people will be programming like we do today in five years
I don’t see why not.
I like programming how I do now. I don’t plan to stop.
People do lots of things manually that machines have been able to do for a long time.
And they generally get outcompeted sooner or latter.
All disciplines evolve over time and those who fail or refuse to keep up will be left behind.
Just because you can have a robot/machine that can efficiently churn out 1000 frozen lasagnas a second doesn't necessarily mean that italian restaurants have been "outcompeted" or "left behind" by not using such a machine in their business.
Sometimes quality and responsibility matter. Even if a machine is really good at producing bug-free code, often someone is going to have to read/understand the code that the machine produces and take resposibility for it, since machines cannot take responsibility for things.
In the end, you'll always need a human that understands the formal algorithmic language (i.e., a programmer), capable of parsing the formal program that will be run, if you want to be able to trust the program, since any way of mapping an informal request (described imperfectly in an informal language) to a formal construct is always going to be "lossy" and prone to errors (and you'll know this if you ever used an automatic code generator). Just because someone is willing to blindly trust automatically-generated code, doesn't mean everyone else is: there are contexts in which you need a person to blame, when things go wrong.
Ok, but to continue this analogy, industrialization and the ability to create 1000 frozen lasagnas a second had an enormous impact on the world. Not only on the economics of production, but ultimately on human society.
Sure, handmade lasagna still exists, but the world looks nothing like it did 200 years ago.
Heh, most cooking these days is like using libraries to build an application.
I don't slaughter an animal, I buy a cut of meat.
It's very rare I make pasta, I rehydrate dried husks.
The cheese comes in some kind of jar or package. The vegetables come from a store.
This has been the general move in applications too. I see companies with very large programs that are just huge sets of node modules joined together with a small amount of code.
AI can be used to support that activity too. Models can just as well used to explain existing code, possibly cranked out by another AI. I bet many companies are thinking about fine-tuning or LoRA-ing language models on their moldy codebases and outdated pile of documentation to make onboarding, refactoring, and routine extensions easier.
To interpret what AI models themselves are actually doing, researchers employ AI models as well.
That's an interesting thought; people "taking responsibility" as a form of labour, for machines that stole their lunch. Probably be around zero applicants for that job.
Responsibility is a complex quality needing capacity and competence. Right now even the manufacturers of "AI" are unable to assert much about behaviour.
Where "responsibility" exists around robotics and ML it will more likely be a blunt legal instrument tied to ownership, like owning a dangerous animal.
People do things because they enjoy doing them. And people will continue to do things they enjoy doing such as programming.
I don’t think it has anything to do with competition or being left behind.
The point is that people won't get paid anymore to do it. It has happened before: many activities that have been replaced by technology have been almost forgotten (for example the newspaper reader in the factory) or are practiced as art or niche crafts only. Careers built on these are either wildly successful or have highly unsteady income since they are literally reliant on the whims, not on the needs, of people.
We can't all run a YouTube channel for the programming equivalent of Primitive Technology, fun though that would be. 99.99% of us will have to adapt to AI being a coworker, who will probably eventually replace us.
Right now we're still OK because the AI isn't good enough; when it gets good enough, doing things manually is as economically sensible as making your own iron by gathering a few times your mass in wood, burning some of in a sealed clay dome to turn it into charcoal, digging up some more clay to make a porous pot and a kiln to fire it in, filling it with iron rich bacterial scum from a creek, letting the water drain, building a furnace, preheating it with the rest of the wood, then smelting the bacterial ore with the charcoal, to yield about 7 grams or iron.
who will probably eventually replace us.
no one is going to be using AI and then just have it 'replace them', they're going to use it to augment their abilities and avoid replacement.
The people using AI to write code aren't necessarily former professional programmers, for the same reasons people using AI to make pictures aren't necessarily former professional artists, and those using aim bots aren't necessarily former professional snipers or olympic shooters.
A manager can prompt a chatbot to write a thing instead of prompting me to write the same thing — for the moment, what (I hope) keeps me employable is that the the chatbot is "only" at the level of getting good grades rather than n-years professional experience.
I have no expectation for any specific timeline for that to change. Perhaps there are enough incentives it will never get trained to that level, but also perhaps it was already trained 4 months back and the improvement to capabilities are what caused the OpenAI drama.
I mean, if we had something that capable, I have zero idea why you think your manager is going to be in a safer position to you? That seems ridiculous.
You could easily flip it around, ask the bot to manage you better than your manager and make the best use of your time, or something like this?
Managers have more business contacts and access to money.
But others your point is valid.
Nice analogy! I saw an estimate recently on the cost of programming and they predicted that automated coding will cost 10,000 times less than human coders. It was all back of the envelope and questionable but still it was food for thought. Will we be 10,000 times more productive or will we be out of work? I think a lot of people will be out of work.
Thanks! :)
It can be both. Automation of farming means we've gone from a constant risk of starvation to an epidemic of obesity, while simultaneously reducing the percentage of the workforce in agriculture.
You can do what you want, you just won't be paid to do it anymore.
I still get paid for project involving Oracle 9i running on Itanium HPUX and Delphi application running on Windows XP. This project is not going anywhere in the next 5 years. And there are numerous other projects which will not go anywhere either. I just don't believe that programming landscape will change much. May be in California startup world. My world moves slower.
I don't think anything significantly changed in my approach to the code since 2013. We will see how 2033 goes, but I don't expect nothing big either. ChatGPT is just Google replacement, Copilot is just smart autocomplete. I can use Google instead of ChatGPT and I could use Google 10 years ago. I can use vim macros instead of Copilot. This AI stuff helps me to save few hours a months, I guess, so it worth few bucks of subscription, but nothing groundbreaking so far.
I think most are going to wait on the side lines watching the bleeding edge. The promise is great but so is the risk of disaster. Imo it's going to be a generational shift.
They’re making fun of your typo, but you’re right. Pretty much every software job in 5 years will be an AI job. This rustles a lot of feathers, but ignoring the truth will only hurt your career.
I think the era of big tech paying fat stacks to a rather larger number of technical staff will start to wane as well. Better hope you have top AI paper publications and deep experience with all parts of using LLMs/whatever future models there are, because if not, you’ll be in for a world of pain if you got used to cushy tech work and think it’s inevitable in a world where AI is advancing so fast.
Have you ever worked in tech and had to deal with the typical illiteracy and incompetence of management and execs?
If LLMs got this good, the brick wall these orgs will hit is what will really ruffle feathers. Leadership will have to be replaced by their technical workers in order for the company to continue existing. There's simply not enough information in the very high level plain english requirements they're used to thinking about. From a theoretical and practical perspective, you very likely cannot feed that half-assed junk to any LLM no matter how advanced and expect useful results. This is very much already the case human-to-human for all of history.
Either that or nothing happens, which is the current state of things. Writing code is not even 10% of the job.
Why don't you think that a sufficiently advanced AI can do the same as what technical humans do today with vague directions from managers?
Indeed.
I can give a vague, poorly written, poorly spelled request to the free version of ChatGPT and it still gives me a correct response.
As correct as usual at least (85-95%), but that's a different problem.
Correct compared to what?
There's gonna be a lot of context implied in project docs based on previous projects and the LLM won't ask hard questions back to management during the planning process. It will just happily return naive answers from its Turing tarpit.
No offense intended to anyone, but we already see this when there are other communication problems due to language barrier or too many people in a big game of corporate telephone. An LLM necessarily makes that problem worse.
Correct compared to what I ask it for.
Previous projects can be fed into LLMs either by context window (those are getting huge now) or fine tuning… but of course it's not a magic wand like some expect it to be.
People keep being disappointed it's not as smart as a human, but everyone should look how broad it is and ask themselves: if it were as good as a human, why would companies still want to employ you? What skills do you have which can't be described adequately in writing?
I feel that issue with AI is similar to issues with AI cars.
AI car won't ever reach its destination in my city. Because you need to actively break the rules few times if you want to drive to the destination. There's a stream of cars and you need to merge into it. You don't have an advantage, so you need to wait until this stream of cars will end. However you can wait for that for hours. In reality you act aggressively and someone will allow you to join. AI will not do that. Every driver does that all the time.
So when AI will try to integrate into human society, it'll hit the same issues. You sent mail to manager and this mail got lost because manager does not feel like answering it. You need to seek him, you need to face him and ask your question, so he has nowhere to run. AI does not have physical presence, neither he have aggression necessary for this. He'll just helplessly send emails around, moving right into spam.
LLMs are cool and will continue to change society in ways we cannot readily predict, but they are not quite that cool. GPT3 has been around for a little bit now and the world has not ended or encountered a singularity. The models are expensive to run both in compute and expertise. They produce a lot of garbage.
I see the threat right now to low-paid writing gigs. I’m sure there’s a whole stratum of those they have wiped out, but I also know real live humans still doing that kind of work.
What developers may use in five years is a better version of Copilot trained on existing code bases. They will let developers do more in the time they have, not replace them. Open source software has not put us all out of jobs. I foresee the waning of Big Tech for other reasons.
And they won't right up until they do. Reason why is that…
…doesn't extend to the one cost that matters: money.
Imagine a future AI that beats graduates and not just students. If it costs as much per line of code as 1000 gpt-4-1106-preview[0] tokens, the cost of rewriting all of Red Hat Linux 7.1 from scratch[1] is less than 1 million USD.
[0] $0.03 / 1K tokens
[1] https://dwheeler.com/sloc/
I like financial breakdowns like this. The thing an LLM cannot do is all the decision making that went into that. Framing the problem is harder to quantify, and is almost certainly an order of magnitude more work than writing and debugging the code. But a sufficiently good LLM should be able to produce code cheaper than humans. Maybe with time and outside sources of truth, better.
I give it two years. Salaries will drop like a rock.
I’ve been programming since middle school. That would be 30 years. Nothing really changed much. C++ is incrementally more convenient but fundamentally the same. Code editors are same. Debugger are same. Shell is same.
I am certain in 30 years everything will still be the same.
The way I write code was fundamentally altered in the last year by GPT4 and copilot. Try having GPT4 write your code, you won’t be so certain about the future of programming afterward I guarantee it.
I have free Copilot due to my OSS work. This week I disabled it for C++ because it is chronically incapable to match brackets. I was wasting too much time fixing the messes.
I use it for TypeScript/React. But it’s just a more comprehensive code complete. Incremental.
Uh huh, try GPT4 and report back. It’s a generational leap above copilot. I use copilot to auto complete one liners and GPT4 to generate whole methods.
GPT 4 does not produce code that I'm ready to accept. The time it takes to convince it to produce code that I'll accept significantly larger than the time it takes to write that code myself.
GPT 4 is fine for absolutely foreign tasks to me, like write a power shell script, because I know almost nothing about power shell. However those tasks are rare and I generally competent about things I need to do.
I am the same, 35 years. I use GPT 4 every day now. It sure is handy. It speeds up some things. It is a time saver but it does not seem to be better than me. It is like an OK assistant.
I would agree, not a fundamental or radical improvement yet.
Will it be? I hope so.
Other than 30 years ago you were writing a whole shitload more buffer/integer overflows. Hell, that's why we've written numerous languages since that point to ensure it's a hell of a lot harder to footgun yourself.
If coding hasn't change much in 30 years, it may mean you have not changed much in 30 years.
Process improved - version control, CI, unit testing. But not the tools. Clang is “recent” but it is still traditional CLI compiler.
I right fundamental software. Chromium, Node. It is good old, largely incremental, C++.
I don't people will be either brotha, I don't people will be <3
Cool
People don’t like it is, but it
People don't think it be like it is, but it do.
I think the biggest blind spot for many programers/coders is that yes it might not change much for them but it will allow many more people to code and do stuff that they were not able to before. As the the models get better and people use them more and learn how to use them more efficiently they will start changing things.
I am hoping we get to the point where the models are good enough that classes in schools are introduced on how to use them rather than just build them as the number of people wanting to or willing to learn programming is a lot smaller than the number of people to looking for ways to do things more efficiently.
It's not like schools have any other option on the table, students will find a way to use all the help they can get like they always have. Embracing it is the only way they can stay relevant in the coming age of one-on-one AI tutors.
It reminds me of the middle ages where only the priest was allowed to read and interpret the bible, mostly through the virtue of knowing latin. Then suddenly the printing press comes around and everyone can get their own cheap bible in their language. You just can't fight and enforce this kind of thing in the face of such insane progress. In 100 years (if we're not extinct then) people will probably look back on mass education where one overworked teacher tries to explain something in a standard way to 30 people (over half of who are bored or can't keep up) as some kind of old age savagery.
I want to see it come out with a cure for a disease that is tough to cure first. Singularity itself is pointless unless it benefits humans which is mainly in health/lower suffering
I'd say advancement in mathematics, computer science, and heck, even art is far from "pointless". Why does it feel like goalposts get moved everytime there is a significant progress in AI?
I read this same empty “goalposts” lament multiples times a day when reading this website, and though I know what the words mean I’m confused what you all think they mean.
https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu...
Is this just some performative grousing or do you really think what has been developed to date is “artificial intelligence”?
These comments all conveniently fail to define their author’s goalposts that apparently have been reached or surpassed. What were yours?
There was a very recent time when passing some version of the Turing Test would have been a fairly commonly accepted goalpost. Many experts thought that was 20+ years away, and were perfectly comfortable saying that it was their "bar", primarily because they thought generating convincing conversational text was so difficult that you'd basically have to solve All The Problems(TM) first.
Notice how nobody is talking about the Turing Test anymore now that it's either already been passed or is very damn close? We can argue back and forth about whether the real stupidity was the earlier expectation that the Turing Test was a useful threshold for AI, but it's impossible to claim that it wasn't a somewhat common and well-known one, so that goalpost really has been moved in a very dramatic way (or rather, removed altogether and replaced with nothing in particular other than a vague "I'll know it when I see it", in most cases).
Exactly so. The posts have been moved safely waayyyy over there at AGI, and at "super-human" or at "critical thinking". And several stages or degrees of AGI have been hierarchized. There is a serious reluctance at accepting how dumb an algorithm can be and still compare with humans.
But it is also true that numerous ground techniques are issue of the field of AI and generally called AI as they come out. It makes for good press. And that too was silly.
On the other hand, we've passed the turing test, what's changed?
You mean practically? What did this achieve? What did we gain now that the turing test is passed?
For actually already done: Actually believable chat-bots? Summarizers and question answerers? Generative text and graphics actually usable for generation of text, graphics and (mostly) photo-realistic renderings? Architecture brainstorming? (And logos, etc.) Kinda working self-driving cars? New Go playing strategies? A super-human Go champion? AI is on a roll these days.
That's not counting the more proprietary and discreet applications being already used all over the place. I fully expect there are already several.
For the sake of this, let's agree ChatGPT passed the Turing test. So you're saying "The posts have been moved" - you don't want them moved, you want "AI" to mean "ChatGPT equivalent, which passed a Turing test". Presumably you don't want progress and research to stop there, so what would be achieved by stopping using the term "AI" so that we "aren't moving the goalposts"? Why is "not moving the goalposts" a thing you care about at all? All it would mean is we make up another term like "artificial superintelligence" or "artificial person" or something, and everything else is exactly the same.
It's like arguing that when a child rides a bike with stabilisers that they have "learned to ride a bike" and then complaining anytime someone suggests the child learns to ride without stabilisers, because they "already learned to ride a bike" and now that's moving the goalposts - but you still want them to learn without stabilisers, presumably, you're just complaining about the term used to describe it, for ... apparently no reason or benefit whatsoever?
Is this an argument in good faith? This is HN so I'll answer it in good faith and without fighting. I mean:
- Passing the Turing test (for some measure of that) is a huge achievement - Poopoo-ing it is unuseful.
- Moving the bar from chat-bot Turing test to wwwaaayyy over there at super-human intelligence is unuseful. There are lots of valuable steps in between.
- There are many valuable steps before super-human intelligence.
- Most humans are nowhere near super-human intelligence. They are still "intelligent" enough for significant effects on the world as well as day-to-day grind.
- You can get plenty of sci-fi-level results without super-human intelligence
- We have already achieved AGI - Artificial General Intelligence because plenty of humans operate "just fine" in the world with bog-standard intelligence and for the ones limited to a keyboard roughly equivalently to an LLM-based chatbot. Top of the line AGI OR top of the line intelligence is not necessary to massively change the world.
- Incremental and bonus goals are a great thing! You are right!
- Many humans will fight hard to reserve the term intelligence to wet stuff. That's unuseful.
- Just because "it's done" doesn't mean all of a sudden that Turing test was not a good test.
But say everyone who posts "stop moving the goalposts" on every internet forum gets their way, the world agreed "ChatGPT passed the Turing Test and that's what AI means now" ... nothing that you said here will change at all? There will still be steps from here to super-intelligence, there will still be the same uses for ChatGPT as there were yesterday, people who think ChatGPT isn't usefully intelligent will still think that, etc. etc.
About the only point of arguing it is if you personally developed Deep Blue to beat Gary Kasparov at chess, when "beating a human grandmaster at chess" would definitely(tm) be AI, and now you feel hacked off that your personal or team recognition has been trivialised by moving goalposts and you've missed out on fame and a place in history. But I'm thinking it can't possibly be that all the people rambling about moving goalposts could be in that position and not mention it.
It would be cool if / when we see any of these advancements solve some real world important problems. Realistic chatbots? who cares?
Today’s chatbots definitely don’t pass Turing’s standard imitation game. You have two humans and one machine. One human gets to help the other human trip up the machine. It’s trivial to trip up today’s chatbots.
Well for it to be Artificial and convey Intellgence. I think that goal has been met!
How do you define intelligence?
I don’t have a exact definition but I will claim that helping me to write code that usually works first time or has a minor bug based on natural language is intelligence.
Back in my CS undergrad we learned Dijkstra's Algorithm in AI class. Once upon a time, that was AI. I think AI just refers to newly discovered useful algorithms.
Why does it feel like people think this is a useful or interesting whine? OK you brute-force solved tic-tac-toe, you built an AI. Congratulations, everyone recognises the dawn of Artificial Intelligence - and truly, enumerating all states of tic-tac-toe is all we ever dreamed of, all we could want, it's really all there is to intelligence. The term "AI" will never ever be used to mean anything else.
The term "AI" means (solving the first problem that was ever suggested to be AI decades ago) - how is that a better state of the world? What has anyone gained from "not moving the goalpoasts"?
In order for an AI to evaluate the effect of a small molecule on the brain, it would have to... simulate the operation of a human brain in a simulated environment. Similarly, to avoid Thalidomide-style disasters, it would have to simulate the conception, development and growth to adulthood of a human.
These things are... physically possible, but have WBE and uploads as a hard requirement. Those are going to affect a hell of a lot of things more than the drug industry!
Amusingly, machine-phase nanotechology and blood nanobots would be easier to evaluate, since simple cell-level mechanical interventions (reading surface proteins on cancer cells and chopping them up, say) will have fewer interactions than a small molecule that diffuses into every cell in the body.
This is how the human doctors who have cured things in the past have done it, is it?
The way this is going to work, when it happens, is that you'll ask the AI for a cure and it will give you a hundred candidates. A human doctor will look at the list and throw half of them out because they're toxic, several of the remainder will be excluded by animal trials, the few remaining will proceed to human clinical trials and one of them will actually work.
Then AI will have no effect on the drug industry at all.
The rate limiting step isn't "thinking up molecules." The University of Bern enumerated all possible molecules composed only of hydrogen, carbon, nitrogen, oxygen, sulfur and chlorine, up to 17 atoms. That produced 166 billion molecules. https://pubs.acs.org/doi/10.1021/ci300415d There are commercial drugs considerably larger than that. We've got molecular structures out the nose. There is no shortage of molecules.
The problem is the clinical trial. Putting drugs in humans and seeing what they do. That's the part that takes years and tens of millions of dollars. Using AI for anything else is like saying Microsoft Powerpoint accelerated drug development. Sure, it made presentations easier, but did it do anything for the problem of putting chemicals in people?
Clinical trials only start after about five years of research and development. While they do represent a large part of the budget (even in the hundreds of millions of dollars), there are countless of other necessary steps before, during, and after trials to ensure that drugs are both safe and effective. The problem is that we still don't understand how most of these molecules behave in the body, and how we can produce them reliably and efficiently enough, which brings me to the next point:
Yes, there are plenty of problems that AI and computational chemistry already help with in the pharmaceutical industry, including predicting solubility, stability, crystallization, granulation, toxicity, pharmacokinetics, developing the formulation, optimizing and scaling up both the synthesis and production process, developing appropriate techniques for quality control, and so on.
In all these cases and more, AI can help reduce the amount of experiments that need to be done in the lab, which require highly specialized equipment, personnel, and a lot of time. Oh and design of experiments is also a very important topic, again aiming at reducing the amount of lab time needed.
Admittedly, most of these things aim at ensuring that we do not put the wrong chemical in people, but they do represent most of the R&D effort spent in pharma, and reducing everything to clinical trials is not correct. There is a very wide gap between "AI will design drugs entirely on its own" and "AI is useless".
Which is useless, because you can't run 166 billion clinical trials.
But you could run half a dozen if there's a strong chance one of them will be a success. Filtering the list down to 100 molecules from 166 billion, some of which can be further eliminated by human evaluation without the expense of clinical trials, is actually useful.
You still ultimately have to do the clinical trial, because there is no substitute for empiricism.
It doesn't matter if it takes tens of millions of dollars if the result is a billion dollar drug.
Google is already doing something like this: https://arstechnica.com/ai/2023/11/googles-deepmind-finds-2-...
This is great if you want to use well understood pathways or make new drugs that you can then patent and mark up.
New pathways are gonna require feeding data into these models in the first place. Your not getting ozempic out of ML without doing the ground work first: https://globalnews.ca/news/9793403/ozempic-canada-scientist-...
Sure, but a lot of the ground work has already been done, or is susceptible to simulation. They're getting a lot of results out of simulating protein folding and things like that.
Humans aren't capable of doing this, but still make useful drug discoveries. AI can be empowered to conduct research in the real world, it doesn't need to simulate everything.
We start by doing them on mice (well, in vitro first, mice as the first in vivo), who have no say in the matter; and as mice are only rough analogues of humans, the human trials are still cautious once the animal trials are over.
We also develop a lot of drugs with a which have side-effects, which will is probably better than no drugs in most cases, the side effects are because it's a lot of educated guesswork.
WBE?
This is impossible, we can makesure that more possible scenario is that most of people lose job and starve. it is not sure whether we can reach to a society have UBI.
AI advancements are why we have affordable genome reading.
AlphaFold was a nice surprise when it happened, too.
“Cure” is a tough bar, but I believe Paxlovid, the anti-viral used to reduce Covid severity, was identified using ML. There’s many companies like Recursion Pharma which are entirely focused on using ML for drug discovery, and from what I can tell seem to have promising results, but drug development is slow enough that nothing will come of it for a while.
Also, while not medicine focused, Google’s GNOME project results announced a few weeks ago was pretty remarkable. They discovered more theoretical new materials using their ML approach than the rest of human history combined, and they are already confirming many of the results in laboratory settings. That has the potential to be a revolution in limitless scientific and engineering applications.
For me it's just another gold rush after dotcom, mobile, cloud, VR.
The first 3 have and did result as of today in trillions in dollars of economic activity. And have changed societies, politics, political participation, access to knowledge etc worldwide for good and bad. So I don't get why you are so dismissive of them.
AI is definitively here to stay forever. It's not a hype, it's 100% here for the long-term.
The hype may be specific to some companies for now, but AI is deeply going to change many industries, especially due to open-source, specialized chips to allow running in local, and new hardware (I strongly hope a clone of H100 A80G comes quickly).
The next step is to add limbs to the LLMs.
Then we get Tesla bot who is going to help you with daily chores, and to execute tasks in a factory.
The bot can ask its own internal knowledge base to know what action to execute next, and because the model can output JSON, the action commands can be sent to motors for in-real-life execution.
Pretty sure we're quite a ways off that stuff yet, especially for consumers. But sure, maybe something like that will be reality in the coming decades.
You mean it isn't overhyped, hype is just what expectations people have it is underhyped or overhyped that says how those expectations related to reality.
I am not dismissive. I think, though, that for a hundred companies engaged in the race one or less might succeed.
I'm not sure what the purpose of the word "just" there is. There indeed seems to be quite a lot of gold to be had by whoever gets a foothold.
"just" is there to distinguish a gold rush from a singularity
This + FunSearch make it seem like Singularity is imminent.
https://deepmind.google/discover/blog/funsearch-making-new-d...
At great risk of sounding completely ignorant, this approach is basically what I thought the point of machine learning was - cleverly using feedback loops to improve things automatically. The thing that sticks out to me as particularly cool about FunSearch is the use of programs as inputs/outputs and the fact that they managed to automate feedback.
I'm pretty naive in terms of granular understanding here as I am barely proficient in Python, to be clear, but when I daydream about things you could solve with machine learning/AI, this is the approach I always think of and I guess is how I thought it already worked. Load it up with the best information we have currently, define the desired results as clearly as possible, implement some form of automatic feedback, and let it run iteratively until it produces something better than what you had before.
Is this a case of "well no shit, but actually implementing that effectively is the hard part"? Is it being able to quickly apply it to a wide variety of problems? I guess I'm trying to understand whether this is a novel idea (and if so, what parts are novel), or if the idea has been around and it's a novel implementation.
The important thing is "how do you change X so that it heads towards the goal". And "how to do it quickly and efficiently".
Otherwise the description is the same as "select randomly, keep the best, iterate".
The goal is also complex. You might be thinking of "find the most efficient program" but that's not what we're doing here iiuc. We're trying to get a program that makes other unseen programs more efficient. That's hard to define as a goal.
That is what they did though. The LLM didn't know what problem it was "solving".
That's not really true unless you're ignoring the rest of my points. The process did not just uniformly randomly create programs.
They also don't just keep the best and search from that point but feed the resulting programs and their scores into an LLM.
They did remove the worst results from the group over time, the others was just uses a seed to generate new examples from instead of starting each function from scratch.
Some speculate that this is what OpenAI's Q* model is about and what caused the Altman/Sutskever split.