return to table of content

Ilya Sutskever: “If you learn all of these, you’ll know 90% of what matters”

klunger
49 replies
2d6h

The headline belies the scope of this task. If you click in to see what these are: - one is a full course in convolutional neural networks https://cs231n.github.io/ - one is a 500 page text book https://www.lirmm.fr/~ashen/kolmbook-eng-scan.pdf - another one is an 80 page text book (?) https://arxiv.org/pdf/math/0406077

and so on.

I would guess it is possible to go through this list if you make it your full time job for a year or so. That might be worth it depending on your priorities. Regardless, it bears mentioning.

sheepdestroyer
44 replies
2d5h

Reportedly, Carmack did it an a week by locking himself in an hotel room

jb1991
25 replies
2d5h

Ah, the freedoms of not having any other responsibilities.

soulofmischief
21 replies
2d5h

If you can't take a single week off of work during an entire year in order to advance your career, then you're satisfied with your current one.

There is no need to direct negativity toward others for making time to learn. Carmack is in this position today because of such work ethic, not in spite of it.

brayhite
18 replies
2d5h

What a strangely narrow way of looking at this.

Work isn’t the only responsibility.

What is the correlation between taking a week away from responsibilities - work, family, etc. - and job satisfaction?

soulofmischief
12 replies
2d4h

I assumed someone would retort with this angle, but I hoped people would interpret my comment charitably without me littering it with a dozen caveats.

Of course work isn't the only responsibility.

Of course some people have kids, pets, parents, medical conditions, etc.

It's one week away from work. It can still be done. You weren't going to be taking care of any other responsibilities during your normal 9-5 hours, so that should have no bearing on the week you take off; treat it as any other work week, except you're working for yourself.

jb1991
7 replies
2d

It's one week away from work. It can still be done. You weren't going to be taking care of any other responsibilities during your normal 9-5 hours

I think you didn't read the parents carefully. The comment being replied to is not about 9-5 hours, it's about "locking yourself into a hotel for week", and if you have kids, this is not exactly a healthy thing to do in the face of other responsibilities. That's the point being made here, and you are changing the goal posts.

soulofmischief
6 replies
1d23h

I'm not moving goalposts. The commenter tried to assert that only someone without "any other responsibilities" gets to spend a week learning something. It doesn't have to be in a hotel room. It doesn't have to be a week. It can be two, three, four weeks. Those are all implementation details.

The negative sentiment itself is what I addressed. Everyone fortunate enough to have internet access, a computer and a functional brain/body can make time, if they really want to, to learn new things that benefit their long term goals.

jb1991
5 replies
1d23h

The commenter tried to assert that only someone without "any other responsibilities" gets to spend a week learning something.

That's what I mean -- you didn't read the parents carefully. The commenter (me) made that reply to this:

Carmack did it an a week by locking himself in an hotel room

Everyone can and should spend time learning, but the little point being made was that locking yourself in a hotel room is not something most people can do due to responsibilities. Somehow you changed this into your more general statement "spending a week learning something."

soulofmischief
4 replies
1d15h

Because the hotel is an implementation detail.

jb1991
3 replies
1d

Spoken like someone without common life responsibilities.

soulofmischief
2 replies
21h18m

Spoken like someone who casts judgement and makes assumptions before considering that different people have different priorities.

It's downright silly to claim that I have no life responsibilities because I prioritize setting aside time for self-growth and encourage others to do the same.

Open your mind a little bit and ponder why you feel compelled to have such a negative and dismissive attitude about this.

jb1991
1 replies
20h16m

You are getting this reaction from people because of the judgmental tone you took in your original comment:

If you can't take a single week off of work during an entire year in order to advance your career, then you're satisfied with your current one.

It’s a pretentious remark, saturated in lofty assumptions about people, and you could learn from these reactions.

Edit: I just noticed your username, “soul of mischief“, and now it makes sense to me. Are you trolling?

soulofmischief
0 replies
19h44m

I don't find that pretentious at all. What I'm conveying is that we have the Things We Think We Want and the Things We Actually Want. The former live in our minds, the latter are actually realized through our everyday actions and choices.

I just noticed your username, “soul of mischief“, and now it makes sense to me. Are you trolling?

It's against the rules of Hacker News to assume bad faith. I certainly am not arguing in bad faith, nor do I intend to make anyone feel bad for the sake of it.

I purposefully wrote an abrasive comment in order to stoke the minds of others and make them think inwardly. There was one time in my life where I also needed to hear those exact words. That no matter what adversity I face, I must make times for the things I really care about.

I must constantly re-evaluate what drives me and check to make sure I'm not letting life pass by without achieving at least some of my goals. I must not make excuses, and I must not make excuses for others. I must not be negative toward those who are in a better station of life.

As for my username, it is derived from a popular hip hop group, Souls of Mischief, who are part of the larger Oakland set, Hieroglyphics. https://www.youtube.com/watch?v=fXJc2NYwHjw

giantg2
3 replies
2d3h

"You weren't going to be taking care of any other responsibilities during your normal 9-5 hours, so that should have no bearing on the week you take off;"

Have you ever been married, had kids, etc? Pretty much any time I take off of work, it involves catching up on projects, chores, or parental responsibilities during the day. It seems the vast majority of people I've spoken to tell of similar situations.

soulofmischief
2 replies
2d1h

Well, I have specifically avoided having kids at this stage in my life so that I may climb out of the poverty class first. I'm one abortion deep already. I was not as fortunate as 99% of the people in this little isolated corner of the internet.

I was homeless from 16 to my early 20s. I suffer from multiple traumas, both physical and mental. I have ADHD and bipolar disorder. I've had sciatica, degenerative disc disease, gout and other issues since my teens. Most days I can barely walk. When I do have kids, I'm not going to get to be the dad I want to be, running around and playing with them. My parents are abusive drug addicts who were not present in my life, so I was raised by other extremely abusive people instead, who kicked me out into the streets for being atheist. So malnourished as a child that my friends would make fun of being able to see the bones in my face and body. I spent years living homeless on scraps of food.

I have to advance my career. Every single thing is stacked against me, trying to make me fail. I don't get the luxury of deciding if I want to take a week off to do so. It's a matter of survival, and I have very purposefully structured my life in such a way that I have a shot in hell at one day raising a child above the poverty line.

Believe me, this is not a matter of me not empathizing with others or understanding the complexities of life. It's a matter of prioritization and commitment. If I can make time to learn, anyone can.

And on top of all of that, I do in fact have to provide for my sister and her four young, expensive children, because she doesn't have anyone either. I don't get much free time, man.

tarsinge
1 replies
1d3h

I'm sorry you had to go through all of this. I think your bias is your responsibility is 100% financial so is aligned with always spending more time for your career. For most people financial/career responsibility is only a part, ignoring the rest is as impossible as you putting your career aside.

soulofmischief
0 replies
1d2h

That's a good point. I do experience the boundaries, I spent the majority of the last 8 years alone and was solely focused on work and hobbies. It was a very productive time for me. Now that I'm in a longer-term relationship again, my productivity has seen a major reduction. It's exacerbated by the fact that I am extremely productivity-focused, but my significant other wants to relax in her free time.

I'm still learning where I should set boundaries on both sides of the field, and this will probably be something I continue to struggle with for some time.

ptero
3 replies
2d4h

If one cannot disappear for a week to think or study he is putting himself at some serious life disadvantage. My 2c.

I do not mean telling family that I am disappearing for a week starting tomorrow, but that I want to disappear for a week in a couple of months and the family would cover responsibilities for me (I should be willing to reciprocate).

deprecative
1 replies
2d3h

Realistically most people just don't have the resources for this sort of thing. It's not necessarily a matter of want but a limitation of needs. Assuming one has a family that could cover, they would still have a job with responsibilities. Not everyone can afford to risk termination or go without a paycheck for a week even if it's the most intelligent option.

therealdrag0
0 replies
1d22h

Most people in the field have PTO. Some people take PTO and travel, others play video games, Carmac has locked himself in a hotel room to work on projects pretty regularly since he was younger.

Many people have a lot more flexibility than they let on. They just make excuses instead of acknowledging their own priorities and commitments or lack of actual interest in change and work.

orochimaaru
0 replies
2d3h

I agree. We travel for work and disappear for a week. I've disappeared for 4-5 days to party with friends and my wife and kids have been fine with it. Obviously the work travel I don't control much. The party part - I'd be getting some serious conversations if it was happening every month. However, if it was to improve my career prospects I don't think anyone will hit much resistance. I mean provided you use the time well and don't get into a space of watching youtube clips of Dune 2.

aworks
0 replies
2d2h

So far, my 2 years of retirement are an experiment in discretionary time. I'm married but no children or pets in the house. So I have lots of leeway in what I do every day. Maybe I average 1-2 hours of responsible time. And there are constraints - health, energy, lack of ambition, family travel, Internet distractions etc.

But in general I can do what I want. This month, it's been relearning awk to do financial data analysis and reading the books of a British author, Diana Athill.

I'll say that my personal project satisfaction level is medium. Life satisfication has a major social component that provides meaning, though. Optimizing for both isn't necessarily easy even in retirement.

secondcoming
0 replies
2d4h

Sometimes it's nice to take a week off to do things like going outside.

jb1991
0 replies
2d

Disappearing into a hotel for a week and locking yourself in is quite a bit different than "taking a week off."

I'm going to guess you don't have kids.

read_if_gay_
1 replies
13h9m

you think john carmack has fewer responsibilites than you? while you’ve wasted enough time on here to rack up 15k karma?

jb1991
0 replies
2h43m

Yes. His kid is an adult now and he is reportedly single now too so that makes it a lot easier to do this. The daily life of a family is a large time-consuming responsibility and many can’t lock themselves into a hotel for a week for that simple reason.

1123581321
0 replies
2d4h

If you're feeling unable to learn, have an honest conversation with your family about your career and your interests. They may be more willing to support you than you think. Just as you'd do the same for them so they could achieve something important to them.

seanhunter
4 replies
2d5h

Tony Stark was able to do it in a cave with a box of scraps.

jareklupinski
3 replies
2d5h

Maria Hill: When did you become an expert in thermonuclear astrophysics?

Tony Stark: Last night.

The Avengers (2012)

belter
2 replies
2d4h

That is nothing..Sam Altman did it, on his way to this podcast... https://youtu.be/4gASfaRWe7s

deadbabe
1 replies
2d4h

On Hackernews some people don’t even do it at all, they just know.

ethbr1
0 replies
2d4h

We do it by reading a thread of such insightful comments that our third eyes are instantly opened.

zyklu5
3 replies
2d5h

If he could've done it in a week then he didn't really need to do it at all.

falcor84
0 replies
2d4h

Sometimes you have pretty much all the prerequisite knowledge but are still missing a crucial piece that would help you "piece it together" and make a significant advance.

admissionsguy
0 replies
2d4h

Yup, prerequisites is all you need. I can go through any course of study in a week as long as I'd mastered it before.

OJFord
0 replies
2d4h

I don't think that's true, thinking back to university, if I 'locked myself in a hotel room' and did nothing but study a given topic, I reckon you could get a decent grasp of more than one lecture course.

Of course not being able to ask someone any questions would be disadvantage (but is that even a 'rule' of this scenario? Surely at least the internet is allowed, i.e. a person to ask who's available to anyone), and it would have been a lot easier while at university than with X years' rust on the background maths etc. - but still, seven full days is a lot of time if you choose to completely dedicate it to some particular thing, we just generally don't do that, so it doesn't sound it.

mtlmtlmtlmtl
2 replies
2d3h

If he really did it in a week, why hasn't anything come out of his AGI startup yet, two years later? Maybe it wasn't as easy as he seemed to imply, despite his towering intellect.

Anyone can read an absurd amount of text in a week. To absorb and understand it all is a different matter entirely.

Seems more likely to me that Carmack simply convinced himself he'd understood it all. He does seem to think very highly of himself.

giantg2
0 replies
2d3h

I would generally agree. Another point could be that even if you understand something, executing on something can still be hard, especially when trying to push beyond existing limits or implementing something complex.

AlchemistCamp
0 replies
2d1h

The guy did pioneer 3D gaming, improve linux networking in the process (to support his games), build vertically landing, reusable rockets years before SpaceX's and play a pivotal role in modern VR.

I would not bet against his understanding on technical topics where he's done a serious deep dive. Also it was more than a year, not just a week.

indigoabstract
1 replies
2d4h

How hard can it be? It's only 30 papers.

Maybe someone should ask Chuck Norris how long it took him. Who wants to go first?

tarboreus
0 replies
2d4h

Chuck Norris ghostwrote those papers while on a motorcycle tour of Brazil.

sage76
0 replies
1d20h

Must be a hell of a hotel room, I should visit.

belter
0 replies
2d5h

Almost forgot about how Carmack is the one that will give the world AGI...Easy to take the eyes of the ball with all these OpenAI and Google distractions...

Keyframe
0 replies
2d5h

Carmack says the darndest things.

ykonstant
1 replies
2d5h

I wasn't aware of the Kolmogorov complexity book; it seems great, I'll be devouring this soon!

mark_l_watson
0 replies
2d5h

I will also. I heard a lecture on Kolmogorov complexity at an AI conference (perhaps AAAI IN Austin in 1984?) and it forever changed my view on complexity and patterns in data.

admissionsguy
0 replies
2d5h

If you make it your full time job for a year or so

Sounds like a good MSc program

Xelaz
0 replies
1d20h

FWIW, Ilya's recommendation for Kolmogorov book in the Arc viewer is "PAGE 434 onward", which lands on the last chapter "Algorithmic Statistics" (p.425 in the book). Appendix 1 starts on p.455 ...

smeej
5 replies
2d4h

How long does it take to learn the prerequisites to learn these if you don't already have a CS background?

Reading through the list, I can barely parse half the titles due to acronyms and terms I've never seen before. Would I be able to figure them out along the way? Or is there a body of knowledge this is assuming I should already have?

(This isn't a criticism of the list, or even the idea of presenting such a list without explaining who the target audience is and isn't. I'm genuinely wondering, if I just start at the top, is there any hope I would figure this out, or do I need to start somewhere else entirely?)

smeej
0 replies
2d2h

I think I may just need to accept that the work it would take to get to this level would mean a whole life change.

It's been 20+ years since I took calculus in high school, and I haven't taken a math or science class since. I reckon that puts me at least five years behind where I'd need to be to start this course and the listed ones.

Thanks for the recommendation, and helping me assess whether it's the use I want to make of my time!

barfbagginus
1 replies
2d1h

1. Bibliography Blues

The papers have bibliographies with citations in text. The knowledge you need to read a given paper usually lies within 3 degrees of bibliographical separation.

You can also use reverse citation search on Google scholar to find newer works that cite a given work. This can uncover a review or textbook, which often makes the concept much simpler to digest. And it can be a more advanced approach which gives you an alternative view or lets you abandon the original paper altogether.

2. Back to tha basics

For math, it would help if you have college level calculus, differential equations, linear algebra, and statistics. If you don't have that, be ready to assign yourself math readings and homework. It would also really help if you had analysis, topology, abstract algebra, and category theory under the belt, just so you can see the mathematical forest for the trees... But that's not really so necessary

You can often find the author's preferred reference for a given work in the bibliography, and use that.

For computer science, it helps to be able to read algorithm pseudocodes, follow complexity and correctness proofs, and understand performance experiments and charts. Skimming Knuth's "Art of Computer Programming" should be very helpful.

You'll need some algorithmic/statistical learning theory as well, since authors often use these concepts to prove that a system can learn. Though TBH, I usually skip these proofs and focus on the performance details. I usually find a higher performance paper before I need to care about the learning proofs. Still it helps to expect them and have an idea of how they work, if only to know when you can safely ignore them.

3. Narrative Arcs

Many AI results build on keystone papers that define eras. We can build on these papers by understanding their philosophy and deeper implications, even if we don't really understand their math. There's a story that you will tell yourself, and it may lead to interesting research or business results.

For example, if you read "Attention is all you need", you set the stage for LLMs. then "LLMs are in context learners", followed by the "chain of thought", "reflexion", and "toolformer" paper will give you a great sense of how LLMs are emerging as more than just stochastic parrots, ready to be developed and deployed in a lot of ways that overcome their issues. But looking at how bad the numbers can be even with our best efforts, you also get the sense of how much work these systems will need to actually be good, and the sense that corner cutting will lead to a lot of pretty awful systems.

It's a fun time to be a computer scientist or software engineer! Lots of cool papers to read!

smeej
0 replies
1d20h

Wow! I'm glad there are people like you out there to be excited about it! I glazed over before I even got to the end of your comment, with the dazzled sort of awestruck glaze where I realize I will never do even a tiny fraction of the work required to know things about this topic you probably take totally for granted.

Thanks for taking the time to expose me to your enthusiasm!

jononor
0 replies
2d3h

Depends on what you mean by learn... And what other relevant background one has outside of CS - programming/math/stats/systems/engineering. Assuming one has a few years of that, and one want to get to the level of being able to apply this to a problem that comes up in a realistic setting (making a product etc). Maybe extreme dedication over many months, or a few years at 20 hours a week? This is assuming that one either is very good at studying alone, or have access to people one can ask alone the way.

If one has no relevant background, then I would 5+ years of full time study.

dennis_moore
0 replies
2d6h

I failed to find any evidence that this person is an OpenAI employee, but please correct me if I'm wrong.

3abiton
0 replies
6h19m

What's the relevance compared to the latest AI advancement including transformers for example?

latexr
4 replies
2d7h

This is just someone’s bookmarks folder. Where’s the proof these were recommended by Ilya?

This looks like a subtle way to advertise a VC funded web browser.

latexr
1 replies
2d5h

That confirms the existence of a list, there’s zero indication it’s this list.

Tenoke
0 replies
2d5h

He posted it when this list was getting viral and acknowledged it without saying it's not the real one. I read it as a tacit confirmation at the time.

dennis_moore
0 replies
2d6h

A canonical list of references from a leading figure would be appreciated by many.

That confirms the opposite, no?

Ringz
4 replies
2d7h

Reading 30 studies does not mean "learned and understood". Especially if you start from zero.

beepboopboop
2 replies
2d7h

Disagree, I’ve already changed my title to “ML professional” on linked in. Aside from the odd hallucination, I’m indistinguishable from an expert to the average hiring team.

seanhunter
1 replies
2d5h

You have to use "Certified ML professional" or noone is going to take you seriously.

rawgabbit
0 replies
2d4h

I changed my title to “OpenAI Certified Professional” and 10 recruiters have already reached out!

barrenko
0 replies
2d6h

People will conveniently disregard the fact that it's meant for Carmack, so not an aspiring coder.

throwaway71271
3 replies
2d5h

Some time ago I also copied the list at https://punkx.org/jackdoe/30.html so its easier to wget (e.g. get all the arxiv pdfs)

        wget -A .pdf -r -H -D arxiv.org -nd https://punkx.org/jackdoe/30.html
A friend and me used pdfunite to join them in one pdf and then printed them in a local printing shop in good a4 paper https://www.printenbind.nl/ as a spiral book, the end result is very good (also there is space around the pages so that its easy to take notes).

Keep in mind there is no confirmation that this is the actual list.

rossant
2 replies
2d5h

How many pages is the spiral book?

msoad
0 replies
2d3h

xLSTM is really really hard to grasp. It's mind boggling how they came up with such a unique architecture. I read the paper multiple times and still not sure if I can say I understood it.

snowfield
2 replies
2d7h

Just learn 30 papers

r0ckarong
1 replies
2d7h

Or keep doom scrolling some other shit. Your choice.

At least this is relevant information for people who are actually interested.

password54321
0 replies
2d3h

You mean for those who will just add it into their bookmark pile. You could have started yesterday and built a NN from scratch if you were truly interested.

js8
2 replies
2d5h

The argument that this is 90% of what matters in ML seems a bit bold. AFAICT it is completely missing reinforcement learning, which has been source of some mindblowing results (from Deepmind) in the past decade. It is also missing other stuff that I as a layman found fascinating (graph neural networks, 1.5 bit networks, ..). Also, I am not sure how much the Kolmogorov complexity actually matters, it's more philosophical than practical. So beware, it is a very opinionated list.

pmelendez
0 replies
2d3h

The argument that this is 90% of what matters in ML seems a bit bold. AFAICT it is completely missing reinforcement learning,

Wouldn't that account for the other 10%? To me it sounds like the quote is not too far off when you consider that he said "of what matters today". There are many other things missing from that list that might become more relevant tomorrow.

PartiallyTyped
0 replies
2d5h

AFAICT it is completely missing reinforcement learning

There's no RL in there, nor diffusion. There are many other very interesting papers that I think have lots more to offer — but I just have a master's degree here. I am not a phd let alone a distinguished researcher.

ausbah
2 replies
2d5h

is there nothing here about attention?

rsfern
1 replies
2d4h

The first link on the list is “the annotated transformer” which goes into a fair amount of detail on attention, it’s an annotated version of the paper that introduced scaled dot product attention

ausbah
0 replies
2d1h

i didn’t see that :)

Woodi
2 replies
2d4h

On the related subject: any way to filter out anything AI/LLM related from HN ?

Woodi
0 replies
2d2h

O, or maybe plugin with regexes will do for 90% use cases ? ;)

taraparo
1 replies
2d7h

Would be nice if someone put those into an llm and generate a single coherent document with all the essential knowledge.

CamperBob2
0 replies
2d3h

That's... pretty much what an LLM is. Dive in, and when you get stuck, ask questions in Claude 3 or ChatGPT4.

rrr_oh_man
1 replies
2d7h

Source?

wartijn_
0 replies
2d6h

I don’t think John Carmack will gain much knowledge by studying a game engine.

gessha
1 replies
2d3h

What people forget is without a proper, tangible goal, no matter how many core or interesting papers you read, you will still remember nothing. At most you will be able to make armchair ML researcher comments on an online forum.

GaryNumanVevo
0 replies
1d8h

I have a career in ML research and I use most of my knowledge to be an armchair ML researcher!

digitalsushi
1 replies
2d3h

oh man my cognitive dissonance is crazy cause i am easily in the top 50% percentile for being able to use popular LLMs but that reading list is outside my comprehension abilities. I cannot synthesize a thing I use

drivebyhooting
0 replies
2d1h

Do you use pencils? I doubt you can synthesize that.

qwertyforce
0 replies
2d5h

Looks fake

pmayrgundter
0 replies
2d7h

Is this product spam? List looks legit, but clicking on one launched some popover that disappeared and then no paper

navs
0 replies
21h25m

Great way to advertise Arc.

mkl
0 replies
2d6h

There are a few past discussions, going back to October 2022. I wonder how out of date the list is now.

Ask HN: What were the papers on the list Ilya Sutskever gave John Carmack? https://news.ycombinator.com/item?id=34641359

396 points by alan-stark on Feb 3, 2023 | 137 comments

--

John Carmack’s ‘Different Path’ to Artificial General Intelligence (dallasinnovates.com) https://news.ycombinator.com/item?id=34637650

378 points by cmdr2 on Feb 3, 2023 | 495 comments

--

Ask HN: Where can I find an updated AI reading list? https://news.ycombinator.com/item?id=33166948

3 points by costant on Oct 11, 2022 | 6 comments

barfbagginus
0 replies
2d3h

This has nothing on it about llms, where in context learning, retrieval augmention, tool using, and multi modality is really blazing ground. Ideally we'd have papers about these methods, because we need these tools to implement stuff, and they're changing how we do business.

Without in context learning, self critique and tool using, for example, it's easy to think that LLMs are stochastic parrots bound by their training distribution, instead of last mile learners that can adapt to out of distribution contexts. And business applications need RAG to overcome hallucinations. Without these concepts, I don't think people will really know 90% of what matters today, or be ready to take on what matters tomorrow.

I could be wrong here, with my focus on so many llm specific topics. Things like in context learning and RAG exist as engineering workarounds for LLM limitations. They are not definite paths to tomorrow's AI, which won't need our cajoling and hand holding. There's a possibility that we'll move past them, so more speculative and theoretical papers are better.

But I think there's also a possibility that the AI scaling is done for this AI bubble, and we're stuck with building larger scale systems out of imperfect buggy LLMs for the foreseeable future. In that case, maybe some next generation Zen of integrating LLMs into a working mind eventually emerges as a way forward.. so we should pay attention to engineering efforts around LLMs!

alabhyajindal
0 replies
1d14h

What has John Carmack produced after reading these?

_giorgio_
0 replies
2d3h

I've seen this list many times. It's years old already.