A generalist AI agent for 3D virtual environments

There were 112 heroes available at that time. It's also worth noting that two of the heros chosen for OpenAI to use, Viper and Sniper, are considered some of the mechanically 'easier' heroes, as they rely primarily on autoattacks to do damage, as oppose to decision-making around when to use spells. Crystal Maiden, Lich, and Necrophos, the other 3 of the 5 OpenAI heros, are similarly considered 'easier' as they have spammable, very forgiving abilities that can be used almost indiscriminately.

Very interesting! Shows how much AI is over-hyped (even though, as you say, it was very impressive anyway). It was even worse in case of Starcraft 2, where the AI had a much wider view than humans, and while the AI was supposed to show its strategic superiority, by limiting the APM (actions per minute), the limit was still very high, inspired by the max APM achieved by humans - whereas this max is achieved only for a very short period of time (a single minute), and consists mostly of insignificant click spam (had APM been limited to half that for humans, the effect would probably be negligible, and very minor for a quarter...). So as a result the AI would win by being able to micro-manage more units, rather than having a better strategy. But again, it was very impressive anyway.

One of the constraints ended up being implemented into the main game: separate couriers for each player. But generally, agree with your point.

But it's very cool how the OpenAI matches ended up making mid players reevaluate how they used consumable regen.

That's a fair assessment but it was also 6 years ago. Back then transformers had recently come out. Tricks to do DL at scale were still brewing. Even achieving what they did for 10 heroes showed DL could work in "non-deterministic-ish" problem settings.

The more interesting question is: can we train a Dota model that plays with all 124 heroes today?

He was a professional gamer, and switched to biodynamic agriculture after seeing how little hope humanity had playing games against machines

A slow transition (over 5+ years) from web/desktop development into HPC and AI/ML.

I still consider myself a 4/10 at best compared to my amazing peers who studied this from the start, but you have to start somewhere!

A big advantage of AIs is instant reaction time. OpenAI programmed in an artificial reaction delay to most skills, but they were still generally much faster than any human would be. Overall strategy is where the AI was lacking, but its technical flawlessness makes up for it.

As I explained in my sibling comment the version of Dota they played was heavily simplified, because the full combinatorial explosion of mechanics was far too much for the AI training to overcome. They didn't even get close to playing normal Dota at a high level, nevermind a hypothetical version of Dota which is twice as complex.

You would need to define what kind of complexity.

I would broadly break it into things that are complex to perform (crazy APMs or accuracy), things that are complex to understand (the stack or layers in MTG), and things that are complex to predict (e.g. time-delayed abilities and the correct time to use them, like Baptiste's lamp in Overwatch).

AI have basically constant performance across a performance complexity curve, because the complexity typically derives from physical interfaces the AI doesn't use anyways. E.g. their APM is not limited by how fast their fingers can physically move.

AIs do very poorly on tasks that are complex to understand. The best Magic: The Gathering AI's I've seen are awful (though also likely far less well-funded). Best-case scenario is basically an AI who makes plays that don't make any sense, but are at least valid plays. It's a crazy difficult problem. E.g. there are various ways to make infinite mana with combinations of cards, and the AI needs to a) realize that it can use those cards in order to create infinite mana, and b) that it can do this multiple times (I.e. it can pay for a spell that costs more mana the loop generates by going through the loop multiple times). That's very hard thing to do; human players somewhat frequently don't realize when they have loops.

Add on top of that that a game of Magic can enter a state where a loop of effects becomes recursive but doesn't result in either player winning. The game is a draw, because it cannot progress anymore. Detecting these can be non-trivial, because they might involve side effects that look like someone should win (I.e. you lose a life and I gain one, then I gain 2 life, then you deal 1 damage to me, then I gain 1 life, then you deal 2 damage to me. Life totals shift around, but net to 0 by the end of the loop).

I think the AI do well at complex prediction tasks as well, by nature of their response times and access to prior information. I would expect an AI to beat humans by a wider margin the more complex the prediction gets. Humans have finite time and thus experience; the AI is going to have more "experience", and be able to recall it at a faster rate.

If you're talking immediate mechanical complexity, then yes. If you're talking delayed strategic complexity, then definitely not.

Was there any control input limit on the AIs? Like if the AI couldn't click buttons or move the mouse faster than a very fast human.

In a similar vein, it would be fascinating if the AI had to also evade bot detection, that is appear (nearly) indistinguishable from a human player.

Dota's yearly competition, TI7 means "The International 2017"

I too was impressed at first, but got disillusioned after learning how they did it. It was much more similar to chess AI with piece-square tables than I first thought.

https://gist.github.com/dfarhi/66ec9d760ae0c49a5c492c9fae939...

No? It's absolutely not what happened. Almost the opposite. The micro intensive heroes were NOT in the pool. AI literally chose Sniper and Sven.

AI's APM was limited to a level lower than pro human players. If they had allowed micro heroes AI would have a big disadvantage.

That was the most interesting part about the AI Dota game. AI isn't just better than humans at mechanical level. Even more surprisingly, AI (at least Open Five) isn't significantly better at last hitting than pro players.

AI in most strategic games are provided massive advantage over human players. In CIV V, for example, the AIs start with several units and techs on higher difficulties.

They can be problematic, but for many they are just a convenient excuse to spend time with friends in a voice chat. It's not really more or less of a time waster than most other group activities.

Now playing alone for the dopamine rush of successfully grinding repetitive tasks: yeah, that's a bit of a time waster. Maybe therapeutic for some, and definitely not the most harmful way to spend time and get validation, but also a bit pointless. But I would argue that if you play an MMORPG alone you're doing it wrong. If you don't have friends at least get engaged in a guild and spend countless hours improving real-life social and leadership skills.

It’s not just about time.

During the teenage years while the neocortex is growing, teenagers are practicing and honing fine motor control. Video games help develop that, as well as learning about social interactions, emergent system behaviour and strategic vs tactical thinking.

I’m not saying they don’t have downsides, but there are some upsides too.

Sure, if you’re an addictive personality using video game escapism to ignore your life problems, that’s a whole different thing (and even without video games this type of person would just find another form of escapism)

So I don’t agree with your generalisation of games as “time wasters” - maybe for you they are, but not for everyone, I don’t play them anymore very much (just a bit of chess every now and then) but they provided me with a lot of understanding in my formative years

They are about as much a time waster than any other hobby really

We’re an irrelevant blip in the natural history of our planet. Everything is a waste of time.

The only tried and true metric that HN always falls down to question universally :

"Why did you do that if it doesn't make money?"

As someone who met his wife in an MMORPG (World of Warcraft) and still actively plays it each week with the same group of friends from 15 years ago, I'm not so quick to hope they die.

It's just one way to get enjoyment out of life. It's worth remembering that the logical conclusion when all human works are automated, we'd be left with play. In that lense, wouldn't play be a good thing? Otherwise why are we fighting so hard for it.

What do you propose as an alternative?

It feels fun to be rewarded for something you accomplish in-game.

In many singleplayer games, you can slide difficulty up or down to change the effort:reward ratio.

In an MMORPG, though, you have different groups of players with different amounts of time. You want to make it fun for both the kid on summer vacation who is happy to spend 80 hours a week on a game (not a choice I want for my kid, but I was a kid once too) and an adult who has a 60-hour work week and exchanges 2 hours of sleep after the kids go to bed to play.

That means the person with more money than time will want to buy things from someone with more time than money. But this causes all kinds of distortion in the game balance and economy.

I don't know that this is solvable, whether you're trying to balance against cheap labor or AI bots.

If the botting is profitable then it is trivial to build an interface for a fake controller and screen grabber to do the work.

disable rendering and keep compute pretty minimal.

So that's what was going on in the Matrix when the humans were staring at all that green text.

the idea of a bot-only MMO would be really interesting

Then you may be interested in Screeps: World

Most of these games are pretty slow paced. I built mining bots for RuneScape 20 years ago. We would program a route to the resource and then look for the right colored pixel on the screen and click on it. Repeat x times and walk the route backward to the bank

Everyone's a skeptic these days.

I'm pretty confident that people will find ways to make fun and interesting games using them, just like they have with every other computer technology over the last 40 years.

Anyway, with AI characters, I expect it will actually be more frustrating and boring than not if they have realistic lives and schedules. All the littler irritations that we deal with and accept form real people just become friction in a game.

I remember Ultima V in the 80s had this. NPCs had their own daily routine and went here and there throughout the day. A couple of side quests relied on this mechanic—you had to learn their schedule and catch them somewhere at some time.

Well, weird movements in games should be a thing of the past in the near future, as we can begin to extract motion capture data from videos of normal people acting normally.

I think it depends on the type of game you have, but I wouldn't underestimate this type of technology for say, open world games where it might make the game more immersive due to convincing realism.

Gabe Newell touches on this exact subject: https://youtu.be/MGpFEv1-mAo

That being said, I've noticed that realism in games has a function beyond gameplay: it makes the game cooler.

For example in Rainworld, the enemies keep doing their own thing even when you aren't around. This has some gameplay value (makes the game less predictable), but I think the main value is that people talk about it and think it's interesting. So it changes people's perception of the game world in a positive way.

That sounds like a game that'd be an instant buy for streamers, though. All the weird, annoying, unexpected interactions? Possible comedy gold in the hands of the right person.

Then we'll make new kinds of games where the unpredictability of the NPCs is a core mechanic.

Seriously, have you no imagination? Why sit around coming up with reasons it won't work instead of ways to make it fun?

You can control LLMs to any arbitrary degree of specificity by a mix of retraining and changing the decoding strategies. They can be as predictable as needed, I think the bigger issue is more like how do you write stories when the possibilities get so much larger.

That sounds really cool, as a genre, actually. You would have to figure out what given NPCs actually want, and how you may be able to use that, to make them do what you want. And not based on some lame script, but complex world and character models. I'm sold!

Some of the most hilarious and enjoyable parts of gaming is manipulating the environment in ways developers didn't anticipate. With more intelligent NPCs there will be opportunities to give them instructions, and let them go to get it done.

Crafting instructions that lead to NPC's working to achieve a goal that wasn't anticipated, and provides you with a resource that couldn't otherwise be had in the game, will be a whole new aspect to gaming, and inspire a desire to experiment and explore.

Maybe give them a “Robocop Directive 4 [HIDDEN]” that prevents significant actions…?

Sandbox games are pretty big. Not my thing at all, but I’m sure there’d be a big market for believable open-world RPGs for example. They’d probably be more simulators than narrative-driven games.

I mean, that's solved with a little prompt engineering:

"context": {"Go about your daily life, but you have an IQ of 50 and have zero ambition."}

What makes cyberpunk 2077 such a great game is that the world doesn't revolve around you, it exists despite you, and you see this in all of the side-quest events that are happening in the background. The world being alive despite you, and not for you adds a whole new dimension to the universe.

Skill or mastery is not what LLMs offer in-game AI though, it's creativity, originality, and agency. Players don't primarily prefer to play against other human players because they're an "easier" opponent, but rather because they're dynamic, creatively proactive and reactive, and endlessly original.

I submitted that one earlier

With a made up title which you should avoid, it's in the thing https://news.ycombinator.com/newsguidelines.html

I don't want to pile on in this subthread, just want to say that the reason I changed the URL on the current thread rather than re-upping your post (as we normally would when the better URL was posted by someone else earlier) is that it was the OP's first submission to HN and I like to encourage newer participants.

@dang doesn't do anything, you should email hn@ycombinator.com

that wasn't the question, but thanks

Single-player AIs are fine-tuned by the developers for fun instead of frustration. I think the GP is talking about undetectable online cheating.

Are real world physics really that much more complicated than video game physics?

Massively, in a game running into a wall is perfectly normal and valid strategy to get close to it, in reality that will wreck you.

Or running on a fist sized rock has no consequences, in reality that destroys your foot. Reality is full of such extreme threats everywhere even in normal homes.

i believe it is as hard as it is for robots to act exactly as they do during simulation

    DeepMind> kill dissidents

I mean they already are, just look at that announcement

Ultimately, [..]

I swear this short paragraph style rounding it off with an “ultimately”, “in conclusion” didn’t use to be so common. :

Ai is already strongly influencing how people write. After being successfully deployed for a year.

And once validated, sell to the military?

Can't wait to see the leaked footage of war crimes showing robots murdering civilians and teabagging their corpses

They’re expensive and ruin our bottom lines.

What counts then?

They intentionally included combat training in the dataset. It is in their Technical Report.

How can combat training not be interpreted as "principal purpose or implementation is to cause or directly facilitate injury to people"?

Do you believe the agent was trained to distinguish game from reality, and refuse to operate when not in game environment? No safety mechanisms were mentioned in the technical report.

This agent could be deployed on a weaponized quad-copter, or on Figure 01 [0] / Tesla Optimus [1] / Boston Dynamic Atlas.

[0] https://twitter.com/Figure_robot/status/1767913661253984474?... [1] https://www.youtube.com/watch?v=cpraXaw7dyc

And variation. Video games cheat at everything.

But humans have already trained on an incredible number of games (including reality) when they play No Man's Sky for the first time. What they say here is that training on N-1 games makes you better at the Nth game. So you just continue to scale this up.

Wow, I guess lots of people saw this coming a mile away

Good point, they are quite different objectives.

Their approach is one that works for simple directives: "Go to ship" or "Pick up iron ore" which lends itself well to sandbox-like games (which seems to be a major focus looking at Deepmind's tech report). Similar research has been done in Minecraft [1].

These instruction following agents are more an RL achievement than a language understanding achievement. On the other hand, Disco Elysium has over a million words of dialogue, and solving the quests requires an agent to understand and reason about language much more extensively. People have looked at text-based game agents, like Microsoft's TextWorld [2], but these are much smaller in scope and not easily adapted for humans-in-the-loop.

My work bridges that gap, focusing on the language aspect, rather than navigating a 3D world. Again, they are definitely different objectives, but as a sole researcher there's no way I can compete with Deepmind's budget and manpower anyway. Just look at the extensive author list in the tech report. So it doesn't make sense to necessarily focus on outcompeting them in producing a better generalized RL agent (in fact I merely use GPT-4). Instead, I made a publicly available experimentation platform that allows others to be able to build upon this work, which is valuable for the community at large.

At least, that's my take.

[1]: https://sites.google.com/view/steve-1

[2]: https://www.microsoft.com/en-us/research/project/textworld/

And for release builds, running the tests across a cohort of gaming profiles so you could get detailed results for various personas instead of just a guess and check in production. It'd be great for indie and hobby projects as well - getting playtest feedback cheaply and quickly would be awesome. Hope this turn this into a service.