I was one the developers responsible for implementing the netcode on Serious Sam. We often slept under the desks in the offices at croteam after lurking usenet. One post in particular described the QuakeWorld prediction system which inspired us. That night we coded a simplistic mvp as a colleague (hi dan) tested it over an old 486 nix machine acting as a router that we could simulate lag with. This was well before the actual game was built around it
Is this the same idea as "1500 archers on a 28.8" for Age of Empires?
https://www.gamedeveloper.com/programming/1500-archers-on-a-...
Yes. Both are “deterministic lockstep” systems. Many many games have used such a system over the years. Although it’s probably less common these days for a variety of reasons.
Is this due to the heterogeneity of client hardware?
Not primarily. There were weirder CPUs in the past. Today's CPUs have largely converged on floating point operations. Basically just sin/cos/tan differ now.
It's possible to write fully cross platform deterministic code today, but it's harder to do and uncommon in libraries.
Writing state synchronization and using client side prediction is generally easier. Determinism is only necessary if there are bandwidth constraints.
Also it's easier to cheat when your local client has all the data.
Not so much easier but enables a different class of cheats in games where players don't have perfect information, even with other schemes there is normally enough information leaked that similar cheats are possible. OTOH using a deterministic simulation prevents a whole class of other cheats.
What class of cheats are prevented in deterministic simulation but possible with a dedicated server running a state synchronization netcode model? I don't think any...
Well, unless you don't trust the server.
Wasn't that even worse before? Games like Warcraft and Doom played across Intel x86, Motorola 680x0 and PowerPC
Did Warcraft 1 and Doom have cross platform multiplayer back then?
I don't know about WC1/Doom1 but WC2/Doom 2 on the Mac also supported IPX to play against PCs in addition to the native AppleTalk support they had. The only time I ever saw IPX on a Mac.
Fun time. Developers essentially have to dig very low into the stack and deal with multiple architectures. The way to build great programmers.
They didn't have to deal with agile/scrum though. Clearly not real programmers ;)
I think another issue with lockstep is that for it to work, each client needs all the input data from all the other clients to be able to simulate the next step, otherwise the simulations become out of sync. So how does the simulation handle both your input, and another clients input that comes in 100ms later? If you just pause until you get their input, your game is effectively pausing and running at 10 steps/second. For comparison Starcraft 1 ran at 42ms a step, or ~24 steps per second. You'd be playing in slow motion.
Games like Starcraft 1 would handle it by adding an artificial delay to your inputs (and all your opponent's inputs). Putting it into a buffer for future steps instead. If anyone remembers the latency option for "Extra high latency", this would increase the delay on inputs so that simulations did not have to pause while waiting for inputs from other clients. And then if any network fluctuations causes your input buffer of other clients to empty, you have to pause again.
In fighting games this is usually called delay-based netcode, and people hate it. It makes your inputs feel unresponsive, and in games that require fast reactions (like fighting games) gives you a smaller window to react.
This is all way above my understanding, but:
Recently I’ve been playing Arma Reforger with friends online on a dedicated server, and when driving vehicles the collision physics are off, especially when driving fast, making the whole scene “teleport” forward, with frames lost and the vehicle reappearing in unexpected positions.
I wonder if this is related to this type of code. The game is technically out of early access, but there’s lots to be added still on the roadmap.
Lockstep determinism has a lot of issues.
First, building a PERFECTLY deterministic sim is brutally difficult. Even for a single piece of hardware. It’s so so hard and expensive. A single bug “crashes” the game with a “desync” error. Making it work cross-platform was never really an issue.
Second, you’re bound by the slowest client. No one can go faster than the weakest link.
Third, you can’t stop cheating. Every client has full and complete world state. This is fine for co-op but a major issue for competitive.
Really the ONLY benefit to lockstep sync is it minimizes bandwidth. In the days or 56k this was a strict necessity. But these days client-server is almost always a better choice.
Indeed, imagine all the cpu / compiler / different platform, all must behave the same.
Factorio is a good example of a modern game that works this way.
This number actually shows how weird it is to have a unit cap in Tempest Rising.
Either you have the resources, or you don't. Don't cap for reasons of capping.
Croteam is such a talented team of game developers. I really enjoyed the Talos Principle (1 and 2) and they were some of the early pioneers of a completely custom Vulkan game engine for the first game.
I found devastating that in The Talos Principle 2 they had dropped their own engine and used Unreal Engine instead.
Me too. I think the main reason is that the lead render developer left, and switched to google to work on Stadia.
Ouch.
So now that lead is looking for a job? :)
it's like every big game (studio?) is either microsoft already, or just not yet
like every gameplay stands as an engine, or a genre, or a meta-genre? like GrandTheftAuto?
but i was i trying to tweet something about getting lost in stocks and tickers and symbols
Probably has buckets of cash, so no? :)
He now works at Roblox.
Damn. No official Linux version :-(
https://www.protondb.com/search?q=talos
Gets a gold rating! I've been playing even the VR versions on Linux (through steam).
Yeah it works flawlessly on Linux, even nvidia RTX works.
Talos 2 was a great game. Except the awful TAA. Made some of the meta puzzles almost impossible for me, because I literally couldn't see certain things.
I had to google TAA:
Q: What is TAA and why should I care?
A: TAA stands for Temporal Anti-Aliasing. It's used to fix/clean up aliasing in games. The different kinds of aliasing in games are edge aliasing, shader aliasing, texture aliasing, temporal aliasing and specular aliasing. Edge aliasing makes edges of objects look jagged (also called as the staircase effect). Temporal aliasing is the shimmer of objects which are either small in nature, or which are far in the distance such as power lines, railings, stairs, balconies, antennas etc... You should get the idea.
https://old.reddit.com/r/FuckTAA/comments/oi0v86/taa_on_vs_t...
I found this disappointing too. I know they don’t have the resources to build their own equivalent of Nanite and such, but… On top of the performance issues, there were a number of things that actually looked so much better in the 2014 game. For example, the forcefields and water ripples.
I just found out the Talos 2 DLC is out on Steam this Friday!
Serious Sam was always a killer LAN party game. Not necessarily because it was the sexiest title of the day, or what anyone had planned. Serious Sam won the LAN party because even when every other game died under some driver issue, thermal issue, update problem, whatever, you’d load Serious Sam and it would just work. This continued through the later series too, hell of someone’s machine was completely dead in the water it would also reliably split screen and handled input well enough for split peripherals. The systemic parts of the game were truly exceptional on the reliability axis.
Serious Sam looked amazing for the shitty hardware it would run (FAST) on.
On a similar note, Counter-Strike never looked good, but was popular for a long time because it ran great on toaster PCs
I think this also explains the continued success of World of Warcraft.
There's a lot to be said for "good enough".
Counter-Strike was/is popular because gameplay is awesome. Players stayed long enough that even "toaster PCs" could run the engine. It was never "cutting edge" even after major upgrades.
It was popular because it was fun.
Also, the many speaker sound of “aaaaaaaaaaaaah” was delightful
I remember getting through some battle and coming on a room FULL of ammo of every kind. First time, it was awesome. But gradually ammo caches started making me say "Yikes!"
noone's mentioned the gargantuan size of the bosses!
playing co-op, running with your mates across the plains, blasting rocket trails into the distant horizon across the gorgeous clear blue skies at the biggest boss you've ever seen, with the pumping music thumping the whole way through, while you're trying to fend off what feels like hundreds of mobs ... I don't think I've experienced anything of that scale and intensity since ... one of my greatest of all time
"aaaaaaaaaaaaah, yourself" was the best response.
In the late 90s, I managed the EA Tech Support website. The support/QA* teams would have massive after-hours gaming sessions playing Serious Sam. It was the only FPS that ran consistently on our work PCs, and it was a ton of fun.
* At least at that time, EA QA and Tech Support had a lot of overlap; support guys would be come in-house beta testers in the Summer when they were trying to get games done for the holidays and do Tech Support in the Winter around Christmas when the calls went up.
Starseige Tribes was massive and ridiculously fun on a 56k connection.
Tribes was brilliant. Skiing down procedural terrain (and watching other people ski up it), big maps, lots of players, in 1999.
Tribes was an absolute riot. Teams keep trying to make new versions but none feel like they capture the pure magic of the original.
The original is out there and free now. Though the community is gone :(
Quite possibly my favourite game growing up (especially Tribes 2!)
In fact, I downloaded Tribes 2 a while back and was playing against bots just a few months ago.
The game is dated but it was still fun... in fact, I've often thought of trying to remake it in Unity or something!
Maybe one day.
I never liked tribes 2 being a tribes 1 addict. The jet sliding in 1 was a happy accident and the developers lost the original physics code that resulted in the sliding being a controllable skill. Tribes 2 had an utterly different movement sensation and a “press button to glide” mechanic that just killed the game for me.
There is a whitepaper exploring similar netcode concepts that was written by the Tribes developers:
https://www.gamedevs.org/uploads/tribes-networking-model.pdf
Every italian knows the game thanks to Zeb.
I'm Italian and I have no clue who Zeb even is.
Zeb89, buddy you werent on youtube 15 years ago?
Always been, never seen/heard of.
I guess there's plenty of popular content on the internet or elsewhere I could name you and you'd be like "never heard of it".
Even extremely popular one.
i guess it depends on how old you are
37
The kind of game where I believe I moved backwards more often than moving forward.
Haha they even made a game about it, and its called "I hate running backwards". It's on Steam. I don't know if its from the same makers but it is in the Serious Sam universe.
Well, you and Netrisca were together, while those thousands of enemies, they were alone.
And the gun that seemed to shoot a fraction of a second before you clicked the mouse button.
...and, IIRC, desparately grasping for generated ammo to pick up while making said retreat.
Isn't this exactly how Doom worked, long before Serious Sam?
But Quake went a different direction, sending the client updates about every entity they could see. Which had some benefits (players don't have to be perfectly synchronized) but also drawbacks (bandwidth proportional to scene complexity).
Quake is vastly superior because the server actually run the simulation, for Serious Sam the server is basically a dumb relay.
The server still runs the simulation for serious Sam and would kick any client getting out of sync (e.g. because they're cheating).
It's not so much "vastly superior", it is more fit for its intended design and context. And certainly more sophisticated.
This is covered at the very end of the article, in the "Comparison with Doom and Quake" section. And yes, that's exactly the comparison and distinction the author draws, and they explain why Serious Sam would have opted for Doom-style networking (the notoriously high scene complexity).
Serious Sam was such a fun game - this was a really interesting write up too. I wish this kind of dedication to low latency and efficiency was more widespread.
totally agree. it was revolutionary at the time for the number of enemies on the screen. even at the time the vibe was very much, “croteam knows what they’re doing”.
really neat to be reading about this so many years later.
It still is revolutionary. Serious Sam 4 still doesn't take itself seriously. Best money i've spent on a shooter since ... Serious Sam 3.
I do miss the sledgehammer.
Agreed, have fond memories of the game! Just pure FUN for the sake of it! Really cool to read about the tech behind it so many years later, this is very well presented
Yet games with 10x the bandwidth struggle to support that many enemies. It only just occurred to me: an increase of technological resources actually has a reversing effect on the efficiency and creativity of computer science. As bandwidth, storage, memory and compute has gotten bigger, there is an inverse response in the software that makes it slower, more bloated, and less capable, per discrete unit of resource. Call it the Benjamin Button software design effect
Do you have a number for "that many enemies"? If article is giving one, I couldn't find it. There are plenty of modern games that is multiplayer and supports enemy numbers that I would consider "massive". Or massive number of players if that is that matters.
I played it last night. There would be maybe a couple dozen enemies on screen. Which, for a dial-up connection, is a massive amount to communicate about in real time. The fights are structured such that it feels like 5x that amount, though.
Better known colloquially as software bloat!
Similarly, Factorio's architecture transmits (almost; notable experience is rail planner) just input events, relying on a lock-step simulation core.
I would love to work on a lock-step thing like that one day. Seems such a satisfying (and testable) design constraint to work with.
Their friday facts are very worth reading, not just for people that play computer games, it is interesting for any developer. If I remember correctly they had issues with parts behaving slightly differently on different platforms, which then lead to sync issues in multiplayer.
I have no experience here, but I guess the difficulty of such a strategy comes from ordering "events" in a game loop.
Impressive writeup. The deterministic aspect is not super popular anymore, I guess, but this entire post does make me wonder how the networking layer compares to a slightly more modern multiplayer game.
It's very common in RTS. StarCraft 1 and 2 also use rolling lockstep.
Both those games are contemporaries of the Serious Sam series though. StarCraft came out three years before the first Serious Sam! But yeah fighting games and RTS games both commonly use deterministic sims.
Do I get it wright - the reason to implement a custom protocol above udp instead of using tcp was the benefit to have a certain amount of packets that didn’t need to be acknowledged?
Packetloss with TCP creates unnecessary latency for games.
The OS TCP stack basically will show data in a sequential fashion to the userspace code, this greatly simplifies things like a HTTP stream but in a game it also means that even if 9 out of 10 packets have arrived, if the lost one is the first packet then we need to wait for another roundtrip or more before seeing all the packets because the first one still needs to be re-sent before anything is seen.
With UDP and custom handling you can just proceed to render and show everything as it shows up, asking for a re-send will at worst cause re-simulation in an engine like SS's but for most parts the player won't notice unless something significant was lost.
For those that played Quake 1 before Quake World the difference was staggering, a Q1 session would freeze on packet-losses when the client tried to catch up(hard when a significant portion of the bandwidth was already used on the modem) whilst QW mostly kept on trucking with some jerks (they probably made other improvements also).
I loaded up the original Serious Sam The First Encounter about a year ago, and that game never ceases to amaze me with how much they were able to squeeze out of relatively low-powered 2002 computers. I had it as a kid, and I had never seen so many enemies on the screen at once.
Even now, the game mostly holds up pretty well and it still kind of impresses me.
Croatia is one of those countries i can't point out on the map but routinely churns out amazing innovation.
without interpolation … It's kind of like playing a modern console exclusive.
Ouch. It’s sad to see console gaming no longer be the place to go for highly tuned software.
To be or not to be, this is a serious question.
I only played Serious Sam for about 10 hours sometime around release, but to this day the long scream of the beheaded kamikaze will cause me immense physiological distress.
Anyone else miss seriouszone?
We used deterministic game play to implement multiplayer on the GB Color port of Vigilante 8.
The GBC Link Cable would pass 1 byte in each direction at the same time. It's a pair of shift registers filling each other up across the cable.
The game was locked to the GBC's frame rate. There was a lot work to update the screen that had to happen in each (effectively) V-Blank and if it was missed the smooth scrolling stuttered.
At multiplayer startup we passed our seed. To run it looked like this:
On frame A it reads the controls, and packs them into a byte and puts that in the transfer buffer. The transfer occurs while it renders frame B. At the start of frame C it has the local controls encoded in the byte sent on frame A and it has the other side's controls in the byte received in frame B.
It applies the controls to the game state and renders frame C. Local and remote controls are applied with one frame delay.
There was no frame delay of the controls for local play so if you ever lost in multi-player feel free to blame lag and me specifically if you need to.
Ahh, I remember playing a PC Gamer demo of Serious Sam as a kid. Even back then it was considered a "throwback" game to the days of old-school DOOM and Quake.
And now literally two decades have passed and its considered a classic in its own right.
That was a fun game, I loved the demo.
I think some of the extrapolation might have also inspired optimizations made in CPMA mod for Quake3, which to this day, I think, has the best netcode of all games. You can actually compete with 100 ping, in it, which is completely unplayable in OSP or vanilla Q3. I don’t think any other game even attempts to be so optimized for competitive online FPS.
I love how there's an article about how some legendary game was made, and someone in the comments casually goes "oh yeah, I built that, fun times". It's great.
HN is wonderful for that. I once commented on an article about Brave Browser that it was an ad extortion racket, and Brendan Eich showed up to call me an asshole. Good times!
Link?
If you know a good way of searching through my entire comment history I'm happy to search for it, but it was some years ago, and I post several comments a day, so it may be a tall order to manually page through my history.
You can use the sort of official HN search engine like this: https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...
Found it: https://news.ycombinator.com/item?id=11230347
https://news.ycombinator.com/item?id=11230347
... via:
https://duckduckgo.com/?q=site%3Anews.ycombinator.com+%22Adm...
Hacker News is so special for that. I posted a question about Django a while back and sure enough, in rolled the framework's creator to answer it.
That's great! Which creator was it? Simon Willison is very active here, I don't know if Adrian Holovaty ever was.
I work at a big company and commented once that some decision was stupid and one of the top two engineers at the company dropped in to tell me I was wrong. I felt so honored. (And he's wrong.)
Hahah oh man, do you have a link? I'd love to have an opinion on who's right.
I once commented something to the effect of “that must have sucked” on a story about debugging a weird error on crash bandicoot and in comes the developer to tell me “yes, it did suck.”
Why did you make the guys with exploding heads who run at you and their scream gets louder as they approach? I can still hear them.
That's like one of the key selling features of the game. Great experience that also scars you for life.
Exactly. See also: first headcrab fight with only a crowbar in Half-life.
This is THE one thing I remember from playing Half-life :)
The other thing is when it comes over the intercom: Forget about Freeman! Anyone still down there is on there own!
The worldbuilding in Half-life was so good. You weren't a character walking into rooms and enemies would attack you, like Doom and Quake. There was a big event happening, and you were just a part of it.
It's quite memorable in VR as well! If you get the chance I highly recommend playing HL Alyx with a valve index setup.
It was on par with my first playthrough of the original HL so many years ago.
Why wouldn't they? It's called great sound design. Games of the 90s and early 2000s put a lot of effort in that. See Thief the dark project. Sound design can bring more immersiveness than graphics alone. It's something forgotten in many games of today that keep beating the ray traced graphics horse
I think it was a rhetorical question. I wasn't a big fan of the SS series, but those damn kamikaze still echo in my mind. I've even had nightmares about them...
the way they start in the distance and blend into the rest of the sounds until... ".. wait what's that sound?"
I love Thief - I am playing new levels (I think its black parade)
One of my favourite games.
I think it was a playful rhetorical question.
The Witch in Left 4 Dead. Whoever cooked up the audio for that deserves an Oscar.
https://youtu.be/C71U4CxD_J8?si=dvaDyhYCKtb5f069
you've no doubt heard it many times, but the split screen game play was very appreciated
That game was legit. You should be proud that all your efforts resulted in something truly great that people loved to play. Thanks!
Awesome.
I loved that game.
I was a huge fan of co-op games and shooters, but all my friends wanted to play Counter Strike all the time.
With Serious Sam I could at least sometimes motivate them to play something I liked.
Thanks for your service! :D
Serious Sam was one of my uncle’s favorite games (along with Duke Nukem 3D). He was a hugely important figure in my life, and playing through his favorites is a good way to keep connected to my memories of him. So, thanks to you and your coworkers! Excellent game, excellent multiplayer, and very good memories.