TeamSpec's Networking ArchitectureTeamSpec's Networking ArchitectureTeamSpec's Networking ArchitectureTeamSpec's Networking Architecture

Update 10/7/2026: A lot changed over the summer, so I went through this whole article again and brought it in line with the code as it is now. The biggest changes are the server's per-tick work moving into C++, the reworked lag compensation, how projectiles work, snapshot rates and interpolation that adapt to your connection, and a few systems that grew around the core (Steam, replays and the kill cam, and voice chat).

Introduction

I wrote about networking once before, back in 2023 ("A Word on Networking in TeamSpec"), when TeamSpec was a lot younger and I had just figured out how Godot 4's MultiplayerSynchronizer and MultiplayerSpawner nodes worked. If you read that article and then looked at TeamSpec's net code today, you'd find that almost none of it is still there. The synchronizers are gone, the spawners are mostly gone, and so is the habit of checking is_server() on every other line. The network transport runs on its own thread, and everything the server tells the clients goes through a hand-written binary snapshot stream that nothing in Godot's built-in multiplayer nodes could produce. I want to lay out where the architecture sits today, because my other articles (like the one about running the integrated server on its own thread) build on top of it and assume you already know the basics. This is the basics article.

Fair warning, this is a long one. Networking in a real-time shooter has a lot of layers, and if I skip any of them, you end up with half an explanation that hides what's actually going on.

Where the ideas came from

Almost none of this is original. Tick-based snapshots, lag compensation, client-side prediction and replaying the player's inputs all come from the Source engine family (Half-Life 2, Team Fortress 2, Counter-Strike). Those games are decades old, and the way they network has been public knowledge for a long time, through the Source SDK and through plenty of writeups over the years. I read a lot of it, and most of the decisions in this article boil down to "what would Valve do", applied to a Godot game. The one big exception is projectiles, where I went with the way Overwatch handles them instead. I'll get to that later.

On the Godot side, the biggest influence was netfox, a Godot addon by foxssake that does tick-based networking with rollback, interpolation and prediction. Once I realized that Godot's stock multiplayer tools weren't going to cut it, TeamSpec's networking started out as netfox almost verbatim, and a few of the early systems (the original tick loop, the way rollback was signaled) still have netfox's fingerprints on them even though the code has been rewritten since. If you're making a Godot game that needs prediction, try netfox before you write a single line of your own networking. It's the right place to start.

The reason TeamSpec ended up with its own stack anyway is performance, and I want to be specific about that, because it's the thing most likely to bite anyone else doing this in Godot.

Netfox is pure GDScript. GDScript is great for game logic and for gluing C++ pieces together into a game, but net code isn't game logic. Every property access, Dictionary lookup, signal and get_node() call in GDScript goes through the interpreter, and every value gets wrapped in a Variant on the way in and unwrapped on the way out. For ordinary game code that's fine, because a frame only does a handful of high-level things, and the heavy lifting (rendering, physics, audio) happens in the engine's C++. Net code is the opposite. It's tight loops over thousands of tiny operations: walk every hitbox, check a distance, restore a transform, write a few bytes, read a few bytes, compare two floats, look up a tick in a ring buffer. Each of those is basically free in C++ and a lot slower in GDScript.

The places I felt this on netfox were, as far as I can remember:

• Lag compensation queries. A single shot has to check every player's hitboxes against where they were in the past, and projectiles do that every tick, 60 times a second, for every projectile in the air.
• History. Rollback means saving the state of every rolled-back entity every tick, and in GDScript that means writing Variant-wrapped transforms into Dictionaries keyed by tick number, thousands of times a second.
• Snapshot serialization. Writing a binary snapshot in GDScript means calling PackedByteArray.encode_u8, encode_float and friends for every field of every entity, which is hundreds of calls through the interpreter per snapshot, 60 times a second. The C++ version writes the same bytes by moving a pointer.
• Transform math. Restoring rollback state and posing hitboxes is matrix math all day long. In GDScript, each one of those operations is a method call that returns a fresh Variant; in C++ it's a few instructions.

Netfox ran all of that at a budget that's fine for a 4-player demo, which is exactly what most Godot multiplayer projects need. TeamSpec needs 24 players plus bots and a lot of projectiles, and those same loops had to get something like 50 to 200 times faster. You can't get there from GDScript by being clever. I found that, at scale, GDScript is only good enough as the glue that holds C++ components together, not for brute force computation.

So the heavy pieces moved into C++ one at a time. The transport moved off the main thread so a slow frame couldn't stall the network, the snapshot codec became hand-written C++, and lag compensation and the clock moved over after it. At some point I was forking more of netfox than I was using, so I pulled the C++ side out into its own library, gameprog-net, and rebuilt the GDScript side on top of it. This September I went a lot further and moved the server's per-tick work into C++ as well, which gets its own section further down. The ideas are still netfox and Source (client-side prediction and server-side reconciliation); the implementation is custom because it has to be in C++ to be fast enough.

Everything else below has been adapted to Godot specifically, which mostly means fighting Godot's assumption that there's exactly one SceneTree running on exactly the main thread (the integrated server article gets into that), and fighting Godot's stock multiplayer API, which was clearly designed for co-op indie games and board games rather than shooters.

Why the default Godot setup doesn't work for shooters

If you look up "Godot multiplayer tutorial", you'll find a very specific kind of demo. It's usually a top-down game or a simple platformer where the client is trusted completely, and each player's position gets broadcast a few times a second by a MultiplayerSynchronizer. Every player runs the same script, and the code is full of if multiplayer.is_server(): and if is_multiplayer_authority(): checks that branch between "do the real thing" and "show the visual thing". That's the typical Godot multiplayer pattern, and it works fine for the demo. For an action shooter it falls apart, for three reasons.

The first is that scattered is_server() checks aren't really a separation between the client and the server. If your code says if is_server(): apply_damage() else: play_hit_sound() in twenty different places, you don't have a server and a client; you have one big tangle of code that runs in two modes. The worst bugs this creates are the ones where the server branch quietly does the wrong thing because the client branch was the one being tested. Nothing stops you from calling a client-only function on the server's copy of a character or the other way around, and you only find out at runtime, often months later, when something acts weird under specific network conditions and nobody can reproduce it. It also gets messy the moment the host wants to play in their own game, because then the server is also a player, and is_server() stops being enough. In TeamSpec this turned into a spaghetti code mess.

The second is that MultiplayerSynchronizer is the wrong tool for character state in an action game. It's fine for cosmetic things and stats that rarely change, but it doesn't delta encode, it doesn't know what a tick is, and it gives you no path to client-side prediction. Without prediction, your character can't move until your input has gone to the server and the result has come back, so at 100 ms of ping you wait 100 ms every time you press W, and the game feels awful. No competitive shooter is made without prediction, and the synchronizer pattern can't get you there.

The third is that a host who takes shortcuts ends up playing a different game than everyone else. This is the sneakiest bug class in the whole space: the host's character moves differently from a remote player's character. It happens because the host's character has direct access to the real game state (the host is the server), so the code that handles the host's input never goes through the prediction, reconciliation and lag compensation that a remote player's input goes through. When you test by playing alone on your own server, everything feels perfect, and the moment a real player connects, their game behaves differently from yours. That's terrible for fairness, and the fix isn't subtle: the host's own input has to go through exactly the same path as everyone else's.

Two copies of every character

TeamSpec is server-authoritative. The server (a dedicated server, a player hosting a game, or a private session on your own machine) is the one source of truth for everything that matters to gameplay, and the clients are responsible for showing the player a good-feeling version of that truth. The two roles are kept apart by where the code lives in the scene tree, not by is_server() checks.

Every character exists as two separate objects with separate scripts and separate jobs. The server's copy holds the real state: position, velocity, health, ammo, the weapon being held, and what the character is doing. Every server tick it takes the next input command from that player's queue (more on those below), runs the same movement and weapon code every other server character runs, and its state goes out in the next snapshot. The client's copy is what you actually see. It plays animations, renders the model, handles the camera, and if it's your character, it predicts your movement by running the exact same movement code the server runs.

Which copies a machine has depends on its role. A dedicated server only has server copies, and a player connected to it only has client copies. If you're hosting, your game has both: a server copy of every character on the server's side, and a client copy of every character on yours. They live in different branches of the scene tree, and on a hosted game those branches are actually two separate scene trees running on two separate threads. The engine won't let the main thread write to anything on the server's side, so client code can't accidentally reach over and change the real game state. That barrier is a big part of why the "the host is just another client" rule actually holds.

Every character also has an entity id. For a human, it's their network peer id, and bots get ids from a reserved range that starts at two billion, so the same scheme covers both. Back in 2023 I wrote that I didn't know how I was going to tell bots apart, since they would all share the server's peer id. This is how that got solved.

Three character behaviors

On top of the server and client split, every character has a behavior attached that decides what runs for it every tick. There are three of them.

The authority behavior goes on every server copy, including the host's own character. It runs the server's tick for that character: take the next input command, run the movement, run the weapons, apply damage, and write the result back. Bots are authority characters too. The only difference is where their commands come from, which I'll get to in a bit.

The predictive behavior goes on the client copy of your own character. When I press W, it runs the same movement code the server will run, and my screen shows me moving right away. When I say "the same movement code", I mean that literally: the server, the live prediction and the replay after a correction all call one single C++ function, just in three different modes. If both sides start from the same state and feed it the same command, they get the same answer, and that's what makes correcting a wrong prediction possible.

The ghost behavior goes on everyone else's client copy, so the other 23 players on your screen are all ghosts. A ghost doesn't simulate anything. Its pose comes straight from the snapshots the server sends, played back through the interpolation I'll describe below, and since it does no simulation of its own, it can't drift. These days the ghost behavior is basically just a label. The actual work happens in the code that receives snapshots, which keeps a short history of snapshots for every character and poses each ghost from it.

The important part is that there's no if peer_id == my_id or if multiplayer.is_server() inside the tick loop. The right behavior gets attached when the character spawns, and from then on, the tick is just "ask my behavior to do its thing". The only time a character switches behaviors is when control of it changes hands: any player's character can be handed over to the AI and back, and when that happens, the client swaps that character between predictive and ghost.

Bots go through the same path as players, which matters more than it sounds like it should. A bot doesn't move its character directly. Every tick, it produces one ordinary input command, the same kind a player's client sends, and that command goes through the same input handling and the same movement function as a human's. The only differences are that a bot's commands don't have to cross the network, so they skip the input buffer, and they're made at the server's present, so nothing has to be rewound for them. To the clients, a bot is indistinguishable from a player.

The host plays like everyone else

When you host a game in TeamSpec, the game doesn't just spawn your character locally and start a server loop. It starts the server inside the same process, on its own scene tree and its own thread, and then your client connects to it like any other client would. The server's side of the connection is a single list of peers that mixes players connecting over the internet (through ENet or Steam's relay network) with a fake in-memory connection for you, the host. Snapshots come back to your client through that in-memory connection exactly like they'd come over the internet to anyone else.

Your input takes a slightly different road. Since the server lives in the same process, your input commands get handed to the server's thread through a thread-safe queue instead of going through the fake connection. That's just a delivery detail, though: they arrive at the same function, go into the same kind of input buffer and get used by the same server tick as a remote player's commands. So every line of client code still runs on the host, including prediction, reconciliation, applying snapshots and interpolation. And every server path gets exercised by everyone, because the host's character goes through the same input buffer, the same lag compensation and the same snapshot path as anyone else's. There's no untested "host fast path".

I have a rule for this: the host's own client has to go through the same paths as a player on another computer, with no "host mode" shortcuts. I keep that rule because every single time I've tried to take a host shortcut, it created a difference between the host's experience and everyone else's, and the difference was never worth the little bit of code or performance it saved.

Snapshots, not synchronizers

Everything the server tells clients about the state of the game travels in a binary snapshot, delta encoded, sent at up to the server's tick rate of 60 times a second. There's no MultiplayerSynchronizer anywhere in it.

A snapshot describes every character in the game at one tick. Each tick, the server builds one compact record per character in C++, with every field encoded once, and every field remembers the last tick it changed on. To build a particular client's snapshot, the server looks at the last snapshot that client confirmed it received and copies only the fields that changed after it. Position, rotation and velocity get compared directly against the confirmed snapshot. If a client hasn't confirmed anything in a while (66 ticks, which is about a second), it just gets a full snapshot. The format is written by hand rather than generated through reflection, so I know exactly which bytes appear in which order for every field.

Not everyone gets the same fields. There's a public set that anyone watching a character needs in order to draw it (its pose, its stance, its weapon, when its reload ends and so on), and a detail set that only goes to the character's owner and to anyone spectating them, with things like ammo and the connection feedback that drives the HUD's network warnings. Other players don't get your ammo count, because there's no reason they should. What TeamSpec doesn't do yet is leave out characters you can't see: every snapshot carries every character.

A few fields change every tick for every active character, like the clock that drives the walk animation, so a snapshot is never empty. On average, a snapshot with 20 characters in it comes out to around 650 to 675 bytes, and with 24 it's around 930 bytes, so it usually fits in a single packet.

Sixty snapshots a second isn't guaranteed for everyone, though. Each connection gets 60, 30 or 20 snapshots a second. The server steps a connection down when it sees 5% packet loss or when the client falls 20 ticks behind on confirming snapshots, and steps it back up after 10 clean seconds. Players on a bad connection get fewer snapshots that actually arrive, instead of a flood of half-delivered ones.

Shots are handled separately, because losing one would be really bad. Every shot gets an identity and travels on its own reliable channel, batched once per tick, and the snapshot only refers to it. If a snapshot gets lost, the shot doesn't.

The reason I don't use MultiplayerSynchronizer for any of this is that its whole data model is "broadcast these properties at this rate". It doesn't know about ticks, it doesn't delta encode against what the receiver actually has, and it gives you nothing to hang prediction or rollback on. To do prediction, you need to know exactly what the server's state was at exactly which tick, and to delta encode, you need the receiver to tell the server which ticks it got. Snapshots give you both. Synchronizers are fine for low-stakes stuff like UI state.

Playing snapshots back

On the receiving end, the client plays the ghosts back a little bit in the past, smoothly blending between the two snapshots on either side of the moment it's showing. How far in the past depends on your connection. It starts 2 ticks behind the newest snapshot (about 33 ms), and the instant it sees jitter, a gap, or a moment where it ran out of snapshots, it adds more delay, up to 6 ticks. When the connection calms down, it eases back toward 2 over about 3 seconds. Playback speed also stretches by up to 10% in either direction so it can drift toward the target instead of jumping, and it only snaps forward if it somehow ends up more than 8 ticks behind.

It never extrapolates. If the client runs out of snapshots, it holds on the newest one and adds a tick of delay, so a bad moment shows up as a brief pause instead of a character sliding off in the wrong direction and snapping back.

Input is data, not RPCs

Your input isn't sent to the server as function calls, since doing that with RPCs would waste a ton of bandwidth. It's sent as a user command (Source calls these usercmds): a small, bit-packed record of one tick's worth of player intent. It has a command number and the tick it was made on, your aim angles, your movement direction, a bitmask of buttons (jump, fire, alt-fire, crouch, reload, use and so on), the weapon slot you want, exactly which moment of the game you were looking at (that's for lag compensation, below), and up to 256 bytes of extra data for game-specific stuff.

Every packet also repeats the commands that came before it. Normally that's the last three, but if the server reports that it ran out of commands for you, the client adds two more at a time, up to eight, and backs off again after 5 quiet seconds. That gives you retransmission for free: if a packet gets lost, the next one carries the lost commands anyway, so as long as one of the next few packets makes it, the server gets your input. An RPC either arrives or it doesn't, and if it doesn't, that tick of input is gone forever. The server, in turn, puts the number of the newest command it applied into every snapshot, so the client knows which of its commands still need predicting. And since the client and the server read commands with the same code, I never have to wonder whether they interpreted the input the same way. They literally read the same bytes.

The other part of a command is the sub-tick input, an idea I took from CS2. The server only runs 60 ticks a second, but a player running at 240 FPS clicks somewhere in between ticks. So instead of just a "fire is held" bit, each command carries an ordered list of press and release events for fire, alt-fire and jump. Each event carries where in the tick it happened, the exact aim at that moment, and exactly which moment of the game the player was looking at when they clicked, and the server replays those events in order. A quick tap between two frames still fires, two shots inside one tick are two shots, and a player who flicks and clicks within the same tick gets the aim they actually clicked with, not whatever aim they had when the tick started. That alone made high-FPS aim feel noticeably tighter.

Client-side prediction

Your character's predictive behavior runs the same movement code the server runs, so when you press W, you move on the very next frame instead of waiting for a snapshot to come back. The server processes the same command a little later and comes up with the real answer.

As long as the two answers agree, you never notice prediction is happening. The snapshot shows up, agrees with what your client already drew, and nothing visible happens. The interesting case is when they don't agree. Maybe the server saw you bump into a player your client didn't know had moved there yet, or you got hit by a rocket blast your client hadn't heard about. Your predicted position is wrong now and needs to be corrected, which is what players usually call rubber-banding.

The correction is rewind, re-apply, replay. When a snapshot arrives, the client takes the server's state as the new starting point at that snapshot's tick, then walks forward through all of its own commands that the server hasn't processed yet, re-running each one in order through the same movement code. The result is a new predicted position that includes the server's correction and all of the input you gave since then, so you don't lose a single button press. Small corrections get blended in over a tenth of a second so you don't see a snap.

Not every difference is worth correcting, and some are too big to replay. Errors under 10 cm are ignored, unless a small drift of 3 to 10 cm sticks around for 10 snapshots in a row. Anything over 2 m snaps immediately, and so does a run of 10 corrections in a row that are each over half a metre. If the client has 90 or more commands waiting to be replayed (it fell way behind, packets were dropped, or the game stalled), it skips the replay and snaps you straight to where the server says you are, because one visible jump is better than spending a whole frame of CPU re-simulating input you've already moved past.

None of this works without two things. The first is movement that's deterministic, meaning the same input from the same state always gives the same result, down to the bit. I check that with a test that drives a character through a long run of movement and compares every prediction against the server; the last time I ran it, that was 2,428 snapshots over 97 metres of movement, and the biggest disagreement was 1 micrometre. The second is a tick numbering scheme that both sides share, which is the network clock's job.

Keeping time

Every tick has a number, and the server decides what tick it is. The server regularly broadcasts its current tick, and each client steers its own clock toward the last tick it heard about. That means a client actually runs a little behind the server, by about the time it takes a packet to travel from the server to the client, and its commands reach the server about one round trip after the tick they're stamped with. That's on purpose.

The usual alternative is to make the client run ahead of the server, so its commands are already sitting in the server's buffer before the server needs them. That buys a safety margin against jitter, but you pay for it in the wrong place: a deeper buffer means the server acts on your input later in real time, so your shots land later and everyone else sees what you did later. It also can't really help the players who need it the most, because one global setting can't fit a player on 20 ms and a player on 200 ms at the same time.

So instead, the server sizes each player's buffer based on what their connection actually does. Each player has a small buffer between the network and the server tick. Incoming commands get de-duplicated (remember, every packet repeats the last few), put back in order if the network shuffled them, and thrown out if they're more than 60 ticks stale or stamped with a tick from the future. Each tick, the server runs one command from the buffer. If no command has arrived, the server simply doesn't run one for that player that tick, and when the late ones show up in a burst, it runs two commands per tick until it has caught up. And if some extra depth sits in the buffer for a whole second without ever being needed, the server drains it. A stable connection ends up with an almost empty buffer and almost no added delay, and a jittery one keeps exactly the depth its jitter needs.

The buffered, one-command-per-tick part is the model CS2 moved to. For most of development, I ran the TF2-style model instead, where the server tried to run each command at the exact tick the client had predicted it for, holding early commands and running a catch-up loop for late ones. It works, but that catch-up loop was a constant source of edge cases (fire a rocket in the middle of a catch-up and enjoy the debugging session).

Underneath all of it is a C++ network clock in gameprog-net, the library I describe at the end. The clock has a thread of its own, but not for network messages (those get handled on the thread that owns the scene tree). Its thread is for scheduling ticks: it sleeps until the exact moment the next tick is due, so the tick rhythm stays steady even when the game hitches on a heavy frame. Clock drift under 10 ms is left alone, bigger drift gets corrected gradually, and if a client is ever off by more than 250 ms for three frames in a row, it just jumps to the right tick. A client that falls behind runs at most two ticks per frame to catch up, so a stall can't snowball. Every scene tree gets its own copy of the clock, which matters for the integrated server, since the server's tree and the client's tree each have their own idea of what tick it is.

Lag compensation

When you fire, the server has a problem. Your screen shows other players where they were a little while ago (the time it took their snapshots to reach you, plus the interpolation delay), and you aimed at what you saw, not at where they are on the server right now. If the server checked your shot against its current state, you'd miss shots that were dead on, because your target kept moving on the server in the meantime.

Lag compensation means the server checks your shot against the world as you saw it. Every command says exactly which moment you were looking at, using three numbers: the older of the two snapshots your screen was blending between, how far along that blend was, and how many ticks away the newer snapshot was. That describes exactly what was on your screen, even if a snapshot went missing or you're on a connection that only gets 30 or 20 snapshots a second. Classic Source estimated this moment from your measured latency and interpolation settings. TeamSpec doesn't need to estimate it, because the client just says it.

On the server's side, recording every player's full skeleton every tick turned out to be way too expensive (that's 21 hitboxes per character, for every character, 60 times a second). So the server records very little per tick: where each character stood, and the handful of numbers that drive its animation. When a shot comes in, the server first throws out every character the shot can't possibly come near, and only for the ones that are left does it rebuild the skeleton at the moment you saw, once per character per tick. In practice that almost never has to happen. In a 20-character fight, the server went from 18 full skeleton evaluations every tick to 0.04 per tick on average.

The blending between the two snapshots happens in bone space, the same way the renderer does it: the character's position and each bone's pose are blended separately, and then put together. Blending the final world positions instead (the obvious way to do it) cuts corners when a character turns quickly, by up to 80 mm on a hand at 60 degrees per tick, which is plenty to turn a hit into a miss.

All of that only works if the skeleton the server rebuilds is the same one you saw. A ghost's skeleton on the client is computed purely from a few animation numbers that come in the snapshot, once per received tick, and the server poses its past skeletons the same way, so the hitboxes the server checks are what was drawn on your screen. I measured the difference at 0.2 mm.

Two more details. A shot fired partway through a tick uses the last completed tick, since the state at that in-between moment doesn't exist yet. And none of this ever moves anything in the live world. The server runs the query against the recorded history, next to the present state, which is faster than the classic "move everyone back, check, move everyone forward" approach and makes it impossible to forget to put something back. Movement is the one place where things do get moved back: when the server runs a player's command, it puts the other characters and moving platforms back where they were at that command's tick, so you bump into what you saw, and then restores them.

There's a limit to how far back the server will go, which is 300 ms. Past that, your shot is checked against the world 300 ms ago instead of what you saw, and the HUD warns you that your rewind got capped, so you know you need to lead your shots. Without a limit, someone on a terrible connection could shoot people who had been behind a wall for half a second already. Even within the limit, the target can sometimes feel like they got shot around a corner (that's the classic complaint about lag-compensated shooters), but the alternative is that players with high ping never hit anything, which is a lot worse.

Projectiles

Lag compensation covers hitscan weapons, but projectiles are their own problem. A rocket takes time to fly, and the shooter's copy of it is ahead of the server's by about a round trip. For this part, I used Overwatch as my reference. In Overwatch, even at 200 or 400 ping, when you shoot something like a grenade, it comes out of your gun instantly. When it hits someone, the projectile just disappears on your screen, and about one ping later the explosion shows up and they take damage. You get instant feedback, and what you see never disagrees with the server.

TeamSpec does the same thing. When you fire, your copy of the projectile spawns and flies at full speed right away, at any ping. If your copy hits a character, it only hides itself: no explosion, no damage, nothing that could turn out to be wrong. The explosion and the damage only happen when the server, which runs its own copy of the projectile, says so. If the server doesn't confirm the hit within a window based on your round trip time, your client decides it guessed wrong, un-hides the projectile and lets it keep flying. Hits on the world are different. Walls and floors are in the same place for everyone, so a projectile that hits geometry explodes immediately on every machine, which is what keeps rocket jumping feeling instant.

I tried another approach first, which was holding the projectile at the muzzle long enough to line it up with the server's copy. At high ping, it looked like the rocket spawned frozen in place and then started moving 200 ms later, so that idea got thrown out pretty quickly.

Making the server fast enough

This is the biggest change since I first wrote this article. In late September, a server tick with 20 characters on it was taking around 9 to 10 milliseconds. At 60 ticks a second, the budget is 16.67 ms, and the machine a dedicated server actually runs on is two to three times slower than my desktop, so that wasn't going to work. It's frustrating that TF2 is almost 20 years old and runs the same player counts and the same kind of combat at 66 ticks a second, and I was struggling to hold 60.

When I looked at where the time went, most of it wasn't the physics or the movement. About 7 of those milliseconds were interpreted GDScript glue code and per-character Dictionaries being created and copied every tick, mostly around building snapshots and running the bots. On top of that, every character's animation was being fully evaluated, and all 21 of its hitboxes recorded, every single tick. Source does all of that in C++ over plain structs, marks network fields as changed when they're written, and only sets up a player's bones when a trace back in time actually needs them. So I restructured the server to work the same way, while keeping the netcode model exactly as it was. Nothing about what goes over the wire or who decides a hit changed, only where the per-tick work runs and how the state is stored.

The pieces, roughly in the order they went in:

• One C++ function runs a movement command in all three cases (the server, live prediction, and the replay after a correction), and applying a command's input happens in C++ too.
• The snapshot record I described earlier, built in C++ once per character per tick, with every field stamped with the tick it last changed. It replaced a Dictionary per character per tick.
• Writing each client's snapshot runs in parallel on worker threads once there are 4 or more clients, and it doesn't create any new objects once the game is running.
• Skeletons only get posed when a shot actually needs them, like I explained in the lag compensation section.
• Clients decode snapshots into reusable typed records instead of Dictionaries, so scripts read one row of numbers per character.
• The bots' decision making (what they see, how they aim, what they do and where they path) moved into C++ too. That's the newest piece, and I'm still testing it.

One thing I deliberately didn't copy is how Source marks fields as changed. In Source, networked variables flag themselves as changed whenever they're written. I measured what that would save over just comparing each field when the record gets built, and it came out to less than a microsecond per character per tick, while costing an extra call on every single write, on every machine. So I kept comparing at build time.

Here are the results with 20 characters. Measured without a profiler attached, the server's average tick went from 9.32 ms to 5.19 ms, and its slowest 1% of ticks went from 14.21 ms to 7.88 ms. Skeleton evaluations went from 18 per tick to 0.04, and sending snapshots to 24 clients went from about 1,063 microseconds per tick to 128. On the client, decoding and applying a snapshot went from 1.70 ms to 0.61 ms per packet. And a dedicated server on my machine now holds a steady 59 to 60 ticks per second with 44 bots on it and no late ticks, where two weeks earlier the same test ran at 26 to 34. (I use 44 bots on my desktop as a stand-in for 24 players on the slower server hardware.)

The gameprog-net library

A lot of what I described above (the clock, the transports, the lag compensation history, the snapshot format, the input commands) isn't specific to TeamSpec. It would work for any server-authoritative action game in Godot, so I keep it in a separate C++ library called gameprog-net that compiles straight into the game's GDExtension. TeamSpec is a user of the library, not the owner of it. The game plugs into it through five hooks: its extra data format for commands, its snapshot field codec, its list of entities, a way for the rollback system to reach the game's own C++ state, and the function that poses a character for lag compensation. The game's own C++ side grew a lot over the summer too, since everything tied to TeamSpec's specific data lives there: the snapshot records and sending, applying input, the bots, the animation sampler, moving platforms and the voice codec.

The library currently has:

• The network clock: the tick scheduling thread, the sync handshake, the tick broadcasts and the drift correction, plus a manual stepping mode that the replay viewer drives.
• An ENet transport that runs ENet's event loop on its own thread, so a slow frame can't stall the network, with per-peer and overall bandwidth statistics.
• Steam transports, both for dedicated servers and for peer-to-peer, that speak the same interface as everything else, so the whole stack runs over Steam's network unchanged.
• The in-memory loopback transport, which behaves like a network connection with no latency and no loss, and a host transport that combines it with ENet or Steam into a single list of peers, so the server doesn't need to know or care what kind of connection a player is on.
• The lag compensation history and its hit queries, including the blended queries described above.
• The rollback bookkeeping used when correcting a prediction: what every rolled-back entity looked like at every tick, so the replay can restore it and re-simulate.
• The snapshot layout and receiver.
• The input command format, the client's ring of recent commands that gets re-sent with every packet, and the server's per-player input buffer.
• The smoother that blends prediction corrections in over a few frames.
• Low-frequency replication for things that don't belong in the per-tick stream, like player names and cosmetics, and for spawning weapons and structures.
• And the small stuff: bit-packing readers and writers, raw packet channels, a fake lag buffer for testing the game under artificial latency and packet loss, and some OS process priority tweaks.

It isn't public yet. My plan is to release it once it's cleaned up enough for other people to use. It's pretty tied to a Source-style, tick-rate driven model, so it won't suit something like a turn-based game, but for the specific niche of "server-authoritative, predicted, lag-compensated, snapshot-replicated action game in Godot", it covers most of the heavy machinery.

What's changed since the 2023 article

Going back and reading my own 2023 writeup is useful, because it shows the natural path any Godot multiplayer game goes down if you let the engine's documentation decide your architecture. Pretty much everything I wrote about in 2023 has been replaced.

The thing the 2023 article got right is that the synchronizer pattern is enough to make a working multiplayer demo. It's not enough for a competitive shooter, and every row in that table is something I learned the hard way.

Everything that grew around it

Since I first wrote this article, a few bigger systems got built on top of the core, and each of them is probably worth its own article someday.

First, replays. TeamSpec has a demo system similar to GOTV: a hosted game records every tick's snapshot plus every player's commands into one file you can seek through, and the replay viewer plays it back through the same snapshot code the live game uses. Recording is on by default for every hosted game.

The kill cam is built on the replay system. When you die, you see the 4 seconds before your death and the 1 second after it. On a dedicated server, a separate worker thread keeps a rolling 8 seconds of history, cuts and compresses the clip when someone dies, and sends it out over its own socket, so it never touches the server tick or the game connection. Your client then builds the replay world a little at a time, over about 40 frames, in a separate scene tree, while the live match keeps running behind it. That's what makes it asynchronous, and it took the frame where a clip arrives from a 770 to 890 ms hitch down to about 10 ms. In a stress test with 24 players dying at the same moment, all 24 clips (about 160 KB each) were delivered in a little over a second.

Voice chat deliberately doesn't use the game connection. It has its own socket (or its own channel when going through Steam), so voice can never compete with snapshots or input for bandwidth. The game connection is only used to hand out a token, and the server just relays the audio without ever decoding it. It's Opus at 24 kbps with error correction, plus noise suppression.

And then there's Steam. Both dedicated servers and players hosting a game can go through Steam's relay network, so nobody has to port forward. There's an in-game server browser that reads Steam's master server list, Steam ticket authentication, parties built on friends-only Steam lobbies, and quick play through a matchmaking service that dedicated servers connect out to.

What wasn't covered

I haven't talked about the integrated server's threading model here; that's its own devlog entry. The same goes for the reconciliation loop in detail (the "how do you actually replay commands correctly across a correction" part, which has a bunch of edge cases around explosion impulses, which tick an effect lands on, and how other players see predicted weapons), the clock's drift correction, which has a lot more depth than I gave it here, and the exact layout of the snapshot format. That last one has some clever tricks, most of which I didn't invent; they come from the Source SDK and from the BR200 reference writeups. Those might get their own articles if there's interest.

If the 2023 article contradicts anything in this one, this one is the current truth. The 2023 article is a snapshot of where I was at the time, when I was still using MultiplayerSynchronizer and learning what its limits were, and a few of those limits are exactly why TeamSpec ended up where it is now. I thank you for taking the time to read all of this!