Update 10/7/2026: brought up to date with the code as it is now. The dedicated server got a schedule of its own and no longer contains any client code, the kill cam and replay viewer added a third scene tree, and I added the threading bugs that showed up after I first wrote this, along with what fixed them.
Introduction
Note: It is highly recommended to read the "TeamSpec's Networking Architecture" article before diving into this one, as it provides the foundational knowledge needed to understand the changes described here.
I've been working on TeamSpec for a long time, and one thing that always bothered me was how the integrated server worked. When you host a game in TeamSpec (the kind where you click "Start Server" and you're both the host and a player), your client and the server run in the same Godot process. That part isn't unusual; Minecraft, Source and basically every game with built-in hosting do it that way. What's unusual is how they share that process.
In TF2, CS:GO and Half-Life, a listen server runs its simulation on the main thread alongside the client, and rendering happens on a separate thread. In Minecraft, the integrated server runs on its own dedicated thread, and the client talks to it over what's basically a loopback socket. That's why Minecraft can run a singleplayer world full of mobs and redstone contraptions and still keep a high frame rate: the server's work isn't fighting your rendering for CPU time. I always thought that was the right model, and I knew TeamSpec would have to do the same thing eventually.
For a long time, my listen server was just the server logic running on the main thread, the same way every Godot multiplayer tutorial shows it. It worked, but it scaled badly. With 20 bots on the map my frame rate tanked, because every server tick (physics, networking, AI, building snapshots) ate into the same frame budget that was supposed to be rendering the game. On top of that, the client and the server couldn't keep their hands off each other in Godot, because every node lives in one SceneTree, and a SceneTree assumes it's running on the main thread. There's no built-in way to say "this part of my scene runs over here, and that part runs over there."
Why this is hard in Godot specifically
Godot's SceneTree, its multiplayer API and its rendering server are all built around the assumption that there's exactly one scene tree and that it runs on the main thread. When you call get_tree(), you get the tree. When you add a node, it goes into the tree. Most node operations have thread guards that require you to be on the main thread, and those guards aren't there for fun; a lot of the rendering and audio code really does assume it's only ever called from main. So just starting a Thread and running server code on it falls over the moment that code touches a node, which is always.
The way to actually fix this turned out to be modifying the engine. I forked Godot a long time ago for unrelated reasons (the lightmapper, mainly), and I extended that fork to support what I call a secondary scene tree. A secondary scene tree is a real SceneTree with its own root node, its own multiplayer API and its own owning thread. A worker thread claims it and creates nodes inside it, and the engine's thread guards accept that worker as the owner of those nodes instead of rejecting it. The worker then runs the tree's loop itself. It wakes up about 66 times a second, a little faster than the server's 60 Hz tick so it never falls behind, and each time it runs whatever the main thread has queued up for it, reads any network packets waiting for the server, and runs the server ticks that are due (at most two at a time, which comes up again later). The ticks come from the server tree's own clock, so the server keeps its own time no matter what the client is doing.
The actual layout
When you host a game in TeamSpec, whether it's over the internet (with ENet or Steam's relay network) or a private session just for you, it runs like this. The main thread runs the regular Godot tree with everything on the client's side: rendering, audio, input, the UI and your predicted character. A worker thread (I call it T2) runs a secondary tree with everything the server cares about: the server's copies of the players, the bots, weapons, projectiles, building and sending snapshots, the lag compensation history, and every character's server tick. A few server-side things still run on the main thread, like the game mode's logic (which ticks in step with the server) and the team rosters.
The two trees talk to each other through an in-memory loopback connection for network packets, the same kind of connection a remote player has, and through a pair of queues (one in each direction) for direct calls from one thread to the other. Your own input commands get to the server through that queue. It's the Minecraft and Source pattern, just expressed as Godot scene trees.
What gets shared, and what doesn't
Getting two trees to run at all is the mechanical part. The part that actually needs thought is deciding what each tree gets its own copy of and what they share. If each tree gets its own copy of everything, you double your memory and you have to sync the world between the trees every tick, which is basically running a dedicated server in the same process and defeats the whole point. If you share too much, you get data races, and you may as well not have split anything.
The biggest example is the map. The server's physics needs to know where every wall and prop is, and so does the client. If each tree loaded its own copy, the map would get loaded into Jolt (the physics engine) twice, doubling the collision memory, and the two copies would have to be kept in sync. So the main tree owns the map, and the server's tree borrows main's Jolt physics space, through a borrowed-space feature I added to the engine's World3D. The server's World3D still has its own rendering scenario (rendering really is main-thread-only in Godot, and the server doesn't render anything anyway), but the physics space, which holds all the colliders and which the server checks to see whether a rocket hit a wall, is shared. There's one Jolt space, it's owned and stepped by main, and T2 queries it at the same time. The client's and the server's characters both live in it, kept apart by collision layers. Navigation is shared in the same way: the server's bots ask main's navigation map for their paths.
Moving platforms are the one place where I did end up with a second copy. The server has to be able to move a platform back in time for lag compensation, and if it did that to the platform you're actually looking at, the platform on your screen would jitter. So the server gets its own invisible, collision-only copy of each moving platform in the shared space, and a check every tick makes sure that copy stays in lockstep with the real one.
"At the same time" is the scary part of all that. Jolt itself is fine with it, since it's built to allow queries while it steps, using per-body locks. The problem was that Godot's Jolt module didn't use the locking interfaces. It used Jolt's NoLock variants for speed, which is correct when only one thread ever touches the space and a trap the moment you do what I was trying to do. I patched the module to use the locking variants and removed the guards in its query code that assumed single-threaded access. That got T2 ray casting and shape casting against the shared space while main was stepping it. It wasn't the last threading problem, though (more on those below).
The other big decision was autoloads. Godot's autoload list creates one instance of each autoload script and puts it under the root. For a lot of my autoloads that's fine; a lookup table like the item list only needs one copy, and sharing it is correct. But things like the network clock or the lag compensation history need their own instance per tree, because each tree has its own idea of the current tick and its own set of characters. So I added a second, per-tree autoload list to the project settings. Everything in that list gets instantiated once on the main tree and once on every secondary tree, under each tree's own root. The really nice part is how GDScript resolves the names: when code on the main thread uses the network clock, it gets main's instance, and when code on the worker thread uses the exact same name, it gets T2's. Just by virtue of which thread runs the code, the name points at the right instance. For autoloads, that meant no call sites had to change anywhere in the game, and it deleted a few hundred lines of "if this is the server side, use this one, otherwise use that one" that had piled up over time. Anything that actually crosses between the trees still has to say so, by queuing a call onto the other thread, and each thread keeps its own list of characters.
Why this is so useful
The visible win is performance. The server does the same amount of work it always did, but it does it on a CPU core that used to sit idle, so rendering doesn't fight snapshot building for the frame budget anymore, and input doesn't wait on bot AI before it can read your next mouse movement. A host still pays something, since it really is running two games in one process. When I measured it in September with 20 bots, in a debug build, hosting ran at 175 to 235 FPS, while playing as a client on a dedicated server with the same bots ran at 262 to 277.
The less visible win is correctness. Once the server's state lives in a different tree than the client's, a whole category of bugs gets a lot harder to write. Before the split, I had a recurring crash where the main thread and a worker thread both touched the same stats Dictionary on a server character, and Godot's reference counting would race. I tried mutexes, but mutexes were a band-aid. The real problem was that the main thread shouldn't have been touching server characters in the first place, and now the engine won't let it: the main thread can't write to anything in the server's tree. That's a much better way to be done with a bug than "added a lock".
It didn't make every cross-thread bug impossible, though, because that guard only works in one direction. The main thread can still read from the server's tree, and in August the same kind of crash came back in a new spot: the main thread was walking a list of characters while the server's thread was adding to it. The fix was the same idea one level up. Each thread now keeps its own list of characters, and there's a test that fails if code reaches for the other thread's list.
For dedicated servers, I skip the separate worker thread entirely. The secondary tree still exists, so all the server code works the same way, but its loop runs on the main thread. There's no host client on a dedicated server to compete with, so putting main to work on the server makes more sense than leaving it idle while a worker does everything. The dedicated server has its own schedule now, with a deadline that moves forward by exactly one tick each time (instead of restarting from "now", which drifted) and a much shorter idle sleep. That took it from 48.9 loop iterations a second, with 22% of them having to run two ticks at once, to 59.6 a second with only 1% doubled. Dedicated server builds also don't contain any client code anymore (no UI, no HUD, no effects), since a server never draws anything. On my machine, a dedicated server now holds a steady 60 ticks a second with 44 bots on it.
The catches
It's not all clean. When more than one thread is involved, Godot's physics server has a command queue in front of it, and any shape changes T2 queues up (like resizing a sphere for a lag compensation query) don't actually reach Jolt until the queue gets flushed, which normally happens on main at the start of a frame. If T2 then immediately used that shape in a query, it could race itself and read a radius of zero. My first fix was to let scripts flush the queue and have T2 flush it right after changing a shape. That turned out to be worse than the bug. Flushing from T2 also flushes everything else in the queue, including bodies waiting to be added to Jolt's broadphase, and if that happens while main is in the middle of a physics step, the broadphase gets corrupted and the game crashes (mine did reliably, the moment bots spawned). The fix that stuck was a small engine patch that makes shape changes skip the queue and go straight through to the physics server from whichever thread made them. T2 sees its own changes immediately, and the dangerous stuff, like adding bodies, stays queued and only ever gets flushed by main. That works because only one thread ever writes a given shape, which was already true in practice.
The multiplayer API needed fixing too. Each tree has its own multiplayer API, and an incoming RPC needs to find its target node in that tree, not in the main tree (which is where Godot always looked). Without that patch, an RPC meant for a server-side node on T2 would get looked up in main's tree and never find its target. Signals had the same problem when they connect across trees, so I added a mode where a signal fired from the wrong tree's thread gets deferred onto the receiving tree's own queue. That way, a Jolt body_entered signal emitted on main can safely call a handler on a T2 node.
The worst ones came later, and they were deadlocks. In July, leaving a server could freeze the whole game at 0% CPU. The main thread was waiting for T2 to finish its queue, and T2 was in the middle of a prediction replay, reading a moving platform's position through the physics command queue, which waits for main. Each one was waiting on the other. In August, the same kind of freeze came back with a different pair: T2 was running a timer callback while holding its tree's lock, and that callback hit a queued physics call that waits for main, while main (the game mode logic, respawning a player) was trying to create a timer on T2's tree and sat waiting for that same lock. What I eventually figured out is that the main thread legitimately grabs the server tree's lock in a lot of places, so there's no way to stop main from ever waiting on T2. The rule has to go the other way: T2 must never wait on main. So physics reads from T2 now go straight to the physics engine instead of through the queue, any physics call that would block from the wrong thread gets reported in the log, and the game has a watchdog that writes a crash dump if a thread really does hang, so I get a stack trace instead of a frozen window.
There was also a nasty one at spawn. When 20 bots spawned at once, setting each one up stalled T2 for a moment, so the server's clock fell behind and tried to catch up by running a bunch of ticks in a row. Those catch-up frames, with 20 bots in them, took so long that they created as much lag as they paid off, and the game sat at about half a frame per second for 10 seconds. The server now catches up at most two ticks per frame. A client has a good reason to replay a backlog, since those are real inputs the player pressed, but the server doesn't: bot inputs get generated when the tick runs, so replaying a backlog would just invent movement nobody saw.
And one engine bug: Godot keeps some internal linked lists per scene tree, and in a rare timing window, a node got removed from the wrong tree's list. The engine's safety check stopped the removal halfway, which left a dangling entry behind, and the main thread crashed on it later, far away from the actual cause. The engine now removes the node from the list it actually belongs to.
A third tree
The best sign that the secondary tree was the right idea is that I've used it again. The kill cam and the replay viewer run in a third scene tree with their own per-tree autoloads, stepped on the main thread. That way a replay can bring up its own world, with its own clock, its own copies of every character and its own lag compensation history, while the live match keeps running right next to it, without the two ever getting mixed up. All it took on the engine side was letting secondary trees have viewports without a window.
There's also some new parallelism inside the server tick itself. Once there are 4 or more players connected, writing each player's snapshot gets spread across worker threads. The characters themselves still tick one after another on T2. I had planned to spread them across cores too, but after moving the server's per-tick work into C++ (that's covered in the networking article), it got fast enough that so far it hasn't been needed.
Conclusion
If I were starting a multiplayer game from scratch in Godot today, I would still pick this architecture. The cost is a forked engine, which is unfortunate, but upstream Godot isn't going to support this any time soon, and the patches I needed are all pretty small. Most of them only kick in on the new code path and fall back to the normal behavior otherwise (the game even turns the whole thing off if it's running on a stock engine), which I've been careful about because every game I work on shares this engine fork. The first lesson I'd pass along from the engine work is to land every piece together and test the dumbest possible version first (two trees on two threads, each printing its own tick number), because every half-finished state where only some of the pieces exist is its own unique debugging nightmare. The second lesson, from the deadlocks, is to decide early which thread is allowed to wait on which, and to never let it go both ways.
I also think Godot would benefit a lot from upstreaming the basic idea of a secondary scene tree. The tension between "one tree keeps everything simple" and "modern games need to do work on more than one thread" is just sitting there, waiting for someone to push on it. For TeamSpec, this is still one of the best decisions I've made, and it keeps paying off in ways I didn't plan for, like the kill cam.