Mon Dec 8, 2014

I had originally planned to use Lua to re-implement D2K’s console, but using GObject Introspection via Lua GObject Introspection was just too hard, because GObject Introspection depends on Python, which requires dozens of patches to cross-compile using MinGW-w64. You can find these patches scattered everywhere, from random GitHub or Bitbucket accounts to the MinGW-Builds tarballs, and what’s clear from the number of patches is that none of it is guaranteed to work at all, let alone work correctly.

So now my plan is to essentially build my own bindings. I can only imagine what’s really involved with this, so it may be that I abandon this as well.

Designing as I write this, I envision something like:

|                                                                     |
|                       xf.Console (in Lua)                           |
|                           ||                                        |
|                           \/                                        |
|                     Widget Library (in Lua)                         |
|                           ||                                        |
|                           \/                                        |
|                       PangoCairo                                    |
|                         /    \                                      |
|                     Pango    Cairo                                  |
|                                                                     |

This is, unfortunately, lots of work. I’m currently planning on not providing full bindings to Pango, Cairo or PangoCairo, although I don’t know how hard it would be to do so. If it ends up that it’s as simple as generating (or C-N-P’ing) the rest of the code, then I’ll go the whole way. I certainly won’t hold up D2K development for 100% coverage here.

I would feel remiss if I left this post without noting my dismay at Python’s lack of MinGW support, which is starting to snowball into a general dislike of Python. Its abject slowness, inability to use more than 1 CPU core, wild memory consumption, total mishandling of Unicode, lack of support for MinGW, and insane Python3 redesign (for essentially no good reason, because the Unicode redesign is broken) rule it out for almost anything I will ever do. I’d rather build a GUI in C++ with wxWidgets or QT, or C and GTK than I would with Python and respective bindings. I’d sooner build a website in Go or Java than I would with Python and its frameworks. It’s CPU and RAM usage rule it out for systems programming, where C/C++/Go/Java shine.

These days, all I use Python for are when I need better scripting than what Bash will provide, and to be honest, I’m thinking about improving my Bash knowledge so I won’t have to do that either. It’s crazy because I have a lot of deep Python knowledge, but it’s just not useful anymore.

Anyway, I think this delays scripting and the console for a while. I was hoping for a quick translation between the existing C code and the (to be rewritten more performantly) new Lua implementation, but it looks like there’s some plumbing I have to add first. That’s a shame because I was excited about this, but now it just looks like 90% grunt work.

D2K Plasma/Bullet Lag

Thu Nov 13, 2014

Rev. 42f4ae5 lessens the amount of data sent for certain types of actors (plasma, spider demon plasma, bullet puffs and blood spots) when they spawn. This was necessary because these actors tend to spawn very rapidly, causing lots of new information to be sent over the wire (delta compression doesn’t help here because the actors are new), which would lag (throttle) clients on bad connections. A little more could be done here, and in fact, the solution could be seen as a hack. I think Valve, for example, has fine-tuned controls about what entity fields are synchronized between client and server, which can be helpful for mods. In D2K, if you add more rapidly-spawning actors, there’s not a great way to tell the engine to only send certain data about them, let alone specify what that data should be, and how to fill in the blanks clientside.

D2K struggled with this more than other C/S ports because of its “send everything” philosophy. Delta compression works because the world usually changes very little, but an actor is 312 bytes, and delta compression can’t leave any of that data out when an actor first spawns. 2-3 new actors over a couple of TICs isn’t a big deal, but the SSG spawns something like 20 bullet puffs; that’s over 6K!

To fix this, we pull the size down from 312 to 48 bytes (44 for puffs & blood). This is still a lot of data for 1 SSG shot (~900 bytes), and we can probably pull it down even more if we get smarter (puffs don’t need angle, pitch or flags, can save 16 bytes, or 320 per SSG shot, there).

D2K Client Sync

Thu Nov 13, 2014

Rev. c6a8550 is my latest attempt at solidifying clientside sync (along with a couple of preceding commits). After trying dozens of different schemes, I finally discovered one that works nearly perfectly.

First of all, clients need to know what commands a server ran between the delta’s start state and end state. This isn’t so important for consoleplayer’s position, but it important in order to get accurate sounds from the other players. The client runs commands from other players (non-consoleplayer commands, or “NCP” commands), during this phase, called “synchronization”, and that’s where the it will start sounds such as rockets/plasma firing, lifts starting, etc. It’s possible for consoleplayer’s commands to affect NCPs’ positions, so they need to run in tandem. Currently they are run all at once; it would be better to have a way to run them the way the server did, or at least stretch them out as much as possible.

Once this is finished, the client loads the new, latest state received from the server. Re-running the commands the server ran between the delta’s start & end state is an imperfect process, so it can’t be relied on to perfectly create the state. After this step, synchronization is complete, and re-prediction commences.

The server won’t always run consoleplayer’s commands in lock-step. The client may send commands 10080, 10181, and 10282 (TIC/Index), but the server may run them all on TIC 101. Let’s stipulate that command 81 triggered a lift at TIC 101.

The client will then (possibly, but let’s say it does for the sake of our example) receive a delta “server received commands <= 82 at TIC 101”. So it synchronizes all commands <= 82, and spawns the lift at 101.

When the client first ran command 82, the lift had already been activated, so this TIC (102) would be the 2nd time the lift had been moved. However, after synchronization, the TICs will still align, but the commands won’t. Let’s say that the client has 3 commands left (10383, 10484, 10585) that the server has yet to acknowledge receipt of. It will now run each of these commands separately, and move the lift 3 more times. However, initially, by the time the client ran command 85, the lift had already been moved 5 times. So when the client initially predicted the world out to 10585, the lift had moved 5 times, but now after re-prediction, the lift has only moved 4 times. This will cause the lift to snap back 1 TIC’s worth of movement.

You’ll also notice that the client is about to run TIC 102, but will then run command 10383. This disparity allows the client to know that it’s fallen behind. When the client notices this, it runs an extra TIC, without a command. This runs the world simulation, but doesn’t move consoleplayer.

New Site

Wed Nov 12, 2014 using tags infrastructure, news

Ever since TotalTrash was converted to a static site, I’ve wanted a way to add pages without copying and pasting tons of markup and updating the navbar. I’ve finally built a new site using Hugo, a static site generator written in Go.

Additionally, I plan to feature more news on TT. I’ve wanted a place to post D2K updates, and I think TT is where that belongs.

I may backdate some posts just for content’s sake. We’ll see how it goes.


Sun Mar 2, 2014 in development using tags d2k

I’ve decided to start working on a new source port called Doom2K or “D2K” for short. My goal is to build a state-of-the-art multiplayer source port capable of being the standard-bearer for multiplayer Doom. This, in my mind, means a couple of things.

Message-Based Netcode Is Broken

ZDaemon, Zandronum, and Odamex all use message-based netcode. For example, there is (historically) a function called P_SpawnMobj which spawns a new actor into the game. Serverside, inside of P_SpawnMobj is a call to a function that broadcasts a “mobj spawned” message; let’s call it SV_BroadcastMobjSpawned. For illustration’s sake, it generally looks something like this:

| ID    |
| X     |
| Y     |
| Z     |
| Angle |
| Type  |

This model “works” for any number of game events. EECS, for example, has 44 messages. Odamex has 112. The way message-based netcode “scales up” is message proliferation, and there are a number of downsides to this.

You have to litter your code with network messages

Ugly, error-prone, and overly complicates extending the engine (scripting).

You have to exempt huge swaths of code from running clientside

Same as above.

Consistency is complicated

Unless you package up all your message into a single packet (assuming it will all fit), consistency requires sequencing and a “tic ended” message.

Doom Scripting Is Busted

Probably the best scripting out there is Doomsday. Otherwise, you can use EDGE/3DGE and COAL, or you can use ZDoom and ACS. I think you can use FraggleScript in Legacy.

The situation isn’t great though.

With a powerful scripting language, it would be possible to implement new game modes (like CTF), convert most of Doom’s physics and original game play, and represent assets and configuration in scripting also. PWO, for example, could simply be a user-provided function in a config.

ASCII Is Out, Unicode Is In

Unless Doomsday uses UTF-8, I’m unaware of any source ports that are Unicode-aware. This has serious implications for Doom’s popularity outside of English-speaking countries.

Software Rendering Is Limited And Slow

Software rendering is much slower than hardware-accelerated OpenGL rendering. In general, people who prefer software rendering do so because it’s more faithful to Doom’s original renderer (which almost no port uses anymore, due to its bugs and limitations). However, with display pixel count rising, software rendering is becoming less and less feasible.

Software renderers also have various limitations: lack of support for portals, 3D architecture (slopes, room-over-room, etc.), bad math (wobbling flats, problems with long walls, inaccurate flat rendering, etc.), and so on.


My major goals are to fix all of this.

Instead of network messages, D2K will use delta-compressed game states, which avoids all of the problems with message-based netcode (at the expense of increased CPU usage and RAM consumption).

D2K will use Lua for scripting. Current plans include:

  • Implement console in scripting
  • Implement new game modes in scripting
  • Move Doom physics, game play, and assets to scripting

D2K will use UTF-8 instead of ASCII.

Regarding the renderer, I haven’t entirely decided to remove the software renderer. My main focus will be on the OpenGL renderer, however, because it’s easier to add advanced renderer features to it. Further, I will work on implementing a shader for the OpenGL renderer that accurately emulates Doom’s lighting.