Sword of the Stars II Dev Diary: Building a graphics engine

PC Gamer routinely features Developer Diaries: tales of what goes on behind-the-scenes in the development studios making your favorite games.

PC Gamer routinely features Developer Diaries: tales of what goes on behind-the-scenes in the development studios making your favorite games. In this particular entry, Darren Grant, Lead Programmer on Sword of the Stars I and II at Paradox, walks us through the process of building a brand new graphics engine from the ground up . For more information, read the first entry, and let us know what you'd like to see developers discuss in future diaries in the comments.

Over the last five years--through two expansions, a weapons pack and a tech pack--the graphics engine that fueled the original Sword of the Stars game (MARS) has done everything we asked of it. More than we'd originally intended, really. Sword of the Stars II seemed like the perfect time and opportunity to look back at our previous engine and build the future from the ground up.

Part of the process of designing a new engine architecture is identifying exactly what can be improved over the previous iteration. For MARS2, we took a number of new approaches to accommodate our goals for the sequel: provide the player with the capability for additional immersion and interaction with the universe. Below, I break down the different elements that we've added/upgraded in order to bring our full vision of SotS II into existence. It can get a bit technical at times, but I hope you find it enlightening!

Direct 3D 10

For most gamers, the benefits of stepping up to D3D10 is shrouded in mystery. That's partly because Microsoft's main goal with this version is mostly to ease the lives of OEMs and developers, by forcing adopters to agree on competent standards. What this means is that, unlike D3D9 (where hardware and drivers delivered a hodge-podge of mixed and matched features for developers to wade through--a compatibility nightmare), D3D10 guarantees a very powerful common baseline that stays the same. For example, as a graphics programmer, I know with near certainty that if the shadow rendering works correctly on my work station, it will behave correctly (albeit at different speeds) on any DX10- or DX11-era hardware and software. There were no such certainties in D3D9, which added to lost development time.

Shader Model 4

One of the more exciting features that D3D10 adopters get is the common Shader Model 4 baseline. A D3D9 game is limited to Shader Model 2.0 on average (which was the shader model target for SotS Prime). Sadly, the lack of basic computing power on the GPUs of that era severely limit the number of features that could be achieved concurrently.

That bottleneck caused us a lot of problems. You want skinned models? OK, the programmer would come back instantly with a solution! Now you want to add 2 dynamic light sources per mesh? Hmm, it's going to take a day for your programmer to rewrite the shaders to fit both features in the given space available. Now you want shadows? Your programmer grumbles, goes away for a couple weeks and comes back with a rewrite of the renderer that jumps through hoops to solve the problem. Just don't try to fit more in after that.

One of the things we learned developing SotS Prime is that if you don't set up the proper tools for the artists initially, our programmers would spend a lot of time attached to the artists, tweaking parameters. And toward the end of a project, when things are getting tight, we'd be forced to drop certain polish features in favor of actually going final. Upgrading our engine let's us avoid that this time around.

Render stack

Render stacks handle and sort the drawing instructions for each frame and optimize them before they reach Direct3D. There are two major categories of 3D rendering in games: Forward and deferred. Forward rendering is a technique where each pixel is drawn to the screen once, with all of the shading values already accumulated. This can be very efficient in situations where there are very few light sources, and is optimal in genres like FPS, where the world geometry is largely static and can therefore be heavily optimized. It doesn't work well with dynamic lighting, though, so there's also deferred rendering, which separates the lighting and material rendering passes, making it optimal for situations with lots of relatively small dynamic light sources.

Our render stack uses a deferred technique to produce some brilliant dynamic lighting and post-process effects, many of which can already be seen in various screenshots and videos. These effects can be tuned based on the class of video card available. For instance, the depth of field blur radius may be reduced, or flat-out disabled based on the player's preferences of image quality vs. performance. The gamut of effects that can be disabled include full scene shadowing, hundreds of dynamic light sources, decaling, refraction, local ambient occlusion, full HDR bright/bloom and image based ambient lighting.

Particle effects

Including the rich particle effects of the previous game in SotS II is big deal for us. MARS2's particle-authoring system has been greatly revised to enable integration with material rendering and control of dynamic lighting systems. With material integration, some complex and dramatic new effects are possible, and with control over dynamic lights, the particles can convey a much greater presence as they skirt past the hulls of nearby ships or fade into scorch marks.

Thanks to our deferred rendering system, the engine an handle hundreds of simultaneous light sources. There's no reason that a particle system can't be authored to provide a beautiful animated glow that casts light on surrounding geometry in the game. Having direct control over the lighting inside the particle authoring tool gives artists very direct control over the integration of emitted light and the particles they want to draw. In other engines this might have to be done by going to two separate tools, which disrupts work flow.

Shared material library

Another key component to ensuring our artists' efficiency is our shared material library, which enables artists to create and change the appearance of multiple models at once. Each material is based on existing shader class, which means that artists do not need to write code or cut and paste parts of cryptic shader trees--with the support of an in-house model viewer, the shared material library lets artists fine-tune the parameters only once and see the effect on all models that use that same resource.

For instance, an artist might create a material called CruiserGlass, which calls for a green-glass shader with all the fancy refractive and reflective trimmings, tailor the colors and light levels, and then share those settings with all of the glass panels in all Tarka cruisers. Without the help of a single programmer or having to tweak the models at all, an artist can change one paramater on the CruiserGlass model and see the changes across all Tarka cruisers in the game simultaneously.

Asset pipeline and multithreading

In order to further reduce load times between major state changes and improve the performance of the game in general, we built a massive asset pipeline. This pipeline combines a new offline build process with an engine-integrated background loader and cache. Most assets are mapped directly from their files by the runtime, and once they're in the RAM, they stick around unless the cache needs to recover space. This eliminates most of the long loading time problems that can happen when moving between encounters and the star map.

MARS2 had been designed to work with modern multithreaded architecture, which will also greatly help loading times. We're using a hybrid approach to target the 2-8 core PC's that are common today, with an eye to 16+ core desktop systems that are promising to emerge. One primary CPU thread does the heavy lifting and secondary threads are assigned to specific support jobs such as loading and processing assets, building render lists, or executing high level game logic. Using this logic, for example, the engine will prepare for an upcoming combat encounter.

Bullet Physics and 64-bit

Another way we're aiming to make the wide array of weapons and ships feel more alive is by adding Bullet Physics, a superb physics engine that has seen successful application in many major games and movies. It lets us model much more complex collision geometry than in SotS Prime. This greater level of detail goes a long way towards conveying the much larger range of scale in the game.

One of the questions I get asked the most is, “Will Sword of the Stars II ship with native support for 64-bit OS?”. The answer is most assuredly, “Yes!” In fact, we aimed the MARS2 engine to target both 32 and 64-bit builds since the first lines of code went in, a little over a year ago.

I hope this overview of the new game engine answers a lot of the questions people have been asking about SotS2 and maybe even a few that haven't been asked yet. As we head towards completion of the game in the coming weeks, we'll get into more of the fine details about the game itself, so stay tuned for our future dev diary entries!