Refactoring an Asset Pipeline (Part 2)

Part 2

In the last post, we covered where the state of things currently were with the asset pipeline.  Parts of it were slow and inefficient and were a bottleneck in our asset generation process.  This was hurting the iteration time of the artists and leading to build times that were unacceptable.  So we drew up a plan to give it a major overhaul.  Below is the overview for what we were hoping to accomplish.

First Steps

We had used distributed building systems to speed up our code building process for quite some time, so we wanted to take advantage of distributing our asset building as well.  We had this going in a small fashion already, where each platform could be built independently.  That only helped a little, however, as there were many parts of the process that were identical between platforms.  So we would want to make it so that the identical parts only happened once and the results could then be shared by the platforms.  With such a massive restructuring, we needed to be able to properly test to make sure that we were getting the same results as when we started so that we could verify everything was still working.  But before we could start breaking our build up we had to make it so that our assets came out the same every time.

You see, in order to satisfy the requirements of the various platforms, it expected data to be organized a certain way.  Sometimes this led to areas containing data that was there only as padding.  If this data was not properly initialized, then it could be random (containing whatever was in memory at the time it was written).  So different runs with the same data could produce different results.  With this being the case, it would be impossible to know if it was our changes that were causing things to be different or if it was just the random data.  So the first step was the ensure binary repeatability, or simply that the output was always the same given the same inputs.  Once that was done then we could have a baseline with which to compare our results and make sure everything was coming out how we expected it.

Similar to how we split out the materials from the model/scene information before, it was also planned to break up the model and scene information and handle them separately.  That way when the designers moved the layout of the scene, they wouldn’t have to go through a full model build.

Distributed for the Win!

Since the goal was to distribute the build as much as possible we had to move any inter-asset dependencies to as late in the process as we could.  That would allow the majority of the building to occur in parallel since an asset wouldn’t have to wait on something else to finish before it could proceed.  In our system, shaders and textures weren’t dependent on anything, but materials depended on both of them.  In turn meshes depended on the materials, models upon meshes, and the scene layout depended on the models.  However, the only time those dependencies came into play were some optimization steps that we did.  One example was that we stripped the model of information from its vertices that wasn’t used by any of the assigned materials.  Say if a model didn’t make use of normal mapping at all, then there was no need for it to have tangent or binormal/bitangent information.

So we decided that we would split the asset building into two separate and distinct steps.  We were influenced by the design of modern compilers and our current code building, where the compiling step happened distributed and the linking step was local.  Using this as a guide, we designed compilers and linkers for our assets that had the same behavior.  So the compiling step would have no external dependencies other than the source data, meaning it could be done completely in parallel.  Then once all of those were done, it would then be linked, where optimization and other platform specific processing could occur.  The final step would be to pack the various assets together in a load-efficient manner.

Don’t Repeat Yourself

As mentioned earlier, a lot of the processing was the same for the various platforms we were developing for.  Since the compiling step could have no dependencies and would output all the info needed for the asset linking (which would then do the platform specific stuff), that meant that the compiling only had to be done once, regardless of the destination platform.  Similar to code compilers outputting an object file, our asset compilers would output intermediate, platform agnostic data that could be used to build for any platform.

One thought that I had while designing this system was given that the compilers were outputting to an intermediate file, it didn’t really matter what the source was as long as the result was the same.  Because of this, we could tailor each compiler to the input file type instead of having a one-for-one relationship with the asset type!  That would allow us to support multiple input types easily to boot.  This was fortuitous since we were considering replacing COLLADA with FBX, which had a little more support in our 3rd party tools.  Once this pipeline refactoring was complete, then we could build a compiler that would read in FBX files and handle both at once.  Then we could make the transition much smoother and less likely to interrupt the art and design schedules.

So our final design had a compiler for each input type and a linker for each platform and asset type with intermediate files going between them.  We could also add additional compilers for other intermediate steps in our pipeline.  One example is that we used Valve’s distance texture algorithm for certain effects, where the input was a texture and the output was a different texture (which contained the distance values).

Packing Files

One of the decisions I made with the original design was to store our assets in two files, one was the header and one was the data.  This was due to the fact that I was told that one of the problems that people had seen before that we wanted to avoid was reading in data to one area just to copy it to its final destination.  So I thought that splitting the files up so that we could avoid that situation was a good idea, and it worked in most cases.  The main point where it failed, however, was when the header and data files became out of sync with each other.  This was a rare case, but happened enough that it was something else that I wanted to fix.  So the final results instead would be output to a single file that contained a basic table of contents followed by the header and data information.

Alongside this file would be dependency information which could be used in a final packing step so that we could group multiple asset types together.  This was very useful as a load time optimization so we could read in a scene quickly in one read and then deserialize it in memory.  However, we would still have loose file support for the times when you just wanted to change one asset and wanted to avoid the final packing step.  This support could easily be removed in shipping builds as well.

Laying This to Rest

So this was a basic overview of the plans that I and several others drew up to optimize our asset pipelines.  I was really looking forward to implementing this and watching our artists and designers be more productive and have an easier time getting the game to look and behave how they wanted.  I think our first designs and implementations were good for what we could do at the time, but I believe this final product would have been truly stellar.  Since I won’t get the chance to actually implement it, writing about what we hoped to accomplish helps fulfill that itch that I’ve had since the realization that none of this would actually come about.  Hopefully this might even spark a few ideas in other people on ways that they can help optimize their workflow and reduce the iteration time.  Thanks for taking the time to read this!

Refactoring an Asset Pipeline (Part 1)

Lost Project

I want to talk about a project that I planned a lot for but never got the chance to implement: Refactoring an Asset Pipeline.  Granted, this isn’t particularly exciting stuff for most people, but it was going to drastically cut down on our asset creation times which would do nothing but good for our company and the iteration speed of designers and artists.  However, it ended up not happening since the company changed its focus and we ended up parting ways.  So, I have decided that I wanted to post instead about what were planning on doing, since I think it was a good design and in this way I sort of get to do the work by proxy.  And maybe someone will even learn a thing or two about it and the world will be a better place somehow.  Even if it is just a little.

The Old Way

On the project we wrote our own engine and the tools to support it.  We had very short development cycles so stuff had to be done quickly.  I was responsible for designing and writing a few of the asset pipelines, starting with the texture and shader pipelines, while the model/scene pipeline was handled by the senior programmer.  This was my first time ever writing pipelines of this sort from scratch, but I think I did a decent job overall trying to make things flexible and quick.  Granted, I would do it a bit differently knowing what I know now, but that is part of the reason for writing about this in the first place!

We were developing for multiple platforms, and it was decided to create a separate pipeline for each platform so that you only had to build the assets for the platform you are working on.  This was accomplished by having a central executable which was responsible for loading and executing the corresponding DLL for that asset and platform.  All the common code for the various asset pipelines was shared and the platform specific code put in separate files to keep the code maintenance to a minimum.  We standardized on a few asset input types so that we didn’t have to write code to support everything under the sun.  For example with textures we were only going to accept DDS (Direct Draw Surface) files since they were very flexible, easy to read, and there were plugins for Photoshop that would read and export them.  Granted, this didn’t stop the artists from later wanting support for more image types, but then it was a simple matter of just running some tools to convert the JPEG, PNG, Targa, or whatever else to DDS before running it through the pipeline and into the game.  For model data we chose to go with Collada and shaders were simply handled by FX (plain text) files split into separate vertex and pixel shaders.

Asset Dependencies

For the first project it was decided that scene information (instances and transformations of them) would be combined with the model information.  Additionally, materials (shader selection and parameter settings) would also be unique to each mesh (which belong to a single model).  This means that we effectively only had three major assets from the rendering pipeline point of view: models, textures, and shaders.  While this decision reduced our flexibility somewhat, it allowed us to get the game started faster and artists and designers to see their stuff in game in a shorter period of time.  Eventually this would lead to some pretty heavy bottlenecks considering that changing a texture or shader parameter would require the entire scene be rebuilt.  However, some real-time communication with the game was set up so that changes could be seen immediately for tweaking purposes to try to improve the workflow.

After the first game shipped, it was petitioned by the artists to get the materials split out from the meshes so that they could be shared.  In addition, our (now much bigger) tools team created a material editor since the one that I made that ran in XSI had some shortcomings (mainly due to weird artifacts of using XSI’s realtime shader API) plus it never really achieved the artists desire of having XSI as a decent previewer for their assets (which it never would be able to since it wasn’t running with our lighting, shadowing, or other visual effect systems).  So more effort was also put into beefing up our in-game viewer instead which could be controlled by the material editor tool

Next Time

This is an overview of what we started with when we set about completely redesigning how assets would be built.  It could take several hours if a new build had to be made of our game, which led to many very late nights for our build engineers and those getting the build out.  Not to mention the wasted time of the artists and designers due to the inefficiencies present in the system anyway.  Next post will focus on how we were planning on tackling the various issues listed here.

Also will try to get a few visual aids to help show how things used to work, so check back for those and if you have any suggestions on (free) tools that I can use besides Paint.NET, I’d love to hear it.