{"id":12,"date":"2011-06-12T11:33:18","date_gmt":"2011-06-12T19:33:18","guid":{"rendered":"https:\/\/blog.sigmel.net\/?p=12"},"modified":"2011-06-12T11:33:18","modified_gmt":"2011-06-12T19:33:18","slug":"refactoring-an-asset-pipeline-part-2","status":"publish","type":"post","link":"https:\/\/blog.sigmel.net\/?p=12","title":{"rendered":"Refactoring an Asset Pipeline (Part 2)"},"content":{"rendered":"

Part 2<\/strong><\/p>\n

In the last post, we covered where the state of things currently were with the asset pipeline.\u00a0 Parts of it were slow and inefficient and were a bottleneck in our asset generation process.\u00a0 This was hurting the iteration time of the artists and leading to build times that were unacceptable.\u00a0 So we drew up a plan to give it a major overhaul.\u00a0 Below is the overview for what we were hoping to accomplish.<\/p>\n

First Steps
\n<\/strong><\/p>\n

We had used distributed building systems to speed up our code building process for quite some time, so we wanted to take advantage of distributing our asset building as well.\u00a0 We had this going in a small fashion already, where each platform could be built independently.\u00a0 That only helped a little, however, as there were many parts of the process that were identical between platforms.\u00a0 So we would want to make it so that the identical parts only happened once and the results could then be shared by the platforms.\u00a0 With such a massive restructuring, we needed to be able to properly test to make sure that we were getting the same results as when we started so that we could verify everything was still working.\u00a0 But before we could start breaking our build up we had to make it so that our assets came out the same every time.<\/p>\n

You see, in order to satisfy the requirements of the various platforms, it expected data to be organized a certain way.\u00a0 Sometimes this led to areas containing data that was there only as padding.\u00a0 If this data was not properly initialized, then it could be random (containing whatever was in memory at the time it was written).\u00a0 So different runs with the same data could produce different results.\u00a0 With this being the case, it would be impossible to know if it was our changes that were causing things to be different or if it was just the random data.\u00a0 So the first step was the ensure binary repeatability<\/em>, or simply that the output was always the same given the same inputs.\u00a0 Once that was done then we could have a baseline with which to compare our results and make sure everything was coming out how we expected it.<\/p>\n

Similar to how we split out the materials from the model\/scene information before, it was also planned to break up the model and scene information and handle them separately.\u00a0 That way when the designers moved the layout of the scene, they wouldn’t have to go through a full model build.<\/p>\n

Distributed for the Win!<\/strong><\/p>\n

Since the goal was to distribute the build as much as possible we had to move any inter-asset dependencies to as late in the process as we could.\u00a0 That would allow the majority of the building to occur in parallel since an asset wouldn’t have to wait on something else to finish before it could proceed.\u00a0 In our system, shaders and textures weren’t dependent on anything, but materials depended on both of them.\u00a0 In turn meshes depended on the materials, models upon meshes, and the scene layout depended on the models.\u00a0 However, the only time those dependencies came into play were some optimization steps that we did.\u00a0 One example was that we stripped the model of information from its vertices that wasn’t used by any of the assigned materials.\u00a0 Say if a model didn’t make use of normal mapping at all, then there was no need for it to have tangent or binormal\/bitangent information.<\/p>\n

So we decided that we would split the asset building into two separate and distinct steps.\u00a0 We were influenced by the design of modern compilers and our current code building, where the compiling step happened distributed and the linking step was local.\u00a0 Using this as a guide, we designed compilers and linkers for our assets that had the same behavior.\u00a0 So the compiling step would have no external dependencies other than the source data, meaning it could be done completely in parallel.\u00a0 Then once all of those were done, it would then be linked, where optimization and other platform specific processing could occur.\u00a0 The final step would be to pack the various assets together in a load-efficient manner.<\/p>\n

Don’t Repeat Yourself<\/strong><\/p>\n

As mentioned earlier, a lot of the processing was the same for the various platforms we were developing for.\u00a0 Since the compiling step could have no dependencies and would output all the info needed for the asset linking (which would then do the platform specific stuff), that meant that the compiling only had to be done once, regardless of the destination platform.\u00a0 Similar to code compilers outputting an object file, our asset compilers would output intermediate, platform agnostic data that could be used to build for any platform.<\/p>\n

One thought that I had while designing this system was given that the compilers were outputting to an intermediate file, it didn’t really matter what the source was as long as the result was the same.\u00a0 Because of this, we could tailor each compiler to the input file type instead of having a one-for-one relationship with the asset type!\u00a0 That would allow us to support multiple input types easily to boot.\u00a0 This was fortuitous since we were considering replacing COLLADA with FBX, which had a little more support in our 3rd party tools.\u00a0 Once this pipeline refactoring was complete, then we could build a compiler that would read in FBX files and handle both at once.\u00a0 Then we could make the transition much smoother and less likely to interrupt the art and design schedules.<\/p>\n

So our final design had a compiler for each input type and a linker for each platform and asset type with intermediate files going between them.\u00a0 We could also add additional compilers for other intermediate steps in our pipeline.\u00a0 One example is that we used Valve’s distance texture algorithm for certain effects, where the input was a texture and the output was a different texture (which contained the distance values).<\/p>\n

Packing Files<\/strong><\/p>\n

One of the decisions I made with the original design was to store our assets in two files, one was the header and one was the data.\u00a0 This was due to the fact that I was told that one of the problems that people had seen before that we wanted to avoid was reading in data to one area just to copy it to its final destination.\u00a0 So I thought that splitting the files up so that we could avoid that situation was a good idea, and it worked in most cases.\u00a0 The main point where it failed, however, was when the header and data files became out of sync with each other.\u00a0 This was a rare case, but happened enough that it was something else that I wanted to fix.\u00a0 So the final results instead would be output to a single file that contained a basic table of contents followed by the header and data information.<\/p>\n

Alongside this file would be dependency information which could be used in a final packing step so that we could group multiple asset types together.\u00a0 This was very useful as a load time optimization so we could read in a scene quickly in one read and then deserialize it in memory.\u00a0 However, we would still have loose file support for the times when you just wanted to change one asset and wanted to avoid the final packing step.\u00a0 This support could easily be removed in shipping builds as well.<\/p>\n

Laying This to Rest<\/strong><\/p>\n

So this was a basic overview of the plans that I and several others drew up to optimize our asset pipelines.\u00a0 I was really looking forward to implementing this and watching our artists and designers be more productive and have an easier time getting the game to look and behave how they wanted.\u00a0 I think our first designs and implementations were good for what we could do at the time, but I believe this final product would have been truly stellar.\u00a0 Since I won’t get the chance to actually implement it, writing about what we hoped to accomplish helps fulfill that itch that I’ve had since the realization that none of this would actually come about.\u00a0 Hopefully this might even spark a few ideas in other people on ways that they can help optimize their workflow and reduce the iteration time.\u00a0 Thanks for taking the time to read this!<\/p>\n","protected":false},"excerpt":{"rendered":"

Part 2 In the last post, we covered where the state of things currently were with the asset pipeline.\u00a0 Parts of it were slow and inefficient and were a bottleneck in our asset generation process.\u00a0 This was hurting the iteration time of the artists and leading to build times that were unacceptable.\u00a0 So we drew […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[7],"tags":[],"class_list":["post-12","post","type-post","status-publish","format-standard","hentry","category-programming"],"_links":{"self":[{"href":"https:\/\/blog.sigmel.net\/index.php?rest_route=\/wp\/v2\/posts\/12","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.sigmel.net\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.sigmel.net\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.sigmel.net\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.sigmel.net\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=12"}],"version-history":[{"count":0,"href":"https:\/\/blog.sigmel.net\/index.php?rest_route=\/wp\/v2\/posts\/12\/revisions"}],"wp:attachment":[{"href":"https:\/\/blog.sigmel.net\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=12"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.sigmel.net\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=12"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.sigmel.net\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=12"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}