Monthly Archives: September 2017

Dev Diary #3: Interaction between XAML and DirectX 11

Introduction

Foot sliding is a common problem in computer animation. In short, for example, if you create a “walking” animation, it is expected that the feet stick to the ground on every other step as the character moves forward. If this character does not move forward at the same speed as the feet move, it gives the impression that the feet are “sliding” over the floor. The solution of this problem is to adjust the absolute translation speed of the character by adding the relative speed of the sliding foot to the floor. In other words, if the foot slides backwards then increase the speed of the character, or, if the foot slides forward then decrease the speed of the character.

A walking animation is a rather straight forward fix. However, for much more complex animations like a spinning hook kick, then the adjustment is not that easy to define.

Most animation tools come with a way to fix this problem automatically… or at least some good developer has come with a solution and he has been kind enough to share it with the world in the form of a script, plug in or add-on. However, since I’m doing my animations with my very own animation studio, then it seems that I’m out of luck.

No matter, though, for if there is a will then there is a way. So I went ahead and created a custom tool to solve this very special need (after all, that is what I do for a living) by trying a very special type of project in Visual Studio where XAML and DirectX coexist together. I won’t lie, this was not an easy experience: The learning curve was very steep and unfortunately the documentation is very difficult to follow. However, once the tool was done, I was able to grasp the main idea behind this kind of projects, and now I can see the benefits of combining XAML and DirectX.

I’m surprised no one has published anything about this type of project. I’m not talking about marketing brochures or news releases (there are plenty of those): I’m talking about the coding side of the story, expressed in terms that are easy to understand. So I decided to create a blog about it.

XAML and DirectX

For those who are really old school (like me), I need to make a quick parenthesis and define XAML as a markup language created by Microsoft, kind of like the language used to create web pages. With this language, we can create quite impressive User Interfaces and implement them using a wide range of technological platforms to create business oriented applications (msdn – 2017).

With the release of the Windows Store and the concept of Universal Windows Platform Applications, XAML has had a huge momentum. It’s been such a success that Microsoft thought about mixing XAML and DirectX. Ever since Visual Studio 2012 was released, if you want to create a videogame or a multimedia application for the Windows Store using DirectX, you will find that this API comes in two main flavors: With and without XAML.

Now, what I learned from the process of creating this custom tool is that the combination of XAML and DirectX is great for Multimedia Applications, but it is my conclusion that this combination may not be quite optimal when building action games.

Before getting to the reasons behind this conclusion, let me show a video that illustrates the interaction between DirectX and XAML so we can have a graphical reference about the topic.

To understand what just happened, we need to look at the block diagram of the inside of this application, and compare it to the “traditional” approach, which I mean, the one without XAML.

Traditional Block Diagram

When you create a DirectX 11 App (Universal Windows) Project using Visual Studio 2015, the template created (excluding the “Game” class – that one is on me) has the following block diagram:

DirectXClass Diagram

  • The App class is the starting object, and it is in charge of interacting with the Operating System. Windows will load, unload, activate and deactivate the game by accessing this object.
  • The Device Resources class is a wrapper that will help us deal with DirectX API. This frees us from dealing with the most obscure objects from DirectX.
  • The Renderer class is the object that will load all our graphic resources and keep them in buffers, and will wait patiently until it gets the chance to draw them.
  • The Main class is, as it sounds, the main object of the game.
  • All the previous classes come included in the DirectX project template. What I like to do is to create a “Game” class and encapsulate all my routines there. I do that for portability purposes so, if I ever need to change versions of Visual Studio, I just need to copy the game class and beyond, knowing that the main hooks will apply for any other template (this was a life saver when I migrated from Metro style apps in Windows 8 and Visual Studio 2012 to UWP in Windows 10 and Visual Studio 2015).

The dynamic here has some similarity with a typical XNA project:

  • There is an infinite loop in the App class that triggers the “Update” routine, then the “Draw” routine and finally the “Present” routine. This is what most people call the “Game Loop”.
  • The “Update” routine will call the “Update” method of the Game class, putting all components in motion.
  • The “Draw” routine will call the “Draw” method of the Game class, rendering all graphics.
  • However, all input data is handled by the “App” class. What I do is that I populate input information in structure variables that the Game class will interpret during the “Update” call.

XAML and DirectX Block Diagram

When you create a DirectX and XAML App (Universal Windows) Project using Visual Studio 2015, the template created (excluding the “Game” class – that one is also on me) has the following block diagram:

XAML and DirectX Diagram

  • The “App” class is still the starting object, although that’s pretty much all its functionality.
  • The “DirectXPage” class is the actual XAML page. This is where we would implement all our user control elements for input data. This class also takes over of almost all input events.
  • The Device Resources class is still the wrapper that will help us deal with DirectX API, freeing us from dealing with the most obscure objects from DirectX.
  • The Renderer class is still the object that will load all our graphic resources and keep them in buffers, and will wait patiently until it gets the chance to draw them.
  • The Main class is still the main object of the game.
  • All the previous classes come included in the DirectX project template. As in the traditional approach, I also recommend creating a “Game” class and encapsulate all game routines there, for portability purposes.

The biggest, meanest, most complex and most important difference that for some odd reason nobody has ever remarked, is that this new approach is a true multi-thread application.

Here is how things get on fire:

  • The “App” class creates an instance of the “DirectXPage” class, which in turn creates and launches two independent threads, each one with its own infinite loop:
    • The first loop is launched to link input events from pointing devices, like the mouse or a touch screen.
    • The second loop is the actual Game loop, triggering the Update call, the Draw call and the Present call.
  • Aside to all this, the DirectXPage, as a XAML page, has its own thread, usually referred as the “UI Thread”.

To describe in colloquial terms the implied challenge, we can compare the threads of a multi-thread application with the spoiled children of a highly dysfunctional family:

  • Every single one of them hates to share.
  • The communication between them is reduced to the bare minimum.
  • If one of them ever breaks protocol, we get such a drama that will last until the next Thanksgiving dinner.

It gets more volatile if we consider the nature of these threads:

  • No one but the UI Thread can interact with UI elements in an XAML page. That’s almost a cardinal rule.
  • DirectX has some roots from the good ol’ COM API. This means that objects created by one thread can only be accessed by this very same thread. If a different thread touches it, the whole thing crashes – I learned that the hard way.
    • Ok, granted, there is a way to make DirectX be able to handle multiple threads, but the procedure to follow is waaaaaaaaaaay above my level.
  • The input thread is a wee bit more flexible. However, this is the one in charge of communicating input data to the other two drama princesses. It’s still doable, but it is like walking on eggshells.
  • The class hierarchy doesn’t help either:
    • The “DirectXPage” class knows the “Main” and the “Game” classes. However none of these last two classes know the “DirectXPage” class. Adding an include entry in the header file as a reference to the DirectXPage creates a circular reference that generates an error at compiling time. I’m sure there is a way to fix this, but at this point I have absolute no idea how.
    • Even if the circular reference is solved, only the UI Thread can access UI elements, so there is not much benefit here.

The solution I found was to abandon the idea of a direct communication between all components and, instead, I implemented an indirect communication through the use of messages stored in a common repository (a bunch of variables known by everybody).

  • The “Game” class is known by everybody. As such, it becomes the perfect place to store our messages.
  • The “Main” class template comes already with a “Critical Section” object, which is the one to use to regulate access to our shared repository to read and store messages.
    • For those who are not familiar with a multi-thread application approach, this “Critical Section” object is also known as “The Bathroom Key” (Purwar – 2013): In order to use the premises, you need to get a hold of this key. Once you finish your business, you release this key for the next person to use.
    • Likewise, if any thread wants to access the common repository, it needs to get a lock of the “Critical Section” object, and release it once it is done.
    • I assume there is no need to say that in both cases you need to clean the aftermath mess on every use.
  • Following this approach:
    • Every time the UI thread wants to say something to the “Game” or the “Main” class, it needs to get a lock of the Critical Section object to avoid any conflict. I have labeled these messages in the diagram as “BG Events”. In my case, I’m using an enumeration. I know, it doesn’t really fit the formal definition of an “event”, but it is close enough, and it kind of works for our purposes.
    • The main game loop needs to get a hold of the Critical Section object every single time it calls the “Update” and “Draw” methods of the “Game” class, ensure that only one thread ever access objects created using the DirectX API. This gives the “Game” class exclusive access to the common repository.
    • Now, on every “Update” call, the “Game” class checks for any messages (or “BG Events”) that need to be processes. If so, it does what the message requests.
    • If the “Game” class needs to send information to the UI Thread, it leaves a message (what I have labeled as a “UI Event” in the diagram) in the common repository.
    • The UI Thread seldom runs as fast as the main game loop. However, it has an event that is called on every Draw call (called “CompositionTarget::Rendering”). This is the perfect place to get a hold of the Critical Section object and process any message coming from the Main Game loop.
    • Likewise, the Input thread will get a hold of the Critical Section object when it wants to communicate input data to any of the other two threads.

Benefits of using XAML

After reading all this rambling, you may be wondering why should we use this template if the implementation of XAML and DirectX is that complex. The answer is in the benefits provided by XAML:

  • A XAML page grants access to some very useful controls, like Textboxes and the File Open dialogs. This saves A LOT of work.
  • XAML runs on its very own UI Thread. By doing so, we don’t need to worry about a stutter in the UI controls, knowing that the Main Game loop will still run at the desired speed of 60 fps.
  • DirectX has always had a problem displaying text: Processing a long string is a tall order, considering that it is processed and drawn 60 times per second. If XAML is left with the task of displaying text, we reduce the risk of dropping from the desired 60 frames per second.
  • Sometimes, informational data doesn’t need to be updated at the same speed as the game loop. For example, the health bar of the player could be updated just a couple of times per second, making this element a perfect candidate to be displayed by the XAML page.
  • As a game developer, if you want to split the screen in two (one half for each player), a XAML page should be able to accomplish that easily… although I have to confess that I have never done this – no one to play with in a one-man team.

There are some reasons that XAML may not be an important component:

  • Based on the diagram shown previously, the game loop will get exclusive access to the computer resources on every “Update” and “Draw” call, leaving only a very small percent to the other two threads during the “Present” call. This could make the UI thread even slower, especially for hi level processing games.
  • Messages sent between the game loop and the UI threads are at least one frame behind. This could be the difference between winning and losing if the controls have been coded in XAML.

Conclusion

After I created my custom tool, I was able to grasp the “big picture” of mixing both XAML and DirectX. It wasn’t an easy project, and the development strategy described here about messages stored in a common repository may not have been the most optimal approach. I worked before with XAML during the Silverlight days and I have to say that I didn’t find much difference when used for UWP applications.

However, I found controls like the “Open File” and “Save File” dialogs to be essential for Multimedia applications (for me, it was a project saver). Also, I can see the performance benefit that RPG games could get when using text and input controls implemented in an XAML page (for example, sending messages between players using a XAML textbox while still rendering their fancy graphics on the background).

Still, for action games where control response is paramount, I’d still code everything within the main game loop, which makes a XAML page kind of an optional component. Then again, that is my personal point of view.

References

  1. “What is XAML?” Microsoft Development Network. 2017. Web. https://msdn.microsoft.com/en-us/library/cc295302.aspx
  2. “Multithread: How to Use the Synchronization Classes” Microsoft Development Network. 2017. Web. https://msdn.microsoft.com/en-us/library/thdxkfx9.aspx
  3. “Understanding Thread’s Synchronization Objects with a Real Life Example”, Priyank Purwar, 23 Jun 2013. Code Project. Web. https://www.codeproject.com/Articles/610566/Understanding-Thread-s-Synchronization-Objects-wit

Dev Diary Entry #2: Using Microsoft’s Graphic Debugger

Introduction

When the DreamBuildPlay 2017 challenge was announced, I immediately put aside all personal projects I was working on at the time (they were all rather boring anyway). I created a new instance of my game engine, copied some animations from my previous games, baked some draft 3D models and used all that to build a quick prototype. Despite of being placeholders, when the main character started kicking butts (literally), I got pretty excited. As I went further on, I created more animation sequences as well as a new “combo attack” manager, and I was in the middle of adding some fancy graphics… when I noticed a small stuttering every now and then. I got somewhat alarmed since having performance problems this early in the development phase is pretty much a dark omen.

Wondering what on Earth I had done wrong this time, I ran the Graphic Debugger feature included in Visual Studio. The numbers I got were unexpected, but they pointed out what my problem was. I fine-tuned my code and soon enough I had the performance issue under control – I hope. The process was so useful and smooth that I thought about writing a quick blog about it to share the experience.

Microsoft’s Graphic Debugger

To give a quick introduction history, the Graphic Debugger is a feature in Microsoft’s Visual Studio, introduced in version 2012 Professional Edition and became widely available with Visual Studio 2015 Community Edition. The early version was a little bit confusing. However, the new version is so easy to use that, in just a few clicks, I got an analysis of the tasks that were run by the computer’s GPU, along with the time it took to execute each and every one of them.

Diagnose Procedure

So, without further ado, let me give a quick account of what was done in hope that this same process could be useful to other indie devs: Having my project loaded in Visual Studio, I went to the “Debug” pull down menu, then “Graphics” and then “Start Graphics Debugging”.

GraphicDebugger01

The Graphic Debugger application requested to run in high elevation mode, and then my prototype was launched. As the game session went by, a number in the top left corner kept me informed about the time it took to draw the current frame. This number never went below 17 milliseconds (equivalent to 60 frames per second), which is consistent with the fact that my game engine synchronizes with the display’s refresh rate (as we all should, for there is no point of presenting frames that the player will never see). I pressed the “Print Screen” button a couple of times to take snapshots of the frame at hand, and then I quit the prototype.

The next screen showed me a summary of the collected data.

GraphicDebugger02

In short:

  • The first graphic is a plot of the time that the GPU took to draw the requested frame. Overall, each frame too around 17 milliseconds to draw, for the exception of six instances:
    • The biggest chunk (which can be interpreted as a drop in performance) happens at the beginning of the session. This is expected since that is when the game loads most of the assets in memory, making the game a little bit unresponsive for about 10 seconds, just before the main menu is shown.
    • The second biggest chunk happens when the game loads all assets needed to execute the first level of the game. Likewise, this drop is expected.
    • The next two chunks, linked to an orange triangle on top, are the screenshots taken for the frame analysis. These two drops are also expected.
    • The last two chunks, and this is something I need to work on, is when the game compiled, at run-time, the animation sequences for the main character. In other words, this is a problem and I better add some code to cache those animations.
  • The second graphic on the Report tab is the same information as the previous plot, but seen as the number of frames per second. In other words, this is the mathematic “inverse” of the previous plot.
  • The bottom section has a list of individual frames that were captured by the diagnose tool.

From the frames listed at the bottom, I clicked on the second frame (the first one usually is not that accurate). The next screen provided a detailed list of steps executed by the GPU for that given frame.

GraphicDebugger03

Those tasks that have the icon of a black paintbrush have a screenshot attached to it, so clicking on them will show, on the right, what was being drawn at the time.

To the right of the “Render Target” tab, there is a tab called “Frame Analysis”. On that tab, there are two options: “Quick Analysis” and “Extended Analysis”. The latter option breaks all the time, but the first one is good enough for most diagnoses.

After a quick analysis, the application shows a report of the collected data for the given frame.

GraphicDebugger04

The first section of this output report puts the collected data per task in a bars graphic, focusing on the time it took to process each of them. When seen like this, it is very easy to spot problematic draws. The second part of the report has the information collected in numbers. Likewise, the potentially problematic draws are highlighted in orange (actually, it’s salmon, however this is technical post – the graphic design ones come later).

On the screenshot shown, in both sections, there are two tasks that pop out: tasks 467 and 470. Going back to the “Render Target” tab and clicking on the suspected tasks, I found out that my little trees shown in the background were being a process hog for my game. Altogether, these two tasks alone were consuming a third of the 17 ms threshold for a 60 fps game.

Data Analysis

Although I did implement a custom routine to have these trees drawn in different colors (part of the fancy graphic feature I was trying to implement), the shaders I created were not in any way complex. Moreover, these trees have a very low number of polygons (as seen on the report, each one has about 174 polygons) and that is why I was so surprised about them being the culprit.

Anyway, long story short, the problem can be summarized as follows: these trees are huge. It’s not quite evident on the screenshot, but these trees are instances of a 3D model, and that is why they look different based on the angle of view. This means that some branches are drawn on top of each other, using some transparency effect, meaning that the color of a given pixel is not final until all branches are drawn (not to mention that these trees overlap each other at some point). Now, given that a pixel shader is executed at least once for every pixel on the screen, and that the current resolution of the screen is 1680 x 1050 pixels (making the trees about 500 x 500 pixels), this is A LOT of calculations, especially for texture sampling. Moreover, as the report shows, the GPU tried to draw 18 trees (9 on each pass), however only 8 of them are actually visible, which means that almost half of the time spent was pretty much wasted (I state that almost half of the time because I still need two pair of trees on each side in case the player needs to go side to side).

Conclusion

Usually, every time I work with custom shaders I run the Graphic Debugger to see how many things I broke in the process. Knowing where the time is allocated when drawing a given frame helps me focus my attention to specific shaders. In this case, after reviewing what I had typed, I did find a way to optimize the background trees’ pixel shader and that took care of the stuttering – for now. To be fair, my development system has an NVIDIA GeForce 8500 GT, which has a pretty poor performance (it has a PassMark of 139, making it pretty far below on the list), so, if my game can perform well on this system then it should run fine on most computers out there.

I will continue to use the Graphic Debugger as often as I can. However, if in the final product you see brick walls instead of trees, then you know what happened.