“As we departed the wizard’s tower we left burdened with more questions than we arrived with.”
- The Quest for Knowledge, M. K.*
Before we begin I would like to lay some “ground rules”. What I propose here is based on experiences I have had with automated testing and software development. Nothing is set in stone. I feel many teams turn off their own decision making ability by referring to some article or book as the “rules” to creating solutions. You will hear “We can’t do that, SomeTechAuthor said ‘that’ was bad”. The following week ‘SomeOtherTechAuthor’ will say ‘that’ is good. There are no clear answers here. I only present questions that you and your team need to answer for yourselves and your projects. Like many things the answer is ‘it depends’. Clearly defining the statement “For our team we define X as this….” is one of the most freeing and empowering statements your team can make. Not following edicts from crystal towers on high but, defining your own path. Ok off my soapbox. Now take everything I am about to say as law. ;-)
Would Someone Think of the Tests?
You are your test’s eyes, ears and hands. When you give someone limited instructions you will get limited results. Your tests cannot assume the state of your application. When you use an application you are looking at it with your eyes, interacting it with your hands, hearing audio prompts and using a lifetime of perspective about how applications work and behave. What does our poor test have to lean on? Nothing. It is trapped in the dark with no eyes, ears, hands or knowledge. You are the only one that can give any of that information to it. Remember the automation tests are to behave like surrogates for your human users.
Query Based Testing
Think of what you do when you use an application. This is the process your tests need to follow. For example, if I handed you a phone with either configuration A or B and said “Click the Start Button”. Could you do it? For you it would be easy. You would look for the button’s location, position your finger and click the start button with confidence. Something we all do everyday many times. Would it matter if the button was positioned anywhere on the screen? Could you still click it? Yes, of course. If the button was visible you would be able to click it. This very basic thought experiment is the key to writing automation tests well. Mirror what your user does, not just the process of physically interacting with your application. Mirror the thought process they need to follow to interact successfully.
Now that we have awarded ourselves an amazing prize for our ingenuity and skill in clicking buttons as higher level primates. Let’s see what a typical approach would be for writing an automated test for the same action. Of course, the test name must be very descriptive
Perfect. Looking at some “pseudo” example code it would be something like this:
Software engineering at its best, descriptive comments to really drive home what is going on, truly awe inspiring. We as humans get all the benefits of millions of years of evolution, huge brains and amazing eyes which can master the universe and we decide to give our tests what? A hard coded set of coordinates. This was coded on a Monday for scenario ‘A’ above. It is now the second Wednesday of the month so therefore the designers now need to change their minds, the app now looks like ‘B’. So the test passed for about a day and a half and now the builds are failing. Unless we want to return to this test every time something changes we need to give our poor test just a bit more brains. Let’s re-code this to let the test ask and see where the button is:
Now we have given our test just a bit more horsepower to do its job. The start button can move wherever it wants and this test will pass as expected. Scenario A,B,C,D, back to B and finally Z will pass.
The ‘AskGameForWhereTheButtonIs’ and ‘ClickThatButton’ are not actual methods but, the intention of what they do is important. We control every part of our application. We know where anything is at any point. In the Assassin’s Creed games our player jumping from rooftop to rooftop does not deftly dodge the oncoming crossbow bolts by inches. We make the bolts miss by inches to make them think it is their amazing skills. Again, we control everything. Why not take that control and give it to your tests. If we can calculate the arc of the parabola of a player’s jump in game we can determine at runtime where in screen space a button is. “We have control of everything…” ok so how do we do that? The detail you may have missed in the code above is the all important ‘StartButtonId’. Having a consistent identifier for the elements of your application are important. No matter where the button exists its ID always stays the same ‘StartButtonId’. This is far better than passing around the Unity hierarchy path which will be invalid the moment something changes. We want to be able to query the state of our application and interact with it easily. Your developers need to be able to write tests that are terse as they need to be, easy to understand and resilient to change. Hard coded values and assumptions like “This only works when the screen resolution is 1080p” will make things brittle and a pain to manage. Design for change. We know it will happen all the time, don’t resist it. Go with the flow, flow with the go.
Obviously, the code above is a bit over dramatic but it does give us a simulation of what we need to do to avoid the perception and potential reality of “automated tests are brittle”. We are writing tests to add confidence to our development process. Yes, automated tests do need to be maintained and curated. At the same time we want to limit the number of tests that need to be modified or adjusted as our application develops and changes.
Setting Up Our Test Project
Before We Start
I am assuming you have GameDriver https://www.gamedriver.io/ installed in your project and your license file is copied into the “GDIO” directory. If you have not done this step please refer to GameDriver’s installation instructions and the license file you received from GameDriver. Some of this article is going to reference specifics to Visual Studio, if you are using a different IDE there are equivalent functionalities providing similar solutions.
The Testing Framework
We are going to run our GameDriver automation tests using the ‘Test Explorer’ window in Visual Studio. There are similar “Test Runners” in other IDE’s VSCode and Rider. Visual Studio supports MSTest, NUnit and XUnit testing frameworks. We are going to be using NUnit to run our tests. If you have used the NUnit tests inside of Unity this framework should be familiar to you.
Unity Project Configurations
In the Unity development community I have seen two project configurations:
1. Teams that keep all code inside the Unity project and use solution / project files generated by Unity. This is a majority of teams and Unity developers, especially in the smaller teams and indie community.
2. Teams that keep their code in external projects and copy the dll's into the Unity project as managed plugins. This is less common. These teams typically have more mature pipelines / CI / CD.
The motivations, pro’s & con’s and implementation details for both could be a series of articles in of themselves.
When building automated tests there is sometimes a drive to make the tests themselves external to the primary project. This presents a number of different challenges for teams
that fall into category #1. Just initially there might be push back to the required modification to their workflows.
1. Where is this new solution / project located in reference to the Unity project?
2. Pulling down the one Unity project is not enough to start working there needs to be an additional directory as well.
3. What are the dependencies between Unity built assemblies and the automation test project?
#3 is where the main issue lies for most teams. If the external automation project is completely independent of the Unity project, then it may not be a concern. For most I don't believe this is the case, most of the time the tests are going to want access to the same assemblies as the project. If the assemblies are built by Unity in the project under test, how does this external project reference the assemblies? From the external project does it reference the dll's in the Unity project's 'Library/ScriptAssemblies' or if they are using constants / values from the Unity libraries they need to find where in the Unity Editor install folders the specific dlls reside. I in the past have been on a team in category #2. We created NuGet packages to wrap Unity assemblies and eventually copying them into a ‘libs’ folder instead. My hope here is to avoid all of these possibilities and have our solution blend smoothly into the "standard Unity" practice of most teams in the #1 category.
We will build on top of Unity’s IDE integration so we can maintain our code all within our Unity project. The issue we will run into is Unity will continually re-generate our projects for us whenever a change triggers IDE project regeneration. To use NUnit we could hand copy in assemblies but using NuGet packages (https://www.nuget.org/) will be easier. No matter which route you decide to go, any changes you make in your test project referencing these libraries will be overwritten by the IDE project generation. To give us more control Unity provides documentation here on the “Managed Code Stripping” page. To demonstrate this process we are going to walk through testing a basic ‘Hello World’ example. The unity package with this code is at the end of the article.
Do We Have To Make the Sausage?
Some of the following may feel like we are doing a lot of work for not much benefit. I would frame the work we are going to as “making the sausage” for your team. Many people do not want to know how the sausage is made but, sure do enjoy it at the breakfast table. Once this work is done it will blend into the background and be completely transparent to your team. It is oddly satisfying to do something that benefits your project and team which is nearly 95% of time never mentioned or noticed. It just works. Once this is done your typical Unity development workflow will not be changed except now you will be writing automation tests.
Simple Button Scene
The code provided gives us a basic scene to interact with and test. Initially it will have one button and a simple class to show the button has been clicked. Once installed you will see a ‘GDHelloWorld’ folder with a ‘GameDriverHelloWorld’ scene and a single code file ‘ButtonClickFader’. Press play, the Text object will become hidden. Click the Button and the Text will become visible and then eventually fade away. See Package Link Below.
The GameDriver Tests
- Create a folder called ‘Tests’ under your project’s ‘Assets’ folder.
2. Create an Assembly Definition file for our Tests folder ‘GDHelloWorldTests’
Ensure it is only enabled in the ‘Editor’
3. Add a new code file and call it ‘GDTests’
4. Double click our new code file and open it in Visual Studio.
5. Starting at the top we are going replace the existing using statements with the following:
a. We are using .NET System, NUnit framework and GameDriver’s API.
6. We will no longer inherit from Unity’s MonoBehaviour.
7. To label our class as one that will hold tests we will add NUnit’s [TestFixture] attribute.
a. NUnit has a well rounded API with all the functionality required to manage your tests as
you expand the test coverage in your application.
b. I recommend taking the time to walk through its tutorials and documentation.
8. Now let’s write our first test, it is just here to show a test running and passing.
- Each test function has NUnit’s [Test] attribute
- We have a basic check ‘Does this int value have the value it was assigned to?’
- When a test fails the result message will follow the convention of the first argument as the “expected value” and the second argument as the “test value”.
Running Our Tests
Great, we have a test now we need to run it. In Visual Studio there is a ‘Test Explorer’ window which will allow us to see, sort, filter and run all our tests.
9. In the ‘Test’ menu go to ‘Windows / Test Explorer’.
10. In the ‘Test Explorer’ window click ‘Run All’
11. Looking in the ‘Output’ window you will see a similar message saying there is no ‘test adapter’. This means there is no framework installed that runs the tests.
12. To fix this issue we are going to install two NuGet packages ‘NUnit’ and ‘NUnit Test Adapter’. Before we install these we need to make Unity and NuGet cooperate. Currently your Unity project has a ‘Packages’ folder where your ‘manifest.json’ is held and any ‘Embedded’ packages live. When installing packages from NuGet the default folder is the ‘packages’ folder in the same directory as the Visual Studio solution. To avoid this folder doing double duty we are going to let Unity use the existing ‘Packages’ folder. We have control to tell NuGet to use a different repository folder.
13. By adding a ‘nuget.config’ file we can specify a new ‘repositoryPath’. As seen below we tell NuGet to use a folder called ‘nugetpackages’ instead.
NOTE: Close Visual Studio and re-open it to ensure the 'nuget.config' is loaded.
14. Now we can install our two NuGet packages ‘NUnit’ and ‘NUnit Test Adapter’.
15. Right Click your Solution in the Solution Explorer and select ‘Manage NuGet Packages for Solution’
16. Under the ‘Browse’ section type in ‘nunit’ in the search filter.
17. You will see various listings, find ‘NUnit’ and ‘NUnit Test Adapter’.
18. Select each one of the packages, check your test project in the right side panel and click the ‘Install’ button.
19. You will possibly get a pop-up to confirm the install unless you have disabled it previously.
20. After installing both packages you can go confirm in your ‘nugetpackages’ folder in the same folder as your solution and see each package has its own folder.
21. You will also see there is a new ‘packages.config’ file that specifies which NuGet packages have been installed.
22. Before we return to Unity, now attempt to run your tests in the ‘Test Explorer’.
23. You will see the very happy green check marks of passing tests, along with a print out in the Output window about the test adapter running the tests.
24. Now return to Unity. Once you give Unity focus it will see there have been modifications to the project and re-generate the Visual Studio projects.
25. Return to Visual Studio and attempt to re-run your passing tests. It will complain there is no test adapter or no test executor.
26. Also, in your test project’s References right click the ‘nunit.framework’ reference and select properties.
27. Note the path to the ‘nunit.framework’ is referencing the ‘nunit.framework.dll’ inside Unity’s ‘PackageCache’ folder.
Modifying IDE Project Modification
We want to override Unity’s default behavior and make some modifications to the generated test project. Unity gives us access to hooks after the generation has occurred to inspect and modify the output. These hooks are part of the ‘SyntaxTree.VisualStudio.Unity.Bridge’ assembly, again if you are not using Visual Studio there are libraries for your IDE. To put in our new code we are going to be creating an Editor only assembly definition. There is a 'VSProjectGenAssembly.unitypackage' package at the bottom of this article, if you don't want to hand enter this.
- Create a new folder called ‘VSProjectGen’.
2. Create an Assembly Definition called ‘VSProjectGenAssembly’.
3. As we did before make sure it is only included in the Editor.
4. In the ‘VSProjectGen’ folder create a new ‘ManageVSProjectGeneration’ code file.
5. In this file we are going to wrap the file in a preprocessor to make sure this code only runs when using Visual Studio. Also include the following using statements.
6. The class called ‘ProjectFileHook’ will be given the ‘InitializeOnLoad’ attribute. This will ensure this class loads when the Editor loads or the code is recompiled.
7. We will then define a static constructor with an event registration on the ‘ProjectFileGeneration’ event. A class’s static constructor executes anytime it is loaded so we will always be listening for this event.
8. The event gives us the name of the project file and the content of the file as a string. I am going to do something only for super advanced engineers like myself, a hard coded string. As the comment below says we could do this many different “cleaner” ways. ScriptableObject, json data asset, naming convention, detect test files in project, external REST api, astrological planet alignment coefficients, remote viewing, essential oils. You get the idea. I leave it to you and your team to do something “clever”.
9. Now that we know the project what we are going to do is modify the XML of the project and insert two ‘Import’ elements to reference NUnit and NUnit Test Adapter. These are inserting ‘prop’ files which are known as PropertySheets in Visual Studio. They are a great way to define and share property settings between your projects.
10. Now we are going to modify the existing ‘nunit.framework’ to reference the newer NUnit assembly we added using NuGet. This will not break anything in your Unity project. The Visual Studio projects are only for our usage as developers. Unity still does all its own compilation inside the Editor. Modifying these projects won’t change the compilation of your code and project.
11. Finally we are going to print out a message that we made our modifications and save the document out as a string to complete the modified project generation.
12. Give Unity an opportunity to compile and re-generate things and you will see our new project ‘VSProjectGenAssembly’ and if you look in the Editor’s Console window we should see our message.
13. Finally if you repeat the process we did early on your test project’s ‘nunit.framework’ reference you will see it has been redirected.
14. You can also rerun your single test and see that the NUnit Test Adapter has been detected and the single test passes.
Our First GameDriver Test
If you are still with me we are finally getting to where we wanted to go in the first place. Setting up and running our first GameDriver test. GameDriver uses a component called ‘GDIOAgent’ to manage communication between your game and your tests. When starting your tests there is a local network connection made between your test runner and your game. The sequence of events are as follows:
- We will run our tests in Visual Studio.
- The Unity Editor will automatically start playing the loaded scene.
- The GDIOAgent starts and connects to your running tests.
- Tests will run.
- On completion of the tests we clean up and close the GDIOAgent.
Once you get this working it provides a very powerful platform to build our query based tests, simulate user input and automate nearly anything in your game.
Setup / Teardown of GameDriver
- In the scene you are going to test add a ‘GDAgent’ GameObject.
2. Add a ‘GDIOAgent’ component. Leave the default settings on the Agent.
3. Now back in your test project. In our ‘GDTests.cs’ file we are going to add a function with a ‘OneTimeSetUp’ attribute. This is run once per test run session. Again, please read over NUnit’s documentation. :-)
4. We will be interacting with GameDriver through its ‘Api’ object. This article is no replacement for reading the docs. GameDriver provides documentation for its Api and active forums. We call ‘Api.WaitForGame’ which triggers the Editor to play and communicates over the localhost ip address “127.0.0.1’.
5. Once a connection is established we tell GameDriver to override the keyboard and mouse input.
6. We will also define the clean up function using the NUnit ‘OneTimeTearDown’ attribute.
7. We are going to disable our overrides on keyboard and mouse input, then we are going to pause to give the Api a moment to clean up then call ‘Quit’ which will stop Unity playing and clean up and close down the agent.
Writing Our First GameDriver Test, For Real This Time
We made it. We finally made it to where we wanted to go in the first place. Let’s write our first true “automation test”. Get ready to update your resume and LinkedIn. When we get this running you can with confidence say “I have experience with automation testing” in your next interview. You are welcome.
In all seriousness let’s walk the steps to define your first automation test with GameDriver.
- Define a function ‘ClickTheButtonAndConfirmClickCount’ with the NUnit attribute ‘Test’. This will specify it is a test to be run in this Test Fixture and be listed in the Test Explorer.
- Naming of tests have many (many, many) different possible patterns, some are just descriptively long, others take the form of MethodName Should When, State When Expect. Look up ‘unit test naming conventions’ and your team can hash it out, let the religious debates begin.
- Implement the following function.
- Let’s walk this line by line.
- First we will trigger the Api to click the object at the unity path “/Canvas/Background/Button” with the left mouse button. GameDriver uses an XPath like syntax to query objects within your Unity project.
- Next we ‘WaitForEmptyInput’ this triggers a wait for the last input action to complete.
- We will now request the value of a field from the object that manages the number of times the button has been clicked using ‘GetObjectFieldValue’. The query translated into plain english is:
- “Look for an object at the root level called ‘Logic’ with a component of type ‘ButtonClickFader’ then on that component return the ‘TextShownCount’ field.”
4. Finally, assert that the ‘expected’ and ‘clickValue’ variable are equal.
5. Now run our test by clicking ‘Run All’ in the Test Explorer.
- Unity will begin playing
- There will be a small dialog panel in the upper left hand corner with the date time and mouse coordinates.
- A black cursor arrow will click your button.
- The display text will become visible and fade out.
- Back in Visual Studio the Output window will show the tests have run.
- The Test Explorer will show you happy green checkmarks with the time it took to complete each test.
- Unity will stop playing after the tests have completed.
5. Congratulations you did it! Your first automation test, wow. You are in the club now, your decoder ring and secret hand shake instructions are in the mail. Things are totally different now.
Let’s now flex our new found skills and demonstrate one of the first issues we will run into when writing automation tests. Testing can take time. Notice in the above example we connect to the GameDriver agent, run a number of tests and then shut down the agent connection and game. In the unit test space the idea is each test is supposed to be run in isolation so you can clearly control the state and focus specifically on testing one thing. The reason that works is many traditional unit tests are code only tests with any external dependencies mocked out. We don’t have or want that luxury. The intention in automation tests is they are to be run as close as possible to what an actual user does when playing the game. This is up for debate as all things are and a decision your team will make for yourselves. In our scenario we don’t have time to start and stop our application for each test. Instead you will be starting the application and running a number of tests.
In the attached package there is also a Component called ‘MoveOnClick’. It will move the button to a new position in its parent Canvas whenever the button is clicked.
- Add the ‘MoveOnClick’ Component to the ‘Logic’ GameObject
- Write the following test:
- Very similar to our last test but, we will be passing in the number of clicks we are expecting to make.
- The NUnit ‘Values’ attribute allows us to define multiple runs of this test each with a unique value passed in.
- The ‘waitBetweenClicksMilliseconds’ is a value we can tune as needed, at times we need to give the application time to finish processing. The Api provides a way to create pauses within your tests with a variety of ‘Wait’ functions. Leave it at 5000 milliseconds, then on the next run turn it down to 100 milliseconds.
- Finally we do the same test as the last test but we are using the passed in value.
3. Looking in the Test Explorer you will see 3 versions of this new test.
4. When running the tests you will see one might pass but the rest will fail.
5. The problem here is we need to keep in mind we are running our application one time and triggering multiple tests. Again, remember our tests don’t know anything about the state of application unless we tell it. You could reset the state of your application between your tests or you could do what a user might do, just look at the state of the application and react accordingly. Avoid any assumptions of state information, query first and take into account what you learn.
6. Modify our test to first get the starting value of the ‘TextShownCount’ then modify your test to confirm your clicks have added onto that value. Now it will pass no matter when your test runs ends up running in your test suite.
One more tip for free. When you start noticing a repeated set of query and operations like our ‘TextShownCount’ and converting it to an integer. Do yourself a favor and wrap it in a function, instead of having to search everywhere for that query string.
This repeated string is all over:
Replace it with:
This has been a great start to adding automation testing to your team’s tool belt. You have the ability to make your automation tests as brittle or as resilient as you see fit. Just remember:
- Query first
- Assume less
- Think of your poor tests :-)
Good luck out there.
- GameDriver's main page. https://www.gamedriver.io/
- Unity’s Managed Code Stripping Documentation
- Unity’s Assembly Definition Documentation
- Unity's Embedded Packages Documentation
- Unity's InitializeOnLoad Documentation
- NUnit Main Page
- NuGet Config File Reference
- Visual Studio Property Sheets Reference
*No, not a real book but, sounds pretty good right? Must be a line from something.