Game Loops ( Firebase ) Command Line Testing

This Blogpost won't be long and I will try to keep it as much simple as I can :). When your team is small but game a little bigger, you are trying to find a way to get into automated testing. True with the manual testing you can cover a lot of more grounds and search for those little bugs that at the specific condition they occur, but you can't perform manual testing always. Reason - It takes too much time to do the testing of the whole game :O.

So what can you do if you want to have a well-tested game but not too many resources to do it?

You can try to implement automated testing for your game :) The are many sites and programs that offer this, but my advice is to look for one that suits your project the best and the people. Currently, I am doing a game in Unity Engine and you guessed it, a game is for Mobile Platform. Ideally, I would like every night, or every Friday a build machine to execute tests, but where?

We don't have too many physical devices, and I can't ask people to leave their phones for the testing, but what I can do is:

• 1) Use Firebase Test Lab and Game Loops functionality
• 2) Create Emulators on the build machine and run tests on those devices
• 3) Use Genymotion and simulators for testing
• 4) Find other services

I decided to go with Firebase Test Lab and make my game tests run on it. There are some good and bad things regarding Firebase Test Lab. One awesome thing that I like in a variety of devices that it has, and video recording of the test execution. What I don't like is the limitation of how many devices can be used daily, and for how much time your test is executed. So on physical devices, you are limited 30mins per device, while on simulator devices you are limited to 1h per device. You could say 30min per device is a lot, but it depends how much you want to test, and how big is your game. Our game is not small, so if I want to test full gameplay pass, a test would surely take more then 30mins. To pass this obstacle I have made it to work a little bit faster, but because of that some things may not seem right or can't be properly tested. So once I got everything running, I decided that I want an option before executing tests on Firebase Test Lab, to run them locally. Luckily they provide an app called Test Loop Apps. And with this app you can execute your test as long you have your game installed and app installed. Pretty good except, what if I want to run those tests on emulators or many connected devices? In that case, I don't want to go to each device, install a game, install a test app, and then run tests through the app. Too many manual steps... Solution using ADB ( Android Device Bridge ) and creating a script that will invoke testing. Right now I have developed a small testing App, that helps me with some fast testing when I need it. It can show me the device, instead to do everything on the device and small screen I can do it on desktop, it can clear cache and data of a selected application, get logs, record screen, and few other utils. The App can be found HERE. What I needed to know is how the app is invoking tests and running apps. After digging a bit in logs I saw it is sending a broadcast message with the following information:

com.google.intent.action.TEST_LOOP typ=application/javascript flg=0x2 cmp=ComponentInfo{com.mycompany.myapppackage/com.unity3d.player.UnityPlayerActivity}

Ok...So, it is broadcasting a message to send an intent with the action of TEST_LOOP to my app and running its Launch Activity, but how does it send which test to execute? That took another digging, but this time in implementation of Firebase package. You can find a Firebase package for Unity HERE. In AndroidTestLabManager.cs there is this part of the code:

private void CheckIntentForScenario() {
 Debug.Log("Getting Intent's action");
 string action = _intent.Call<string>("getAction");
 Debug.Log("Checking for test loop");
 if( action != "com.google.intent.action.TEST_LOOP" ) return;
 ScenarioNumber = _intent.Call<int>("getIntExtra", "scenario", NoScenarioPresent);
 Debug.Log("Fetched scenario: " + ScenarioNumber);
}

This is where it checks, which "scenario" ( test ) to execute. Ok, so it expects an integer to be sent within a broadcast message with a key scenario and then scenario number. To achieve this with adb I had to make a command like this:

adb shell am start -a com.google.intent.action.TEST_LOOP --ei "scenario" 1 -n com.mycompany.myapppackage/com.unity3d.player.UnityPlayerActivity

Whoa, :O And game run with testing option! But the victory is not yet achieved...Usually, if you would run a test through Test Loops App, it has a "timeout" for test execution, so if your test stuck somewhere a game would be killed and other tests could proceed with execution! With running through adb< I can't make that but there are some options what I can do to achieve the same functionality :

• Hard code in the test execution for how long it will be executed!
• Add a script that would ping a game and ask for test execution status, so if a test is "executing" the same step for the last couple of seconds/minutes, kill the app and stop test executing.
• Recreate functionality into my little testing app.