Game Testing on Android Emulators without x86 Support

As you know from the previous blog if you have read it, if not you can still read it HERE.
I am trying to find the best possible solution for testing games when there is not enough resources and how to automate that process.
For current engine ( Unity ), Firebase Game Loop testing framework does wonders but it requires a development team to implement the support for it. Usually also coding a bot to perform any game related tasks… This approach works well, if the game is small bot can be easily made, but if the game is quite large then another solution would be needed. I am working on a mobile game, it has a fair chance of features that having bot to do will be enough, but I also want an option to have an “exploratory” testing method. With “exploratory” testing methods, I mean to have games receive input events, sensor events, or any kind of events / command and perform them in that order.
So question how to achieve that?
To achieve that, depending on the platform and engine there are some solutions, and I won't go into details too much how I will achieve them, at least not into this post, here I am dealing with what happens after you have your tests ready but you are unable to execute them on emulator as most emulators only except x86 builds.

First thing that came to my mind when I got into this situation was: “Cool, I could use Firebase, but Firebase is limited to only 15 tests per day...What next…?”
Damn, yeah 15 tests per day may sound much, but actually it is not, to test for example 10 different languages on phone, that would take 10 devices - 10 tests.
To test different aspects at least 5 one, would take 5 devices - 5 tests.
What about sub tests: such as playing a game with a specific character, or play without sound, or watch for performances :/
Yeah on free-model not gonna work, but then again what about creating multiple accounts, then multiple “projects” then switching and running them.
So, If I needed 60 tests per day, I would need to create 4 different emails, to have 4 different projects and then through CL to manage that.
Still could work but it is a hassle.

My dream is to not be limited by the number of devices, but by the number of hardware, so in my case before we execute tests on real devices, to execute them in a simulated environment and have them executed at least each night. Now you may ask why without x86 support. Recently Unity removed x86 support for Android with version 2019.3 so you are no longer able to build for that architecture, therefore I am unable to use android emulators as they are all x86. It is true you could run their armeabi images but they are extremely slow..
I tried to run Genymotion, as in back of my mind I was convinced it uses armeabi architecture, but sadly they also use x86 architecture so that approach didn't work well.
Then I started to dig around and found some possible solutions:

  • Bluestacks
  • MEME
  • Dolphin
  • Andy Android Emulator
  • LDPlayer
Not bad, there are solutions...
Just you need to research what performances each of these emulators can do, what you can simulate on them as in ( OS, Language, Hardware ), and how much of them can you run on your build / test machine to create a farm.
So far all of these emulators are locked on specific OS version such as: 5.1.1 or 7.1.2 or 4.4 It is not bad but then I can forget for now testing about OS compatibility in this way. I can choose which resolution to run, which is ok, at least some aspect checking can be done, I can change language, and adjust hardware settings.
For now this has to do, and I went with implementing a solution for LDPlayer.
LDPlayer supports running multiple instances and it doesn’t eat up a lot of resources.
Currently on my machine which has 32gb RAM and solid i5 process works great, and I am able to run 3 instances fine, and continue my working on other things :)
So I have started to code an option to run tests. Currently I am interested into invoking Firebase testing a.k.a Game Loops since I have coded bot to respond to events.
First thing first, I need to have an apk to select to install, to clear game data, and let my tool work its magic of invoking those tests. Since I can’t showcase the current game implementation tests, I will show an example on an already existing apk. For this apk I will just do “exploratory” testing basically just to install an apk on all connected devices and verify it is installed.

For that I want simultaneously on all connected devices, apk to be installed, and after installation to be run if all goes well, if not then to report what was the error!
This is an easily achievable job using adb and it is just a matter of a few commands, but the goal is to write as few commands as possible and still have some options of editing applications and tests :)
So right now I have coded a system to do this, and it involves making a wrapper around an adb and creating “test” cases documents that look like this.
app_installed {args[1]}
remove_app {args[1]}
install_apk {args[4]} {args[2]} {args[1]}
run_app {args[5]}

It is a simple test, to check if an app is installed, if it is it will attempt to remove the app if it exists, it will install apk, and then run apk. Arguments for this command are received depending on apk, and obb and rest what tester is selected.
In the end the test did work, just I haven’t downloaded obb so technically the app did run just wasn’t able to detect the obb file :)

Comments

    Leave a comment