Automation Testing for Games

Introduction

Game software like any other software requires a proper amount of testing before its release. This document introduces some of the technologies that are used in game development and describes the testing (automation) solution for games developed using those technologies.

Game Development and Testing

Game software has two major facets:

Functional Automation (UI)
Message/State (Non UI)

The UI consists of the actual interface with which a user interacts and it also forms the area of the game software where a bulk of the bugs occurs. The user interaction with the UI, especially for high-end graphic intensive and complex gameplay oriented games, presents a potential for input combinations that cannot all be estimated when developing the game or writing game software code. The UI can hence be tested with a combination of repetitive (regression) testing, standard procedural testing and exploratory testing. Combinatorial testing is one way to achieve the balance between omitted and tested.

Types of Technologies Used for Games

Because of the innumerable technologies used in game development the UI can be developed in many different ways. Depending on the type of game, the UI can be made using some of the following technologies

Browser based games – flex/flash, HTML5, CSS, PHP, Javascript etc

Downloadable games – Java, C/C++ etc

Some combinations of the above are also possible, for example game clients with embedded browser window or embedded flex/flash content.

The standard ways for game UI automation testing include:

Using off the shelf tools and frameworks (especially for flash games) to write repetitive tests that are mainly used for regression testing

Writing artificial intelligence code, eg: bots that can play the game multiple times and faster than a normal player. These bots are typically run on a server that runs the game every few hours to check its functionality. The comprehensibility of these tests varies from simple repetitive “Happy path” use cases to complex gameplay simulations. The choice of test cases depends on the type of the game and the aim of the automated testing.

The interactions include the message flows between the game client and the game servers and the various components of the client. The automated testing of these interactions is rare and not usually carried out in-depth. The automated testing of the interactions allows the QA to circumvent the UI which would normally slow down the testing process, thereby making this way of testing faster and more efficient. The main aim of interaction testing is to discover integration and integration path related bugs in the game software.

The messages that can be used, in a game environment, range from standard HTTP requests to messages from servers written in JAVA or other languages.

Another context of testing game software which is quite popular is unit testing of the game code. This can be accomplished in any way that the development team approves of, including the use of automated unit test generation tool to generate the test cases.

The tools used depend on the language that is used for development, for example, JUnit is used for unit testing JAVA code.

Solutions

Build v/s Buy

There are many good tools in the market like IBM’s Rational Functional Tester, Seetest by experitest, and monkey talk by gorilla logic, that handle a lot of the scenario’s that one faces when testing games. The solutions described here, though, are custom built solutions that have been designed keeping in mind the specific requirement posed by the game software. The decision to build the custom solutions was based on the fact that covering most of the scenarios that are faced while testing game software are usability intensive and require unique handling capabilities which cannot be fully achieved via standard tools or frameworks.

The solutions that are described here make use of many open source technologies like selenium, guava and other assistive technologies like MSAA, JAB etc.

Functional Automation (UI)

There are a range of technologies used to develop game clients. Most game clients can be classified into two broad categories – downloadable and no-download.

Downloadable

The solution for the functional automation of these clients is a combination of JAVA and accessibility.

The frame work is designed using JAVA and the interactions with the various objects/elements of the game clients are done using JNI calls via Microsoft Active Accessibility. The flow of control is depicted in the Fig 1.

Application Under Test
NATIVE LIBRARY
JAVA FRAMEWORK

The JAVA framework is the driving engine of the tool. It is responsible for the reading and interpreting of the tester’s input, managing of the results and the generation of the final report of the execution for the tester.

The Native Library is the repository of the methods that are used for finding the objects on the UI of the software clients and performing actions on these objects.

Handling Custom Objects

In order to handle custom/non-standard objects in the client software a custom wrapper was created that could be implemented and filled with methods with custom handling mechanism to ensure that the properties required for automation are available for the test framework.

These wrappers can be enabled/disabled as per the requirements of the build.

Handling of flex/flash objects

The flex/flash content, in downloadable as well as no-download client software is made automatable by using the APIs provided by Adobe.

SYSTEM MANAGER

tom Agent
Automation Tool
Automation
Automation Manager

The assets are compiled with or without the custom agent classes in order to enable or disable automation for the flex content.

Handling embedded web pages

Another aspect of the testing of downloadable clients is handling embedded web pages for example the login page. There are many types of embedding engines that let you embed web pages and provide functionality specific to a browser Eg. Trident (Internet explorer) and Chromium Embedded Framework (Chrome).

Some frameworks like Trident and Gecko (to some extent) support the accessibility APIs in windows but the same is not true for CEF (for version 1.x).

For CEF the solution proposed is to use a combination of the wrappers written for the custom objects and the javascript integration provided by CEF.

Web based / no-download

The functional testing of web applications ranges from simple flex/flash based games to complex social games. There is also the need for compatibility testing i.e. to test the application on multiple browsers.

Internet explorer supports all the accessibility APIs in windows and can be automated in the same way as other software. Selenium has been integrated into the framework in order to provide the ability to run automated cases on multiple browsers.

JAVA (applet etc.) based web applications can also be handled by using the JAVA Access Bridge which exposes the JAVA accessibility API which enable Java applications and applets that implement the Java Accessibility API to be visible to the automation tool.

Mobile game testing

Mobile application can be either native apps that are installed on the mobile device or mobile web apps that can be played on the browser of a mobile device. The operating systems for mobile device like android and IOS have custom emulator for enabling the testing of applications made for these OS environments. Automating the test case execution is possible with the help of proprietary software like Seetest, ranorex etc. and with open source software like Monkey Talk (for Native applications) and selenium android driver and IOS driver (for mobile web applications).

Summary

The framework supports the automated testing of application made using host of different technologies that are used for game development. A quick snapshot of the handling is given in Fig 4.

AUT
Adobe CA
JAB
MSAA
FRAMEWORK

Message/State Testing (Non UI)

Another important aspect of testing games, both downloadable and no-download, is the testing of the integration paths and the interactions or message flows between the various services/components of the game-engine.

The solution is a framework designed around JMeter to handle server messages along with HTTP request/response pairs. The extension to JMeter allows the handling of asynchronous server messages that can be used in a functionally valid manner in order to execute a functionally valid flow in terms of messages and HTTP requests. A snapshot of the working of the framework.

JMETER
Trade-off

Designing custom solutions may provide the added usability and customisability but there are a few draw backs to not using standard/off the shelf frameworks. One such drawback is the lack of record- and-play capabilities which results in the object maps and test scripts to be written by hand. Another trade-off is the lack of a UI for editing all the input in one place.

Gaps and short-comings

As mentioned earlier there are a few trade-offs that come with creating custom tools for automation like the lack of record-and-play capabilities leading to difficulties in creating test scripts and object maps. There is also no UI editor to manage all the input data and configurations in one place. The testing process is also longer due to the sequential nature of the automation testing.

What’s next?

One of the most popular forms of gaming today is social gaming, i.e. playing games over the network or on social networking sites. These games take advantage of the features of social networking sites to tailor make a game environment that fits its players.

Some of the features of social gaming are: Real time support, Asynchronous gameplay, Community i.e. leveraging the social network of the players, No victory conditions or endless games.

Though most social games are designed using standard technologies like flash, javascript, PHP etc. the automated testing of these games is made difficult as it can present scenarios that need support on the above mentioned aspects, which is both very interesting challenges and enriching learning, way to go !!! But yes will be there soon.