Jul 19, 2013
Gaming has changed. While the blockbuster titles have gotten larger and more complex, there has also been a diversification in business models which has allowed smaller companies with social, mobile or indie games to thrive. All of this change has required rethinks in how QA fits in to the gaming industry, and what we teach the next generation in academia. These were key themes at the European Gaming QA & Localisation Forum, the world’s first senior-level B2B event dedicated to QA and localisation in the gaming industry.
The first day started on an interesting note; Gaming IQ boss & founder of the forum Arran Oakes asked the room to raise their hands if they worked in Quality Assurance – unsurprisingly for a conference on QA & Localisation all of the hands went up. Then he asked for hands to remain in the air if they intended to stay in QA, and around half lowered their hands. This, he said, showed how far we still have to go to get QA seen as a respected discipline, and not just a stepping stone into the industry. Throughout the day however it became increasingly apparent that more and more people were beginning to see QA in this light and that standardisation, training and communication would be essential components in this change.
The first presentation of the day was given by Chris Rowley, Senior Director of Certification at Electronic Arts. His presentation was an excellent example of how people has seen changes in the industry in recent years with QA developing from a resource management job “getting enough staff in to do the job” to a much more technical specialty where specialist engineers are required. He also outlined a change which now seems to be industry wide, from a traditional model where development happened first and the developers handed the game over to QA to find the bugs. This model as he showed has changed so that QA are made integral to development and brought in and embedded in the process from the very beginning. With the new model of software as a service, the job of QA also does not finish with the street date, but continues with support of the online platform from the moment the game goes live until it is taken offline.
To continue on the theme of how the field has changed Chris handed over to David King, Technical Director (Quality Engineering). David spoke about the emphasis that has been place on automation at EA using the example of Need For Speed: Hot Pursuit. Automation, he said, was great for performing repetitive tasks to “take the robots out of humans”. They key to a good operational QA system was a clear and comprehensive error logging and collating program which captures everything about a crash even while it is happening. The results he presented were striking to say the least with logs showing the lowest bug count of any Need For Speed game ever. The results from this presentation underlined the importance of constant meaningful feedback to QA and how this can produce large effects in the development of a game.
When faced with a sea of bugs where does one start? The next presenter was Ghulam Khan, Head of QA & Localisation for SEGA. Ghulam’s presentation focused on the bug triage and prioritisation system at SEGA and how this can produce more efficient workflows in QA. The system at SEGA takes many factors into account when deciding the weighted priority of a bug: Severity, Priority and Encounter Risk.A number is assigned for each of these factors and they are combined together to give a combined weighting. The bugs can then be presented in a simple list order with the top priority fixes at the top, ensuring that resources are intelligently directed where there is the greatest need. Bug weighting can also be used to discover interesting things about the product such as ‘on which platform does the game have the biggest problems?’ ‘which weight of bug requires the longest fix time?’ etc. These systems ideally work together to ensure that time budget and resource constraints can be set knowledgably and met properly.
One of the most interesting panel sessions of the day followed the morning coffee break, it was a panel discussion on the need for standardisation in QA chaired by Feike Dijkstra Vice President Interactive Entertainment at GlobalStep the panel was formed of Chris Rowley, Ghulam Khan, Dave Parkinson Director of Global 1st Part QA at Sony Computer Entertainment Europe and Eddy di Luccio, Head of QA at Codemasters. One of the main things to come out of this session was that all were agreed that there needed to be a professional certification for QA and that industry leaders needed to work alongside academia to include QA in syllabi and also look at the possibility of creating masters courses in the subject. All of these steps, they suggested, would help to solidify QA as a recognised discipline. Other suggestions that were mentioned were creating a standardised and consistent lexicon with the intention of having standard documentation within the industry, it was pointed out that with increasing amounts of work done in conjunction with outside parties, this standardisation could ease co-operation and lower ramp up times for projects in partner companies. Although many agreed that working together was important, the different requirements in different companies could make such standardisation difficult.
The next session with Giles Davies, Technical Specialist, Microsoft Visual Studio and AJ Grand-Scrutton, CEO, Dlala Studios was one of the most entertaining of the day. Giles talked about the kind of support that studios need in game development and QA and then handed over to AJ to talk more about how these needs were met by Microsoft’s tools. AJ is a born entertainer and his presentation (rated ‘E’ for Essex) wasn’t the usual conference fare. He spoke of his experiences in the industry and how in his various experiences, QA were almost an afterthought and how since they are the ones who ensure the game is playable, this is an illogical standpoint. Now the CEO of his own indie company Dlala Studios, he talked about how he has made QA central to his process.
Chris Hind, Test Manager for GTO Europe, Microsoft Studios was up next and he spoke about the increasing role of automation in Microsoft’s QA process. He emphasised the things which automation is great at doing: tasks which humans might find boring or repetitive, stress/load tests, compiling information about faults and providing informative performance telemetry. He also outlined the major challenges in automation such as encouraging adoption or communicating the benefits of automation.
In the unique afternoon breakout sessions attendees could choose which session to attend one of which was titled Managing International Coordination, a self-managed session about working in a global environment. A second session facilitated by Andressa Dantas and Stuart Crocker, Technical Leads (SDET), Microsoft focused on a hot topic of the conference Automated Testing and Advanced methodologies. The third session was an unconferenced session where people could create and manage their own interactive roundtable and direct it to any topic which they thought was important and stimulating.
Jesse Penning, Content manager at Ganz Studios gave his presentation after the afternoon break and focused on the human issues in QA under the title ‘Building a Better Team’. His startling statistic that ‘70% of all testers are under the age of 27’ highlighted the employee retention problem that QA has. He discussed how to make these employees feel valued, how to motivate them. Fitting in with the theme of ‘creating a discipline’ the human factor will certainly be a very important aspect of making this happen. Jesse’s interesting interview technique of asking applicants to test a box of tissues became somewhat of a meme of the conference.
The final session of the day was a panel discussion on educating students in QA, the panellists involved were Iain Donald, Institute of Arts, Media and Computer Games, University of Abertay Dundee and Bob Hobbs, Senior Lecturer, School of Computing, University of Staffordshire. The focus of this session was that what is needed to create good syllabi to educate the next generation of testers is reciprocity with the industry. Collaboration can allow the industry to state what skills are important to instill in students and then universities can begin delivering programs to teach these skills.
Overall it was a very varied day which primarily underlined the huge shifts in perspective which are underway in the industry. QA is changing and at the Gaming QA & Localisation conference, it was a thrill to hear from those in the industry who are making this happen. This meeting was a true first – a unique gathering of senior QA and localisation professionals in the gaming industry, and everyone is already looking forward to next year’s event!