Life As a Tester: Part 2
By:
Schuyler Lystad
|
April 13, 2010, 8:04 pm

The same problems I mentioned in Part 1 (which you can read here) presented themselves when working for Microsoft (excuse me, a separate company that did software testing, and whose only customer was Microsoft's Xbox 360 division), also in Redmond. After coming back from college, I applied for the testing division, which they call “Functional.” They offered me as high as $9.00/hour, but with a college degree, I really felt like I should be making more and politely declined. They called back the next day, and said they had a higher paying position, $12.25/hour, for something similar, but not altogether the same. This is how I worked in “Compliance.”


See, at the company in question there are three tiers to testing. “Compliance” is the highest. During testing, there are particular test cases that a game must meet before it can get the final approval. These test cases can take the form of things like time limits on loading screens (15 seconds if non-interactive, 45 if interactive). Other test cases can be things like making sure save files stay uncorrupted, checking for missing “storage devices” (places to save, like a hard drive or memory card), and assuring that games sport basic features like pause menus, different resolutions, multiplayer modes that work offline and online, etc. I thought these were really admirable things for a company as big as Microsoft to set aside an entire division for, until I realized that failing to meet these standards would not result in the cancellation or delay of a product, but would merely lead to a mandatory fee paid by the developer to the publisher (guess who) if the developer chose not to fix it. I can think of at least one big name game developed by a first party studio that flaunted these requirements almost entirely. I promise you've played it.
 

The tier directly beneath "Compliance" was referred to as “Functional.” The responsibilities of this role are less structured than they are at Nintendo (I worked a few off-days there, and I still don't know how to log their bugs), but the hours are just as irregular. The only difference is here you don't get called the night before. Instead, you get added to a nightly e-mail for the following day saying how many people they think they will need. When you come in the next day, sometimes the list is accurate. Sometimes it's too small. More often than not the list is too large. I guess they just expected us to cross our fingers and hope for the best.

The next tier down was “Hardware.” These are the people who sit in front of a console turning it on and off all day long, or who have some other menial tasks. I'm not exaggerating. Their pay reflects the menial nature of their duties. While on the job, I heard that these "Hardware" drones didn't shower and apparently stole food from the fridge even if it was marked. Generally, it was understood that their position was so volatile that they just grabbed what they could during the few times they worked and quickly tried to move up the tiers or find another job.
 
"Compliance" was a much more intimate setting than either of the other tiers. In a space with less square footing than some large living rooms, as many as twenty compliance testers were packed, each having up to two television screens, two computers complete with monitor, and at least two dev kits. We were side by side, almost like in a canister. Rather than work on one project for weeks, we performed our test cases on as many as ten games a day. Technically, for most games I could take care of all of my test cases without even entering gameplay once. Our project leads knew that we were all desperate for hours, and wouldn't turn a very strict eye towards what we were doing, so long as everything got done. Three hour days became eight hour days. The guy that trained me had it down. He'd take one test case and stretch it out to as many as six hours. I thought he was just lazy at first.
 

This "Compliance" job had the best structure of all of these differing tester roles. It meant that bad games were off your plate in an hour if you were quick, and there was a way to hold onto builds of a game you actually liked and play them for months if you chose. Considering these games make their way to Compliance several months before the stores, I was the first of my friends to play and beat Resident Evil 5, for example. Naturally, with a non-disclosure agreement, I couldn't message these friends and tell them about it and what I thought of it compared to RE4, although I could engage in particularly heated debates with them in my head when I got bored.

There were two offices for Compliance: one in Redmond, Washington, and one in England, which had fewer people than the American branch. Every game released on a Microsoft console or in the Live Aracade goes through one of these two offices. At Nintendo, we worked solely on North American games. Here, we took builds of games in Japanese, French, Spanish, German, Mandarin, one in Cantonese, Italian, Norwegian, Swedish, Danish, Finnish, Dutch, one in Czech, Russian, one in Hungarian, Korean, Vietnamese, and maybe a couple others I'm forgetting. You may be thinking, “less than forty people, and all those languages? What do you have to speak to be hired there?” Truth be told, the only languages we ever had covered in any competent manner were Japanese, English, Spanish, German, Korean, and Mandarin when we hired this one girl (yes, we had a girl). They learned a little later that I could choke through some French and Norwegian, at which point I became the expert for all Scandinavian languages and French, but for the most part, speaking one of these languages meant you were practically hired. I actually like trying to get by with limited knowledge of a language, but understandably, most people didn't, and when we got every build of a worldwide game in for testing, especially a big budget one, we had to go through as many as ten different languages. Most people got the English build on one console and the one they were assigned on the other. Checking error messages for content was a hoot.
 
To be honest, you don't even really have to know that much about languages to be hired here. One of our leads said to anyone who was listening, “Whoa, if you select French, they put the game in German? That's pretty insensitive.” I took a look, and then asked him to do it again for me. Everything was working fine: he had just been selecting Dutch the whole time.
 

One caveat that I will throw out there for those interested in pursuing careers in testing: If you're allergic to dust, don't get this job. When I joined, the testing room hadn't been dusted in more than a year, probably two. During my stint there, when the janitors finally did take up the task most of the way through the year, none of us could recognize the newly cleaned tables. 

At first, I had a couple dozen test cases I had to execute, and the first few weeks, I had to keep reminding myself what each was. Eventually, I could predict which cases would be assigned to which games. With this knowledge, I could do several at once, and I could do others' test cases as well. This is a job where you apply a formula and sorting system to creative works that have, as one of their many goals, the desire to elude your classifications. Quite often, whether a game passed or failed boiled down to my judgment rather than any real formula or criteria.
 
One thing that's undeniable about testing is that the products you work on will make it out the door in some form. I remember a huge blockbuster title (I promise at the very least you know someone who has it) that flawlessly passed every test case except a few of mine, which I failed without thinking about it. The downloadable content was corruptible in a certain way and the game didn't handle it that well. A new build came in the following day, and my main project lead told me it had come back in cause of my failure, the only one on the game. I had to try every one of my test cases before a certain deadline, I think it was lunchtime, though everyone's attention was on only the one I failed the day prior. If the game didn't pass, it would be delayed several months. If it did, their time table would stand, and the game would ship as scheduled. Lucky for me, it passed my requirements, and I deftly avoided earning the ire of a famous development studio.
 
Another one of my test cases involved corrupting my save game file for a certain first-person shooter (you've heard of it). I was tasked with trying to get the game to load the corrupted file to see if the game handled the situation “properly”. What this particular shooter did was inform the player that their save file was damaged, and then open up a stage select for every level in the game, asking the player to designate where they were in the story. Technically, it didn't fail the test case, so technically, I couldn't fail it. We were the final checkpoint before mass production, so I have no reason to believe the game didn't ship like this.
 

In the sea of irritation and annoyance that is testing, one of the more frustrating things was all of the issues and resulting work-arounds we had to employ in the "functional" stage of quality assurance. Since Functional was a different department, it was a low priority for us to report bugs that didn't specifically fail one of our test cases. Another big budget game (you've heard of it too), which was as broken as possible in its first iteration, had a multiplayer that only worked on one level in one mode with three players, with the time limit set below a certain threshold and only with certain characters chosen. Technically, even though it took an excruciating amount of time to match up, all the multiplayer test cases had to be run within that window, including the ones that took longer than the time window available, or ones that kicked the player back to the main screen every other step.

The truth was, even management knew every tester there was actively looking for another job, and having another job interview was always a valid reason for requesting time off. There was a high turnover rate, and people cycled in and out constantly. Yes, it was fun. Yes, it was easy, and yes, I liked it there. I even got free pizza some nights, barbecues in the summer, and a couple other lunches while I was there. However, I had worked there for eight months before I actually got to log 40 hours during one single week. Some weeks I had as few as 12, and there was no way to tell a given day's work schedule until the nightly e-mail the day before. All said and done, after jumping on every overtime opportunity afforded me, I made $14,000.00 that year. It would have been less painful if I wasn't living in Seattle proper, but that's still barely above the poverty line. At Nintendo, part of orientation was establishing that it is extremely rare to advance from the testing position, and that the company does not look at the pool of testers when selecting permanent employees. While working there, I had a couple of friends in the call center as Customer Service Representatives, both of whom advanced and gained raises, things for which I was not eligible. Admittedly, the situation at Microsoft was a little better. The guy I trained, who I am still friends with, advanced just after I left and eventually became a lead. Admittedly, the department head at Microsoft worked his way up from Hardware, a tidy little example of the American Dream that hard work and perseverance can yield great reward. Mostly, people there want to work on games, and hope they can shoehorn themselves into a development studio's testing staff with their experience, and then work up from there.
 
Both companies made a point of telling me during the interview process, “We don't just play games here” and that's very true. There is some work involved, occasionally very laborious, tedious work, and I can think of more than a handful of games I never want to touch again. I'm glad I had the experience, and I had a lot of fun, but there are very serious downs to those ups. In the end, it's just another job. Being a tester is not something that should be an ultimate goal or a prize. You do it for a while, and then you move on to something better.

Page URL:
blog comments powered by Disqus