 |
| |
|
|
yin26
|
 |
This space reserved for Darkstar.
Apr 1999
time: 14:12
|
|
What follows is an article written by the guy who is in charge of Quality Assurance for Civ 3. Just thought it might be interesting to see what the community has to say about it as we ponder Civ3's eventual release...Is he right? Your thoughts, please, ESPECIALLY as we anticipate a Civ3 that will no doubt have SOME problems with it. Will the community actually be happy to have these problems...?
Written by: Jeff Morris, Programmer / QA manager - Firaxis Games
Published: August 2, 1999
Born in the Los Angeles suburb of Santa Monica, Jeff Morris lived in San Francisco before moving to Austin to work at Origin Systems in June 1995. While at Origin, he worked on such games as ATF, Nato Fighters, USNF '97 and ATF Gold. After leaving Origin in February 1998, Morris moved on to help form Firaxis Games, where he worked on Sid Meier's Gettysburg!. Having been an avid computer gamer for over 15 years, Jeff led the Quality Assurance testing on Alpha Centauri, which involved coordinating professional and public testers. He also programmed the install module, the autorun menu and other miscellaneous portions of the title. So is there such a thing as the bugless game? Morris explores that question for the Adrenaline Vault in his guest editorial.
------------------------
One of the most frequent complaints made against modern computer games is that they are buggy. I don't care what title you think of, there is most likely a group of consumers out there evangelizing to the developers about the need for just one more patch. Try it. Visit the powerful Usenet search engine deja.com, search for any PC game that sold over 10,000 units and add the keyword BUG. I guarantee you'll find at least one heartfelt testimonial about how the customer was cheated by the "unfinished" product and a dire warning to any who might consider purchasing the title.
It's impossible to ignore the fact that games with significant bugs have been released. I strongly recommend that if you feel something you've bought is "unplayable," for whatever reason, you should return it. If the store won't take it back, try the publisher (raise a stink since that almost always works), and if not them, the developer themselves. Massive returns will have a far greater impact on future products than holding onto the title and venting on the Internet. The second option may feel better, but the developer gets to keep your money and, thus, has no real motivation to alter their behavior.
An interesting extrapolation of the perception that "most games released today are buggy" is the inverse statement that "some games released today aren't buggy." Even more interestingly, it seems what the public wants, if not demands, is a 100 percent bug free title. At first, this sounds absolutely plausible, something all developers should strive to achieve and all consumers should expect to get. Unfortunately, reality doesn't quite live up to this ideal. Even worse, it seems all parties concerned -- the developers, the consumers and even the machines -- actively combat any effort to achieve this goal. In short, on the PC platform as it currently stands, there can be no such thing as a bugless game.
Let's approach what at first seems the easiest to believe -- that computer diversity will never allow a bugless game to exist. This statement leads to some interesting questions about what qualifies as a "bug." Now, don't worry. I'm not going in the direction you might think. For the purposes of this discussion, let's say a bug is anything a player considers a bug. For instance, a hardware problem we've encountered on Sid Meier's Alpha Centauri (SMAC) relates to the fact that some customers never selected their monitor type, instead leaving it set to Standard VGA. SMAC, however, requires either an 800x600x8-bit or 1024x768x8-bit screen resolution, so the ceiling of 640x480x8-bit set by this configuration will prevent it from running. Now, is this a bug? Well, maybe.
The programmers could have anticipated this particular problem and displayed a helpful error message suggesting a solution. A great idea, but it is almost impossible to predict every possible incorrect configuration a system can experience (if you could even consider this example incorrectly configured). Even worse, as new hardware becomes available or additional features are added to existing hardware via driver updates, configuration problems that didn't exist during development and testing can create new "bugs." And this problem is only getting more pronounced. The great game testing challenge over the next few years is how to even begin approaching this massive diversity of not only hardware, but also the drivers that make it function. The standard "hardware lab" model of compatibility can't simulate it well, so they are wisely focusing on "off-the-shelf" systems and configurations. This at least gives the developers the knowledge that on a stock Gateway or Compaq, the game will perform as expected. But the innate customization a PC offers ensures it sure won't remain stock for long.
My point is this: The developer cannot realistically tell a customer they haven't experienced a "bug." Other games worked on the consumer's system, and they paid money for the new one to work on their system, but it doesn't work. It is a rare person faced with this prospect who will consider the possibility that his computer may be to blame. Technical support avenues of various flavors can solve the problem, but the fact of the matter is that the customer had problems, and in their minds and the minds of whoever they discusses it with, it will be portrayed as a bug.
Another reason I believe a bugless game will continue to elude reality on the PC is the simple fact that creating and selling computer entertainment is a business, first and foremost. Game developers draw their living wages from this industry and in almost every instance, their checks are cut by a company. And it is well known that companies have a persistent habit of striving for profitability and survival.
Now, I'll admit this is unfortunate and even go as far as saying it was closer to ideal before gaming became a multi-billion dollar industry. I truly wish game development were an altruistic hobby of millionaires, one in which they created vivid interactive entertainment for fans out of sheer generosity. It's a sad reality, though, that genre clones are cranked out because investors like safe bets. Hype machines are sometimes a more reliable method of nailing a sales forecast than the voodoo involved in producing a first class title. And, tragically, some products are rushed out the door because if they don't hit that final milestone, the company will tank and the game will never be released.
This financial angle can sometimes hit the hardest in QA. Requests for additional time by a title's testing staff may be completely justified, but equally impossible to entertain. Extra time required for development typically comes from the "QA padding" at the end of a project, while final deadlines remain unaltered. Unbelievable amounts of money can be spent on securing retail displays, magazine and Internet advertising, and duplication facilities, all of which must be done months ahead of time. Miss your date and not only do you lose the dough, but you have to scramble to secure even the most rudimentary of advertising real estate (read: even more expensive). Salaries are an exceptional way of burning through even a generous advance and a completion bonus can sometimes make the difference between an employee paying their rent or walking out the door, rarely leaving everything in a state of completion. Slip gold by a few weeks, even when it's to fix a critical problem, and the product may not be finished for many months, if ever.
These pressures aren't right or wrong, they're simply reality. If it comes down to a contest between a quasi-repeatable fatal or slipping by four months, laying off a third of your staff and kissing your Christmas bonus goodbye because you missed your window, I know the choice many developers would make. It's an easy choice for the consumer, but a heart wrenching one for those who got into this business to make cool computer games. There's a saying that a title can only be late for awhile, but it can suck forever.
My final and most radical point is that the online gaming community, which most frequently identifies and evangelizes bugs, have strong interests in games not being 100 percent bug free. Let's say a title avoided major hardware conflicts and is produced by a company with the fiscal resources of Midas. The offering is highly anticipated and has a roll out of a quarter of a million units and 20 active fan sites eagerly digesting every bit of news. In this situation, the final product is actually an ending, and those who spent the last year or two following it are faced with saying goodbye to the friends they've made online and moving on to another title.
Why? The reason is that, for a majority of the hardcore, Internet savvy gamers out there, playing a title is secondary to the community around which it forms. This usually gestates well before an offering is available in any playable form, with the seed typically being a news release or trade show unveiling. Fans of previous genre efforts have lots to discuss and even more to hypothesize about. How will it be different from previous titles? What worked and didn't before and why? Here's this crazy idea I had for a game once, what do you think? These vibrant social activities absolutely thrive in a lack of concrete information. Even though the upcoming product is a link that brings these souls together, it is often the least interesting thing to talk about.
As the product nears completion, this community often has existed for a year or more. Mores, legends and leaders (and even development staff pinch hitting as celebrities) are all fulfilling rewarding roles. A genuine Internet society begins to reach a feverish pitch as more and more information about the game is exposed. Finally, the day comes when it hits the shelves. Everyone ejects from their forums/newsgroups/fan pages and plays it for a few hours -- and then guess where they are? Back on the 'net discussing every little nuance. But then, except in rare cases, the fun is over. Unless it's buggy.
Even the best title can't sustain a dialogue as fascinating as that of this society, it being just a computer game. By focusing on negative aspects, real or fabricated, the customer can initiate a dialogue with the developers. While this is a potent and useful tool for improvement, the more gratifying use is to generate news, which in turn fuels the society. In the SMAC forums, there is no easier way to generate a million post thread that for a Firaxis employee to post. This interaction boosts a flagging supply of conversation and delivers what the customer has come to value much more than the game. The fan base as an entity has become something with which they strongly identify and to which they belong. Very few titles can compete with that.
As a heavy consumer of computer entertainment, I expect products to work flawlessly. I want every game for which I cough up 50 bucks the day it's released to install itself, teach me how to play it and then let me win. I expect it to seamlessly deal with whatever witches brew of drivers, hardware and applets I happen to have. As a QA veteran, I know that, regardless of how many 24-hour days or hundreds of thousands of dollars are invested in a game, someone, somewhere, at some point in time will find a "bug." It's inevitable.
This doesn't change a thing. Developers who want to stay in business need to deliver a product that works as advertised and satisfies the customer to the point they tell all their friends to buy it. The computer gaming business in that sense is like any other professional trade. Rushing "buggy" products out the door seriously damages that company's ability to stay viable, and since that is the primary goal of most companies, software or otherwise, it isn't as done as often as it's cited. The goal always remains the same -- to release a bugless game. But even should that occur, it still won't be safe from being labeled buggy.
[This message has been edited by yin26 (edited March 05, 2001).]
|
|
| |
|
| |
|
| |
|
|
|
ShiningOne
|
|
New Zealand
Jan 2001
time: 05:12
|
|
I wonder if we can turn JKMs arguement around and look at it from the point of view of a QA team having strong interests in games not being 100% bug free.
A QA test is payed to find and identify, in detail, bugs in the game. Whether real, hard ones like "SMAC has caused a page fault in..." or unwanted 'features' like infinite range missiles and reactors that don't increase hps to 40.
Now, one would sometime suppose that, aside from being poor and expendable (the Morlocks of any gaming company) that just occasionally being a game tester would be a hell of a lot of fun, as you get to play something really good, for months at a time, all during working hours!! This, surely, is the joy of a game testers life and makes up for all those occasions where the thing runs badly or not run at all.
In this sense, the public, really, is just the ultimate level of play testers. A bug here or there might give a few lone individuals something to crow about, or make a mediocre player into a good one for a few weeks if they can exploit it, but these are one off flare ups that benefit a very few people and are negatives for the great majority, who just want to play and enjoy the game instead of take programmers to task over the flaws in their product.
Hence fan level QA has a strong vested interest in their being NO bugs at all in a game on release  .
|
|
| |
|
| |
|
| |
|
|
|
I'll grant that a 100% bug free game is practically impossible. BUT strategy, mod and development ideas go on for a long time if the game is of civ2 calliber. Check out the CTP2 forums. Most would rather be developing senarios and improving gameplay than trying to fix the Activision mistakes that are fixable. Anyway, I'm willing to accept some bugs in civ3 as long as the commitment to patches is close to that of Space Empires IV and far from Activision's level of commitment.
|
|
| |
|
| |
|
| |
|
| |
|
|
|
Darkstar
|
 |
Huntsville, AL, USA
May 1999
time: 23:12
|
|
That was JM's "Covering my ass" fluff speech, wasn't it? Yep.
Developer point of view: You can make a 100% bug-free game.
Yep. It just won't tend to happen. Why? JM even admitted THAT... cause the game is going to get bounced out to meet it's deadline.
We talked quite a bit with JM on the old Alpha.owo forums about all the bugs in SMAC (Over 200+ code based bugs, many inherited from it's base Civ2 code). He had a great interest in how other software companies and organizations, whether in game/entertainment or satellite controllers or ICU equipment monitioring, combatted REAL bugs. Eventually, he apologized as he felt there would be no way to implement those means while old coding hands (ie Brian) and game gods (ie Sid) worked there. Many of the code bugs in SMAC are due to poor code discipline (not reusing the same code segment everywhere--- instead, having multiple code segments that did the same thing, which results in the same bug having to be irradicated multiple times or different rules being followed by the different sections as they get individually modified (ie, computer can borehole beside each other, player can't in SMAC, etc).
Publisher point of view: There are no bugs. Those are just undocumented features or last minute changes to the product and therefore did not make it into the documention.
Nothing is a bug to a PUBLISHER. Although something might become an "issue" if it causes significant loss of revenue. At which point, they may pressure for the developers to patch the product, as the cost of patching will be more then made up for in the long term revenue stream that would result in fixing the "issue".
QA point of view: You CAN achieve 100% defect free product.
I've worked QA. That's QA's goal. Any good QA (meaning, any competent QA) can do so. The obstacles are time and money. Jeff gave a truthful answer in his fluff about QA. They are the first thing cut from the schedule when a product "overruns". Whether on money or time. The reason? QA isn't viewed as generating money. They are viewed as an evil fostered on the various industries keeping them from getting the product out the door and making money. With software development, you have to keep the code monkeys to get the product. But you can let the code monkeys "debug" and "test" their work, and cut down on QA cycle. (It's not true, it's just one of those Management Myths that is commonly followed.)
It is true that a small code team can actually debug their code. If they have the discipline. But code monkeys are generally working at getting the product done. They do their work, test it, fix it, repeat. However, in a project of decent size, it's easy for a coder to make a change or enhance a section and have what he uses it for still work, but impact other parts of the application that touch it or use it that the coder doesn't know about (or remember how it works). And that's what QA catches. Chop down QA's time to do that, and you get a lesser product for it.
Misconception "bugs": Misconception bugs are those things that the application/system does, but the user thinks it does it wrong. A good tech support can and will handle these. Games have a harder time TRAINING their users on what they are SUPPOSED to do, but that just means you need a good Tech Support to handle that problem.
100% Bug Free Games are a myth? Nope. It is just that the industry, from it's parts to it's whole, have no interest in providing that to their customers.
The publishers want to pay the least amount of money out, and have the development take as little time as possible, to get the game to market. If they publish 100 games, a few of those will be a good, solid hit, and cover the loses on all the rest while turning a profit for the publisher. (Same as in music, books, and movies.)
The developers will never make the game they WANT to. They have contracts to meet, publishers to please. At some time, that game is going to be popped out the door. Or the shop will damage it's credibility with the publishers. For those of you who aren't experienced coders and developers, there is always things the developers would like to improve. And there is always a piece or three left out. And there is always something they wanted to tear and and rebuild it properly. It's the nature of the beast, and of the fact that there is always more then one way to have done that, software wise.
The QA and testers will never be given the time to chase down all the paths and interactions, and check everything. Businesswise, the companies are always looking for where money and time into testing equals the most customers that will be content with the product going out the door. After that, it's a point of dimishing returns.
The users will CONTRIBUTE to amount of actual bugs in the product with general LUSER stupidity. It's quite truthful that many users will report particular behaviors as bugs, when in fact that's the design and is documented in the manual. (Example, SimTex recieved email bug reports siting that their various "engineer" units in the game of Master of Magic couldn't found a city. In Master of Magic, engineers build ROADS. That's all they do, and it's documenterd.)
Tech Support, when it exists, is often little more then a FAQ document. Tech Support isn't a strong "value added" area in most publishers opinions. It's not like the customers are going to stop buying the products they peddle. While some development companies are very prideful and try to help their customers, eventually even they have to stop the bleeding of their money, or at least cut back on it. Typically, if a game has been out the door for 3 years, you just won't find the support for it like a newly released one. Try looking up the support on some of the old "classic" games. You are lucky to find an ftp server with the last patch from the company (released 6 months to a year after the game) sitting on it's publisher's site.
There is one FINAL factor in all of this... New customers.
Computer usage is growing. With that growth comes new customers that have no experience with any of these development shops (which go out of business or split up and start anew quite often) or their publishers, so they don't realize that BorgCraft is just a clone of StarCraft, or that Civ2 is just Civ with updated graphics and a little more tech/wonders/units. And as long as there are more NEW customers then OLD ones, there isn't one reason for any publisher or any developer to worry what their customers THINK of them or their products. They should be instead dumping their money into marketing and hype, to alert the new customers that they should be buying this product when they see it.
Buggy games. We get them because the industry really doesn't care about your OPINION. The only REAL way to show your outrage about a buggy game is to return it for your money, and warn all your friends off of getting it. But unless ALL the games are unprofitable, the industry will not care and will not change. Not until then will they start examining why they aren't reaching new customers, and why they are not getting any repeat customers.
-Darkstarr
|
|
| |
|
| |
|
|
|
yin26
|
|
This space reserved for Darkstar.
Apr 1999
time: 14:12
|
|
Darkstarr:
So shall we say, then, that if Civ3 comes out with "enough" bugs to ruin the game, we shall return it rather than cry about it? I agree with you 100% Of course, the tricky question is: What could a patch fix vs. what it couldn't? I suspect this is where a lot of the discussion comes in. If Civ3 has issues that can be patched, why return the game?
But it's that 'waiting for the patch' game that begins to sour people.
For the record, I have never had anything but enjoyment from a Sid game. Civ3, I'm certain, will be an amazing game. But there WILL be bugs. Will those bugs, then, become the focus of this forum? Will they be enough to make us return the game?
How, in sum, should we handle this issue once we have Civ3 in our hands? Perhaps a separate forum full of polite requests for a patch? Close repetative threads yelling about bugs? Sorry if this seems like "trolling" Mr. Secret Message Person, but us moderators actually have to think about these issues.
|
|
| |
|
|
|
Jon Miller
|
|
I think all bugs and the like should be kept to a specific thread which is moderated(all other created threads dealing with these issues would be closed and what inside moved to the official thread).
Jon Miller
|
|
| |
|
|
|
ShiningOne
|
|
New Zealand
Jan 2001
time: 05:12
|
|
It's a two way thing.
The complexity of modern computer systems, especially those put together by uber geeks running strange software configurations on ancient hardware, make it economically impossible to test for all possible equipment side bugs.
The degree and depth of interactions between various things in a strategy game makes it extremely difficult to test in depth all possible combinations and options at all times during the game. Additionally, finding testers who are good enough to be able to innovate numerous different playing styles and test them all thoroughly, as a product is continuously changing, is very difficult to do at all, let alone economically.
Users will of course invent their own bugs and demands for features. I have personally heard the D-gun in Total Annihilation, the EMP in C&C2, and the 'lack of cheats in multiplayer' in Starcraft, all referred to as bugs. Moreover, this is a very difficult area for a company to police, since an unbalanced feature, even one integral to the whole game design, can make a game just as unplayable in the medium term as an amazing black screen of death show stopper.
Demanding a game be 100% bug free, in the modern internet ready climate, is silly. Demanding that a game arrive fully playable, fully installable, and reasonably fun, is not. Patches that fix bugs should be small and obscure, and are more a vessel for making improvements instead of making a game playable.
|
|
| |
|
| |
|
|
|
Jon Miller
|
|
I don't think any game we get will be bug free. But I also don't think bugs are the only life a forum has, if the game is deep enough people will talk strategy.
Jon Miller
|
|
| |
|
|
|
A person who posts private message can not be a moderator on these forums. I'm disappointed to say that as of this moment yin26 is no longer the moderator of the civ3 forums. Dan and I would like to thank him for his (organazing and moderating, not... teasing) services all this time.
------------------
Markos, Apolyton Civilization Site
|
|
| |
|
|
|
Imran Siddiqui
|
 |
The Potterverse
Jan 1970
time: 00:12
|
|
Well, Yin, as you know from SMAC, I don't really like bug talk (but then again we were trying to get you newbies to leave  ). Actually, wait. What I said in paranthesis is important. We believed that if the bug talk was gone, you'd all leave, and we'd have our OT community back, basically. So, the same concept.
I ain't saying bugs topics are good, I'm saying they are usually the lifeblood of a game forum for a LOOOONG while. I also remember the Baldur's Gate board, where it was either: bugs, OT crap, or how do I... (and then they leave). Bugs definetly keep a certain kind of forum community running, and I do think that Mr. Morris is right on target here. We don't live in an idealistic world. If we did, then I'd say, yeah, there can be talk of strategies and game mechanics (well, not so we can find out what is causing a bug). I don't believe that to exist.
If you make a bug forum, that forum will have more posts that the General in the long run, that I believe fully.
|
|
| |
|
|
|
and yin, if you wanted to discuss how the community will handle the bugs discussion, you would post only the specific quote of the article. posting the whole thing meant that you created a discussion on the entire issue of bugs and what developers say about it. i fail to see how an experience poster like you made such a mistake....
therefore, you're now free to speak to whatever tone you like(always according to the rules of course  )
|
|
| |
|
|
|
and Lung, there are 1500 more threads archived from the CTP-General forum. so it's not smaller than the multiplaying one...
|
|
| |
|
|
|
if you were frequently in the OT, you might remember when someone got a "vacation" partially cause he posted the private message of another
|
|
| |
All times are GMT. The time now is 05:12.
Apolyton Time is 00:12. |
top of page
|
| archivepost |
|
Forum Rules:
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts
|
HTML code is ON
vB code is ON
Smilies are ON
[IMG] code is ON
|
|
|
|
|
|
05-03-2001 19:16
for discussion, there are strategies and mod making to discuss instead, which are far more interesting and gratifying than bugs.
|