It's Not Just Abusive. It's Stupid.
By now, we've all read that cathartic LiveJournal entry by an angry EA widow who has had her husband, her family life, and her own career co-opted by the hellish product development environment that has become the norm at Electronic Arts. (If you haven't read it, follow that link above, read it, and come on back.) Most of us in the business know, right down deep in our ulcers and migraines, exactly what she's talking about. Too many of us have been caught in "normal" development cycles that require overtime as a matter of course; and have been at the mercy of abusive managers who ratcheted us up to several months of 13-hour-a-day/7-day work weeks. Perversely, these managers always claim that this is what's required to make the schedule -- and (the mendacity of this part is always breathtaking) to prevent our work hours from expanding even more in the future.
These stories are nothing new to me. I spent my 20s living them -- and my 30s figuring out how to avoid ever doing that again.
Let me begin by establishing my bona fides. I’ve been building software for more than 20 years. Fifteen of those years were in the games business; half of those years were spent at EA's Bay Area offices as an external developer and an employee. I've held just about every technical position from tool programmer to director of engineering. As a programmer I’ve worked by myself, and on teams of almost a hundred engineers. As a manager at a Fortune 100 company (Adobe) and elsewhere, I ran teams of up to 25 people, working on up to five projects at once. I’ve managed multimillion dollar art-intensive games, single developers, and core technology teams responsible to as many as eight clients (all with different requirements and all on different shipping schedules). Over the course of my career, I've been “in charge†(i.e. the senior engineering or project manager) on more than a half-dozen published titles, and held up the technical direction or project management end on over two dozen more.
In all that time, for all those titles, no project I was in charge of has ever missed its ship date or overshot its budget.
Yet I absolutely refuse to work the kind of death march hours ea_spouse describes. And I have never, ever asked or allowed my employees to do so.
Her story -- and others that have been shared in the industry-wide conversation that her post provoked -- make it clear that EA's management believes, as a matter of institutional principle, that only way to make money at games software is to create tight schedules, and the only way to make a tight schedule is to work your employees harder.
Decades of software engineering research and best practices -- and my own experience -- prove conclusively that this belief is complete bullshit.
Accepting for a moment that EA’s goal is to make software that sells well regardless of inherent value or quality, I'll also concede the assumption that tight scheduling is a fact of life. Nothing will ever change the fact that you have to make Christmas (or the start of a sports season) or die. I’ll even concede a claim that increasing team size is usually impossible for budgetary or other reasons. There's no point in doing a project if you're not going to make a profit on it.
Even with these stipulations, though, death marches are almost never necessary. In fact, if you're worried about short timelines and long profits, they are your mortal enemy. Let's put it more directly: with very few exceptions that do not include game development, death marches are ALWAYS an indication of gross failure on the part of management to do their jobs. They ALWAYS result in reduced productivity, and usually result in the release of significantly inferior product. In all cases, overworking employees for extended periods is ALWAYS counterproductive. It doesn't just damage employees -- it damages the entire company.
Know Your Workers
Programmers, artists, audio engineers, and others who make computer games are knowledge workers -- that is, people who work with their brains, not their butts.
The world is full of jobs that mostly require butts. (Woody Allen said that 80% of life is just showing up). But programming computer games is not one of them. Good programmers have to have good brains. When you hire one, that's what you're paying for.
The Crunch Mode philosophy that ea_spouse describes is a typical case of caring more about worker’s butts than their brains. The manager she describes is obsessed with how many hours the worker's butt is in the chair, but doesn't give a good goddamn about the quantity or quality of the work actually getting done during those hours. He's valuing butts, not brains. Bad manager, no tee-time!
Knowledge workers need their brains engaged and working at high efficiency to do their jobs well. Once his brain shuts down, a lead engineer is just a butt in a seat. And brains start to shut down pretty rapidly if you don’t take good care of them.
Long experience has taught me that the average programmer can work 40 hours per week pretty much indefinitely. At 50 hours, significant performance degradation begins after two to four weeks. At 60 hours, degradation begins at well under two weeks. I call this degradation “burnoutâ€, although most people don’t start calling it that until they discover the victims spending their entire days playing Quake III instead of programming.
Regardless of what you call it, the degradation is real. As the brain goes, the team gets less useful output, higher defect rates, more time wasted fixing those defects, and a rapid reduction in the wit and will available to come up with effective and creative solutions to the growing list of problems. Not long after that, you lose the butt as well, as stress-related medical and family problems begin to exact a much more tangible toll.
In other words, working people too many hours lowers their productivity. And if you keep it up for very long -- four to six weeks at the outside -- you can actually reach a point where your team is making zero progress. Their fatigued brains generate problems faster than they can fix them. Bad decisions get made. Dwindling resources are wasted on stupidly chosen priorities. Before long, it's literally true that the more hours they put in at work, the farther behind they get. (You laugh -- but I've actually had to turn around teams in zero-progress mode more than once.)
It seems obvious. But, evidently, there's something about being promoted to upper management that makes even well-meaning, caring people forget how easily this happens.
The Better Way
So if working more hours isn’t the solution, what is?
First, improve the work environment. Elementary management books like Fred Brooks’ The Mythical Man-Month, DeMarco & Lister’s Peopleware, and Edward Yourdon’s Death March all make it plain that working environment is a critical factor in programmer productivity. Simple things like enclosed offices, phones that turn off, quiet periods, adequate development equipment, and sufficient time to think and recharge are not luxuries: they're absolutely essential to getting the most productivity out of your people. If you don't provide a supportive work environment, you can expect huge amounts of time leakage.
Second, find the most productive people, and make sure nothing disturbs them unnecessarily while they work. Skip the “pep talks.†Excuse them from all unnecessary meetings. Keep upper management out of their offices entirely. Make sure they don't downgrade their own productivity by getting too little sleep, exercise, or food.
Remember that if their work is impinging on their home lives, they're getting stress from two sides, which will double the rate at which their output degrades. I have occasionally told eager beavers to limit their work time (no weekends, out of the office by 8pm) because I knew that I wouldn't make my ship dates if my team members couldn't maintain a functional and happy personal life.
Third, measure output -- any way you can. You can't control what you can't measure. At the same time, accept that whatever measurement systems you adopt, your programmers (if they're any good at all) will be smart enough to game them -- so understand the limits of your numbers, and don’t trust them unconditionally.
One of the more trustworthy gauges is reputation. Everybody on a development team knows who's a hero, and who's a slacker. Be smart about how you deploy the heroes. Get rid of the slackers (it's amazing how much you can add to a team's productivity by simply rooting out and pruning the deadwood that weighs them down). And give special attention to the middle-of-the-road programmers who quietly and reliably get the work done, because these are the people who are actually going to get you to the finish line.
(A small digression: Years of studies have found that there are large (up to an order of magnitude) differences in programmer productivity that are not directly related to years of experience. But measuring programmer output is such a difficult and controversial subject that most organizations don’t even attempt to deal with it -- though if you're managing programmers, you should at least have a basic comprehension of the issues and theories involved. At EA, they've apparently settled on a binary output metric: the product either shipped on time (1!) or it didn’t (0).)
It’s fashionable to consider one programmer just like another. Don’t believe it. Some people are better at creating new code; some are better at debugging. Some can’t code their way out of a paper bag, but are system testing gods. It sounds obvious, but evidently a lot of managers didn't get the memo: you'll get farther faster if you try to match the tasks to the available talent.
Fourth, plan your development as well as you can. Avoiding crunch mode isn't complicated or esoteric. All it takes is the willingness to seriously, realistically and competently plan out the development process at the outset, and the discipline to revisit those plans periodically. There are decades of best practices that make this pretty straightforward -- and nobody should be promoted to a software management job unless and until they have at least a passing familiarity with them. There are also terribly expensive analytical tools available to help, but if you can’t do the planning with a word processor and a spreadsheet, you don’t belong in project management.
Good planning doesn’t mean creating a billion-page design document that specifies every algorithm and variable name. You need room for the creative muse to work; and besides, most organizations won’t allow themselves to be hogtied this way. But it does mean taking a few weeks or months at the outset to do the 90% of the design that takes 50% of the time. At the very least, allow time to adequately scope systems and tasks over a certain size.
Fifth, prioritize. Competent managers know how to balance the so-called "development triangle" -- features, resources, and time. In any construction environment, you have features (also called "requirements" or "the things you are building"). Invariably, you'll start with a bigger list of features than you can possibly build within the limitations of your resources. Matching features to resources tells you how much time the project will take. If you don't like the answer, change the balance. For more features, allocate more resources or allow more time. If you don't have the time, allocate more resources, or reduce the feature set. And so on. This is the origin of “cheap, fast, good -- pick any twoâ€.
Many software development organizations can’t keep this basic idea in their collective heads. Or if they do, they consider “work more hours†to be adding resources. As you've seen, it’s not that simple.
Part of this initial planning is prioritizing what can be left out and what can’t. Do what must be in the product first, and be prepared to let the rest go. If there are more first-priority requirements than you can create with the allotted time and resources, it’s not your fault. It’s your management’s fault. That won’t stop them from blaming you, but at least you'll have the satisfaction of knowing exactly where they got it wrong.
Sixth, hold managers accountable for mismanagement. Any manager who leads his team into a death march has failed his employees, his superiors, and his company's investors. He has not done the job he was hired to do, which is to deploy the company's resources intelligently and profitably. We've all been to those triumphant post-ship meetings where these so-called "leaders" got big praise and a bigger bonus check as a reward for wreaking havoc on the health and personal lives of their employees. That's a perverse feedback loop, which perpetuates an abusive system. Any manager who put her team through more than a month of pre-ship crunch to get the product out the door should be getting a pink slip, not a promotion.
The Challenge
How confident am I in this way of working? Very. I've made this speech more than once at the Game Developers' Conference, written articles on it, and successfully put it into practice for several companies. But I'm ready to take it one step further. Here's my challenge, issued to upper management of any game company that would like to get off the death march treadmill, but worries about the risk to their profitability.
I’ll take on any existing, approved and staffed pre-Alpha project you have, and bring it in on time and on budget with the team you have already assigned to it. Furthermore, I’ll do it without putting any employees on more than two consecutive weeks of crunch mode within any six-week period, plus an additional four weeks at the very end. The vast majority of the project will be done in 40-50 hour work weeks, with no weekends. There are a few stipulations we'd have to work out regarding approvals, benchmarks, lines of authority, and mediation, but you wouldn't find them excessive or unusual.
I’d do this for a salary of $1 for the first year. If I succeed in meeting our prearranged benchmarks, my reward is a five-year contract to extend these methods to the rest of your studios. The contract will include a generous compensation package commensurate with the work I’ll be doing, and bonuses based on the increased efficiency and profit you’ll see when you’re not turning over half your talent every year. (At Adobe, management worried when the turnover rate hit 18%. EA’s 50% is evidence of a grossly dysfunctional culture.) If I can’t do what I say I can, you’re only out $1 for a year of adult supervision.
Bold? Yeah. That comes as no surprise to anybody who knows me. But those same people know that I've got the tools, the talent, and the experience to back it up. All you need is the the good sense to hire me.
It's About All of Us
The attention we’re all giving to ea_spouse's post shows that it's way past time for this industry to be having a serious discussion about how we're managed, and under what terms we consent to give our time and talent to the major houses.
Some of you are afraid of seeing your jobs off-shored to India. Frankly, many major software companies (including one of my former employers) has learned the hard way that this is not the solution it promises to be. And, for a lot of cultural reasons, game software is even less exportable than, say, graphics software. It's not a threat I'm going to lose sleep over.
And I'm intrigued by the suggestion that we do what Hollywood's trades people did in similar circumstances, and form a union.
We might pay more attention to the number of bright young newbies who graduate from the growing number of game development schools each year. EA is obviously depending on this river of eager young talent to keep its turnover machine churning. (They're hardly the first to adopt "eat the young" as a profitable business strategy: Bill Gates built Microsoft in much the same way.) Warning these kids, as we've begun to do, is a good first step. If the best and brightest in each year's new crop are avoiding EA -- and telling them why they're going elsewhere -- the company may eventually get the message.
Our business runs on Cool. The last thing EA wants to be is Uncool. Becoming a pariah among the hot young wannabees would be incredibly bad PR. But, even worse, it wouldn't take long for the stench to be picked up by the "opinion leaders" in the game-buying audience and disseminated throughout the market. Since those of us who build the games have more credibility to influence both the newbies and the end users than our corporate masters do, all we'd have to do is get serious about spreading the word.
And then there's the possibility of a shareholder lawsuit. When EA is spending many millions to produce a game that could be done for half the price (which may well have been the case with ROTK, based on what I've heard), flirting with methods that raise the risk that critical games won’t ship on time, and running its affairs in such a way that the industry's top talent won't even take their phone calls, the shareholders might be inclined to take an interest. I know there are plenty of experts (me included) who would testify that the described management practices are in gross violation of commonly accepted professional engineering standards. As a first step, there might be enough stockholding former employees out there that we could pool our shares and make our point to the board.
EA was not always this way (though I've been around long enough to see that the potential was always there, right from the beginning). After 20 years of patching up the wounds inflicted on myself and others by this escalating abuse, I'm glad to see that we're having an open conversation about this at last, and am looking forward to seeing where it goes.