Posts filed under "Lessons Learned"
August 26, 2013
Randy’s Got a Podcast: Social Media Clarity
I’ve teamed up with Bryce Glass and Marc Smith to create a podcast – here’s the link and the blurb:
Social Media Clarity - 15 minutes of concentrated analysis and advice about social media in platform and product design.
First episode contents:
News: Rumor – Facebook is about to limit 3rd party app access to user data!
Topic: What is a social network, why should a product designer care, and where do you get one?
Tip: NodeXL – Instant Social Network Analysis
August 23, 2013
Patents and Software and Trials, Oh My! An Inventor’s View
What does almost 20 years of software patents yield? You’d be surprised!
I gave an Ignite talk (5 minutes: 20 slides advancing every 15 seconds) entitled
“Patents and Software and Trials, Oh My! An Inventor’s View”
Here’s some improved links…
I’ve created ip-reform.org to support the “I Won’t Sign Bogus Patents” pledge.
Encourage your company to adopt Twitter’s Inventor’s Patent Agreement
Sequestration has delayed a bay area PTO office, support this bill
I gave the talk twice, and the second version is also available (shows me giving the talk and static versions of my slides…) – watch that here:
July 25, 2012
Forward looking statements
People say things to other people all the time that are misinterpreted or misunderstood; this is a normal part of life as a social animal. But this is especially true of things people say about the future, what the securities business calls “forward looking statements”. Statements about the future are marvelous sources of chaos and confusion because the future is intrinsically uncertain. The inevitable divergence between what someone said at one time and what actually happened at a later time invites all kinds of reinterpretation and second guessing and finger pointing, well beyond the usual muddle that is an ordinary part of human social interaction.
Because people in an organization are trying to coordinate purposeful and often complex tasks over time, forward looking statements make up a large fraction of intra-organizational communications, a larger fraction than I think is typical in purely social or familial interactions. Over the years I’ve learned that I often have to train people I’m working with on the distinction between three related but very different kinds of forward looking statements: plans, predictions, and promises. In my experience, somebody treating one of these as one of the others can be a significant generator of interpersonal discord and organizational dysfunction.
In particular we make a lot these kinds of statements to people to whom we are in some way accountable, such as managers and executives up the chain of command, but also, notably, investors. We also make these kinds of statements to peers and subordinates, but somehow I’ve found that the most chaotic and damaging effects of misunderstandings about what something really meant tend to happen when communicating upward in a power relationship. Consequently, reinforcing a clear understanding of these distinctions has become part of my standard routine for breaking in new bosses.
The distinctions are subtle, but important:
- Plans are about intention
- Predictions are about expectation
- Promises are about commitment
A plan is a prospective guide to action. A plan can be wrong (moreover, it can be known to be wrong) and yet be still useful. A plan is often an approximation or even wild a guess. However, if you are in a high state of ignorance and yet trying to take purposeful action, you have to start somewhere. As George Patton famously said, “A good plan, violently executed now, is better than a perfect plan next week.” Plans can readily change, because over time you learn things, particularly as a side effect of trying to execute the plan itself. In fact, in my line of work, if your plans aren’t changing relatively frequently you’re probably doing something wrong. Plans typically concern matters that are within your own sphere of control: “first I will do this, then I will do that”.
A prediction is declaration about what you think will happen. A prediction may very well encompass elements that are beyond your control. A prediction will often incorporate, if only implicitly, some model or theory or idea you have about how some part of the world works. When making a prediction, you are offering somebody else the benefit of your knowledge and analysis, so it can be beneficial to articulate your reasons for believing as you do. Like plans, predictions can change, but the reasons for change are different: a prediction can change if external facts change, or if you discover some shortcoming in your analysis. Thus it may also be important to be explicit and articulate when you change a prediction: explain why. Unlike a plan, a prediction that is just a wild guess is largely worthless, though a prediction that is the product of an inarticulable intuition may still be useful (but if so, in some sense it’s not really a wild guess — though it’s a valuable and rare skill to be able to reliably distinguish the times you are going with your gut from the times you are just stabbing in the dark).
A promise is a statement that you grant other people the right to treat as a fact that they can rely on, as they figure out their own actions and make plans, predictions, and promises of their own. A promise is a positive assertion that you will or will not do something specific. A promise is not something that generally changes; a promise is either kept or not kept. A change in circumstances may render a promise unkeepable or inappropriate, however. People put a lot of moral weight on promises, because accepting a promise requires trust. Because trust is involved, a broken promise can have emotional and organizational consequences that go beyond the direct practical effects of whatever contrary thing was or was not done. I could go on at length about the moral and emotional dimensions, but it would be a digression right now. The short version is: promises carry a lot of baggage.
On their face, these three kinds of things are all simply declarations about the future, and there’s nothing innate that necessarily marks which of these a given statement is: “I will mow the lawn tomorrow” could legitimately be taken as any one of the three. The differences have to do not with the form of the statement but with the intent. The reason the distinctions remain important, however, is because serious trouble can result when somebody says something intended as one of these categories, and somebody else interprets it as one of the others. The reasons for this sort of misinterpretation are varied and probably infinite: the person who said it was unclear what they meant, the person who heard it wasn’t paying attention, or misunderstood, or had different background assumptions, or was simply clueless. Sometimes the misinterpretation is deliberate and willful; this is especially destructive.
These categories are not pure. That is, a single statement is not necessarily 100% one of these things and 0% either of the others. A statement can be a mixture. However, having the parties at both ends of the communication be clear on what was intended is still essential.
There are many different ways trouble can result from interpreting a statement of one of these types as one of the others.
Treating a plan as if it were a prediction invites confusion and mayhem if the plan changes. The normal evolution of a project can be seen as evidence of problems where none actually exist: “You said you were going to do A and instead you did B. Why did you say you were going to do A? Do you really know what you are doing? Please explain.” A lot of time and energy can be dissipated accounting for changes to people for whom the changes weren’t actually important.
During the last year of the Habitat project, we reached the point were the product was fundamentally complete but it had a lot of bugs. The bug list became our main planning tool: each bug was assigned a priority and a rough time estimate, and the bug list was the thing that each developer looked to to decide what to do next. I call it a bug list, but not everything on it was, strictly speaking, a bug. Some things were tasks that we’d like to get done that needed to be balanced against the debugging work, and other things were just stuff that could be made better if we spent the time or resources. Since the world is constantly changing and we are constantly learning, a fairly common pattern was for a task to be identified and put on the list, and then gradually drift into some form of irrelevance as the shape of the system evolved or operational experience gave us feedback about what was really important. This kind of drift and accompanying deprioritization is a process that every developer should be familiar with.
Perversely, we found ourselves keeping multiple uncoordinated bug lists. As the project matured we acquired a product manager, who was a well intentioned but ultimately useless detail freak. In an attempt to track the status of the project, in hopes of answering management’s eternal question, “when is it going to be done?”, she’d convene status meetings wherein she’d try to use the bug list as checklist. Every couple of days we’d spend several hours going through these items, and all the dross that we’d been ignoring because it was irrelevant or pointless became a topic for discussion, and “never mind that” was never an acceptable way to dispose of these items. Her reasoning was that if something had been important enough to get put on the list, it shouldn’t be taken off without due consideration. Since she wasn’t the person doing the work and so didn’t understand a lot of the particulars, everything had to be argued and debated and explained, wasting many hours of time. Plus, she’d be adding up all the time estimates for these random and vague things and freaking out because the total was wildly unreasonable — never mind that the estimates were engineers’ guesses to begin with and many of these tasks would never be done anyway. And on top of all that, a lot of these status meetings were teleconferences with our partners at QuantumLink, where each of these irrelevant items got unfolded into even more useless discussion and became the basis for lots of interorganizational dispute. So we found ourselves developing the defensive habit of keeping private todo lists of tasks we’d identified that we didn’t want to have to spend hours debating, and everybody made up their own plan.
The consequence of all of this was that a whole lot of planning activity was taking place off the books, so when the work got done it meant that lots of resources were spent on things that never showed up in the official project plan and could not be accounted for. It also meant that each of us had much fuzzier than necessary picture of what the others were doing, and management had a worse picture than that.
Nearly every experienced developer I know has his or her own variation of this story. Many of us have several.
A plan that is treated as a promise is even worse than one treated as a prediction. A normal change of plan can become an invitation to recrimination or outright hostility or even punishment. Plans treated as promises are at the root of many of the most awful cases of organization dysfunction I’ve ever experienced.
One of the projects I worked on at Yahoo! (to protect the guilty I will refrain from naming names) actually kept two schedules: the official schedule, for showing to upper management (the promise), and the real schedule, for day to day use by the people doing the work (the plan). As the project evolved, these two diverged ever more sharply, until the picture that upper management was getting became a complete and utter fantasy. At one level, the problem was that the person running the project was a craven coward, afraid of telling the truth to his superiors because he knew they wouldn’t like it (the real schedule said that things were going to take a lot longer than the Potemkin schedule said — funny how it never seems to go the other way). But at another level, the deeper problem was that the higher echelon people persisted in treating any forward looking statement by their subordinates as a promise, which made planning impossible.
Treating a prediction as a promise holds someone responsible for the consequences of their analysis rather than for the quality of the analysis itself. Even if someone has some control over whether a prediction comes true or not, the mere act of making a prediction should not carry with it the obligation to intervene to ensure the outcome. Many predictions are conditional, statements of the form “if A happens then B will happen”; this does not mean that someone who says this is now committed to making A happen. Indeed, as with plans, changes in circumstances can render a prediction wrong or irrelevant. It may be more constructive to adapt to the changed reality than to try to bend reality just to preserve the prediction.
Treating a prediction as a promise often leads to people being held responsible for things they have no control over. Putting people in this sort of bind is a classic cause of various forms of mental illness. Aside from being basically useless and stupid, this is great a way to make people hate you, and you’d deserve it. Nevertheless, how many of us have experienced a boss refusing to hear that something can’t be done, even when it really couldn’t?
Treating a prediction as a promise abdicates responsibility. If you are obligated to produce some outcome and fail because some prediction you relied on turns out to have been wrong, it is still your fault. It was you who chose to rely on the prediction. Government and big business both do this all the time, trying to duck accountability for mismanagement or malfeasance by pointing at external estimates or projections gone wrong (indeed, at times it seems like the Congressional Budget Office was established principally to enable politicians to use this particular dodge).
The failure modes just discussed are the worst, because each, in one form or another, imputes causality that isn’t really there. The other possible category confusions can still be disruptive, however, by jumbling the mental models people use to make sense of the world.
If you treat someone’s promise or prediction as a plan, it means you are pretending they have a plan when they might not. You are confusing ends with means. Sometimes, of course, you don’t care what their plan is, and sometimes it’s none of your business anyway, but in such cases you should know that you are banking on the quality of their analysis or of their commitment, and not on a fantasy model of what they are doing.
If you treat someone’s promise as a prediction, you risk using the wrong grounds to assess the validity of their statement. You consider the trustworthiness of a prediction by looking at the predictor’s knowledge and analytic ability, whereas a promise is evaluated by looking at the promisor’s incentives and their ability to execute the relevant tasks. These two pathways to assessment are wildly different, and so if you use one when you should use the other you are in danger of getting the wrong answer.
There are already plenty enough ways for organizational relationships to go off the rails without adding the various nasty species of communications failures I’ve described here. However, I don’t think it’s sufficient to just exhort everyone to try to be clearer. Managers, in particular, need to be aware of these failure modes and press for clarity when somebody says something forward looking and the category it belongs in is uncertain. Because humans tend to like certitude, many managers have a bias towards interpreting the things people say as constraining the future more than they actually do. If they do this a lot, it teaches their subordinates to be stingy with their knowledge, timid in their public outlook, and even sometimes to lie defensively. All of these things are corrosive to success.
March 23, 2011
SM Pioneers: Farmer & Morningstar – How Gamers Made us More Social
Shel Israel has just posted @Global Neighbourhoods the latest in his series of posts from his upcoming book Pioneers of Social Media – which includes an interview with us about our contributions over the last 30+ years…
Many of us often overlook the role that games have played in creating social media. They provided much of the technology that we use today, not to mention a certain attitude. Of greatest importance, is that it was on games that people started socializing with each other in large numbers, online and in public. It was in games that people started to self-organize to get complex jobs accomplished.
We had people meeting and sharing and talking and performing tasks several years before we even had the Worldwide Web.
We’re honored to be amongst those highlighted. Shel says about 100 folks will be included. There won’t be enough pages, but we eagerly look forward to the result none-the-less.
January 15, 2011
Requiem for Blue Mars
Another virtual world startup (Blue Mars) is dying. At The Andromeda Media Group Blog Will Burns [Aeonix Aeon] writes:
Looking Back at the Future
The really interesting part about all of this is that in order to see the future of Avatar Reality, and subsequently Blue Mars (or any virtual environment today), we need not look into the future but instead look to the past…
[many interesting insights about 1990s era worlds]
In 1990, the solution was given by two people to all of this madness. Chip Morningstar and F. Randall Farmer, authors of Lessons Learned From Lucasfilm’s Habitat. Strangely enough I had asked Mr Farmer about Linden Lab and he informed me that he was actually called in as a consultant in the early days, and not surprisingly, ignored.
That’s a bit of a harsh summary, but more than five years ago I did write The Business of Social Avatar Virtual Worlds: Or, why I really like Second Life, even if their business is most likely doomed. Clearly I wasn’t 100% right about them – they found a business model that at least got them to cover their run-rate. But many of the things in there may to apply to Blue Mars…
November 16, 2010
Quora:What lessons of Social Web do you wish had been better integrated into Yahoo?
On Quora, an anonymous user asked me the following question:
In hindsight, what lessons have you learned from the Social Web that you wish you had been more successful at integrating into Yahoo before you were let go?
I considered this question at length when composing this reply – this is probably the most thought-provoking question I’ve been asked to publicly address in months.
If you read any of my blog posts (or my recent book), you already know that I’ve got a lot of opinions about how the Social Web works: I rant often about identity, reputation, karma, community management, social application design, and business models.
I did these same things during my time for and at Yahoo!
We invented/improved user-status sharing (what later became known as Facebook Newsfeeds) when we created Yahoo! 360° [Despite Facebook's recently granted patent, we have prior art in the form of an earlier patent application and the evidence of an earlier public implementation.]
But 360 was prematurely abandoned in favor of a doomed-from-the-start experiment called Yahoo!Mash. It failed out of the gate because the idea was driven not by research, but personality. But we had hope in the form of the Yahoo! Open Strategy, which promised a new profile full of social media features, deeply integrated with other social sites from the very beginning. After a year of development – Surprise! – Yahoo! flubbed that implementation as well. In four attempts (Profiles, 360, Mash, YOS) they’d only had one marginal success (360), which they sabotaged several times by telling users over and over that the service was being shut down and replaced with inferior functionality. Game over for profiles.
We created a reputation platform and deployed successful reputation models in various places on Yahoo! to decrease operational costs and to identify the best content for search results and to be featured on property home pages [See: The Building Web Reputation Systems Wiki and search for Yahoo to read more.]
The process of integrating with the reputation platform required product management support, but almost immediately after my departure the platform was shipped off to Bangalore to be sunsetted. Ironically, since then the folks at Yahoo! are thinking about building a new reputation platform – since reputation is obviously important, and everyone from the original team has either left, been laid off, or moved on to other teams. Again, this will be the fourth implementation of a reputation platform…
Are you sensing a pattern yet?
Then there’s identity. The tripartite identity model I’ve blogged about was developed while at Yahoo an attempt to explain why it is brain-dead to ask users to reveal their IM name, their email address, and half their login credentials to spammers in order to leave a review of a hotel.
Again we built a massively scalable identity service platform to allow users to be seen as their nickname, age, and location instead of their YID. And again, Yahoo! failed to deploy properly. Despite a cross-company VP-level mandate, each individual business unit silo dragged their heels in doing the (non-trivial, but important and relatively easy) work of integrating the platform. Those BUs knew the truth of Yahoo! – if you delay long enough, any platform change will lose its support when the driving folks leave or are reassigned. So – most properties on Yahoo! are still displaying YIDs and getting up to 90% fewer user contributions as a result.
That’s what I learned: Yahoo! can’t innovate in Social Media. It has a long history in this, from Yahoo! Groups, which during my tenure had three separate web 2.0 re-designs, with each tossed on the floor in favor of cheap and easy (and useless) integrations (like with Yahoo! Answers) to Flickr, Upcoming, and Delicious. I’m sad to say, Yahoo! seems incapable of reprogramming its DNA, despite regular infusions of new blood. Each attempt ends in either an immune-response (Flickr has its own offices, and a fairly well known disdain for Sunnyvale) or assimilation and decreasing relevance (HotJobs, Personals, Groups, etc.).
So, in the end, I find I can’t answer the question. I was one of many people who tried to drive home the lessons of the social web for the entire time I was there. YOS (of which I helped spec in fall 2007) was the last attempt to reshape the company to be social through and through. But, it was a lost cause – the very structure of the environment is personality driven. When those personalities leave, their projects immediately get transferred to Bangalore for end-of-life support, just as much of YOS has been…
I don’t know what Yahoo! is anymore, but I know it isn’t inventing the future of social anything.
[As I sat through this years F8 developers conference, and listen to Mark Z describe 95% of the YOS design, almost 3 years later, I knew I'd have to write this missive one day. So thanks for the prodding , Anonymous @ Quora]
Social Media Consultant, MSB Associates
Former Community Strategy Analyst for Yahoo!
[Please direct comments to Quora]
October 12, 2010
First! Randy to be the kickoff guest for new Community Chat podcast series.
Bill Johnston and Thomas Knolls are launching a new live podcast series: Community Chat on talkshoe.
I am so honored to be the lead-off guest on their inagural episode (Wednesday 10-13-10):
The kickoff episode of Community Chat! [We] will be discussing the premise of the Community Chat podcast with special guest Randy Farmer. Will also be getting a preview of Blog World Expo from Check Hemann.
I’ll be talking with them about online community issues developers and operators all share in common – well, as much as I can in 10 minutes. :-) Click on the widget above to go there – it will be recorded for those who missed it live…
July 7, 2010
RealID and WoW Forums: Classic Identity Design Mistake
Update #3, July 14th 4pm PST: GamePro interviewed Howard Rheingold and myself for a good analysis piece in which I add some new thoughts, including a likely-to-be-controversial comparison to a certain Arizona state law…
Update #2, July 9th 1pm PST: KillTenRats.com just posted an email interview on this topic that I did for them yesterday. There some potentially useful business analysis in there, and more specific suggestions, even if it now feels a bit like residual heat from a flamethrower fest…
Hey Blizzard! I’m a freelance consultant! Just sayin’ :-)
Update #1, July 9th 10am PST: Blizzard has had a change of heart and will not require RealID for forum postings. This is a big win both for the community, and I believe, for Blizzard! The post below remains only as a historical footnote and perhaps a cautionary tale…
Talk about a crapstorm…
Here’s my latest tweet:
@frandallfarmer Quit World of Warcraft. New policy of RealID for forums - stupid beyond belief. #wow #fail #realid #reputation #identity #quit #copa #coppa
That’s too terse, given the magnitude of the error that Blizzard is making, so here’s a longer post…
Identity as Defense?
Blizzard has announced that the upcoming Starcraft II forums will require posts to be attributed to the user’s read-life name, taken from their billing information. As if this wasn’t bad enough, they’ve also said that the World of Warcraft boards will start this requirement soon as well.
They also announced a posting rating system, which sounds like they haven’t read anything from Building Web Reputation Systems, or at least about the massive disasters from combining real names and social ratings at places like Consumating.com, but that’s a post for a different blog. :-)
The idea Blizzard has is a common initial misconception – that people will “play nice” if they have to show their real names to each other. I’m sure they are using Facebook as an example – I often do this in my consulting practice. There is no doubt that Facebook users are better behaved in general than their YouTube counterparts, but the error Blizzard made is to assume that their player relationships are like those of Facebook.
This is critical misconception, and the community is responding with the longest threads in WoW history, and blog posts everywhere.
There are a lot of valid (and invalid) complaints and fears about this change – I’m not going to list them all here. What I want to do is point out the fundamental flaws in this model, for WoW in particular.
My 35+ years in building online communities (with and without RealID-like systems) screams out that Blizzard is going to be very, very disappointed with the results of this change. Specifically:
1: Names != Quality
Though this is nominally meant to improve the quality of the community, by civilizing conversation through revealing true names, it won’t because the interesting conversation will simply stop or move elsewhere. Many women (including a Blizzard employee) have already clearly stated that they won’t post anymore. This kind of thing has happened many times before as communities move from Yahoo Groups to Ning or wherever. As John Gilmore said:
“The Net interprets censorship as damage and routes around it.”
2: Brain Drain or “NetNews died for our sins”
Some say that getting rid of (bad) people is what Blizzard wants, so point #1 is a plus. But hold on there! Just owning the problem of driving customers into silence or away doesn’t help either.
Consider the case of Usenet/Netnews, where all the great internet community was until 1994 – when the environment became inhospitable to types of discussions the natives wanted to have, and they left en masse to form private mailing lists, and eventually webblogs. The assertion that a community of those who will reveal their names is somehow better does NOT hold up to any reasonable scrutiny (see next point…)
A shocking number of people who leave will be amonst the best users Blizzard has – and that could kill the quality of content on the forums, just as happened with NetNews. Sure, less trollish posts, but less great posters too. I’m betting there are less trolls to remove than there are good users who’ll leave/not post.
3: Facebook Status != Message Board Participation
I approve my Facebook Friends. None of them are trolls/spammy – or if they are, I block their events and no harm done. All of them can see my real name, status postings, comments, and other personal information. If it turns out I’m sharing too much, I can turn down the disclosure. It’s all optional.
Message boards are public. Readable by God, Google and Everyone. This model requires me to disclose sensitive information to everyone. Completely different.
Here’s the deal. We’re talking gaming here. People will get pissed at each other for stolen kills, breaking alliances, and the price of components – and they want to – no, they need to – have a safe place to express this, to play.
This is my spare time. It’s no other player’s business where I work, where I live, who my family is. Just as it’s no business of my boss, who knows how to Google my name, what I dedicate my off-hours energy to. The Facebook-analogy of Real Identity = Quality Contributions falls apart when applied Gaming. Google + Friends + Foes + Bosses + My Real Name + The fact I have 6 80th Level Characters = Too Much Information.
Facebook does NOT leak this much information, and the US Senate is looking into their privacy practices.
This has also happened many times before. Every time someone new to the net starts a LiveJournal, they don’t know about friends locking until they get asked into the boss’s office to discuss something they read on the journal while ego-surfing. This is how many LiveJournals get owner-deleted!
It is completely unreasonable to expect that people will understand the risks of using their real names on a message board – and if they DO understand, I contend that most people won’t bother posting anything at all.
- The trolls now get more information to harass
- The best players will leave
- The casual players will panic when they realize that their private-time activity is now public.
This is lose-lose. The worst kind of change. The only upside I see is the ability to lay off board moderation staff as traffic (good and bad) plummets.
An Alternative Everyone Can Live With
There was/is an alternative – described in the Tripartite Identity Model post from two years ago: Implement Nicknames!
Sure, have a top-level social identity, but present it as user-controlled Nickname and allow users to share a variant of their real name – but don’t require it! Sure, if the Nickname is the same as their RealID, feel free to show an indicator, like Amazon.com does with their Real Nametm markers. Allow users to reveal what they wish – even provide incentives for them to do so, but don’t bind full disclosure on them. Even Facebook doesn’t do this!
It’s never too late.
P.S.: I can’t stop being amazed – Asking for help on a forum requires disclosing your real name to God, Google, and Everyone? Come on! You’ve got to be kidding!
February 24, 2010
Grizzled Advice from Business & Legal Primer for Game Development
[Two years ago, I wrote up a few lessons for inclusion in Business & Legal Primer for Game Development. I'd always meant to cross-post it here and was surprised to see I hadn't already when I went looking for it to share with the folks over at PlayNoEvil in reply to a recent post. - Randy]
Here are three top-line lessons for those considering designing their own MMORG or latest Facebook game for that matter…
1. Design Hubris Wastes Millions
Read all the papers/books/blogs written by your predecessors that you can – multi-user game designers are pretty chatty about their successes and failures. Pay close attention to their failures – try not to duplicate those. Believe it or not, several documented failures have been repeated over and over in multiple games, despite these freely available resources.
If you are going to ignore one of the lessons of those who went before, presumably because you think you know a better way, do it with your eyes wide open and be ready to change to plan B if your innovation doesn’t work out the way you expected. If you want to hash your idea out before committing it to code, consider consulting with the more experienced designers – they post on Terra Nova (http://blogs.terranova.com/) and talk to budding designers on the Mud-Dev (http://www.kanga.nu/) mailing list, amongst other places. Many of them respond pretty positively to direct contact via email – just be polite and ask your question clearly – after all, they are busy building their own worlds.
2. Beta Testers != Paying Customers
One recurring error in multi-user game testing is the problem of assuming that Beta users of a product will behave like real customers would. They don’t, for several reasons:
A. Beta testing is a status symbol amongst their peers
“I’m in the ZYXWorld Limited Beta!” is a bragging right. Since it has street-cred value, this leads the user to be on their best behavior. They will grief much less. They will share EULA breaking hacks with each other much less. They will harass much less. They won’t report duping bugs. The eBay aftermarket for goods won’t exist. In short, anything that would get them kicked out of the beta won’t happen anywhere near as often as when the product is released.
B. Beta testers aren’t paying.
Paying changes everything. During the Beta, the users work for you. When you release the game, you are working for them. Now some users will expect to be allowed to do all sorts of nasty things that they would never had done during the Beta. Those who were Beta users (and behaved then) will start to exploit bugs they found during the test period, but never reported. Bad beta users save up bugs, so they could use them after your product’s release to gain an edge over the new users, to dupe gold, or to just crash your server to show off to a friend.
So, you’re probably wondering; How do I get my Beta testers to show me what life on my service will really be like and to help me find the important bugs/exploits/crashes before I ship? Here are some strategies that worked for projects I worked on:
Crash Our World: Own up to the fact that Beta testers work for you and they do it for the status – incentivize the finding of crash/dup/exploit bugs that you want them to find. Give them a t-shirt for finding one. Put their portrait on the Beta Hall Of Fame page. Give them a rare in-world item that they can carry on into general release. Drop a monument in the world, listing the names of the testers that submitted the most heinous bugs. Turn it into a contest. Make it more valuable to report a bug than to keep it secret.
Pay-ta: Run a Paid Beta phase (after Crash Our World) to find out how users will interact with each other socially (or using your in-game social/communications features.) During this phase of testing you will get better results about which social features to prioritize/fix for release. Encourage and/or track the creation of fan communities, content databases, and add-ons – it will help you understand what to prepare for, as well as build word-of-mouth marketing. But, keep in mind that there is one thing you can never really test in advance: How your user community will socially scale. As the number of users grows, the type of user will diversify. For most games, the hard-core gamers come first and the casual players come later. Be sure to have a community manager whose job it is to track customer sentiment and understand the main player groups. How your community scales will challenge your development priorities and the choices you make will have you trading off new-customer acquisition vs. veteran player retention.
3. There Are No Game Secrets, Period
Thanks to the internet – in-game puzzles are solved for everyone at the speed of the fastest solver. Read about “The D’nalsi Island” adventure in Lucasfilm’s Habitat where the players consumed hundreds of development hours in only tens of minutes.
The Lesson? Don’t count on secrets to hold up for long. Instead, treat game walk-thru websites as a feature to be embraced instead of the bane of your existence. “But,” you’ll say, “I could create a version of my puzzle that is customized (randomized) for every user! That will slow them down!” Don’t bother; it will only upset your users.
The Tragedy of the Tapers
Consider the example of the per-player customized spell system in the original Asheron’s Call (by Turbine, Inc.): Each magic spell was designed to consume various types of several resources: scarabs, herbs, powders, potions, and colored tapers. The designers thought it would be great to have the users actually learn the spells by having to discover them through experimentation. The formula was different for every spell and the tapers were different for every user.
One can just hear the designer saying “That’ll fix those Internet spoilers! With this system, they each have to learn their own spells!” But, instead of feeling enjoyment, the players became frustrated with what seemed to be nothing other than a waste of their time and resources burning spell components as they were compelled to try the complete set of exponential combinations of tapers for no good reason.
What was interesting is that the users got frustrated enough to actually figure out the exact method of generating the random seed to determine the tapers for each user as follows:
Second Taper = (SEED * [ Talisman + (Herb + 3) + ((Powder + Potion) * 2) + (Scarab - 2) ] ) mod 12
[Modified from Jon Krueger’s web page on the subject.]
The players put this all into a client plug-in to remove the calculation overhead, and were now able to correctly formulate the spells the very first time they tried. Unfortunately, this meant that new users (who didn’t know about the plug-in) were likely to have a significantly poorer experience than veterans.
To Turbine’s credit, they revised the game in its second year to remove the need for most of the spell components and created rainbow tapers, which worked for all users in all spells, completely canceling the original per-player design.
Hundreds of thousands of dollars went into that spell system. The users made a large chunk of that effort obsolete very quickly, and Turbine then had to pay for more development and testing to undo their design.
Learn from Turbine’s mistake; Focus on making your game fun even if the player can look up all the answers in a database or a plug-in.
Don’t start a secrecy arms-war with your user. You’ll lose. Remember: There are more of them than you and collectively they have more time to work on your product than you do.
December 9, 2009
Creatures Of Habitat (@1up.com)
There is a loving historical tribute to the role that Lucasfilm’s Habitat played in the history of MMOs at 1up.com:
Creatures of Habitat
What modern day MMORPGs borrowed from Lucasfilm’s ahead-of-its time adventure — and what they still could learn from it.
By Scott Sharkey
After another year of massively multiplayer online game crib deaths, we can’t help but be reminded of the MMOG that started the whole thing back in 1985 — well over a decade before the genre even had a name. Lucasfilm Games’ Habitat remains an unaccountable anomaly in the history of videogames, a multiplayer online world from the days long before the advent of the World Wide Web. It’s the sort of historical oddity that stands out as dramatically as, say, the discovery of a fossilized dinosaur holding a machine gun: Incredible, but pretty damn cool.
Hell Is Other People
In addition to being perhaps the earliest example of a graphical MMO, Habitat was one of the first games to embrace the concept of emergent gameplay. Habitat’s designers threw a bunch of strange people into a huge space full of a whole lot of weird toys and items and just watched to see what would happen. It was a kitchen sink approach, in line with their philosophy that “[c]entral planning is impossible. Don’t even try.”
Of course, some of the things that happened were murder, theft, bug exploitation, and runaway currency inflation. The game’s designers advocated a hands-off approach to administrating the world, encouraging players to administer themselves, but they did intervene on occasion. The solutions to those problems (and the debate over whether they even were problems) were enlightening glimpses of the kinds of things that other designers would have to wrestle with decades down the road…
The two-page article is worth the read if you’d like a great short summary of what’s possible when no one tells you that chasing your dreams is a fools errand…
[Comments disabled on this post, please leave them at 1up.com.]