Posts filed under "Lessons Learned"
June 6, 2008
Randy Interviewed on Community Management
JAKE: Hi. Hello. Thanks for joining us on this OCRN interview.
RANDY: Hi.
JAKE: Tell us who you are and what you’re up to and what you’re working on these days.
RANDY: Hi, I’m Randy Farmer. I’ve been building online communities for over 30 years. I’ve built one of my first multiplayer games, all text-based, 300 BAUD [30 characters per second], one of the first chat systems and message boards. I’ve dedicated my entire career to helping humans interact with each other as the computers as the mediating technology. Most recently, I was working at Yahoo as their community strategic analyst and I’m now independent as a consultant helping many early and mid-stage startups with their online community strategy and I’m also writing a book on the topic.
JAKE: Excellent, excellent. I’m hearing more and more people are writing books these days about online community and there’s nothing wrong with that.
RANDY: And they’re getting much better too.
JAKE: Yeah, that’s true. That’s true. Well, today’s topic is focused on online moderation and sort of the role of good community management and online moderation as part of any good community strategy and part of the reason that I wanted to bring up this topic even though it seems sort of obvious and standard is because more and more businesses and brands are getting into doing online community stuff whatever that means—some sort of user involvement, user engagement, using the web and social tools to do stuff and do stuff together. But it’s really surprising to me how much this discussion of maybe outsourcing online community moderation has played a role and in the desire many brands have to try and filter content, right? So get rid of the spam, get rid of the hate speech, get rid of all those basic things you need to do, it seems very common that they tend to lose their footing on what good moderation is about which is about setting culture and I’d love to hear your thoughts on what the right way to do online community moderation is.
RANDY: Sure, I can talk a little bit about the kind of taxonomy I developed for Yahoo in this area and one of the confusions that a lot of product managers have and a lot of people building [communities] is they don’t realize that community moderation covers a broad range of tasks and you shouldn’t confuse them. I’d like to make this dichotomy – that your referring to – I’d like to call it the difference between Terms of Service enforcement and Quality of Service. I used to refer to a QoS. This is a QoS issue, not a ToS issue. In a large company like Yahoo, ToS issues were actually handled by a different group of people. Those ended up being the Customer Care agents, the people who have the power to suspend your account, and there are also the ones that if you violated certain Terms of Service agreements, would forward your information to the law enforcement in the appropriate country. I’d like to think of community moderation primarily as Quality of Service management so you need to be able to tell the difference between an illegal or violating post, let’s say, when you consider text posts, for example, where just one that violates the community guidelines, or just needs to be highlighted as a good example. When we bought Flickr, that was a good injection of people and moderation staff who were focused on Quality of Service, finding good ways to reinforce the behaviors and specifically model the behaviors that you want users to take because they will follow the example. I’m a big believer in online communities with a concept of broken windows which is if you have a community which – let's use the example of text posts on message boards or on a group or blog and there’s just garbage lying around because people haven’t been cleaning up the Quality of Service or kind of online gardening, then people feel free to throw garbage around. Windows are broken and there’s graffiti, people think it’s all right to break windows and leave graffiti. Community managers are in the business of helping to keep the windows clean and nice and graffiti off the walls. Some of that ends up being really problematic and some is just appropriate window dressing and Flickr has been a really good example of that.
JAKE: That’s great. That reminds me of the story from the Tipping Point, of course, where in the turnaround of the New York City subway system. One of the things that they focused on to try and solve that was repainting the subway cars. Every time they get sprayed with graffiti, immediately they’d wait for the people to finish spray painting them and then they’d walk up with these big buckets of gray paint and as these guys were walking away, they’d start to clean it up and before you know it, it just wasn’t worth the time [for the vandals] anymore.
RANDY: So we did that with Yahoo Answers using mechanism. For that Terms of Service violating problem: one of the things is to not use [paid] humans, or [instead] to distribute it. Make it a distributed human’s problem. With Yahoo Answers, I worked with their team on the reputation model that we applied to get rid of trolls and spammers on Yahoo Answers.
The way Yahoo Answers works is a front … kind of like when you think of a forest fire. It spreads outward consuming everything leaving ash behind in the middle. Supposedly, you want the questions and answers that are in the activity of the past to be useful. That’s a Quality of Service issue but as the fire spreads, there’s more and more questions than answers and on the outside of experience on the home page where people interacting with it, that’s where all the hot activity is. That’s where all the trolls go. They don’t care about being left there forever. They just want to irritate people that are trying to tweak and so what they’ll do is they’ll post really horrendous questions so leave their spam there, just really garbagey questions to the home page. What would happen with those that have fallen with the standard customer care model, they get a report and they go into a queue and there’s a human being somewhere on the planet takes it out of queue and looks at it, decides it’s a violation, removes the content. Typically, this would take over 12 hours. After twelve hours, it is back in the ash somewhere. The flame has long since moved on. The graffiti did all the damage it was going to do. It was too late. It’s closing the barn door after the horses have escaped. We worked to build a reputation model that let the users who were paying attention to this and reporting it, suspend it immediately and when we compare the reporting percentage accuracy of the people who reported the error if ever there was a violation against the reputation of the host since trolls typically have no reputation. This allowed them to suspend the content immediately and then we implemented an appeals process so that we would contact the person and say, “Hey, your post has been taken down. If you want to appeal, go here.” Appeals then went to customer care where a human being would look at them. They said the net effect of reducing the 12-hour, 95% of the stuff that was suspended. It used to take 12 hours to suspend 95%. It got down to less than 30 seconds. So what’s happening is, you’ll be put out as soon as… the same thing about your truck, your story about paint. As soon as it was visible, it was taken down. An interesting thing happened, which is the trolls left. After a while, the reporting count went significantly down. It was because they stopped trying, the graffiti guys paint somewhere else. The place that’s always being abated is no longer the interesting place, it's [the story of] broken windows.
JAKE: Wow, that’s very cool. Well, so when a brand manager or a client comes to you and say, we just don’t have the time or the resources but we can’t hire people. We don’t have the head count to hire people to do what you’re talking about but we’ve got a check we can write to some firm. Is there a good way to outsource this kind of stuff or is this a… How do you build a great culture when you can’t necessarily involve the people who are putting the tool in place?
RANDY: So I actually recommend the opposite. I have no problem if someone can put together this split I made between community management and customer care meaning handling the dregs. If they can specify it cleanly in such a way that they can farm out the latter, that’s fine. I’m not sure they can but if they can, great. Yahoo runs a whole department which is separate from each one of the properties that handles that. The community management, you can’t outsource. In fact, most of the clients I’m working with, I recommend they increase the dialog with their customers directly. If they don’t have a community manager, that they get one and if they have a community manager and they’re just doing customer care, I suggest they promote them and have them run the company blog. This is not a big insight for anyone [OCRN memeber] that’s watching this interview. Corporate dialog is critical for at least all the clients I’m working with. If you’re talking about having an online community, you have to have a dialog between the company and the community. Otherwise…
JAKE: (overlay) Perhaps more importantly, how do you get that conversation to happen between colleagues? A lot of times, one brand versus another may not really communicate but there’s great lessons to be learned in what somebody’s learning about how they’re developing reputation system. How does that get across you and smaller companies?
RANDY: Well, the good news is, smaller companies are small. Hopefully, so the way I always used to put it – even before I worked for Yahoo – is when working with online games, there will be a very similar split as there is now on social sites. You would have that product people were designing a product, operational staff were kind of keeping machines running and making sure controlling – throttling logins, that's where you handle a lot of the trolls then we’d have the engineering staff. Typically, those would get separated and I always saw it as the kind of community stewardship role which usually falls in a senior community manager to actually be connected to all three groups to attend critical meetings and do communication. They are an advocate for the user to the company. And who needs to know about what’s going on with users? Well, pretty much everybody at the senior level of the company. I know it’s been a long term discussion on the Online Community Research Network. I’ve never had a problem with it because wherever I work, there’s someone like me. At Yahoo, I played that role. I was a free-floating strategist. I was in the platform group but I was both defining… I was meeting with the direct customers (the individual properties) and writing specifications for platforms which I would then meet with the engineers and making sure it was being designed properly, implemented properly and then when they were deployed, we were tracking all the metrics all the time to see if they were actually working. So in a situation, like I said, whenever you have communities as an important part of your revenue stream, direct or indirect, you need a community manager because you need that dialog and you need to facilitate all those interactions that are generating the virtual circle for you to be reflected up into the product groups and engineering.
JAKE: Got you. I couldn’t agree more. Okay so we’re about out of time but I’m curious, what is the one thing that you feel just doesn’t ever get paid attention to enough in these kind of conversations about designing a good online Quality of Service or Terms of Service moderation system? You find yourself saying over and over again but it’s new for everybody you talk to but not necessarily new for the conversations you’re having with them.
RANDY: Believe it or not, it’s the dialog between the company and the users that almost always gets forgotten. You’re right. The community managers are often thought of as customer care agents and they’re just taking care of the worst but they don’t focus on quality of service and when they’re gonna start talking about quality of service, what’s the kind of stuff we want to happen on the site, you need the word we to mean something, we as the company. What is the company looking for? I like to describe three participants in the win-win-win cycle. If you want a successful social site, you have those that are producing the content, so those are the bloggers or the videographers or the life streamers, or the people who are reviewing restaurants or whatever it is that they’re doing and interacting with each other. We have these consumers, those people who just got there through a search link or reference or they clicked on a video in a blog comment that was now hosted in YouTube and they all need to find benefit. They need to find the strong benefit, but the company also needs to find benefit. Everyone seems to think, well, as long as I make the consumers and the producers happy, everything’s fine. We crashed the dot-com era on that. We’re about to crash another one on it. Users don’t have a problem with participating. If you look at all of the crowd-sourcing stuff, if you set it up right, people are not gonna have a problem with the company earning enough money to keep doing the cool thing that they want to. And the company needs to spend some of that money on keeping those three groups happy, having someone that communicates between them.
JAKE: Excellent. Well, thanks for your time. Where can we find more about you or more about the pending book later on?
RANDY: Sure. I blog at a blog called Habitat Chronicles. If you search for that on Google or Yahoo search you’ll find it. Yeah, that’s probably the best place to track me. There's going to be several news items coming out soon.
JAKE: Habitat Chronicles. All right.
RANDY: Habitat Chronicles.
JAKE: All right. Well, thanks for your time.
June 5, 2007
Gnome Rain: BlockChat™ moves to Worlds of Warcraft
As I wrote in a recent post:
By hook, or by crook, customers will always find a way to connect with each other.
As with all worlds, this is now demonstratively true in Worlds of Warcraft, where a new text-filter on broadcast chat messages was recently installed to supposedly prevent gold farmers from spamming users about their websites.
It worked for a few days, until yesterday, The Day it Rained Gnomes.
Watch the video and duck…
I often have a conversation with online community/product managers telling them that text-filters are a symptom of deeper problems in your product and are not a solution. They usually don’t belive me – but now I have the video.
(For the comments section: How many design flaws/weaknesses are involved in the chain of events that lead to the world raining gnomes? Here’s a hint about where to start counting…)
March 1, 2007
The Untold History of Toontown’s SpeedChat (or BlockChattm from Disney finally arrives)
In 1992, I co-founded a company with Chip Morningstar and Douglas Crockford named Electric Communities. We initially did a lot of consulting for various media companies that were looking to leverage the emerging online gaming industry. One of those companies was Disney.
Disney had formed a group to look into taking the brand online, including a full-fledged multiplayer experience as early as 1996, when the were considering a product called HercWorld, which was to leverage the upcoming movie franchise Hercules. Having built Lucasfilm’s Habitat and WorldsAway, we were clearly amongst a handful of teams that had successfully constructed social virtual worlds that’d made any real money, and Crock had media connections from his days a Paramount, so they brought us in to discuss what it would take to build a kid-safe virtual world experience.
They had hired their own expert to lead the project, a former product manager for Knowledge Adventure – a kid’s software company that’d done some 3D work as well as their own online project KA-Worlds, which was meant to link sick children in hospitals together using computers and avatars.
Disney makes no bones about how tightly they want to control and protect their brand, and rightly so. Disney means "Safe For Kids". There could be no swearing, no sex, no innuendo, and nothing that would allow one child (or adult pretending to be a child) to upset another.
I found myself unable to reconcile the idea of a virtual world, where kids would run around, play with objects, and chat with each other without someone saying or doing something that might upset another. Even in 1996, we knew that text-filters are no good at solving this kind of problem, so I asked for a clarification: "I’m confused. What standard should we use to decide if a message would be a problem for Disney?"
The response was one I will never forget: "Disney’s standard is quite clear:
No kid will be harassed, even if they don’t know they are being harassed."
"So much for no-harm, no-foul," Chip grumbled, quietly. This requirement lead me to some deep thinking over the coming weeks and months about a moderation design I called "The Disney Panopticon", but that’s a post for another day…
"OK. That means Chat Is Out of HercWorld, there is absolutely no way to meet your standard without exorbitantly high moderation costs," we replied.
One of their guys piped up: "Couldn’t we do some kind of sentence constructor, with a limited vocabulary of safe words?"
Before we could give it any serious thought, their own project manager interrupted, "That won’t work. We tried it for KA-Worlds."
"We spent several weeks building a UI that used pop-downs to construct sentences, and only had completely harmless words – the standard parts of grammar and safe nouns like cars, animals, and objects in the world."
"We thought it was the perfect solution, until we set our first 14-year old boy down in front of it. Within minutes he’d created the following sentence:
I want to stick my long-necked Giraffe up your fluffy white bunny.
KA-Worlds abandoned that approach. Electric Communities is right, chat is out."
That was pretty much settled, but it felt like we had collectively gutted the project. After all, if the kids can’t chat, how could they coordinate? It’d end up being more like a world where you could see other players playing but you couldn’t really work with them much. [Side note: Sadly, a lot of MMORPG play is like this anyway, see Playing Alone Together.]
As I starting daydreaming about how to get chat back into this project, we moved on to what activities the kids might do in the now-chat-free HercWorld. It was standard fare: Collect stuff, ride stuff, shoot at stuff, build stuff… Oops, what was that last thing again?
"…kids can push around Roman columns and blocks to solve puzzles, make custom shapes, and buildings.", one of the designers said.
I couldn’t resist, "Umm. Doesn’t that violate the Disney standard? In this chat-free world, people will push the stones around until they spell Hi! or F-U-C-K or their phone number or whatever. You’ve just invented Block-Chattm. If you can put down objects, you’ve got chat. We learned this in Habitat and WorldsAway, where people would turn 100 Afro-Heads into a waterbed." We all laughed, but it was that kind of awkward laugh that you know means that we’re all probably just wasting our time.
HercWorld never happened.
Once again, into the breech
Electric Communities moved on, renamed itself Communities.com (which has nothing in common with the current company/site using that name and url.) and did some wonderful design work on a giant multimedia 3D kid’s world for Cartoon Network, which ended up being much too ambitious to fund, but I mention it because the project was headed by Brian Bowman. Brian eventually left Atlanta for Disney, where he was in charge of the online experience for Zoog Disney, a pre-teen programming block. Brian remembered his work with us and asked us to help build a world for the Zoog audience. Nothing so extravagant this time, just something simple, like The Palace (which, by then had been acquired by Communities.com.), a no-download, in-browser, 2D graphical chat with some programmed object capabilities.
"The Disney Standard" (now a legend amongst our employees) still held. No harassment, detectable or not, and no heavy moderation overhead.
Brian had an idea though: Fully pre-constructed sentences – dozens of them, easy to access. Specialize them for the activities available in the world. Vaz Douglas, our project manager working with Zoog, liked to call this feature "Chatless Chat." So, we built and launched it for them. Disney was still very tentative about the genre, so the only ran it for about six months; I doubt it was ever very popular.
Third time’s a charm
But the concept resurfaced at Disney a few years later [2002] in the form of SpeedChat in ToonTown. It was refined – you select a subject and then from a submenu of sentences, each automatically customized to the correct context. Selecting "I need to find …", would magically insert the names of the items you have quests for. For all walk-up users, all interactions would be via SpeedChat.
They added a method to allow direct chat between users that involves the exchange of secret codes that are generated for each user (with parental permission). The idea is that kids would print them out and give them to each other on the playground. This was a great way for Disney to end-run the standard – since Speed Chat was an effective method of preventing the exchange of these codes, and theoretically the codes had to be given "in-person", making the recipient not-a-stranger. Sure, some folks post them
on message boards, but presumably those are folks who 1) are adults, or 2) know each other, right? In any case, as long as no one could pass secret codes within Toontown itself, Disney feels safe.
The Ghost of BlockChattm past
Soon after ToonTown opened its doors, they added Toon Estates – a feature that gives you a house with furniture, initially just a bed, gumball machine, chair, and armoire. Then they added the ability to buy more furniture of all shapes and sizes from catalogs, and then you could invite people to visit your house to see how you have arranged all your cool stuff.
Sure enough, chatters figured out a few simple protocols to pass their secret code, several variants are of this general form:
User A:"Please be my friend."
User A:"Come to my house?"
User B:"Okay."
A:[Move the picture frames on your wall, or move your furniture on the floor to make the number 4.]
A:"Okay"
B:[Writes down 4 on a piece of paper and says] "Okay."
A:[Move objects to make the next letter/number in the code] "Okay"
B:[Writes…] "Okay"
A:[Remove objects to represent a "space" in the code] "Okay"
[Repeat steps as needed, until…]
A:"Okay"
B:[Enters secret code into Toontown software.]
B:"There, that worked. Hi! I’m Jim 15/M/CA, what’s your A/S/L?"
It seems that many of The Lessons of Lucasfilm’s Habitat still ring true.
I’ll consider this as The SpeedChat Corollary:
By hook, or by crook, customers will always find a way to connect with each other.
P.S: Brian tells me that Cartoon Network is actually resuming the project, more than ten years later; "… now that is being ahead of your time."
[Thanks to the legendary Robin Hood of Neopets for telling me about this Secret Code exchange prototcol.]
[Yes, the BlockChattm brand is a joke.]
December 22, 2006
Smart people can rationalize anything
One of the things we were able to do at Electric Communities was to attract one of the highest density collections of scary-smart people I’ve ever seen gathered in one place before. There are a lot of nice things about working with smart people. For one thing, they’re not stupid. Working with stupid people just sucks. Smart people are good if you need to do a lot of really hard things, and we did a lot of really hard things. But it’s not all upside. For one thing, smart people tend to systematically overestimate the value of being smart. In fact, it is really valuable, but they still tend to weight it too heavily compared to other virtues you might also value, such as consistency, focus, attentiveness to the emotional needs of your customers, and so on. One of the problems with really smart people is that they can talk themselves into anything. And often they can talk you into it with them. And if you’re smart yourself, you can talk them into stuff. The tendency to drift and lack of focus can be really extreme unless you have a few slower people in the group to act as a kind of intellectual ballast.
Why do less when you can do more?
Smart people can invent solutions to problems you don’t actually have yet. The problem is, it’s easy to think of problems you don’t have yet. Stopping to solve them all now is a recipe for paralysis. Furthermore, while it’s easy to think of all kinds of potential future problems, it’s much harder to forsee which of those you will actually have, much less all the ones that you are going to have that you didn’t anticipate. People who are less smart manage to avoid pouring resources into unnecessarily solving future problems because they aren’t able to figure out how to solve those problems anyway. So they just ignore them and hope they don’t actually come up, which in a lot of cases turns out to be the way to have bet.
Programming sage Donald Knuth taught us that “premature optimization is the root of all evil.” It turns out that this doesn’t just apply to coding.
You can’t sell someone the solution before they’ve bought the problem
Smart people can invent solutions to problems folks actually do have but don’t know it yet. These solutions are usually doomed. This ties in with the whole You Can’t Tell People Anything principle. It is nearly impossible to solve a problem for someone if they don’t believe they have the problem, even if they really, really do.
For example, one of the deep flaws in many distributed object schemes, such as the CORBA standard, is that they make no effective provision for distributed garbage collection. This is a major pain, because if storage management is annoying to get right in a non-distributed system, it can be brutally so in a distributed system. Java’s Remote Method Invocation standard is somewhat better in that it does do DGC, but it still can’t cope with unreferenced distributed cycles. One of our wiz kids, Arturo Bejar, devised for us a truly elegant DGC algorithm, which is not only efficient but gracefully handles distributed cycles. (To my eternal shame we patented it.) Since to work well in Java it really wanted to be in bed with the Java Virtual Machine, we tried to sell it to JavaSoft, who were literally next door to us in Cupertino (actually, we tried to give it to JavaSoft), but they weren’t interested. They hadn’t bought the problem yet. So a small piece of great technology that could make the world a slightly better place sits on the shelf.
Generalitas gratia generalitatis
(For those of you who, unlike me, lack a co-worker who spent 7 years studying Latin, whom you can bug for stuff like this, that’s “Generality for Generality’s Sake”.)
Smart people love to think about the general case scenario.
For example, at Electric Communities we ended up making a big investment in developing an orthogonal persistence mechanism for our object infrastructure. For those of you who are unfamiliar with it, orthogonal persistence is one of the ultimate examples of highly generalized technical coolness. Basically, the idea is that you abstract away the file system (or any other persistent storage, like a database) by keeping everything in memory and then playing tricks with the virtual memory system to make processes immortal. The example we were inspired by was KeyKOS, a highly reliable OS for IBM mainframes that was developed by some friends of ours in the 1980s, in which you could literally pull the power plug from the wall, and, after plugging it back in, be rebooted and running again — including transparently resuming all the running processes that were killed when you cut the power — in about 8 seconds (this was actually one of their trade show demos). You gotta admit, that’s pretty cool. Some commercial installations of KeyKOS have had processes with running times measured in years, surviving not only power failures and hardware malfunctions, but in some cases actual replacement of the underlying hardware with newer generations of equipment.
Orthogonal persistence was attractive to us not just because of the reliability enhancements that it promised, but because it would free programmers from having to worry about how to make their objects persistent, since it abstracted away all the serialization and deserialization of object state and associated design questions. Anything that made programming objects simpler seemed like a big win. And so we built such a system for our object framework, and it worked. It wasn’t quite as awesome as KeyKOS, but it was still pretty awesome.
One of my favorite catch phrases is, “The difference between theory and practice is that, in theory, there is no difference, but, in practice, there is.” Orthogonal persistence was a great idea — in theory. Of course it cost months and months of development time, and it introduced a number of subtle new problems, any one of which would have a made a good PhD dissertation topic. If you are trying to produce a commercial product in a timely and cost efficient way, it is not good to have somebody’s PhD research on your critical path. For example, it turns out that for most kinds of objects, the amount of state you actually need to save persistently is a small fraction of the full run-time state of the object. But in an orthogonal scheme you save it all, indiscriminately. The problem is not so much that it’s bulky, though it is, but that the volume of semantic meaning that you are now committed to maintain in near-perpetuity is vastly increased. That turns out to be very expensive. The problem of schema migration as new versions of objects were developed (due to bug fixes or feature enhancements) proved effectively intractable. Ultimately we fell back on a scheme that compelled programmers to be much more deliberate about what state was checkpointed, and when. That proved more practical, but in the meanwhile we lost literally years of development resources to the blind alley. Less sophisticated developers would not have gone down this blind alley because they wouldn’t have a clue that such a thing was even possible, let alone be able to figure out how to do it. They would have been saved by their own simplicity.
Keep in mind that all of the foregoing is not an argument against being smart. Rather, it’s a recognition that human rationality is bounded, and even really, really smart people must necessarily fall far short of mastering the complexities of a lot of the things we do, as engineers, as business people, and as ordinary human beings. What works the kinks out of a thing is often just the passage of time, as the shortcomings and subtleties gradually emerge from practice. Because stupid people work more slowly, they get the benefit of time for free, whereas smart people have to work at it.
September 20, 2006
Social Computing: From Message Boards to Blogs & Beyond
Back on Valentine’s day I was honored to be on a panel on the history of social computing that I was able to help structure. My co-panelists were Usenet guru Erik Fair, LinkedIn founder Reid Hoffman, and Six Apart co-founder Mena Trott, together with our moderator Wall Street Journal columnist Kara Swisher.
Since I don’t come off as clumsy as usual on camera, I’ve embedded the video. It is and hour and a half long, so you might want to download it from the Computer History Museum directly instead…
I thought we covered some good ground getting some of the history on record, but think I’d like to try this again with a tighter focus on lessons learned. Perhaps, next time, I’ll moderate. :-)
March 19, 2006
Resilience is better than anticipation
“In preparing for battle I have always found that plans are useless, but planning is indispensable.”
— Dwight D. Eisenhower
Paul Saffo, favorite futurist of every journalist on the tech beat, famously said, “Never mistake a clear view for a short distance”. An important corollary is: being able to see the destination doesn’t necessarily mean you can see the road that gets you there. One lesson I take from this is that trying to plot a detailed map of that road can be a big waste of time.
This tome represents half a million (1993) dollars of Grade A, USDA Choice, prime Vision, happily paid for by our friends at Fujitsu. This lays it all out, in loving detail. This is the document that sold the venture capitalists on funding Electric Communities’ transition from a three-guys-who-do-cyberspace consulting partnership to a full bore Silicon Valley startup company. We had a regular business plan too (albeit one that was a little vague in the “and this is where the money comes from” part; see It’s a business, stupid), but what the VCs were really buying into was this: the utopian dream. It was exhilarating, it was brilliant, it was some of the best work I’ve ever done.
It was doomed.
It was at once too detailed and too vague. This is the nature of big complicated plans: they have lots of details (that’s what makes them big and complicated) and they leave lots out (because, the world being the complex thing that it is, no matter how much detail you give, it’s never enough to completely describe everything relevant). Plus, the more details and complexities there are, the more opportunities you have to make mistakes. As the number of elements you are juggling grows large, the probability of significant errors approaches certainty. (One notable VC who declined to invest in Electric Communities told us, “This is one of those Save The World plans. We don’t do those; they never work.” I want him for an investor the next time I try to start a company.)
Big utopian plans are doomed by their very natures, because they are always wrong. Being the huge fan of F. A. Hayek that I am, I should have figured this one out a lot sooner. I no longer believe in big plans that try to comprehensively anticipate every requirement and every contingency. Instead, I believe in resilience. Resilience is better than anticipation (a formulation for which I need to credit my friend Virginia Postrel).
It is better to have a simple plan and be resilient in its execution.
By resilience, I mean the ability to quickly and inexpensively adapt to the inevitable changes in circumstance, unforseen events, and surprising discoveries that any significant undertaking is bound to encounter. Unless what you are doing is completely trivial, the awful truth is that you must do it in an environment that is filled with uncertainty.
Most people will acknowledge the value of being prepared for unexpected external contingencies like earthquakes or downturns in the market. Fewer will take into account the broader, more diffuse, but ultimately more important phenomenon which plagues most long-term projects, namely that the nature of the world shifts between the time you start and the time you plan to be done, so that a plan that might have been ideal when you started might be disastrous by the time you finish (Freeman Dyson has some interesting things to say about this in Infinite in All Directions). Very few indeed appreciate what I think is the biggest source of uncertainty, which is that you don’t really understand exactly what to do and how to do it until you are well on your way to having already done it. It is in the doing of a thing that you end up discovering that you need to do other things you hadn’t originally counted on or that you now need to do some things differently from how you’d originally intended. Indeed, you may discover that you’ve already done some things wrong that now need to be done over or just thrown away.
You might reasonably ask how I can possibly reconcile this extreme skepticism about the value of (or, indeed, the possibility of) planning with what I mainly do for a living, which is to develop large, complex software systems. These undertakings would seem to demand exactly the kind of comprehensive, large-scale planning that I’m criticizing here. Indeed, this is how the world of software engineering has usually approached things, and they have the long history of schedule overruns, budget blowouts, and general mayhem and misery to prove it. Accepting the limitations of human rationality with respect to planning and forecasting is merely bowing to reality.
My new religion, in the realms of project planning in general and software development in particular, is what I guess I’d call “hyperaggressive incrementalism”:
Do really small steps, as small as you can manage, and do a lot of them really, really fast.
Don’t do anything you don’t have to do, and don’t do anything that you don’t have an immediate need for. In particular, don’t invest a lot of time and effort trying to preserve compatibility with things that don’t exist yet.
Don’t try too hard to anticipate your detailed long term needs because you’ll almost certainly anticipate wrong anyway. But be prepared to react quickly to customer demands and other changes in the environment.
And since one of the dangers of taking an incremental approach is that you can easily drift off course before you notice, be prepared to make sweeping course corrections quickly, to refactor everything on a dime. This means that you need to implement things in a way that facilitates changing things without breaking them.
Don’t fix warts in the next rev, fix them now, especially all the annoying little problems that you always keep meaning to get around to but which never seem to make it to the top of your todo list. Those annoying little warts are like barnacles on a ship: individually they are too small to matter, but in aggregate their drag makes it very hard to steer and costs you a fortune in fuel.
Simple is better than complicated. General is better than specialized. But simple and specialized now is better than general and complicated some day.
With respect to software in particular, adopt development tools and processes that help you be resilient, like memory safe, strongly typed, garbage collected programming languages and simple, straight-ahead application frameworks (i.e., Java very good, J2EE very bad). I also favor rigorous engineering standards, ferociously enforced by obsessive compulsive code nazis. I sometimes consider it a minor character flaw that I’m not temperamentally suited to being a whip-cracking hardass. Every team should have one.
Writing things down is good, but big complicated specifications are of a piece with big utopian plans: too ponderous to be useful, usually wrong, and rapidly obsolescent.
In general, it is better to have a clear, simple statement of the goal and a good internal compass, than to have a big, thick document that nobody ever looks at. That good internal compass is key; it’s what distinguishes a top tier executive or developer from the second and third stringers. Unfortunately, the only way I’ve found to tell whether someone is good in this respect is to work with them for several months; that’s pretty expensive.
My bias at this point is to favor productivity over predictability. It’s OK to have a big goal, possibly even a really big, visionary, utopian goal, as long as it’s just a marker on the horizon that you set your compass by. Regardless of what plans you may have (or not), a productive process will reach the goal sooner. Though predictability is elusive, productivity, in contrast, is actually achievable.
December 20, 2005
The Robin Hood of Neopets [circa 2000]
Over on my Yahoo! 360° blog I have a new post recounting how my son was The Robin Hood of Neopets, at least breifly back in mid-2000.
April 5, 2005
Stretching the Lessons
A response to Marc Hedlund’s Reading Yahoo! 360° through “The Lessons of Lucasfilm’s Habitat”
First, I must say that I’m personally flattered that Marc Hedlund and Clay Shirky think that The Lessons of Lucasfilm’s Habitat is “fantastic” and “Best. Essay. EVAR.” I’ve never been called a hero before. :-) Frankly, the paper is showing its age, so I was quite surprised that Marc chose it to frame his thoughtful and lengthy critique of the Yahoo! 360° service. You see, in the spring of last year, Chip and I started this blog: Habitat Chronicles in part to archive and transcribe the “Habitat Redux” (ppt) presentation, where we take our original paper to task and talk about the new lessons learned since then. Chip and I haven’t transcribed everything yet, so that Marc may have missed some of the hindsight, reinterpretations, corrections, and retractions is understandable.
Despite our best efforts at clear communications, The Lessons of Lucasfilm’s Habitat has become a bit of a social software Rorschach Test: In it, people see some wisdom that was not intended, or was quite accidental. Marc’s critique does suffer a bit from this effect and in a few places even acknowledges it, but he goes further as he re-interprets several of the lessons.
I welcome all thoughtful critiques of my work (new, and old), and think that several of Marc’s prescriptions are correct for Yahoo! 360°. But, since his critique used The Lessons as framework in ways that I find personally challenging, this seems like the right place respond.
Identifying the Customer
Marc says:
… Yahoo! Mail or MSN Passport is a bad idea, since the service’s goals will diverge from yours over time … an email address in your own domain will serve you better than one @yahoo.com. Great stuff, especially now that domain name registration is effectively free.
This statement typifies an attitude that separates us digirati from the rest of the world.
My mother [sister/nephew/etc.] can barely manage her web-based email. How on earth would she manage her own domains, servers, and what would motivate her to cough up the additional money to pay for these services? I run a server farm from my garage but can’t get AOL to accept a connection from my mail server. Likewise, pobox.com bounces all mail from my machine as coming from a dynamic domain. I get spam from people trying to trick me into using their domain registry with deceptive mailers that look like bills. Why was my own email domain good idea again? There is a definite cost/benefit trade-off calculation here.
It seems presumptuous to tell everyone that the free web-based email provider that they use is not good enough – that they should pay for so called ‘effectively free’ services that are significantly more expensive and complex – when all they want to do is send baby pictures to their relatives. The presumption that you understand other people’s cost/benefit calculus better than they do was at the heart of the final and most important of the Lessons: Get real. For millions of people, web mail is plenty good enough.
Stretching The Lessons
How better, then, to look at Yahoo! 360°, than to take these lessons learned nearly two decades ago, and apply them to the brave new project? Yes, let us arrange the deck chairs to spell Habitat, and see how they feel.
- A multi-user environment is central to the idea of cyberspace.
On the surface, Yahoo! 360° seems to have learned this in kindergarten. But the world is different than it was then, and there is not one Habitat, but many. It connects you to your friends with Yahoo accounts, and not to any other friends you might have…
We’d like to believe that eventually we’d be able to share with any friend the same way we can email with any friend, but look at Yahoo Messenger and its competitors, and that’s not the future you’ll see.
Yahoo! 360° is built fundamentally on the idea of sharing – what is more multi-user centric than that?
During Marc’s visit to preview the project, the development team shared that more integration with non-Yahoo! services was consistent with the goals of Yahoo! 360° and already on the product roadmap. Look at the recent release of web service APIs, deep support of RSS over all its services, acquisition of Flickr, and initial availability of RSS feeds on public blogs for evidence of this commitment. It is not a matter will or understanding that it needs more integration. It is only a matter having the resources and time to implement.
- Communications bandwidth is a scarce resource.
Lessons talks literally about modem speed when it talks about bandwidth, but it also talks about attention bandwidth…Today, the problem with attention bandwidth is the number of applications that want my daily attention…RSS comes out of 360 only through blog postings, one feed at a time (no “show me blog posts by all of my friends,” as Flickr would have it); other system messages aren’t available at all without logging in…
The original lesson doesn’t mention attention at all, but I’ll take the bait anyway :-).
Yahoo! 360° is designed with the idea of reducing to one the number of places you have to go to get the latest photos, blog entries, reviews, favorite songs, group postings, messages, etc. That it currently fails for his case is a concern, but already there are customers that are telling us that we have significantly reduced the number of separate sites that they have to ‘hand out’ to their friends and family.
- An object-oriented data representation is essential…
…360 represents data in HTML only; no access allowed unless you’re logged in through a browser…[The Lessons] hint at a web services API. My usage of my data might be arbitrarily elaborate; allow me to communicate with the service at a behavioral level rather than through your presentation…
Stretching this lesson to talk about web services APIs is probably appropriate, but not exactly the original context. I still believe that online services should offer object-level semantic interfaces, where it makes sense. And it makes sense for Yahoo! 360°.
Yahoo! 360° will have more feeds and APIs as soon as we can get them working. I can’t think of a service this comprehensive that had full web services APIs on their first day of Beta. Even Flickr took the better part of a year.
- Detailed central planning is impossible; don’t even try.
Relationships between people in 360 — the social networking part — are bi-directional, as with Friendster and Orkut…
Sharing and the free flow of communication are not built around awkward social questions. Let me browse around, see what there is to see, and choose the things I like enough to want to see more. Let me send out my ideas and my pictures and whatever else to people who haven’t joined and haven’t linked to me, and may never…
The original lesson was a call to let the users create their own content, instead of depending on a single content creator (aka the bottleneck) to meter out the experience in a linear form. As originally written, Yahoo! 360° exemplifies this goal: There is only user-generated content.
As to the critique of bidirectional links: there is much debate on this topic – there are some things (such as permissions management, gathering recommendations by degree, etc.) that are simplified for users by establishing these kinds of relationships. Though no one in my family, nor any of my strong ties, has any problems with this connection structure, the desire to observe other’s public activity anonymously is a valid request and is currently enabled for public blogs via RSS feeds. The product team is considering other one-way connection schemes as well.
- You can’t trust anyone.
A central point of 360 is its controls for permissions and… the privacy interface creates an expectation that it will actually do something to protect my privacy… The idea that a site like Yahoo can give me actual control over the distribution of my ideas and photos with a popup menu is absurd on its face; if the idea is sharing, then sharing will find a way. You can’t trust anyone but the promise of the site is that you can.
The original lesson explicitly limits its scope to client-server interactions, like an API. It is a software security lesson, not a distribution of information lesson.
As to the claim that Yahoo! 360° protecting your information is “absurd on its face”, I beg to differ and propose a challenge to prove my point: Marc – Please send me a copy of pictures of the gift my daughter made last Christmas for a friend. They are available through Yahoo! 360° and Yahoo! Photos and are protected only by “popup menu” settings. You do not know who has permissions, and even if you can guess, they don’t trust you.
Do Linux file permissions (set from an even more primitive command-line interface) mean that those files aren’t protected? Come on.
Marc closes with…
What 360 gets right is that it isn’t about anything in particular; it’s a blank piece of paper with some wrinkles and lines. What users do with it, even in its closed-garden, api-less, feed-free incarnation, is their own choice, and they may well find a way to make the tool better than its designers originally intended.
This is no accident. Yahoo! 360° did learn Lessons from Habitat (and hundreds of other projects, papers, and products). What we didn’t do was get every feature/integration we wanted working before we put it out into the market.
But, the proof is in the pudding – watch the way Yahoo! 360° evolves and see for yourself.
F. Randall Farmer
October 29, 2004
SOP II Presentation Slides
As I feared, many at State of Play were speaking about virtual objects as if they were user’s personal real-life property. The general reasoning is:
Secondary Virtual Item Market implies Real World Value and that implies that it is Real World Property and, by extension, subject to real world property laws.
Refuting this position is why I prepared the KidTrade position paper and released it two weeks early. I didn’t want to spend my time at the conference talking about KidTrade proper, but use it as one of several proof-case(s) that the the formulation above is an insufficient to rationalize the Real World Property hypothesis. I was able to achieve this goal.
The powerpoint slides I used as notes are available, and as an added bonus the deck includes the “History of Habitat” slides I presented at the evening event at the Museum of the Moving Image, complete with screen shots.
UPDATE 11/12/04: The video archives are up and you can watch my panel.
July 11, 2004
It's a business, stupid
Everybody says this but this time we really mean it
Like we said, we really don’t want to be in the platform business. It’s like trying to sell PhD research to kindergartners: it doesn’t matter how smart you are or how great your stuff is, because that’s not what they are about (your competition is SpongeBob; you’re doomed).
When we started Electric Communities, we did understand that we needed to be in business. We wanted to do some awesome stuff, and awesome stuff requires awesome bucks, and awesome bucks generally come from a revenue stream. But the reality is that venture capitalists don’t invest in stuff because it’s awesome. At least not any more. They invest in stuff because they think they’re going to make a lot of money. If your plan is about making awesome stuff rather than about making a lot of money, they don’t want to talk to you. This is business 101, but it’s easy to fall into the following trap: “Here’s some awesome stuff, surely somebody will want to pay us a lot of money for it, it’s so awesome!” This is sort of The Business Plan of The Damned. But this is another one of those You Can’t Tell People Anything things. People were telling us this the whole time, and we didn’t really get what they were talking about. I mean, we thought we did, we thought were in business to make a product and sell it, but really we were more interested in making what we were making because it was what we wanted to make than because it was what people wanted to buy. Eventually we figured this out, but by that time the ride was over. I think I know what to do now, but unless one of you out there has 8 or 10 million dollars you want to bet on another spin of the wheel it may be a little while before we get to try again.
So what we’ve got to show for our troubles is a menagerie of amusing business fallacies whose fallaciousness we understand in an intimate and personal way:
User generated content isn’t free
Who’da thunk? You get a couple million people generating stuff, that’s way more stuff than you could ever afford if you had to actually pay people to create it. What’s not to love? Well, the cost of editing and filtering, for one thing.
Sturgeon’s Law famously says, roughly, that 90% of everything is crap. But that proportion is so low only because most of what we see is highly filtered. Over the centuries, vast industries have evolved to filter most of the crap out. It’s kind of like the mess we face today with spam. It’s not that any individual piece of crap is so hard to deal with, it’s that the producers are so damnably prolific.
Context equals Revenue
Advertising! We were going to make money selling ads. So was everybody else. In the early days of the Internet boom, a lot of the advertising business was basically just startups taking in each others’ laundry. Larry Samuels, one of our CEOs (the good one) at Electric Communities, described the dotcom advertising game this way: “Here is $5 million dollars. Your job is to spend this money in a way that generates $3 million dollars in ad revenues. Think you can do it??“
We actually managed see this for the shell game that it was and avoided getting pulled into it for quite a while. We made the plunge into advertising when we took over The Palace, Inc. and unexpectedly found ourselves with a working product on our hands. Faced with the need to now make money from this situation, we talked ourselves into selling ads. In 1999, just as ad prices dropped through the floor.
One of the big things that popped the advertising bubble is that ads need context. Real advertisers, as opposed to dotcoms playing Let’s Pretend We’re A Business, are control freaks. They’re very sensitive to what their ads appear next to (just ask Janet Jackson and MTV). User generated content is unpredictable — not something that gives control freaks that warm fuzzy feeling that encourages them to spend money on you. Context is everything for them. This is one of the reasons why Google has been able to make substantial money selling really quite modest and unintrusive ads: they can deliver killer context.
Another thing about ads is, you’re serving two masters, the advertiser and the end user, and their wants and needs don’t always line up. Getting the value proposition right is tricky. Traditional media businesses generally resolve the conflict of interest by kissing up to the advertisers, but in the online world it’s harder to hold on to fickle users.
Extraordinary Popular Delusions & the Madness of Crowds
For those of you who don’t recognize the heading, it’s the title of a wonderful book by Charles MacKay all about the Internet. It was published in 1841. I wish I had read it in 1995.
A million free users does not a business make
Another business plan: “Hey! We can get millions of people to use our service. Millions of people, that’s a lot! Surely there must be a way to make money from such a large population!” This is another variation of The Business Plan of The Damned.
An important wrinkle on this is: “We’ll get millions of people to start using our service, then, once we have millions of users who know they like our service, we’ll start charging them money for stuff.” In the real world: turn on the money, watch them all go switch to the other free service who haven’t used up all their venture funding yet. Or watch them prove just what a powerful engine for innovation the Internet is, as they figure out countless clever ways to keep using your service while not paying you.
Economists have what they call the Principle of Revealed Preference, which is the idea that the only way to find out what people really think is to watch what they really do, especially when it comes to spending money on something. If you instead simply ask them what they think, you’ll get a different answer, because giving you an opinion does not require them to commit to anything. The same principle explains why it is hard to convert a free beta tester into a paying customer. As long as things are free, the starving student, the middle-class yuppie, and the millionaire are pretty much interchangeable. Once you put a price on things, the student can’t afford it and the yuppie now needs to be persuaded to spend. In fact, it’s worse than that, because the one resource they were expending during the free period was their time, for which their budgets likely follow completely the reverse hierarchy.
Boundless Potential
So we had this myth of the boundless potential of the Internet. I love this quote from the economist Herbert Stein:
“Things which cannot go on forever eventually tend to stop”
First Mover Advantage
Another lie to be suspicious of is the so called “first mover advantage”. This notion was most notably promoted by McKinsey Associates, the business consulting firm whose big Silicon Valley success was Apple Computer (or least, that’s the way it got spun). The idea is that a competitor who gets in first can establish such a dominant position that the barriers to entry to later competitors are huge and so the dominant position is locked in. This is just nonsense. Our experience is, first movers get slaughtered. And we always knew this; back in the days when IBM occupied the post of Great Satan now held by Microsoft, one of their widely quoted mottos was: “Never first. Always second.” The second mover has the advantage of being able to see which of the first movers didn’t get eaten by the sharks.