December 22, 2006

Smart people can rationalize anything

One of the things we were able to do at Electric Communities was to attract one of the highest density collections of scary-smart people I’ve ever seen gathered in one place before. There are a lot of nice things about working with smart people. For one thing, they’re not stupid. Working with stupid people just sucks. Smart people are good if you need to do a lot of really hard things, and we did a lot of really hard things. But it’s not all upside. For one thing, smart people tend to systematically overestimate the value of being smart. In fact, it is really valuable, but they still tend to weight it too heavily compared to other virtues you might also value, such as consistency, focus, attentiveness to the emotional needs of your customers, and so on. One of the problems with really smart people is that they can talk themselves into anything. And often they can talk you into it with them. And if you’re smart yourself, you can talk them into stuff. The tendency to drift and lack of focus can be really extreme unless you have a few slower people in the group to act as a kind of intellectual ballast.

Why do less when you can do more?

Smart people can invent solutions to problems you don’t actually have yet. The problem is, it’s easy to think of problems you don’t have yet. Stopping to solve them all now is a recipe for paralysis. Furthermore, while it’s easy to think of all kinds of potential future problems, it’s much harder to forsee which of those you will actually have, much less all the ones that you are going to have that you didn’t anticipate. People who are less smart manage to avoid pouring resources into unnecessarily solving future problems because they aren’t able to figure out how to solve those problems anyway. So they just ignore them and hope they don’t actually come up, which in a lot of cases turns out to be the way to have bet.

Programming sage Donald Knuth taught us that “premature optimization is the root of all evil.” It turns out that this doesn’t just apply to coding.

You can’t sell someone the solution before they’ve bought the problem

Smart people can invent solutions to problems folks actually do have but don’t know it yet. These solutions are usually doomed. This ties in with the whole You Can’t Tell People Anything principle. It is nearly impossible to solve a problem for someone if they don’t believe they have the problem, even if they really, really do.

For example, one of the deep flaws in many distributed object schemes, such as the CORBA standard, is that they make no effective provision for distributed garbage collection. This is a major pain, because if storage management is annoying to get right in a non-distributed system, it can be brutally so in a distributed system. Java’s Remote Method Invocation standard is somewhat better in that it does do DGC, but it still can’t cope with unreferenced distributed cycles. One of our wiz kids, Arturo Bejar, devised for us a truly elegant DGC algorithm, which is not only efficient but gracefully handles distributed cycles. (To my eternal shame we patented it.) Since to work well in Java it really wanted to be in bed with the Java Virtual Machine, we tried to sell it to JavaSoft, who were literally next door to us in Cupertino (actually, we tried to give it to JavaSoft), but they weren’t interested. They hadn’t bought the problem yet. So a small piece of great technology that could make the world a slightly better place sits on the shelf.

Generalitas gratia generalitatis

(For those of you who, unlike me, lack a co-worker who spent 7 years studying Latin, whom you can bug for stuff like this, that’s “Generality for Generality’s Sake”.)

Smart people love to think about the general case scenario.

For example, at Electric Communities we ended up making a big investment in developing an orthogonal persistence mechanism for our object infrastructure. For those of you who are unfamiliar with it, orthogonal persistence is one of the ultimate examples of highly generalized technical coolness. Basically, the idea is that you abstract away the file system (or any other persistent storage, like a database) by keeping everything in memory and then playing tricks with the virtual memory system to make processes immortal. The example we were inspired by was KeyKOS, a highly reliable OS for IBM mainframes that was developed by some friends of ours in the 1980s, in which you could literally pull the power plug from the wall, and, after plugging it back in, be rebooted and running again — including transparently resuming all the running processes that were killed when you cut the power — in about 8 seconds (this was actually one of their trade show demos). You gotta admit, that’s pretty cool. Some commercial installations of KeyKOS have had processes with running times measured in years, surviving not only power failures and hardware malfunctions, but in some cases actual replacement of the underlying hardware with newer generations of equipment.

Orthogonal persistence was attractive to us not just because of the reliability enhancements that it promised, but because it would free programmers from having to worry about how to make their objects persistent, since it abstracted away all the serialization and deserialization of object state and associated design questions. Anything that made programming objects simpler seemed like a big win. And so we built such a system for our object framework, and it worked. It wasn’t quite as awesome as KeyKOS, but it was still pretty awesome.

One of my favorite catch phrases is, “The difference between theory and practice is that, in theory, there is no difference, but, in practice, there is.” Orthogonal persistence was a great idea — in theory. Of course it cost months and months of development time, and it introduced a number of subtle new problems, any one of which would have a made a good PhD dissertation topic. If you are trying to produce a commercial product in a timely and cost efficient way, it is not good to have somebody’s PhD research on your critical path. For example, it turns out that for most kinds of objects, the amount of state you actually need to save persistently is a small fraction of the full run-time state of the object. But in an orthogonal scheme you save it all, indiscriminately. The problem is not so much that it’s bulky, though it is, but that the volume of semantic meaning that you are now committed to maintain in near-perpetuity is vastly increased. That turns out to be very expensive. The problem of schema migration as new versions of objects were developed (due to bug fixes or feature enhancements) proved effectively intractable. Ultimately we fell back on a scheme that compelled programmers to be much more deliberate about what state was checkpointed, and when. That proved more practical, but in the meanwhile we lost literally years of development resources to the blind alley. Less sophisticated developers would not have gone down this blind alley because they wouldn’t have a clue that such a thing was even possible, let alone be able to figure out how to do it. They would have been saved by their own simplicity.

Keep in mind that all of the foregoing is not an argument against being smart. Rather, it’s a recognition that human rationality is bounded, and even really, really smart people must necessarily fall far short of mastering the complexities of a lot of the things we do, as engineers, as business people, and as ordinary human beings. What works the kinks out of a thing is often just the passage of time, as the shortcomings and subtleties gradually emerge from practice. Because stupid people work more slowly, they get the benefit of time for free, whereas smart people have to work at it.

25 Comments

But surely if these folk were really really smart, they would see these potential pitfalls and sidestep them. Strikes me you are talking about people who are academically bright, but not bright enough to know that you also need to be street wise, and think strategically.

Two other ideas share a similar problem. Intelligent Design and Communism. The difference is that Communism posits that we’re smart enough to “manage” the economy where as ID says that nature is too beautiful and was therefore created by the thing in who’s image we were created.

Just by chance I have “The Mythical Man Month” sitting next to a Hayek book. They both basically say the same thing but in reference to different topics.

Some people would say: “A.D.D.” I think the issue is how we define intelligence. We could go there, but, generalitas gratia generalitatis.

“But surely if these folk were really really smart, they would see these potential pitfalls and sidestep them. Strikes me you are talking about people who are academically bright, but not bright enough to know that you also need to be street wise, and think strategically.”

I think this is a good point. I think this all hinges on how we define “smart”.

I think it’s common for people who are “smart” or “intelligent” foremost in the traditional sense to cling to the traditional definitions. Many people have suggested instead talking of different types of intelligence (eg http://en.wikipedia.org/wiki/Intelligences). Although difficult to assess and categorise, I think it’s safe to say there are many seperate traits that comprise the overall performance of the mind, many of which aren’t correlated (indeed some may argue there are traits which are negatively correlated!)

I think it’s partly the fault of the academic system that so much bias is placed on this one (albeit imporant IMHO) trait. I’ve seen academic stars with what I’d describe as skewed intelligence profiles perform miserably in the workplace because their communication and people skills were so poor. Conversely I’ve seen people with less academic prowess excel outside academia, I’d argue that they had a more balanced profile. This isn’t to say that there aren’t people who are strong in all traits, just that they’re a lot more rare!

YOUR ARTICLE (SMART PEOPLE) IS THE MOST INFORMATIVE I HAVE EVER READ.I’VE BEEN READING RICHARD BANDLERS BOOK (THE STRUCTURE OF MAGIC)I KNOW HE WOULD AGREE. THANKS SO MUCH DONALD STAY PS RICHARD BANDLER’S BOOK SUPER AS HE TALKS TO YOU.

for smart people, i must say you have a pretty crappy website.

Interestingly enough, I used to be a smart person. I remember in high school being a full three years ahead of everyone else in understanding derivatives and integrals (i.e. I had thought of and grasped on my own what people were being tought in class three years above me). On top of that I was in the french system, which is notoriously good for math (all proof based, practically no computations).

I remember exactly how I used to think… this weird super calculating pattern machine. Potential solutions to problems would keep on popping up in my mind, almost like picking at a combination lock, until I found one that worked. This all happened almost unconsciously and almost perpetually.

I soon after started working as a programmer. And after a 4 year push during the dot-com era, I burned out and incidentally quit my job. I’ll skip the details of my life, but what I can tell you is that somewhere along the way, my life and brain diverted drastically towards the arts and subtly I made a huge mental shift: I became mentally lazy.

This mental lazieness has a weird implementation. It’s like lazy initialization. I would read problems, or mathematical proofs, or articles and documents, but I wouldn’t even bother looking at the example equations given in the opening paragraph and would just read on until I felt it was absolutely necessary to consult the reference examples.

In contrast, my prior brain would have attacked those equations without even beginning to read the problem text itself… trying to find patterns. Trying to find the solution to the problem before even having read the problem. It was almost a question of pride.

I still consistently score above 160 on IQ tests that I take, despite the fact that I’m always actually bored and uninterested when I take these tests and I’m half heartedly doing them. It’s as if somewhere along the way, my brain suddenly made it a priority to not over exert itself.

Back in my real life, I currently manage the technology for one of the companies which I own. It’s still in its infant stages as far as profitability etc. goes. I also do quite a bit of random contract work here and there and I’ve begun to see a pattern emerging in my professional habits. Whereas before, I was all too eager to jump on a client’s problem, or even the problems he didn’t have yet, now I’m always holding the handbrake and making absolutely sure that we need change before I go ahead with new code. When new code must be put in, my predicate for what is better solution to the problem at hand is a cross product between ‘the amount of code to be modified/created’ (less is good), and ‘the elegance of the change’ (is it patchwork or not) – ultimately though, the second question is nothing more than first one extended over a timespan of several years.

Which reminds me of one of my favorite people: Miles Davis. He once said something like so: “it’s not about playing all the notes. It’s about chosing the right ones”.

I partially agree with some of the commenters as to ‘types’ of intelligence, but only insofar as saying plain IQ doesn’t really matter. IQ and in general any kind of intelligence (EQ etc) smacks of western(?) thought model which is to attribute a scalar value to anything. Which we all know doesn’t work… for things even as simple as a 2 dimensional real plane.

When thousands of years ago Lao Tzu wrote the Tao, he was onto something. That “no-mind” (mu shin) can be better at accomplishing perfection than intelligence. Does he argue that we should be dumb – I would doubt that and don’t personally think so. But from what I’ve understood of it and life in general is that intelligence is a tool. That you have to use it as such and not become its slave. Most incredibly intelligent people I know (including my self from not so long ago), are slaves to their intelligence. It is in fact their biggest impediment to truely achieving perfection… or the divine… or the tao. Or whatever else you want to call it.

Thanks for reading, you’ve been a great crowd.

Ps. sorry if this post comes off as self adoring, but just read it for what it is. Nobody knows who I am, and I will not know who reads this. It’s a perfectly candid and anonymous post.

I’m smart, and you’ve convinced me.

Great article, and insightful comments. I concur with Mo in many topics, and also on the fact that I’ve become ‘mentally lazy’ myself.

I sometimes wonder if the apathy I feel is some sort of illness…

I know this wasn’t the main point of your post, but in under 600 words, you have summed up several of the key problems in persistence.

Interesting article, and Mo is right on target with his story. Thanks, guys.

I hope everyone gets it soon.

This “orthogonal persistence mechanism” gets to the soul of the matter.

“…intelligence is a tool. That you have to use it as such and not become its slave. Most incredibly intelligent people I know … are slaves to their intelligence. It is in fact their biggest impediment to truely [sic] achieving perfection… or the divine… or the tao. Or whatever else you want to call it.”

Wow, that was the most insightful thing I’ve ever read in a technical blog, the most insightful I’ve read anywhere in a long time. Thank You.

I call it Brahma, or just It. “Tao” is a fun one too.

Manojit (This is my Yogi spiritual name, it means, “One who is victorious over the mind” or “A mindful victory”, but the former is more apt here.)

How come you smart guys don’t live next door to me, so you could help me work out the kinks in my screenplay. I am trying to deal with a cool, relatively novel, bit of Quantum stuff, and I am slogging through the same puddle again and again. Frustrating me, but you could help!!!

))_^^((Cheers!!

I like ice cream! BONES IS SMART PRETTY LADY! EIN RAND WAS A SMART!

I used to be an architecture astronaut, but it drove me bonkers and eventually I burn-did out. now I just try to perfect my FIFA 09 game and play Empire: Total War! So much more satisfying.

Yeah,

I am not as smart as I wish/thought I was, but that’s OK, because I am a lot wiser than many smart people.

Where did the wisdom come from? I don’t really know but I frequently see it’s absence.

Humility is part of it, but even talking about it makes a claim of humility a hypocrisy.

A key component is taking the time to think about things in depth and a willingness to shred your own arguments (rationalizations) if they are weak.

I just accept that I am schizophrenic, that there are two minds at battle all the time. The ego and the skeptic.

This is why working in a small company is great – you have very smart programmers doing strategy, management and sales, who are constantly telling you off for “being too clever” by going off and solving problems that don’t have useful practical outcomes. Smart strategic programmers (SSPs) overseeing slightly less smart technical programmers (SLSTPs) seems one workaround. And/or you could just verse your new smarties to constantly asses the usefulness/profitability of their own work.

Hi. I noticed Habitat and EC.

I played/experienced WA (DS) in 95′ and upwards.

I’d like to think I’m one of those smart people.

Any way back in? (on a professional level.)

e-mail me.

Yes, I have seen all of these, I guess I was blessed with working in silicon valley which probably has a much higher average IQ than anywhere in the world. It’s the Mount Everest of IQ.

One of those smart people I worked with mentioned a quote by Albert Einstein to me once, and it’s worth keeping in mind at all times: Make it as simple as possible, but not simpler.

In software development, I think it’s best to be pragmatic, which is why I like the Pragmatic Programmers so much. Stupid coders will cause a huge mess because they pretty much make things that don’t work, and never were going to work. They are too naive. But you can also go too far in the other direction and over-engineer your solutions to the point of hopelessness. For me, my main principle is to avoid complexity. Complexity in the solution is usually a failure to find a simpler one.

The orthogonal storage is a good example of that, however, if you were really clever you’d have known that committing to the semantics of your objects is a huge mistake. Yeah I found this out the hard way too, using Java byte serialization. Maybe everyone needs to go through this once – it’s hard to foresee all the little and often extremely complex problems that can arise from that. Ever since then I have avoided Java object serialization like the plague.

Hi,

I really liked this article. My own experience as a programmer and as a human being is that for some things I need to be stupid, really stupid, and that’s good!!

The good thing about being smart is being able to be smart and stupid. A stupid person can’t be smart.

If you need to kiss a woman,talk with her, you need to be like children, stupid, let it go, stop thinking. I had friends really really smart that have problems with that!!

I love theatre acting too and it has helped a lot in my live to change my personality. If I need to focus, I temporary change my personality, if I need perfectionism, I change my way of thinking, if I need a fast solution, the same thing applies.

Some parts of a program needs to be done fast, and some perfect, and some cheap, you know what you want to get, you change your mood accordly .

e.g I have done a program that recognizes handstrokes, some parts could be done with an easy algorithm, a function and ten lines of code, but I decided to spend a month doing it different and debugging. Now it goes a MILLION times faster. Takes a lot of work, but makes possible what before was impractical to do with a computer.

When I start selling it I thing is going to be the only one that really works.

Smart people can rationalize anything really explains the choices of Alan Greenspan, Ben Bernanke, Barack Obama, etc. This whole recession is the culmination of a perfect storm of smart people guided by really bad theory.

“intellectual ballast” haha, that made my day.

In the early ’80s, I worked for a startup, which consisted of some very smart people who hadn’t finished college, and me, who had, who they needed do the actual design work. Of course, there was banter about this, and the fact every so often an MBA would come through trying to sell themselves, and just fail miserably, not seeing the trees for the forest. Anyways, the older fellow there owned several disparate successful businesses, and had various homespun favorite sayings, such as “if he’s so smart, why ain’t he rich?” To which one day I replied, “well, the richest guy in Japan is an economics professor.”

This was before the Japanese economy collapsed and went into deflation for a decade, negating all “japanese economic miracle” stories. Funnily enough, I read an article in some mass magazine that predicted it a few months before it happened, thought it made sense, and happened to mention it to my boss (not the startup mentioned earlier, which changed direction when they realized it’s easier for a startup to sell a specific tool than enterprise software), who had previously worked in the securities industry and wound up thinking I was some kind of genius. So he introduced me to my [now-]wife, who is scary smart supermom with a doctorate and a clinical practice and teaches grad students.

Rationality may be bounded, but stupidity isn’t. Genius comes from timing.

“intellectual ballast” what a great concept.

Very well written article. Thanks for that. I’ve often wondered if intelligence was like height or strength or speed or bone density, etc. etc. There are downsides to being too tall (blood pressure) or too strong (lots of muscle mass means you need more food to survive). We tend to think of intelligence as “more is better” but I wonder if there is an optimum amount of intelligence – there seems to be for the activity known as “money making”. Most really wealthy people I know have one thing in common; a real lust for money. Most really smart people I know have a lust for solving problems, not so much the money making. Smart people tend to dream of making millions doing something really, really clever, then focus on the clever thing and forget about the money.

I’ve noticed this happening in several projects, mostly with Computer Science post-docs. I think that academic CS schools train people to look for the clever, theoretical solution, and not think about time and clarity as important factors. After all, you don’t get a PhD for doing something trivial.

Paraphrasing a previous poster, you can be very smart and still not be very wise.

Professionalism breeds familiarity blindness. For instance, I have read Hamlet over 20 times and students who have read it for the first time are able to locate intricate patterns I had never considered. It all goes back to Socrates – “I know that I am intelligent, because I know that I know nothing.”

Post a comment

(If you haven't left a comment here before, your comment may need to be approved by the site owners before it will appear. Thanks for waiting.)