Being Right

A few months ago I learned what is, so far, the most fascinating thing I've learned about human nature: the difference between being right, and being able to look someone in the eyes.

If they practice, people can learn to look you in the eyes even when they're lying to you. But it's kind of like a fake smile; there are involuntary muscles up there. If you know what you're looking for, you can still tell.

But what does it mean if they're looking you in the eyes and they mean it? It means that, at least in that moment, they're doing what they really believe is right. That's the definition of integrity.

That part is easy. That's not the surprising thing.

The surprising thing, to me, was that someone can have integrity and still be completely evil. It's kind of obvious in retrospect; the super-villain in an action movie can always look the hero in the eye, and he always does, just to prove it. He has integrity. Evil with integrity is more respectable, somehow, than plain evil.

All it takes to have integrity is to do what you think is right, no matter how stupid that may be.

Beware of people with integrity.

A note on drowning in work

Glub.

I Am Of Literary Significance

Oh yes. Fear me.

Stupid internet.

(The rather dull ancient high school essay that started it all. But this is better. Or maybe this.)

Weird things about git, #1: premature optimization

I've been hanging out on the git mailing list for the last little while, and a few things have been striking me as weird. There are hundreds of people on that list, and lots of them are active contributors. Why? It's just a stupid version control system after all.

But there must be some reason. I've thought of a few. Here's one for today:

The git developers are completely obsessed with performance. Apparently nobody there has ever heard that "premature optimization is the root of all evil." (Incidentally, the ACM has a recent article explaining why that statement is a fallacy.)

It's enlightening just to watch the git developers optimize every possible part of their system: syscall overhead, memory allocation, path trees, file size, network turnarounds, etc. And even while they optimize the heck out of everything, they still complain about how inefficient it all still is. Which it is, of course, if you follow the discussions.

Also interesting is that large parts of git are written in sh and perl and it's still fast, because while they obsess about performance, they also know which parts actually matter to performance. It helps to be a Linux kernel developer sometimes, I guess.

There's no doubt that git's optimization is premature: it started out way faster than svn, and it gets even faster with each release. Is that really necessary? Of course not. Everyone was doing just fine with things like svn. But life is just so much better when programs go unnecessarily fast. It's a very strange sensation; pain you didn't realize you were having just goes away. I've been doing a lot of Windows development lately, and let me tell you, Windows developers have a whole lot of pain that they don't realize they have, because it's not overt. Windows development is mostly fine, really. But what kills it is all the little random delays, the crashes, the giant memory-leaky .Net-based IDEs that make you point-and-click fifty things sequentially once a week because you don't have a way to write a script to do it for you. Just give me a text editor and 'make' any day, thanks.

What was my point again? Oh right, unnecessary optimization. While I've been writing this article, I've been waiting for svn to finish committing 20000 tiny files to a fresh svn repository. It's still not done. (Watching a strace of this is hilarious. Among other things, it's creating an XML file for every single one.)

...

And for comparison, git just did the same thing in 23 seconds. (About half that time was spent running my shell script to create 20000 files.)

Who cares? How often do you create a repo with 20000 tiny files in it anyway? Almost never, of course. It's not an important optimization goal.

...

svn finally finished. See? That wasn't so bad.

How to tell the gcc maintainers are all (still) lunatics

Just got bitten by this insanity today.

There's not being backwards compatible, and then there's... that. What was the point of upgrading, again?

Weird things about git, #2: no bug tracking system

The git developers don't track bugs. If you find a bug, you can write about it on the mailing list. You might get flamed. And then probably someone will ask you to fix it yourself and send in a patch.

This is unlike almost all other open source projects. Virtually every project out there has a basic bug tracking system attached, courtesy of SourceForge or Google Code or whatever. It's a key part of the development process, right?

Well, not exactly.

I'm also following the development mailing list for Mono. It has much less traffic than git, which is weird, because at least by lines of code (and probably number of users), Mono is a way bigger project. One of the reasons for the quiet is their bug tracking system. It's a conversation killer. Like this:

Someone reports a bug on the mailing list. "Please file it in the bug tracker." "I can't figure out how to create a bugzilla account." "Do it like this." "Okay, I created the account, but I can't figure out how to file a bug." "Do it like this." "Okay, I filed the bug. Now what?" "Now hopefully someone will fix it someday! Bye!"

So people aren't supposed to discuss bugs on the mailing list, which invites the question: what are they supposed to discuss on the mailing list? Nobody really knows. So the conversation dies. And not everybody sees every bug that comes in. Which is probably good, because there are thousands of bugs filed, and as with most open source projects (and commercial ones, for that matter), most of the bugs never get fixed, because volunteers just aren't actually very interested in fixing every last one of your problems. (This isn't to say the Mono developers don't fix tons of bugs. Mono is awesome. But it seems to be a cardinal rule of bug tracking that the bug database only grows, it never gets smaller.)

Let's be honest: bug tracking sucks the joy out of software development. git developers just work on whatever scratches their itch at that particular moment. If the same question comes up too often on the mailing list, they update the documentation. If people report a bug, they quiet those people by fixing the bug (or convincing those people to fix the bug). If they feel like implementing a new feature today, then they just do. And with multiple examples of this happening every single day, it's easy and fun to subscribe, get a feel for what's happening, and join in.

When you work to a bug tracking system, you're just doing what you're told. Some poor bug tracking janitor spends hours each day shoveling bugs around inside the system and assigning them to people and prioritizing which ones go into which release. Then the developers line up and take their bugs, and work on them in sequence so that the release can go out at the pre-ordained time.

git developers just make a code freeze branch sometimes and let it simmer for a while, then release it. Junio C Hamano is the equivalent of the bug shoveler for git, except he doesn't shovel bugs: he shovels fixes. I expect that's a much more gratifying job.

Update (2008/06/30): YCombinator News has some commentary on this article.

A serious case of JustOneMoreBug

I just read Waking Up from the Economy of Dreams, a pleasant essay about how Webmind, Inc. went bankrupt in 2001 while trying to invent artificial intelligence.

The essay is an interesting read, mainly because the author seems to have come out of the experience so... innocently. We were 3/4 of the way to a real, working AI, he insists near the end of the story. It was just bad luck that we ran out of money right as the .com bubble burst...

You might almost be able to believe him, except he also told the rest of the story. The original development was supposed to take a few months. To build the first sentient AI! Unsurprisingly (to me), it took longer than expected. They actually ran out of money multiple times in the process, but managed to land financing at the last minute each time. (Incidentally, that sounds to me like a long string of good luck that finally ended, not any bad luck at all.)

But okay, lots of startups almost run out of money, and just about everyone underestimates the amount of work involved in their project. So what?

Well, the problem is that these guys felt sure, at each failure along the way, that they were just one step away from having it finally work. And when they finished that step and it still didn't work? Well, now we're just one step away. We're almost there. We can feel it!

At NITI, we called this the JustOneMoreBug syndrome, spelled using WikiCaps because it was a page in our wiki. It seems now that we must have been the ones to invent that term, since I can't seem to find it in Google anywhere. So with that in mind, I guess I'd better tell you what it is:

First of all, the fact that this was featured in our internal wiki probably tells you something about our experience with it: ie. we had plenty.⁽¹⁾ JustOneMoreBug is exactly what it sounds like: the feeling that if you can just fix this one last bug, your software will work perfectly, customers will come flocking to you, and riches will inevitably ensue.

The frustrating thing about JustOneMoreBug is that every successful project does have a moment exactly like that: the transition from unreleasable to releasable, from unusable to usable, from unsellable to sellable. And so if you're an optimist, you, like us, would prefer to believe that this bug is your one last bug. And maybe it is.

Unless it's not, which becomes clear one way or the other shortly after you fix it. Then what?

Then, if you're experiencing the JustOneMoreBug syndrome as opposed to the JustOneMoreBug major life-changing event, you realize that there's just... one more bug holding you back.

Fool me once, shame on you. Fool me twice, shame on me. You can get away with claiming JustOneMoreBug once or twice per project - but if you go past that, you have to stop and rethink. Why are there so many bugs that you don't even know about yet? Believe it or not, really good programmers will write programs that mostly work the first time, and the parts that don't work can fairly easily be uncovered and solved. If your program is fooling you about how stable it is, it's because your program isn't very good. You need to stop and find a new strategy: often, by doing some heavy testing and using the results to build a brand new multi-month development schedule. That's bad news, because the project was supposed to be done tomorrow. After all, there was just one bug left.

Webmind was on a multi-year marathon of JustOneMoreBug. I feel sorry for them, because advanced research has a strong tendency toward that; structured engineering principles just don't work for software research projects. But it seems like their investors didn't realize they were doing software research, and their researchers honestly didn't realize - repeatedly - that the JustOneMoreBug feeling is just a feeling, not a reality. And if the essay is any indication, they still don't realize it; they still think they were almost there.

At least they didn't come out cynical.

Footnote

⁽¹⁾ NITI had two kinds of JustOneMoreBug experience. First, with certain internal software projects (which is normal occasionally in any software company, although if we'd been smarter we would have caught them sooner). And second, with our business model: even if your product works, if it's not selling as well as it should, there's a strong tendency to believe that you just need to tweak one more thing about your business and it'll take off. But just like with software, you shouldn't fall for this trick more than once or twice. We fell for it repeatedly, and that was our multi-year marathon. (Eventually the winning fix wasn't a tweak. It was a complete re-analysis and redefinition of our target market, and a corresponding total change in the way we dealt with customers and resellers.)