apenwarr - Business is Programming

A bit more language meditation

Sat, 24 Jul 2010 07:34:01 +0000

After the last couple of bits on C++, I thought I would offer something a little more melodramatic: human languages, as experienced in Montreal. Back in 1887.

Old Montreal, 1887, via the McCord Museum's Flickr Feed

Notice anything funny about this picture?

I'll give you a hint: Montreal is a primarily French-speaking city. Looking at the 2006 census, 13% of the population spoke English as a first language, compared to 54% with French as a first language.¹

...

...Yet every single sign in that photo is in English! If you looked at the same street today, you would see every single sign is in French.²

Why? Because of Bill 101 from 1977, informally known as the Quebec "language law." Among other things, that law says any public sign has to be primarily in French; no other language can be "more prominent" than French.

The mere existence of the language law is, itself, fascinating. It's a flagrant violation of the free speech rights guaranteed by the Canadian Charter of Rights and Freedoms.

But Canadians always hedge their bets, so in addition to free speech, the Charter of Rights and Freedoms also has a section called the Notwithstanding Clause.³ That clause basically says the government can enact any laws they want that violate your rights, as long as they comply with a few basic rules, such as refreshing said laws every few years. (Unlike other Canadian laws, rights-violating ones expire automatically.)

As you might imagine, laws that very literally violate human rights can cause a bit of a fuss. This one certainly does - and it has continued to do so since it was first brought in. The need to refresh it every few years guarantees that it gets back into the news every few years, which is both healthy and stressful.

I'm a native English speaker myself, so this law comes down to racism against me. But you know what? I think it's a good law. 54% of Montrealers speak French as a first language; almost all the rest (even me) can speak at least basic French when necessary.

As the story goes, the reason the rule was needed in the first place was this: while almost every French speaker had learned basic English - after all, the people bordering Quebec in every direction are largely anglophone, so there are lots of chances to learn - the much smaller English population didn't bother to learn French. Because if all the French people are willing to speak English anyway, why bother? And if you're making a sign - even if you're a French person making a sign - are you going to make one that 54% of people can understand, or one that 99% of people can understand? That's right. If you're a wise French business owner serving primarily French customers, you'll make your sign in... English.

Those lazy English people have a point. It really is a lot of work to learn French, just so you can speak French in this tiny little enclave of non-English on a whole continent of English. I find it completely believable that English people are so lazy; my own crappy French skills are testament to that. And quite simply, Quebec's French speaking majority called us on it. They demanded justice: they demanded the right to be served in the language of the majority.

And in order to give people that right - a right not guaranteed by the Charter of Rights and Freedoms - we had to violate another right, namely, the right not to talk to people in French. If it weren't for the magic of the Notwithstanding Clause, the government would be enforcing civil rights... but the wrong ones.

By the way, if you're American, and you come to Montreal and people pretend they can't speak English, you're absolutely right: they are pretending. Because you just rudely jabbed them with hundreds of years of cultural history.⁴ (Note: it's not a pretense in other parts of Quebec, where people often speak exclusively French. And obviously some people in Montreal really don't speak English, but it's fewer than it seems.)

So what does all this have to do with programming?

Well, about that C or C++ or Java. Do you really use it because it's better? Or because that's the one thing everyone can understand, even though it's not actually the best choice for most people? If someone made a law forcing everyone to write stuff in a particular language - say, Objective C - in order to prevent the oppression that is, say, Flash - is that a violation of your freedoms or is someone out there actually protecting you?

Okay. It's a stretch.

Footnotes

¹ I tried to look at the 1861 census (okay, it's a few years off, but whatever), but the statistics defeated me. They weren't surveying people's mother tongue at the time, though they did survey people's birthplaces. At least 48% of the population at the time was of "Canadian - French origin" birth, with about 26% from Britain/Ireland/United States. However, that doesn't account for an additional 25% of "Canadian - Not of French Origin." How much of that is anglophone? I don't know. Perhaps there were more anglophones than francophones in Montreal back in 1861? I don't know. How did it change by 1887? I don't know. This is the sort of information I would like to retrieve from Wolfram Alpha if only it weren't a useless piece of junk.

² You would also notice that Montreal's winter road conditions are about the same as ever.

³ There's also the "Limitations Clause," which says the government can violate your rights, but only if they're consistent and there's a good reason. And don't do it any more than necessary ("minimal impairment"). From the outside, it's hard to believe weird stuff like this works, but the emphasis on using power responsibly instead of blindly following the letter of the law is what Canada is all about.

⁴ Tip: I warn my American friends who visit Montreal that one simple change in behaviour will make your experience vastly more enjoyable. When you start a conversation, any conversation, just saying hi in a store or ordering in a restaurant - do your utmost to start it in French. You know, Bonjour, parlez-vous anglais, mispronouncing stuff off the French side of the menu, whatever. It doesn't matter if you suck at French. You probably won't get more than 5 words out before the person switches to flawless English. Why? Because you acknowledged that they have rights. Imagine if some people from France flew to New York, walked into a restaurant, and refused to speak anything but French. Would you think that was cute? Acceptable? Remotely reasonable? Of course not. You'd think they were idiots. But if you know some French, and they came in and tried their best at English, but had a terrible accent and awful grammar, you'd switch to French as a favour to them. Because they're not being idiots, and you're a nice person. Etiquette really is that easy.

How to design a replacement for C++

Thu, 22 Jul 2010 16:19:54 +0000

My last article on the ugliness that is C++ didn't actually receive this complaint, but it should have: I offered a lot of criticism, but no constructive criticism.

I feel a little guilty about it, so let me try to resolve that here with some actual, constructive advice to language designers, for anyone who cares to listen. (Maybe nobody cares to listen, and in fact this will be much less interesting than the blind ranting of my last article. Too bad. Stop reading now if you're bored.)

The first thing you need to know about C/C++ is that they're only barely worth fixing anyway.

C has too few features, and C++ has far too many awful ones. Reasonable people might disagree on which features C is missing and which C++ should lose. But most people would agree at least that C could be usefully extended, and C++ could be usefully simplified (and maybe have a few cleanups, like my earlier suggestions of operator[]=, a sensible method pointer, and sensible standard strings).

We also know that neither change will happen. The C people, having seen what happens when you extend your language willy-nilly (ie. C++) are deathly afraid of it and will never ever change again. The C++ people are well set on their path (ie. ultimate salvation is right around the corner if we can just add a little more crack to our templates) and will never let it go.

But anyway, that doesn't really matter. C and C++ both get the job done in their respective niches. And those niches are shrinking dramatically. Once upon a time, you'd surely write all your apps in C or C++; nowadays, almost everything is better off written in a language with more built-in stuff. My personal tool of choice nowadays (when appropriate; I'll get to that in a minute) is python for most stuff, with C modules added on for the parts that have to be fast. It works excellently, as judged by my favourite metrics of fewer lines of code, increased readability, and maximum performance.

You might prefer ruby or C# or something intead of python. That's fine, although python seems to be the winner so far when it comes to a super-easy and efficient C extension system. (C#, including mono, makes me especially angry because C extensions often run slower than native C#. There's a massive and stupid overhead required to escape from the runtime down into native space and it often outweighs the speed gained from C. Duh. In python the overhead of calling into a C module is essentially zero.)

To a large extent, the reason you can get away with using "higher level" languages like python or ruby or C# is that computers have gotten faster and have a lot more memory than they used to. You need the faster computer to run an interpreted language, and you need more memory because you have garbage collection instead of manual memory management. But we've got the horsepower now. Might as well use it.

That means C and C++ are on the decline and they're just going to get smaller. Good. The world will be a better place for it.

But there will always be programs that have to be written in a language like C and C++. That includes kernels, drivers, highly performance-sensitive code like game engines, virtual machines, some kinds of networking code, and so on. And for me in particular, it also includes new plugins to existing C-based legacy systems, including Microsoft Office.

These programs are never going to go away. So deciding that they will, forever, have to suffer with the limitations of either C or C++ is kind of disappointing. And yet there is still no language - not even the hint of a beginning of a language - that can seriously claim to replace them. Here are the key "features" you will absolutely need to avoid if you want any chance at replacing C.

Things you absolutely must not do if you want to replace C

Do not remove the ability to directly call into (and be called by) C and ASM without any wrapper/translation layers. When I want to call printf() from C or C++, I #include stdio.h and move on with my life. No other language makes it that easy. None. Zero. Do not be those other languages.
Do not remove the cpp preprocessor. Look, I realize you are morally opposed to preprocessors. Well fuck you too. Your moralizing is getting in my way. If you take it out, I can't #include stdio.h, and I can't implement awesome assert-like macros. (Note: see update below.)
Avoid garbage collection. Garbage collection is fine as a concept, but you will never, ever, be able to write a good kernel if you try to use garbage collection. This is non-negotiable. Also, plugins to existing C programs won't fly with garbage collection, because you won't be able to usefully mark-and-sweep through the majority of non-garbage-collected memory, and you can't safely pass gc'd vs. non-gc'd memory back and forth between C and your language. Maybe your language can have optional garbage collection, but optional has to mean globally disabled across the entire executable.
Avoid mandatory "system" threads. If you're writing a kernel, you're the guy implementing the threading system, so if your language requires threads, you're instantly dead in the water. Garbage collection often uses a separate mark-and-sweep thread, which is another reason gc just isn't an option. But it's even more insidious than that: what happens when you fork() a program that has threads? Do you even know? If the threads were created by the runtime, will it be sane even 1% of the time? You can't invent Unix if you can't fork().
Avoid a mandatory standard library. People can - and do - compile entire C programs without using any standard library functions at all. Think about a kernel, for example. Even memory allocation is undefined until the kernel defines it. Most modern languages are integrated with their standard library - ie. some syntax secretly calls into functions - and this destroys their suitability for some jobs that C can do.
Avoid dynamic typing. Dynamic typing always requires some sort of dictionary lookups and is, at best, slightly slower than static typing. To replace C in the cases where it refuses to die, you can't have a language that's almost as fast as C. It has to be as fast as C. Period. Google Go has some great innovations here with its static duck typing. Objective C is okay here because the dynamic typing is optional.
Avoid support for exception handling. It's just too complicated, and moreover, C people just hate exceptions so they will hate you, too. And since C doesn't know about exceptions, you will make a mess when C calls you, then you throw (but don't catch) an exception. Just leave it out.
Do not make it harder to do things in your language than they would be in C. Maybe this isn't even worth mentioning. But the upper bound on the lines of code it takes to do something should be whatever it would take in C. Making your language backward-compatible with C is one way (not the only way) to achieve this.

All this sounds terrible, right? Why even bother if you can't have these obvious features? But actually, there are a bunch of things you can add and make things much, much better than C without making your language unacceptable in C's niche.

Things you can add to your language to make it better than C without ruining your chances to replace it

Deterministic constructors/destructors (RAII). This is, quite probably, my favourite feature of C++ and the primary thing that makes me hate going back to C. (The lack of it is also what makes me hate almost every other high-level language. Python, thankfully, has this, although they claim that it's an implementation detail that could go away at any time. And IronPython can't do it. Bastards.) Deterministic constructors and destructors make smart pointers and automatic refcounting possible (and delightful!) and let you write things in one line of C++ that would take 10 lines of C. No exaggeration. And it compiles down to the same thing that C would, so there's no runtime cost.
Closures and anonymous functions. In fact, Apple has already added these in an incompatible variant of C. Maybe you like them, maybe you don't, maybe you think they're God's gift to programming and any language without them is an infidel. But adding them would be harmless, anyway. (Update 2010/07/21: I mean harmless in that it wouldn't bloat the compiled code; it compiles down to the same ASM as the equivalent verbose C code, and if you don't use it, you don't pay for it.)
Implicit user-defined typecasts. These are a tricky feature of C++ and some C people hate them because they hide stuff they think should be explicit. But you need this if you want to implement non-gross smart pointers and user-defined string objects.
Operator overloading. You have to be seriously tasteful about this one. If you don't think you can handle the pressure, leave it out. But in the name of God, at least make operator== do something sane by default.
Automatic vtable generation. It doesn't have to be full-on OOP, and you don't need multiple inheritence and any of that stuff. But a huge number of lines in C programs are taken up declaring things that are basically vtables. Make it better. Google Go has some great ideas here. This one feature is probably the only good thing about Objective C.
Some sort of generics so you can make type-safe containers. Note, I'm not saying templates here. C++ has made templates a dirty word; you want to copy precisely none of their template stuff. But C# (up to, but not including, C# 4.0) has some very nice (and highly optimizable in native code) generics ideas that you can steal. Also note: I'm not saying generics are necessary in a language that replaces C. C doesn't have them and it survives. Most attempts at a C replacement leave this out of version 1 and add it to version 2, and that's perfectly okay.
One-time declaration/definition of functions. In C or C++, you have to declare your stuff in a header file, then define it in an implementation file. Your header file then gets compiled over and over again by everyone who uses your functions. (In C++ it gets even worse: your templates have to be defined in the header, so compiling every file ends up compiling half of your bloody program.) This is awful, and is the primary reason compiling C and C++ is slow. The problem has also been completely solved since the 1990's. Check out Turbo Pascal sometime. C# and Java, for all their flaws, have also thoroughly solved this. (Update 2010/07/21: Just because you absolutely must not remove the preprocessor doesn't mean you have to use it for declaring functions. The preprocessor is valuable for macros, not for function declarations.)
Standardized string handling. Actually I don't think this is very important; much more important is the ability to keep letting people define their own string types. As I mentioned in my previous article, I disagree with the conventional wisdom that allowing user-defined string types was a major mistake of C++. Strings are often the slowest part of your program. Making them possible to optimize or replace is a good idea; adding some sugar to construct compile-time string literals directly in a user-defined data type would be even better. However, even so, having a decent default string type couldn't possibly make things worse (as long as you can ignore it when it gets in the way, ie. in a kernel).
Implicit pass-by-reference. I'm totally addicted to the way python passes objects by reference, and only by reference. (Pedants would say it actually "passes references by value." I know the difference. I don't care.) This is probably hard to pull off without garbage collection support, but if you can do it, you'll be my hero. At the very least, let us use reference syntax wherever we might normally use pointer syntax, because requiring us to manually dereference pointers all the time was a mistake. And once you've done that, maybe remove pointer syntax altogether, because it's kind of redundant in C++ to have both. (The only exception is that in C++, you can't reassign what a reference points to. But that's only because they're idiots. Just let me do that, and pointer syntax is entirely obsolete.)
Typesafe varargs. C++ totally failed at this, with utterly awful results (ie. lots and lots of templates that define every version of a function with 1 to n parameters). C varargs are great, but they're not typesafe, and while that's great sometimes, it's less great other times. A simple varargs syntax that coerces all the arguments into a particular type (presumably using your implicit user-defined typecasts from above) would be easy and highly useful.
Lots of other things. This is not a complete list of features you should add to your language. Go crazy! Language design is an act of creativity, and most language features will not make your language unacceptable as a C replacement. Just don't break any of the "must not do" rules up above.

Current C and C++ alternatives and why they aren't popular

Apple/NeXT have been single-handledly pushing Objective C since, I don't know, maybe the 1980's or at least the 90's. It makes none of the "must not do" errors (since its dynamically-typed objects are optional). I personally suspect the reasons for its slow adoption are simple: a) Objective C isn't enough better than C and adds nothing if you don't use its dynamic typing; and b) the syntax is infernally ugly. Think about this: for all we know, the Linux kernel is actually written using every feature in the kernel-compatible subset of Objective C. Basically it neither wins nor loses. Mu.

The D language started out as a good idea, but they went crazy in version 2. Also, they require garbage collection, so they're instantly disqualified.

Google Go has tons of great stuff inside and meets almost all of the above requirements. Unfortunately it is also garbage collected, so it's instantly disqualified. (This one hurts me to the bottom of my soul, because the other stuff looks so great. But I'm not disqualifying it because I'm subjective, I'm disqualifying it because it just won't do the job as long as it requires garbage collection.)

C# is a rather nice language overall and, in fact, has very little in it that prevents it from being natively compiled. (Mono actually has a way to compile it natively nowadays, called their "AOT" (ahead of time) compiler.) However, it requires a big huge gunky runtime and garbage collection and at least one system thread and it parses XML at startup time - strace it and see! - so no luck. (I left XML out of the "must not do" list because I thought it was obvious. Don't make me regret it.)

Java actually fails at every single point in this article. Okay, not really. But they did manage to botch most of it in rather spectacular ways.

Any others that I've missed?

Note that C++ meets all the above requirements. That's why it was able to replace C for so many things. The main reason C++ doesn't replace C for a bunch of other things is that it's just too crazy and it encourages you, as the developer, to also be crazy. See my previous rant for all about that.

P.S. No, I am not planning to make my own C replacement language. When python isn't appropriate, I will continue using and complaining about C++, while desperately attempting to use it tastefully, if that is even possible. I will, however, switch to your language if it meets all my requirements. So you'll have at least one user.

Update 2010/07/21: Wow, this hit the front page of news.ycombinator.com in less than 10 minutes. Thanks, guys. But I see there is some confusion about where I stand on C vs. C++ specifically, and why C++ is not the answer if my question is how to replace C. Good question!

The problem with C is that it works but is missing stuff; the problem with C++ is that it tried to add stuff, but the result is hideous. That's a totally subjective evaluation of C++ (see my previous rant for some concrete examples) but it's one that a lot of people seem to agree with. The goal here is to identify the "necessary but not sufficient" rules for creating a C replacement that has a chance of winning. You may hate C++, but it met those criteria, and so it became massively popular, hideous or not. I just want more options; please make me a language that is necessary, sufficient, and not hideous.

Update 2010/07/22: People are taking issue with the swearing - and my love of the C preprocessor - in point #2 above. I rephrased it slightly but I'm not removing the swearing. I almost never swear. But this time it matters: removing the preprocessor is treated as a moral issue, but it's also the reason your language design will never replace C, and as a moralizing language designer, you need to know that and make a conscious decision about it. Yes, the C preprocessor is used for all sorts of egregious hacks. I have bad news for you: that's what it's made for. A language that prevents me from making egregious hacks is a language that will, eventually, prevent me from doing my job. Egregious hacks can be used to create portability where there was none; functions where there were none; typedefs where there were none (I'm looking at you, C#). It lets you transparently replace a call to one function with a call to another. It lets you do "#define private public" (one of my favourite C++ tricks) when the maintainer of a library turns out to be an idiot. Yeah, you don't want your language to depend on these hacks in order to let you write a good program; C fails here, C++ fails worse. You want to keep these hacks to a minimum in production code. But if you think all production code should have zero hacks, you are an idealist, and your language design will never win in the niche where C and C++ win. (There is most certainly a niche for languages without hacks. That's not what this article is about.)

You can't make C++ not ugly, but you can't not try

Tue, 20 Jul 2010 03:56:30 +0000

...everything that's wrong with C++ comes down to that.

Background: I've been programming in C++ since about 1993; that's 17 years now. As late as 2009, I chose C++ to write our Windows client for EQL Data. If I were to make that decision today, I would still choose C++, because, quite simply, nothing else would work. (Okay, C would work, but it would be at least 5x as much effort. So no thanks. And for making plugins to legacy Windows apps, there's just nothing else out there.)

So, okay, I know a fair bit about C++. I've managed 30-person development teams building huge stuff in C++. Successfully. I have some context here.

And my context is: even if there's nothing better for the job, the truth is that C++ is incredibly ugly and misdesigned. C++ is a trap: they tell you that you can do anything you want in C++. Anything! C++ isn't a language, they say, it's a language construction kit! Build the language of your dreams in C++! And it'll be portable and scalable and fast and standardized!

And this is so close to true that even after using it for 17 years, I still almost believe it. I used to actually believe it. But see, some recent experience with the "amazing innovations" in other programming languages have convinced me otherwise.

First of all, if you haven't done much C++, you need to realize: most of the stuff in there is utter putrid boneheaded crap. This includes the RTTI and exceptions stuff; C++'s versions of those were enough to convince a whole generation of programmers that introspection and exceptions were outright evil and should be avoided. But as it turns out, that's only true in C++.

If you've heard anything about C++, you've probably heard that there's no standard string class and everybody rolls their own. That's not actually one of the bad things, in my opinion. As a person who's done a lot of coding in C++, I've actually come to understand that there really are good reasons to use different string objects at different times. In python, my current language of choice whenever it's appropriate, there's only one string class, and it's mostly okay, but every now and then you really want to just replace one character in the string with one other character (an O(1) operation) but you can't, so you instead construct a new string (an O(n) operation) and your program is vastly slower and less scalable. (Happily, python makes it pretty easy to create string-like objects, particularly using C extensions, so if you really need it, you can make it still go fast. But effectively thats just exactly creating your own string class, like so many people do in C++. See? Not always evil.)

The problem isn't interchangeable string classes. The problem is that the default C++ string class is so utterly godawfully stupid. No garbage collection? Check. No refcounting? Check. Need to allocate/free heap space just to pass a string constant to a function? Check. No support for null strings? Check. Horrendous mess of templates that makes tracing in a debugger utterly painful? Check. Horrendous mess of templates that makes non-ultramodern compilers unable to optimize them so that, for years, your toy homemade string class was 5x faster? Check. Totally unclear what character type it uses (actually you can use whatever you want at different times)? Check. Totally missing a sprintf-like formatter so you have to use something, anything, oh god please save me from iostreams just to produce a dynamic string? Check. Can't append to a string without allocating a whole new one? Check. Using the "+" append operator produces more temporary objects than you can count? Check. Using the "+" operator with two string constants gives a weird compiler error about adding pointers? Check.

In contrast, let's take, say, python's strings: refcounted, passed by reference, nullable, compatible with string constants, no templates, trivially easy debugging, always the same character type (although they changed it in python 3, sigh), include a sprintf-like operator, the + append operator works fine and multiple appended constants can be optimized at compile time (python's interpreter compiles to a metalanguage and can do basic optimizations like this). They even have an optimized non-constant append operator in newer versions of python that's more efficient than making a whole new copy every time.

How many of these string features required us to use an interpreted language? Precisely zero. An imaginary, fictional version of C++ could have had a string class with all these features and been just as fast and efficient. And I bet a lot fewer people would have written their own if that had been the case. There's actually no excuse for the crap that is C++ std::strings; they aren't better. They're just, somehow, the standard.

Another C++ problem that's close to my heart is function pointers. Not even lambdas or anonymous functions - let's not get all fancy, here. Just plain pointers to existing named functions. C++, being a superset of C, has function pointers, of course. And while the syntax for them has always been a little funny, they actually work fine and don't make you want to kill people too often. (Everywhere C function pointers are used they should always have a void *userdata parameter, and when people don't do that (like in qsort()), then you do want to kill things... but that's not C's fault, and sensible programmers can avoid that mistake.) So ok. C++ has function pointers.

But here's the thing: they utterly failed to extend this concept to include pointers to methods of an object.

Okay, that's not really true. In fact, it's a little-known fact that C++ - the language, not the insane libraries or templates - has built-in support for function pointers that call member functions.

The bad news is, this feature is so horrendously ill-conceived that absolutely nobody uses it for anything. Seriously. Nobody. I tried my best. The feature really is actually useless. The article I linked to tried desperately to make them look like maybe they have a purpose, but no. They just don't. (You can see the main problem in the linked article under the section "Member Function Pointers Are Not Just Simple Addresses." You might think, oh, of course not. They're a "this" pointer plus an address, right? Ha ha! Ha ha ha ha!! No they're not! They don't have a this pointer! You still have to provide your own this pointer when you call it! But it does store all kinds of crazy other stuff instead so it can do call-time vtable lookups on multiply-inherited objects! Ha ha!)

Utterly useless. But the bad thing isn't so much that it's useless - although maybe someone should have noticed that and killed the feature before it somehow passed the standards committee. The bad thing is that there is an obvious way to do it that wouldn't have been useless: just make a member function pointer be a struct { obj, funcaddress }. Everybody knows that calling a member function obj.f(x,y,z) in C++ is actually done by calling f(obj,x,y,z). There would be nothing to it. Since you know 'this' at the time you create the function pointer, you can resolve the funcaddress from the name 'f' at that point - the same way you would when making any method call, including vtables, multiple inheritance, and everything - and the code receiving the pointer would always just run it as (*funcaddress)(obj, ...). So easy. Nothing to it. So very much terrible C++ code would never have been written if this feature existed.

But it doesn't. There are alternatives, of course - numerous ones, and all terrible, and all incompatible, because the language designers simply failed utterly to do their job. The boost (now TR1) one has the cutest syntax, but God help you if you make a typo using it, because you'll get pages of template gibberish.

Stop and think about that for a second. Template gibberish. For a simple function pointer! Every language not designed by idiots in the last 20 years, including Turbo Pascal, has some kind of function pointers. ASM has function pointers. C has function pointers. This isn't hard. It has nothing to do with making fancy type-independent efficient data structures, for which templates/generics are actually justified. It has to do with a trivial operation that's a basic part of every compiled language: pushing some parameters on the stack and jumping to an address.

While I'm here, no, strings are not "generic" data structures either. The fact that std::string is a template is also incredibly insulting.

Okay, one more example of C++ terribleness. This one is actually a tricky one, so I can almost forgive the C++ guys for not thinking up the "right" solution. But it came up again for me the other day, so I'll rant about it too: dictionary item assignment.

What happens when you have, say, a std::map of std::string and you do m[5] = "chicken"? Moreover, what happens if there is no m[5] and you do std::string x = m[5]?

Answer: m[5] "autovivifies" a new, empty string and stores it in location 5. Then it returns a reference to that location, which in the first example, you reassign using std::string::operator=. In the second example, the autovivified string is copied to x - and left happily floating around, empty, in m[5].

Ha ha! In what universe are these semantics reasonable? In what rational set of rules does the right-hand-side of an assignment statement get modified by default? Maybe I'm crazy - no, that's not it - but when I write m[5] and there's no m[5], I think there are only two things that are okay to happen. Either m[5] returns NULL (a passive indicator that there is no m[5], like you'd expect from C) or m[5] throws an exception (an aggressive indicator that there is no m[5], like you'd see in python).

Ah, you say. But look! If that happened, then the first statement - the one assigning to m[5] - wouldn't work! It would crash because you end up assigning to NULL!

Yes. Yes it would. In C++ it would, because the people who designed C++ are idiots.

But in python, it works perfectly (even for user-defined types). How? Simple. Python's parser has a little hack in it - which I'm sure must hurt the python people down to the cores of their souls, so much do they hate hacks - that makes m[5]= parse differently than just plain m[5].

The python parser converts o[x]=y directly into o.__setitem__(x,y). Whereas o[x] without a trailing equal sign converts directly into o.__getitem__(x). It's very sad that the parser has to do such utterly different things with two identical-looking uses of the square bracket operator. But the result is you get what you expect: __getitem__ throws an exception if there's no m[5]. __setitem__ doesn't. __setitem__ puts stuff into your object; it doesn't waste time pulling stuff out of your object (unless that's a necessary internal detail for your data structure implementation).

But even that isn't the worst thing. Here's what's worse: C++'s crazy autovivification stuff makes it slower, because you have to construct an object just so you can throw it away and reassign it. Ha ha! The crazy language where supposedly performance is all-important actually assigns to maps slower than python can! All in the name of having language purity, so we don't have to have stupid parser hacks to make [] behave two different ways!

...

"...Well," said the C++ people. "Well. We can't have that."

So here's what they invented. Instead of inventing a sensible new []= operator, they went even more crazy. They redefined things such that, if your optimizer is sufficiently smart, it can make all the extra crap go away.

There's something in C++ called the "return value optimization." Normally, if you do something like "MyObj x = f()", and f returns a MyObj, then what would need to happen is that 'x' gets constructed using the default constructor, then f() constructs a new object and returns it, and then we call x.operator= to copy the object from f()'s return value, then we destroy f()'s return value.

As you might imagine, when implementing the [] setter on a map, this would be kind of inefficient.

But because the C++ people so desperately wanted this sort of thing to be fast, they allowed the compiler to optimize out the creation of x and the copy operation; instead, they just tell f() to construct its return value right into x. If you think about it hard enough, you can see that, assuming the stars all align perfectly, m[5] = "foo" can benefit from this operation. Probably only if m.operator[] is inlined, but of course it is - it's a template! Everything in a template is inlined! Ha ha!

So actually C++ maps are as fast as python maps, assuming your compiler writers are amazingly great, and a) implement the (optional) return-value optimization; b) inline the right stuff; and c) don't screw up their overcomplicated optimizer so that it makes your code randomly not work in other places.

Okay, cool, right? Isn't this a triumph of engineering - an amazingly world class optimizer plus an amazingly supercomplex specification that allows just the right combination of craziness to get what you want?

NO!

No it is not!

It is an absolute failure of engineering! Do you want to know what real engineering is? It's this:

map_set(m, 5, "foo");
char *x = map_get(m, 5);

That plain C code runs exactly as fast as the above hyperoptimized ultracomplex C++. *And* it returns NULL when m[5] doesn't exist, which C++ fails to do.

In the heat of the moment, it's easy to lose sight of just how much of C++ is absolutely senseless wankery.

And this, my friends, is the problem.

As with any bureaucracy, the focus slowly shifts from finding a simple, elegant way to solve your problem to just goddamn winning this one battle with the system so that you can get the bloody thing working at all. It would have been easy, at any time, for the C++ committee to have just added a new operator[]=. It would have been totally backward-compatible: any object without an operator[]= would keep working just like it always has.

But they couldn't do that. Doing that would be admitting defeat.

They could have made up a new syntax for sensible member function pointers, any time they wanted. Again, no concern about backwards compatibility - if you don't use it, it doesn't affect you.

They could have written a sensible string class. In fact, people did. Lots of people! But for some reason, they standardized on the non-sensible one. Now C++ users are forever cursed: either you use std::string, and pay endlessly for its suck, or you use your own string class, and be one of those people who constantly gets criticized for designing their own string class.

It is possible to write C++ that's not crap - in theory. This is because it's possible to write C that's not crap, and C programs will compile as C++. Then, you can add a sprinkle of the non-sucky parts of C++ - deterministic construction/destruction (RAII) is one of them - and you'll have a program that's undoubtedly better, more readable, and easier to debug than it would have been in pure C.

But you can't stop there. You should, but you can't. Nobody can. It would be superhuman. Because you'll see something that should be a little clearer, a little easier. Maybe it's string concatenation, maybe it's member function pointers, maybe it's operator[]. But you'll see it, and you'll start trying to solve it. And 1000 lines of code later, you'll have made your life - and the lives of everyone who has to maintain your programs - much worse.

For me it was function pointers. Over the years in wvstreams, I tried doing them so many different ways - using C-style function pointers with wrapper functions, using inheritance and virtual functions, using the insane C++ member function pointers, using templates and the insane C++ member function pointers. Finally, nowadays, function pointers in WvStreams use boost's new functor stuff, which has been standardized by TR1. And every single time I use one, I have to look up the syntax.

For my own library that I've spent the last 12 years building. I have to look up the syntax to declare a callback.

I should have just stuck with plain C function pointers.

Let this be a warning to you.

A Programmer's Code of Ethics

Sun, 23 May 2010 19:45:28 +0000

My programs encode the rules of modern society. I will take full responsibility for the programs I write.
I will not write a program that intentionally fails to operate.
I will not write a program that refuses to do tomorrow what it was able to do yesterday.
I will not create a single point of failure, whether technical or political.
I will not encode foolish rules just because someone paid me to do it.
I will not give people what they want if what they want is not good enough.
I will not stop people from taking my program's ideas and making them better.
I will write programs to help each person produce their best, not to help the masses produce mediocrity.
I will correct those who believe my program's failure is anyone's fault but mine.
I will write programs to benefit even the people who don't deserve it.

Condensed "New Testament" Version

Don't write for others a program you wouldn't want written for you.

Commentary

Do I always follow all the above rules perfectly? Certainly not. In fact, I think I've broken every single one of them.

But thinking over all those situations and knowing what I know now, I'm pretty sure that in every case, it would have been better if I'd done the right thing. The exceptions don't feel like the right move; they just feel dirty.

That's how I know I'm on the right track.

Update 2010/05/22: Based on a suggestion from Chris Frey, slightly rephrased point #3.

At last, the circle is complete

Wed, 19 May 2010 16:33:18 +0000

My twitter search RSS feed (yes, I have one, so shoot me) for "apenwarr" returned a hit today in which the only usage of the word "apenwarr" was an URL hidden behind a bit.ly link.

Oh yes. That means twitter search is now decoding bit.ly URLs as part of the indexing process, but of course it *still* serves you the original stupid bit.ly links.

Thank you, oh great technology gods, for inventing new uses for excess CPU that I never could have imagined.

In other news, SEOs can now increase the keyword relevance of their twitter links; just have bit.ly resolve to something like http://whatever.com/stuff?magic-keywords=fuzzy-wuzzy-chickens-multiplied-by-gargantuan-apple-google-flash-ipad-porn-naked

Tell me what surprised you: iPad Edition

Tue, 18 May 2010 23:31:35 +0000

If someone is about to tell you a long story about a trip they were on, you should make just one request: "Tell me what surprised you." That simple query changes the whole nature of the conversation.

For example, we all know the basic stuff about Paris. It has French people. The food is good. It's pretty. But what surprised you about Paris? Now there's something we can talk about.¹

So. Yes, I got an iPad. And I'll do you a favour: I'll tell you my surprise.

What surprised me was iBooks.

No, no, iBooks looks and works exactly like in the pictures and ads. It really is just like that, for better and for worse. That's not the surprise.

The surprise was that it wasn't installed by default.

Think about that. I had to go to the app store, painfully convince it I was a U.S. resident, search for "iBooks" ("books" is definitely not good enough), and download it, all just to get started.

Meanwhile, I downloaded a bunch of other apps. Some of them had ads. Many of those ads were for the Amazon Kindle app, which is also in the app store, and also free, and doesn't require me to be American. And I could click on any of those ads and get straight to app store. Two more taps, and I'm done.

There weren't any ads for the iBooks app. Anywhere. Thus it was harder to find out about iBooks, and as hard or harder to download it, than the Kindle app.

I've been in the computer world for a long time. I've observed Microsoft and how they do things. Heck, I've observed Apple and how they do things. And one thing I've seen for sure: bundling and cross-selling work. If this were Microsoft, they wouldn't have hesitated for a second to give iBooks a boost by including it with the OS.

But Apple deliberately left it out. iBooks has to compete with Kindle in the very same app store, with no free publicity (other than being a "featured" app in some iPad ads and PR).

I can imagine the iBooks team being told that this is it, yes, you can do your bookstore however you want, but we're not going to make it any easier on you. You have to be the best bookstore in the world all by yourself, not just because you tagged along with something that was already great without you.

Now that is surprising.

For the record, iBooks is doing pretty well so far: it absolutely beats the snot out of Kindle for the iPhone/iPad in pretty much every way (except book prices, which are much higher than Amazon's).²

Also interesting to consider is why they allowed this competition with books, but not with music, movies, and phone calls. Have they had a change of heart? A secret contractual obligation? Does Steve Jobs really just not care about books, as he previously claimed?

You might also ask why their Pages, Numbers, and Presentations (or whatever it's called) apps aren't bundled or cross-sold either; anybody making a word processor is on equal footing with Apple's iWork team. And there's no Weather, Stocks, Voice Memos, Clock, or Calculator app included on an iPad either, even though they were all on the iPhone. The iPad has less bundled stuff than ever before - the diametric opposite of what Microsoft has done in any version of Windows, ever.

The rest of the iPad? It's pretty much as expected. I'll spare you.

Footnotes

¹ What surprised me about Paris was that, at their fruit stands, every fruit is arranged with absolute care and precision. Compare to a typical grocery store in Canada, where fruit is typically dumped into a bin so you can sort through it yourself. When I think about how much more time it must take to do it the hard way, yes, it surprises me. How can they afford to do that? It's magic. (I also had other related observations at the time.)

² I won't bother describing the Kindle app's failings in detail. To get you started, I have just two words for you: page numbers. Compare them in Kindle vs. iBooks. Someone at Amazon needs to be shot.

Update 2010/05/18: Hmm, jordanlev wrote to tell me that on his iPad, it popped up a message right away asking whether he wanted to download iBooks. So maybe they're not playing all that nice after all. He also linked to an interesting article about the ebook market by Charlie Stross.

Mailing lists are cheap...

Tue, 11 May 2010 20:06:03 +0000

...but I still didn't think I'd bother with one for sshuttle, which was just intended to be a weekend toy project. Seems people are actually using it, though, and it's picked up quite a few github followers already. (62 followers isn't that much, but the thing is only 10 days old.)

So okay, here you go: the sshuttle mailing list.

By the way, it seems to not be common knowledge that you can subscribe to googlegroups mailing lists without having a Google account or ever using their web interface. The secret is to send an email to "groupname+subscribe@googlegroups.com", where groupname is the name of the group, in this case, sshuttle. Note that plus sign. It's not a minus sign.

sshuttle 0.30: automatic route and hostname discovery

Sat, 08 May 2010 19:56:28 +0000

My fancypants new sshuttle transproxy VPN could already work even if all you had was a ssh session to the other side. And it avoided the TCP-over-TCP trap. And sure, I even made it upload itself to the other end automatically so you wouldn't have to. And it apparently works on MacOS clients now, except Snow Leopard which is not-so-shockingly buggy, and maybe even on FreeBSD. And it manages latency, even under heavy use, so performance doesn't start sucking when you transfer a big file.

So in all those ways, it was already much better than the old Tunnel Vision, which among other things, you had to install by hand on both ends of the connection, and after that the performance was a bit random.

But Tunnel Vision still had a few tricks that sshuttle missed. The first one is automatic route guessing. When TV connected to the other end, the server would tell the client what subnets it was able to reach, and then the client would automatically set up routes for those subnets to go through the tunnel. Neat, right? But with sshuttle, you had to tell the client what to route by hand. No more:

     sshuttle -N -r username@servername

The new -N option enables automatic network determination. You can still add additional subnets (like 0/0 for people who want to route "everything") if you want.

Another fun feature of Tunnel Vision was automatic hostname mapping. You know what sucks about connecting to a remote VPN? You probably don't, so I'll tell you. What sucks is DNS. Your local DNS server doesn't know anything about the hostnames on the other end, and of course they're private so they're not in the public DNS either. So when you try "ssh internalserver", and "internalserver" is some server on the remote internal network, you get an error.

This one is a lot trickier to solve. After all, there's no good way to get a list of hostnames for you to replicate. And once you do, there's also no good way to add them to the local DNS. But does that stop us? Certainly not. It merely confuses us.

     shuttle -H -r username@servername

The new -H option tells the remote sshuttle instance to start prodding around wherever it can (currently, that means at least the local /etc/hosts file, samba nameservers and browse masters, and a bit of DNS) to try to find good hostnames and their matching IP addresses. As it finds them, it beams them back to your client, which adds them temporarily to your local /etc/hosts file. Gross? Oh boy, is it ever! But it works. More or less.

It would be kind of neat to have it get browse lists from things like mdns (aka "zeroconf" aka "bonjour") but I have no idea how to do that.

The old Tunnel Vision sort of had this feature, but it didn't have sshuttle's amazing Name Prodding Technology(tm). You had to configure the names yourself. As it happened, our proprietary Nitix servers had some very scary code to automatically track local hostnames and configure Tunnel Vision appropriately, so the name mapping worked pretty well there. And Nitix servers were usually acting as your DNS, so they could set that up nicely too. Sadly, Nitix's old name prodding is mostly obsolete due to the way modern networks are run (mdns, domain controllers, names-by-dhcp, switched ethernet, and so on). But life marches on. And we all still want the same things.

Anyway, anybody who knows how to get a good list of hostname/ip pairs out of mdns, ideally in a portable fashion, send me an email :) You might also want to look at hostwatch.py and see if you can think of any other interesting sources to scan for names.

Gunless

Sat, 08 May 2010 19:25:40 +0000

Movie recommendation. There's not much to say about it that the reviews haven't already. But if you're looking for something with a little more intelligence than a Hollywood Extravaganza, but not so intelligent that it bores me to tears like most independent films, this one works.

The idea that the Wild West of the United States didn't have any law is completely bogus. There was law. They were settled by laws like we were. And the idea that we had nothing but law and had no weaponry is also ludicrous. Of course we did.

-- Paul Gross on the Film's Historical Accuracy

So it's also useful if you want to learn totally wrong but funny things about the Canadian "wild west."

Uploading yourself for fun and profit (plus: sshuttle 0.20 "almost" works on MacOS)

Wed, 05 May 2010 22:50:00 +0000

After trying out the initial version of sshuttle that I produced this weekend, a few people asked me whether it would be possible to make it work without installing a sshuttle server on the server machine. Can it work with *just* an sshd, they wondered?

Good question.

Some people pointed out ssh's -D option (dynamic port forwarding using a SOCKS proxy). If we just used that (ie. the sshuttle client transproxies stuff into ssh's SOCKS server), then there wouldn't need to be a server side for sshuttle, and that would solve the problem. But sadly, sshd's latency management is pretty thoroughly awful - among other things, it sets its SO_SNDBUF way too high - so if you have a few connections going at once, performance takes a dump. sshuttle has some clever stuff to make sure that doesn't happen even if you've got giant ISOs downloading over your VPN link. I'd like to keep that.

So then I said to myself, hey, self, what if we just uploaded our source code to the remote server and executed it automatically? It works for viruses (technically worms), after all.

You know it's never a good thing when I start talking to myself. And yet the result is surprisingly simple and elegant. Here's a simplified version of what the "stage 1 reassembler" looks like:

   ssh hostname python -c '
   	import sys;
   	exec compile(sys.stdin.read(%d), "assembler.py", "exec")'

Where "%d" is substituted with the length of assembler.py. Assembler.py, by the way, is the "stage 2 reassembler," which looks like this:

    import sys, zlib

    z = zlib.decompressobj()
    mainmod = sys.modules[__name__]
    while 1:
        name = sys.stdin.readline().strip()
        if name:
            nbytes = int(sys.stdin.readline())
            if verbosity >= 2:
                sys.stderr.write('remote assembling %r (%d bytes)\n'
                                 % (name, nbytes))
            content = z.decompress(sys.stdin.read(nbytes))
            exec compile(content, name, "exec")
            # FIXME: this crushes everything into a single module namespace,
            # then makes each of the module names point at this one. Gross.
            assert(name.endswith('.py'))
            modname = name[:-3]
            mainmod.__dict__[modname] = mainmod
        else:
            break
    main()

Yeah, that's right, I gzipped it.

You know the best part? When the server throws an exception, it still gives the right filenames and line numbers in the backtrace, because we assemble each file separately.

If anybody knows the right python incantation to make it import each of the modules as a separate actual module object (rather than just dumping it all into the global namespace, as the comment indicates) please send a patch.

Grab the latest version of sshuttle VPN on GitHub.

"Almost" works on MacOS

Responding to popular request, I thought I would try to get the sshuttle client working on MacOS. (The sshuttle server already works - or at least it should work - on just about any platform with an sshd.)

MacOS, being based on BSD, uses the same ipfw stuff as FreeBSD seems to use. So it "should" be just a matter of having it auto-detect whether the current system uses iptables or ipfw, then run the right commands, right?

Well, almost. I did all that stuff, and I've *almost* got the rules working, but I just can't make it work right. I'm using MacOS X Snow Leopard on my laptop. I checked it in and pushed it anyway in case anybody wants to take a look; the final fix is probably a one liner.

For more information on my conundrum, see my (as yet unanswered) question on ServerFault. If you can contribute an answer, you'll forever be my hero. Even if you don't know anything about ipfw, if you could run through the steps on your version of MacOS or BSD and tell me what happens, it could help narrow things down.

Enjoy!

Update 2010/05/05: Thanks to Ed Maste and drheld for helping confirm. It seems that FreeBSD and MacOS 10.5 Leopard work fine - which means sshuttle 0.20 should work for you, yay! - while Snow Leopard consistently does not.

Really, that restores a bit of balance to the universe; since everyone I know who's upgraded to Snow Leopard went through huge pain and suffering, I was a little shocked to find that my own upgrade had been harmless, and Erin's really did make her computer go much faster (boggle). Now that Snow Leopard has finally screwed me after all, I once again feel like things are as they should be.

sshuttle: a new kind of userspace VPN

Sun, 02 May 2010 18:57:23 +0000

I just spent an afternoon working on a new kind of VPN. You can get the first release, sshuttle 0.10, on github.

As far as I know, sshuttle is the only program that solves the following common case:

Your client machine (or router) is Linux.
You have access to a remote network via ssh.
You don't necessarily have admin access on the remote network.
The remote network has no VPN, or only stupid/complex VPN protocols (IPsec, PPTP, etc). Or maybe you are the admin and you just got frustrated with the awful state of VPN tools.
You don't want to create an ssh port forward for every single host/port on the remote network.
You hate openssh's port forwarding because it's randomly slow and/or stupid.
You can't use openssh's PermitTunnel feature because it's disabled by default on openssh servers; plus it does TCP-over-TCP, which has terrible performance (see below).

This is how you use it:

git clone git://github.com/apenwarr/sshuttle
on your client and server machines. The server can be any ssh server with python available; the client must be Linux with iptables, and you'll need root or sudo access.
http://apenwarr.ca/log/sshuttle -r username@sshserver 0.0.0.0/0 -vv

That's it! Now your local machine can access the remote network as if you were right there! And if your "client" machine is a router, everyone on your local network can make connections to your remote network.

This creates a transparent proxy server on your local machine for all IP addresses that match 0.0.0.0/0. (You can use more specific IP addresses if you want; use any number of IP addresses or subnets to change which addresses get proxied. Using 0.0.0.0/0 proxies everything, which is interesting if you don't trust the people on your local network.)

Any TCP session you initiate to one of the proxied IP addresses will be captured by sshuttle and sent over an ssh session to the remote copy of sshuttle, which will then regenerate the connection on that end, and funnel the data back and forth through ssh.

Fun, right? A poor man's instant VPN, and you don't even have to have admin access on the server.

Theory of Operation

sshuttle is not exactly a VPN, and not exactly port forwarding. It's kind of both, and kind of neither.

It's like a VPN, since it can forward every port on an entire network, not just ports you specify. Conveniently, it lets you use the "real" IP addresses of each host rather than faking port numbers on localhost.

On the other hand, the way it *works* is more like ssh port forwarding than a VPN. Normally, a VPN forwards your data one packet at a time, and doesn't care about individual connections; ie. it's "stateless" with respect to the traffic. sshuttle is the opposite of stateless; it tracks every single connection.

You could compare sshuttle to something like the old Slirp program, which was a userspace TCP/IP implementation that did something similar. But it operated on a packet-by-packet basis on the client side, reassembling the packets on the server side. That worked okay back in the "real live serial port" days, because serial ports had predictable latency and buffering.

But you can't safely just forward TCP packets over a TCP session (like ssh), because TCP's performance depends fundamentally on packet loss; it must experience packet loss in order to know when to slow down! At the same time, the outer TCP session (ssh, in this case) is a reliable transport, which means that what you forward through the tunnel never experiences packet loss. The ssh session itself experiences packet loss, of course, but TCP fixes it up and ssh (and thus you) never know the difference. But neither does your inner TCP session, and extremely screwy performance ensues.

sshuttle assembles the TCP stream locally, multiplexes it statefully over an ssh session, and disassembles it back into packets at the other end. So it never ends up doing TCP-over-TCP. It's just data-over-TCP, which is safe.

Useless Trivia

Back in 1998 (12 years ago! Yikes!), I released the first version of Tunnel Vision, a semi-intelligent VPN client for Linux. Unfortunately, I made two big mistakes: I implemented the key exchange myself (oops), and I ended up doing TCP-over-TCP (double oops). The resulting program worked okay - and people used it for years - but the performance was always a bit funny. And nobody ever found any security flaws in my key exchange, either, but that doesn't mean anything. :)

The same year, dcoombs and I also released Fast Forward, a proxy server supporting transparent proxying. Among other things, we used it for automatically splitting traffic across more than one Internet connection (a tool we called "Double Vision").

I was still in university at the time. A couple years after that, one of my professors was working with some graduate students on the technology that would eventually become Slipstream Internet Acceleration. He asked me to do a contract for him to build an initial prototype of a transparent proxy server for mobile networks. The idea was similar to sshuttle: if you reassemble and then disassemble the TCP packets, you can reduce latency and improve performance vs. just forwarding the packets over a plain VPN or mobile network. (It's unlikely that any of my code has persisted in the Slipstream product today, but the concept is still pretty cool. I'm still horrified that people use plain TCP on complex mobile networks with crazily variable latency, for which it was never really intended.)

That project I did for Slipstream was what first gave me the idea to merge the concepts of Fast Forward, Double Vision, and Tunnel Vision into a single program that was the best of all worlds. And here we are, at last, 10 years later. You're welcome.

Update 2010/05/02: Oops, maybe it works a little too well. If you're one of the people who was surprised to see eqldata.com where apenwarr.ca should have been this morning, that's because I left my sshuttle proxy running - connected to the "real" server on eqldata.com - as a stress test. Seems that even my DynDNS provider thought my unreliable home PC was part of the eqldata.com network :) (Also, it failed the stress test: some sort of file descriptor leak after a few hours. Will fix.)

Three types of distributed system designers

Mon, 26 Apr 2010 06:13:39 +0000

1. Paranoid privacy nuts. Systems designed by these people never become popular because paranoid people don't have any friends. (Examples: ZKS, Freenet, GPG.)

2. Redundancy leeches. These people want to back up their files (encrypted) to your computer for added redundancy. Unfortunately, you gain nothing by doing this for them; there's no way to force a leech to contribute space back to other leeches. So these tend to end up as for-profit services. (Examples: AllMyData, Dropbox, S3.)

3. Sharers. These people have data they want to share with other people. They benefit by giving you the data; you benefit by receiving the data, and if you like it, you'll feel nice by sharing it further. (Examples: Debian, Wikipedia, BitTorrent.)

(Free) distributed storage systems in groups 1 and 2 don't seem to ever succeed, because there's no network growth effect.

Systems in group 3 succeed regularly. And they don't need encryption.

Why alienating developers is a winning strategy

Wed, 21 Apr 2010 00:05:55 +0000

I love this new iPhone SDK rule - the one that says you have to write all your apps in pure Objective C with no translation layers. Not because it's good for me (it isn't), but because it's fun to watch an old-school titan - Apple - play the platform game like they mean it. There hasn't been fun like this since Bill Gates left Microsoft.¹

Here's the story so far:

Lots of people bought iPhone 1.0 (without apps) because it was inherently awesome.
Lots of people developed apps for it because there were so many users *and* it was awesome.
About 10x more people bought it because it was even more awesome because of all the apps.
Every other platform (hi, Android) is inferior because it has fewer users and developers and thus apps.²
Go back to step 2 and repeat.

(You might have seen this story before; see Crossing the Chasm and its highly-relevant-to-this-discussion sequel, Inside the Tornado.)

Then, suddenly, Apple clamped down even further on its SDK and app store.³ Shocking! People argue that this breaks the positive feedback cycle: this will mean fewer developers, which means fewer apps, which means less awesome, which means fewer users, which means other platforms can compete, and so on.

No.

Here's the paradox. Apple does need developers to maintain the cycle of awesomeness. What Apple's doing is not good for developers. And developers will continue the cycle anyway.

This is why:

Apple owns the platform; if their platform wins, they win bigger than anybody else. All they need to do to win is to continue to deliver awesomeness to end users as fast as possible so that nobody can catch up. Because they're the biggest platform, they have the most money, so this isn't that hard to do.

The other players are the end users. They have to buy Apple stuff in order for the cycle to continue. The awesomeness must be there or they won't buy stuff. Java and Flash are the opposite of awesomeness. Thus, Apple rejects them outright.⁴

Developers are not part of the strategy. As a developer, you don't make decisions based on awesomeness at all. You might think you do, and the first iPhone developers ("early adopters") did, but that's not you. The modern developers, the ones with the pre-made market and fully-debugged SDK and profitable clients paying you to write iPhone apps, build for iPhone only because it's the leading platform.

In the long term, developers like you would be better off if they would boycott Apple and only develop awesome apps for something more open, like Android or even Blackberry.⁵ Users, as they do, would rapidly switch to the platform with the widest variety of awesome stuff, and everybody would win.

But you aren't going to do that, are you? Because nobody else will either. Unless all the developers switch, the only developers who switch will be suckers. You don't want to be a sucker. You want to make as much money in the short term before the whole thing inevitably implodes. Because it will implode, right? ...right?

Microsoft won on the desktop by being developer-friendly and Apple won in mobile by being developer-hostile. Developers never had anything to do with it.

Footnotes

¹ The closest we have right now is Google vs. Yahoo vs. Microsoft, ie. a bunch of clueless losers shaking their fists at each other. Google isn't winning in search/advertising because of their awesome strategy; they're winning because the competition keeps producing crap. Which is an okay reason to win, but it's not thrilling. Apple vs. World is thrilling.

² Actually #4 isn't even strictly true; last I heard, there are still way more Blackberries than iPhones in active use. But people believe it's true, which is all that matters for this discussion.

³ To be honest, the Apple app store was kinda fascist from day 1, so we're just comparing on a relative scale here. But people get upset anyway.

⁴ All apps written in Java or Flash are ugly and stupid, so end users benefit directly from this restriction. Another reason for Apple to reject such cross-platform apps is admittedly self-serving: if you own the leading platform, you will always get the app. So if you make it hard to port apps between platforms, you're sabotaging the other platforms, not yours.

⁵ Ha ha, I just called the Blackberry "open," even though the only language you could use to develop apps for it for years has been Java. Somehow Java people manage to spin horrible restrictions as features. "100% Pure Java!" and so on.

I may be internet famous, but I am not a primary source of original research

Sat, 17 Apr 2010 04:51:58 +0000

Particularly if you're thinking of footnoting my high school English essay as one of your references in a book about high-bandwidth network communications.

Yeah, I remember my teacher gave me a 10/10 on that one. 15 years ago. But seriously.

(Thanks to Eduardo for pointing out what happens when you search for "apenwarr" or "Avery Pennarun" in Google Books. Answer: silly things.)

Open source is stupid

Thu, 08 Apr 2010 21:58:00 +0000

Not the software. The people. And not the people who make the software. The people who comment about it. I guess now that includes me, which is appropriate, since writing this post will obviously have no positive outcome.

Background: I love Linux. I've written far more Linux software (commercial and open source) than software for any other platform. Back in the 1990's, I even wrote a Linux kernel driver and poem that's (to my ongoing dismay) still in use today.

So yesterday's random post, in which I said something nice about Apple, naturally tagged me instantly as an "Apple Fanboy." (Of course, the accuser here is "somewhat of an exception" because he "really loves Linux." Uh huh. Yay you.)

By the way, yes, thank you for asking, I do have a Blackberry, and I have used the SDK, and the Blackberry simulator is 100% pure crap compared to the iPhone simulator. This is obvious after 0.5 seconds of comparing the two, after which the iPhone one has finished loading your app and the Blackberry one hasn't even made its window appear yet, let alone booted the simulated Blackberry OS.

Nevertheless, it's true. I am probably officially an Apple fanboy now. I mean, I've had an iPod since 2005, which is nearly the beginning of time. I even upgraded to an iPod Touch recently. Also I have a Mac laptop because they're the only ones where power management actually works. Plus I totally downloaded their SDK last week.

Speaking of which, XCode sucks.

Anyway. Fanboy. Yes. Probably. I do have to admit that it's interesting following along with the whole Evil/Artist dichotomy. Or is it a dichotomy at all? I mean, how can you be a serious artist and then let people use Java? Yes, Android, I'm talking to you.

And oh, speaking of Android. While I'm flaming people needlessly, let me just add one more thing:

Top 10 Paid apps in the Android App Store

See the list here.

Abduction! World Attack: 2D game in 1980's style, but with more colours.
Power Manager: because multitasking kills your battery.
Baseball Superstars: they're afraid to show any screenshots.
Open Home: Replace your home screen to further reduce usability.
Jewellust: a Bejewelled clone. Guess Bejewelled guys didn't bother.
MyBackup Pro: like backing up with iTunes, only not free.
Tangram Pro: something Chinese.
aHome: Yes, another one. Replace your home screen replacement.
dxTop: Oh look! 3/10 of the top paid apps replace the home screen!
Aevum Obscurum: the ancient game of Risk, only now you get to pay for it.

Looking at the top 10 free apps is even more apalling. The top 10 free arcade/action games include Pac Man, Tron, Rescue Copter (with realistic 1980's 4-colour graphics!), BrickBreaker, Nibbles, and worse. I'm not kidding about worse.

Remember, folks, Google is the worldwide expert at showing the very best stuff at the top of your search list. Just think what the next 10 look like!

Guys. This is what happens when you let Java people write apps for your platform. Who's evil, again?

Top 10 Paid apps in the Apple App Store

The Simpsons(tm)
Mega Man(tm) II
Scrabble(tm)
Monopoly(tm)
The Sims(tm) 3
Bejeweled(tm) 2
plus some garbage

Oddly, the ridiculously addictive and awesome Space Miner isn't even in the top 10.

You will note that Apple actually explicitly prohibits Java developers from coming anywhere near their SDK. It's part of the license. I'm not even kidding. This is not a coincidence.

There. Glad I got that out of my system.

apenwarr. Lowering the quality of discussion since... the 1990's sometime. Aw, who can remember exactly when. Whatever.

How to run an iPhone app in the simulator without using XCode

Thu, 08 Apr 2010 19:08:52 +0000

I spent a lot of time looking around on the Internet for this answer, and the results were basically nonexistent. The answer is: iphonesim on github. (Despite its name, iphonesim isn't an iPhone simulator; you still need the iPhone SDK to be installed so it can use their simulator.)

The bad news is there's no obvious way to run your app in a Debugger using this system. Hopefully someday it'll be added or I'll figure it out, and then I'll be rid of XCode for good.

Why the iPhone Simulator is Awesome

While we're here, I'm very impressed by the whole concept on which the iPhone simulator works. Most embedded devices (including Blackberry, Android, and other phones) use software to emulate the embedded CPU, which then runs the embedded OS, which then runs your app. This kind of sucks, because the emulator has to work really hard (it often runs at only a fraction of the speed of a real device), and if you crash it you have to reboot it. Plus loading apps onto a simulated device is extra crappy, because you have to simulate a slow USB connection, and so on.

The iPhone simulator works nothing like that. Instead, you compile your app for your native CPU, and they made the iPhone simulator just a native program that runs on your workstation and provides the iPhone API (using native libraries). You simulate and test your program, and when you're finally happy with it, you recompile your app for the target CPU that actually runs on an iPhone. Then it won't work on the simulator anymore.

The result is that the simulator starts instantly and there's no insane two-layer debugging scheme in which you're running a native debugger and decoding non-native (and usually JITted) CPU instructions.

Some people would argue that this method is "less accurate" than precisely emulating the target CPU, and thus the simulator doesn't add much value, since you'll have to test the native app in the end anyhow. It's true that simulating this way is inaccurate and you should do final tests on a real device. The misconception, though, is that the old, annoying, slow, "emulate everything" method is any more accurate. In fact, it's worse.

The fact is, emulators are never perfect. CPU/hardware emulators are really hard to get right, especially if you're trying to make them run fast. If you're not trying to make them run fast, you have a whole different set of problems, because now your simulator is way slower than the real device, so all the animations/etc will be wrong. Try debugging an OpenGL app when your framerate is 1/10th what it should be.

By contrast, the iPhone simulator's method seems magically wonderful. Since the iPhone OS is MacOS, all the kernel APIs are the same. The natively-compiled frameworks, libraries, and display engine are built from the same source code, so you know they're the same too. And your Mac's CPU is a lot faster than the iPhone's CPU, so the simulator can slow down your program to iPhone speed, which is a lot easier than speeding it up (although admittedly imperfect).

In fact, with this method, the only potential sources of incorrect simulation are a) speed (which they seem to have gotten right); b) cross-platform bugs in gcc (I don't know of any); or c) differences in memory layout making memory corruption behave differently. (c) could be a problem, but they seem to provide a lot of debugging tools and you shouldn't be depending on memory corruption anyhow.

Incidentally, this design justifies the fact that you have to have a Mac to do iPhone development, and you have to have the latest MacOS (Snow Leopard) to run the latest SDK. This annoyed me when I first heard of it; I thought Apple was just trying to lock more people into buying a Mac. But now it totally makes sense: iPhone OS *is* Snow Leopard, so if you want to run the native simulator, of course you need Snow Leopard, or the simulator can't possibly work.

That's a really brilliant design tradeoff with huge benefits. And they get to lock more people into buying a Mac.

Update 2010/04/08: A few people have pointed out that the Blackberry "emulator" is apparently not actually an "emulator" but in fact runs a natively-compiled version of the Blackberry JVM. Okay, I guess, but that's not really the point. The point is that it still spends upwards of 30 seconds booting the "virtual Blackberry" before it even gets to the point where you can run your program. (And you have to do this every time you want to run your program.) This is annoying, slow, and pointless, and the (apparently native??) JVM still runs everything horrendously slowly - slower than a real Blackberry. So if it's not a native device emulator, then congratulations, it's somehow even stupider. Yes, I've done real Blackberry development, and the difference between the Blackberry and iPhone simulators is night and day.

Dear StartupCampMontreal:

Wed, 24 Mar 2010 18:55:49 +0000

You are not a "camp." You are a "conf."

The xxxCamp naming scheme appears to have originated with O'Reilly's "FooCamp," which was an informal event where a bunch of famous techie people came to discuss stuff. The more commonly known "BarCamp" series was created as a backlash to the invitation-only FooCamp, but in the same "unconference" style. Thus, the "xxxCamp" series of events is defined primarily by its unconference attributes, to wit:

Poorly organized (in a good way)
Informal
No prearranged agenda
More conversations than presentations
No cabal of unelected organizers who decides who gets to speak
No keynote presentations
No direct financial upside for any participants

StartupCampMontreal claims to have an "unconference component," but this component is obviously a second-class citizen. Among the many reasons that this is obvious (from my observations when I attended it last year), one is especially egregious: the unconference portion is from 1pm to 6pm in the afternoon... on a weekday.

It's reasonably possible for many people to get a day off work to go to a conference; company-sponsored "professional development" days are common. Getting a free day off to go to an unconference is very unlikely. And you know this, because you put the "core" (non-unconference, ie. conference) part of your so-called camp in the evening, when people will be easily able to attend.

I don't actually like "conferences." I find them tedious. Keynote speakers, no matter how famous, are almost universally cliches. Presentations selected by a cabal of organizers are almost never the presentations that would have been selected by an audience (as they do at StartupCampWaterloo, which is a real "camp") - they are therefore boring. And a highly structured set of presentations leaves little time for socializing and unstructured discussion, which is the main purpose of an unconference.

Basically, StartupCampMontreal is not actually interesting to me, because it is not actually a StartupCamp. Your name is essentially false advertising.

Your conference is well organized, well marketed, smoothly running, and attracts a lot of people including numerous sponsors, reasonably famous keynote speakers, and an ever-increasing number of attendees. I'm sure that many of those people are happy to go to StartupConfMontreal, and I'm not suggesting that you change your format just for me.

But what I want and expect from a StartupCamp is an unconference. This is why I will not be attending StartupCampMontreal in the future.

Best of luck and have fun,

Avery

bup is now more popular than all my other projects combined

Wed, 03 Mar 2010 21:27:20 +0000

...at least if you measure it by counting the number of watchers on github.

It is also featured in the top 100 or so interesting github repositories, although there's no particular indication of what they mean by "interesting."

(Previously: bup: it backs things up. It's matured quite a bit since then, though, and is now usable for backing up real work that you care about.)

Things that are best lost in translation

Sun, 24 Jan 2010 08:52:14 +0000

A clever Toronto lawyer was deep into a technical argument before the Supreme Court. His position was dependent upon a close reading of the legal text and turned on the letter of the law.

Suddenly the chief justice, Beverley McLachlin, leaned forward and asked if his argument also worked in French. After all, the law is the law in both languages and a loophole in one tends to evaporate in the other. Only an argument of substance stands up. ... Reality seen through two languages can protect us from the demeaning of justice by technical acrobatics.

-- John Ralston Saul, A Fair Country (p.128)

Parallel with the IETF's "at least two independent and inter-operable implementations" rule.

The problem with the iPad is it has no ethernet ports or DVD drive

Mon, 01 Feb 2010 21:20:08 +0000

...just kidding. But those were the main reasons people gave when they decided the MacBook Air wouldn't be successful a couple of years ago.

It's very interesting that in a laptop, those two features are considered critical. In a "tablet," they apparently are not.

This sounds like classic "positioning" in marketing: since you don't know what a table computer is, you don't have expectations. Since you know what a laptop is, you do. I have a MacBook (not Air). I don't remember the last time I used the ethernet port or the DVD drive. But I wouldn't have bought it without them.

Or maybe the difference is just that the tablet is cheap, and the Air was expensive.

A git-subtree tutorial

Thu, 04 Feb 2010 19:43:11 +0000

Jakub Suder has a nice tutorial on how to use my git-subtree tool to manage git repositories that track other projects in subdirs.

It's quite nicely written, and unlike my own documentation, it has pretty diagrams.

Democracy - the hard way

Sun, 24 Jan 2010 08:52:26 +0000

The great fear of the founders of the United States was that, having created their country out of a violence that papered over the country's fundamental internal contradictions ... it would break apart again. And it would do so violently. They were right to be afraid, as that was preciely what happened some 75 years later.

That original fear of the founders still lies at the heart of nationalism in the United States and still drives the country's idea of citizenship expressed through loyalty.

-- John Ralston Saul, A Fair Country (p.139)

Democracy - the easy way

Sun, 24 Jan 2010 08:17:55 +0000

Canada was the first colony in any empire to extract full democracy from the central power without having to go to war.

-- John Ralston Saul, A Fair Country (p.132)

Definitely enjoying this book.

More on prorogation

Tue, 26 Jan 2010 17:39:58 +0000

In response to my previous post about prorogation, someone emailed me this comment:

My participation in our democracy is limited to voting when there's an election and mostly ignoring everything else. [...] Despite that, I can tell from the rumbling that there's something unusual about this particular prorogation. If it was just normal boring governing, nobody would be talking about it.

I actually think this comment is very insightful, because it gets right to the heart of the issue I was trying to address. Other people sent responses that were more like, "But what gives them the right to manipulate it in such an evil way?" It's almost the same question, but those comments were not insightful. They missed the fact that there may not be any manipulation at all, and therefore turned it from a question of fact into a question of opinion.

I'm not a professional journalist. Yesterday's attempt at factual reporting took all the restraint I had. So this post will surely not meet my own high standards for journalistic integrity. I'm just some guy on the Internet. You've been warned.

How I came to know what "prorogation" means

Full disclosure: I personally don't like the Conservatives. I think if they had a majority government, Canada would be worse off. I think the fact that Harper does most of his public relations through a "spokesperson" is a total embarrassment. I also think our current batch of other political leaders, with the possible exception of Gilles Duceppe, are even worse, and I sorely miss the days when Jean Chretien used to beat people up with statues. But I wouldn't vote for Ignatieff, precisely because he pulls the current kind of crap. I wish I still lived in Quebec so I'd at least have a party worth voting for.

That's the perspective with which, not knowing anything about the current prorogation debate (or even what "prorogation" means), I returned to Canada from my vacation in Mexico and was met by an angry Internet mob complaining about our upcoming dictatorship.

See above: "My participation in our democracy is limited to voting when there's an election and mostly ignoring everything else." Me too. You know why? Because I think the system works. That's the beauty of representative democracy. But I figured, okay guys, I'm pretty smart, I can figure this stuff out. If there's really a dictatorship coming, I want to be on the winning team. So I thought I'd better look into it.

The actual mob of complainers were no help. They all figured that someone else knew what was bad about what was going on, or else they figured that just suspending parliament at all made the government a dictatorship.

This left me to do my own "research" (a word that is in quotes because I did all this "research" in bed using my laptop).

First stop: The National Post (via Google), in an article titled, "Thousands turn out at rallies to protest proroguing of Parliament."

Intermission: A note on how to read political news

any headline about politics

always

nobody

every single day

...shockingly, the National Post headline is phrased in an anti-government way (since the government is Conservative). Reading on, we see lots of quotes from political leaders (ignore them; rule 1). The whole article is also really a big quote from an angry mob of protesters (no fact checking was done; rule 2), since it merely reports that there was an angry mob, not that what the mob was angry about even exists. And we don't know if the mob was really 10 people or a million people (rule 3).

So that article, although it used many words, was in fact 100% pure unenlightening.

Nevertheless, I could feel myself thinking anti-Conservative thoughts despite the total lack of facts. I thought I'd better go find something biased in the opposite direction so that I could balance things out a little. I realized that I couldn't think of any actual Canadian mainstream newspapers that are Conservative-friendly; I'm probably forgetting something obvious.

So I resorted to blogs, which made things easier. Next stop: Alberta Ardvark (via Google). I have to admit that I assumed they were Conservative-friendly just because they're in Alberta, which means I'm a racist. However, I was not disappointed.

That article was the usual politico-blogger nonsense, giving handy advice on how to win a political argument not by arguing about the issue, but instead by turning the conversation around to character assassination whenever possible. (In this case, it's about the fact that Ignatieff supported prorogation last year, so he's not allowed to be against this particular prorogation this year; that makes him inconsistent, therefore a liar, so it doesn't matter what he says, etc.) Nicely done as always, Internet. But there was one intriguing quote: "...the ones who have been convinced by the media that this prorogation is not a routine event..."

Wait... routine event? Surely this is Conservative propaganda. I was intrigued, so I followed the link.

It points out that Chretien prorogued parliament 4 times. And Pierre Elliot Trudeau, supposed Canadian hero... 11 times?! Holy crap! Who's the evil one around here, again?

My brain's magical pattern-detectors then kicked in and I thought: hey, wait a minute. Those prorogation counts seem to be roughly proportional to the amount of time a particular Prime Minister was in office. Perhaps there's a pattern here.

And then, at the bottom of the article: "In our 143 years of existence as Canada, Parliament has been prorogued 105 times."

Oh dear.

Maybe I'd better go learn what prorogation is. The answer is: it's an all-too-fancy word for the end of every parliamentary session.

And it's a word I didn't know the meaning of until now. A word that every news article and blog entry I've read so far has not bothered to define. A word that, in some tenses, has the word "rogue" in it.

Okay, this sucks. Reading biased articles in a search for truth is getting me nowhere. Is there not any news source that will just give me the facts and not try to spin it the way they want? Well, no, I guess there isn't. But there's something close: the CBC.

Because it would be weird if they didn't, the above-linked CBC article has quotes from politicians; try to ignore them (rule 1). If you fail to do so, you will discover that the first apparent use of the word "despotic" in this context was by Ralph Goodale, Liberal House Leader, whom you should not vote for because he is thus automatically a lunatic.

However, the non-politician-quote parts of the article seem to be well balanced and, notably, identify several reasons why Harper might have wanted to prorogue government right now as opposed to some other time.

So what have we learned? First, that prorogation of parliament is totally normal, and that the length of the just-ended session isn't even unusually short; and second, that Harper might very well have chosen this particular date in order to benefit himself or his party. Gasp! Let us look at these possible reasons in more detail.

The CBC's suggested reasons for the current prorogation

"Muzzle parliamentarians amid controversy over the Afghan detainees affair." Don't know about you, but parliament has been shut down for a whole month already and I haven't noticed any of those parliamentarians not talking. I wish. But maybe you have a point; when parliament next starts up, I bet the opposition parties will have completely forgotten about the whole thing, despite the obvious political leverage they could gain from bringing it up. Harper has totally outfoxed them on this one.

"To consult with Canadians, stakeholders and businesses as it moves into the 'next phase' of its economic action plan amid signs of economic recovery." Well, I guess theoretically, if you're in parliament you don't have as much time for consulting with Canadians. But don't we have Royal Commissions for that or something? Maybe they just wanted a longer Christmas holiday. (Aside: the reason the prorogation "doesn't start until January 25th" is that they've all been on holidays since sometime in December. Seriously.)

"Strategically, prorogation also prevents question period criticisms from the opposition parties during the Olympics." Hey, not bad. Avoid the bad PR for Canada from discussing our idiotic foreign affairs policies at the same time as we're in the global spotlight. Critically, this allows Mr. Harper, who (let's be honest) doesn't look all that lovable, to hide in the cellar for the whole time the Olympics are on, letting someone cuter represent us to the world. This seems to be a wise strategy no matter which side of the fence you're on.

"By proroguing Parliament, he is unilaterally making a decision to stop any kind of disclosure from happening." As if information can't be disclosed just because nobody's making any laws right now. Remember: parliament is the legislative branch of the government. It's for making new laws. No other part of the government is suspended just because parliament is. (Note: see update below.)

"Gilles Duceppe wrote that prorogation has become 'a tradition for Harper.'" Duceppe has an awesome sense of humour. I had to read this one a few times before I realized that he managed to give them a sound bite while simultaneously making fun of the fact that prorogation is totally normal, ie. a "tradition."

"By the time Parliament resumes, Harper would have had time to ask Jean to name five new senators, which would give the Conservatives a majority on the newly formed Senate committees and greater control for passing their own legislation." (Notably: nobody was quoted saying this. CBC had to look it up on their own.) "Soudas confirmed the prime minister will seek to fill the Senate vacancies between now and March." This one is actually a great example of a real political reason to prorogue parliament; to get more control of the senate in time for the next session. But the Canadian senate system is designed (on purpose) to work like that. That's why the current senate is mostly Liberal even though our elected representatives are mostly non-Liberal. Senators are appointed for life, at which time the Prime Minister selects new ones. No surprises here.

(In case you don't like that system: the only party in favour of senate reform in Canada is the Conservatives. They'd rather you could elect your senators. How "anti-democratic" of them. I actually think such reform would be a change for the worse, but that's just my opinion.)

"Shortly after Soudas' announcement, the government sent out an email saying it would reintroduce, in original form, the consumer safety bill and the anti-drug-crime law that the Tories claimed the Liberals 'gutted' in the Senate." This shows significant political maneuvering. However, bills take multiple rounds through both houses before they (might) get passed anyway, so this isn't as bad as it sounds. The "gutted" version might never have been passed anyway. Also, interestingly, this was pointed out in an email from a Conservative MP. Apparently they don't think it's evil. At least not evil enough to cover up.

Conclusions

Guys, I did my homework. But I'm just not seeing it. The actual facts are:

1) Prorogation is perfectly normal and the recent parliamentary session wasn't abnormally short.

2) We won't have any more new laws getting made for a month or so longer than usual. (Remember: they were on vacation until January 25th anyway.) But being unable to do stuff doesn't make you a despot, it makes you a eunuch.

3) If Harper is really evil, the first thing to happen in the new parliament in March is that there will be a vote of non-confidence followed by an election. If this doesn't happen, it's because the angry non-Conservative parties didn't actually believe he was evil either.

4) There are some valid political reasons why it's better for the Conservatives if they prorogue parliament right now instead of later. However, they aren't very exciting reasons.

5) There is at least one actual reason (Harper is scary-looking and the Olympics are coming) that it's better for Canada if they prorogue parliament right now.

6) All mainstream media that I read - which was quite a bit - failed to properly define the term "prorogation" or to mention that it's perfectly normal. This seems a rather critical thing to know. Its omission suggests to me that they're trying to make news out of non-news.

Epilogue

"If it was just normal boring governing, nobody would be talking about it."

Unfortunately not true.

That's textbook mob mentality: he must be guilty, because otherwise my friends wouldn't be burning down his house.

The only cure for mob mentality is thinking for yourself.

Updates

Some helpful people have emailed me to clarify or correct or question various parts of the above.

Information release on the Afghan torture investigation: make no mistake, this investigation, and the demand for release of information related to it, is very important. It will also be delayed (for about one month) because of prorogation. Because of "parliamentary immunity," the interesting testimony won't be released during the delay. However, you need to think about two key points: first, will the end result of the investigation be any different because of a one-month delay? And second: will the Conservatives benefit because of the delay? Keep in mind that if the results came out now, they would be largely overshadowed by news about the Olympics. If they come out later, the Olympics will be over, the opposition parties could force an election, and the results would be headlining right as we're thinking about who to vote for. And yet the Conservatives have chosen the latter, not the former.

Precise timing of prorogation: Several people responded by claiming that it's not prorogation that's the problem, it's the particular timing of Harper's use of prorogation. This is an insidious line of argument because it's impossible to disprove; if Harper had prorogued parliament back on the 9th of September (9/9/9 is the British equivalent of 9/1/1), you could have accused him of using numerology to choose prorogation dates, and it would be impossible to refute that claim, even if it had been a perfectly sensible date to end the parliamentary session. Thus, to demonstrate any wrongdoing, you really have to be more specific about why the current timing is so evil. I have discussed several possible reasons above. Please feel free to suggest more. But "the timing is evil!" is not specific enough.

With that in mind, however, a lot of bad statistics are being spread with regard to the lengths of various parliamentary sessions. Here are all of the sitting days per session since 1980:

Turner and Trudeau (Liberal): 591, 116
Mulroney (Conservative): 308, 389, 11, 308, 271
Chretien (Liberal): 283, 164, 243, 133, 211, 143 (avg: 196)
Martin (Liberal): 55, 159
Harper (Conservative): 175, 117, 13, 128 (avg. excluding outlier: 140)

Eyeballing it, Harper's numbers are very slightly lower than typical (except for last year's prorogation, which was indeed an interesting event). Ignoring the outlier (13), Harper's average is just 29% lower than Chretien's. However, Harper has also managed to hold together the longest-running (by a large margin) minority government in Canadian history. I find it unsurprising that the effort required to do so would result in somewhat shorter sessions.

Proroguing parliament

Sun, 24 Jan 2010 07:11:07 +0000

"Prorogation" is the term we use to describe the end of any session of parliament in Canada.

Or as the Canadian Parliament web site says, "Each session of a Parliament ends with the prorogation of Parliament by the Governor General, on the advice of the Prime Minister."

Since Canada's confederation in 1867 (143 years ago), parliament has prorogued 105 times. That's an average of about 1.4 years per session (some are longer, some are shorter).

See the Government of Canada's complete list of prorogations.

Use of the term "despotic" in this context makes you automatically a lunatic.

That is all.

The Google Phone

Wed, 06 Jan 2010 23:55:58 +0000

they think they're the good guys!

-- Fake Steve Jobs

Dot dot dot.

Dealing with "tumultuous" economic times

Sun, 03 Jan 2010 19:39:44 +0000

Canadian Economics Superhero Larry Smith has a new article out, addressing burning economic question questions such as:

Why is Silicon Valley so shortsighted?
Why do so many people write iPhone apps and so few write medical software?
How to deal with a shortage of engineers.
Volume of information vs. value of information.

All great stuff as always. You should read it.

bup 0.01: It backs things up

Sun, 24 Jan 2010 02:15:41 +0000

I just spent a few days of my Christmas vacation writing a new program, bup.

bup is a program that backs things up. It's short for "backup." Can you believe that nobody else has named an open source program "bup" after all this time? Me neither. It also has almost no other meanings.

Despite its unassuming name, bup is pretty cool. To give you an idea of just how cool it is, I wrote you this poem:

Bup is teh awesome
What rhymes with awesome?
I guess maybe possum
But that's irrelevant.

Hmm. Did that help? Maybe prose is more useful after all.

Reasons bup is awesome

bup has a few advantages over other backup software:

It uses a rolling checksum algorithm (similar to rsync) to split large files into chunks. The most useful result of this is you can backup huge virtual machine (VM) disk images, databases, and XML files incrementally, even though they're typically all in one huge file, and not use tons of disk space for multiple versions.
It uses the packfile format from git, so you can access the stored data even if you don't like bup's user interface.
Unlike git, it writes packfiles *directly* (instead of having a separate garbage collection / repacking stage) so it's fast even with gratuitously huge amounts of data.
Data is "automagically" shared between incremental backups without having to know which backup is based on which other one - even if the backups are made from two different computers that don't even know about each other. You just tell bup to back stuff up, and it saves only the minimum amount of data needed.
Even when a backup is incremental, you don't have to worry about restoring the full backup, then each of the incrementals in turn; an incremental backup *acts* as if it's a full backup, it just takes less disk space.
It's written in python (with some C parts to make it faster) so it's easy to extend and maintain.

Super quick example

(The README actually has a more detailed example.)

Try making a remote backup:

 	tar -cvf - /etc | bup split -r myserver: -n my-etc -vv

Try restoring your backup:

 	bup join -r myserver: my-etc | tar -tf -

(On myserver) look at how much disk space your backup took:

 	du -s ~/.bup

Make another backup (yes, that's exactly the same command):

 	tar -cvf - /etc | bup split -r myserver: -n my-etc -vv

Look how little extra space your second backup used on top of the first:

 	du -s ~/.bup

Restore your *first* backup over again (the ~1 is git notation for "one older than the most recent"):

   	bup join -r myserver: local-etc~1 | tar -tf -

What's next?

I have lots of plans for this lovely program, in the event that I actually get time to implement them. But if you think it's cool, please feel free to git clone it, hack away, and send some patches! Read the README for a list of some deficiencies in the current release.

I'm sure there are also more deficiencies that I don't know about, of course.

(Previous poetry-related adventures.)

Update (2010/01/05): Commentary at ycombinator news and reddit programming. To answer the most common question: it's different from most of those other apps you mention because: a) bup backs up really huge files rather than silently ignoring them or running out of memory; b) bup is a backend, not a GUI, while most of those apps are GUIs (which could use bup as a backend if they wanted); c) bup stores its backups in big packfiles, rather than a one-file-per-file model, and thus can be much faster (but 0.01 isn't optimized yet).

Update 2 (2010/01/05): By popular demand (well, nonzero demand from the populace, anyway), I've created a mailing list. You can subscribe by sending an email to bup-list+subscribe@googlegroups.com (note the weird + character in the email address).